Usage

Running Server

# Locally
minyma server run

# Docker Quick Start
make docker_build_local
docker run \
    -p 5000:5000 \
    -e OPENAI_API_KEY=`cat openai_key` \
    -e DATA_PATH=/data \
    -v ./data:/data \
    minyma:latest

The server will now be accessible at http://localhost:5000

Normalizing & Loading Data

Minyma is designed to be extensible. You can add normalizers and vector db's using the appropriate interfaces defined in ./minyma/normalizer.py and ./minyma/vdb.py. At the moment the only supported database is chroma and the only supported normalizer is the pubmed normalizer.

To normalize data, you can use Minyma's normalize CLI command:

minyma normalize --filename ./pubmed_manuscripts.jsonl --normalizer pubmed --database chroma --datapath ./chroma

The above example does the following:

• Uses the pubmed normalizer
• Normalizes the ./pubmed_manuscripts.jsonl raw dataset [0]
• Loads the output into a chroma database and persists the data to the ./chroma directory

NOTE: The above dataset took about an hour to normalize on my MPB M2 Max

[0] https://huggingface.co/datasets/TaylorAI/pubmed_author_manuscripts/tree/main

Development

# Initiate
python3 -m venv venv
. ./venv/bin/activate

# Local Development
pip install -e .

# Creds
export OPENAI_API_KEY=`cat openai_key`

Datasets

https://huggingface.co/datasets/TaylorAI/pubmed_author_manuscripts/tree/main

Notes

• https://docs.pinecone.io/docs/openai
• https://docs.pinecone.io/docs/langchain
• https://docs.pinecone.io/docs/langchain#creating-embeddings
• https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
• https://medium.com/@abhishekranjandev/building-a-speech-recognition-app-with-deepspeech-word2vec-and-pinecone-1e5907d103e2
• https://medium.com/@mishra.thedeepak/doc2vec-simple-implementation-example-df2afbbfbad5
• https://cookbook.openai.com/examples/semantic_text_search_using_embeddings

TODO:

• Build this with Word2Vec / Doc2Vec: https://docs.pinecone.io/docs/openai
• https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html#sphx-glr-auto-examples-tutorials-run-doc2vec-lee-py
• https://webcache.googleusercontent.com/search?q=cache:https://medium.com/@rubentak/unleashing-the-power-of-intelligent-chatbots-with-gpt-4-and-vector-databases-a-step-by-step-8027e2ce9e78