How to build a RAG app with groq (llama-3) and llama-index, Hugging Face without OpenAI API keys and with minimal compute

Building a RAG App with Groq (Llama-3), Hugging Face opensource model and Llama-Index: A Love-Hate Story

Setting Up the Environment

pip install Flask requests llama-index==0.10.18 llama-index-llms-groq==0.1.3 llama-index-embeddings-huggingface==0.2.0

from flask import Flask, request, render_template
import re
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
StorageContext,
ServiceContext,
load_index_from_storage,
)
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.groq import Groq
from llama_index.core.base.llms.types import ChatMessage, MessageRole
import warnings
warnings.filterwarnings('ignore')

Initializing Flask App and API Key

GROQ_API_KEY = "<insert your API key here>"
app = Flask(__name__)

Defining the RAG Chatbot Function

def chat_with_rag(message, history):
		reader = SimpleDirectoryReader(input_files=["./paul_graham_essays.txt"]) 
		documents = reader.load_data()
		text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=200)
		nodes = text_splitter.get_nodes_from_documents(documents, show_progress=True)
		
		embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
		
		llm = Groq(model="llama3-70b-8192", api_key=GROQ_API_KEY)
		
		prompt = "enter your system prompt here"
		
		service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm, 
		system_prompt=prompt)
		
		vector_index = VectorStoreIndex.from_documents(documents, show_progress=True, 
		service_context=service_context, node_parser=nodes)
		
		vector_index.storage_context.persist(persist_dir="./storage_mini")
		
		storage_context = StorageContext.from_defaults(persist_dir="./storage_mini")
		
		index = load_index_from_storage(storage_context, service_context=service_context)
		
		query_engine = index.as_chat_engine(service_context=service_context,
		chat_mode='condense_plus_context')
		
		resp = query_engine.chat(message, chat_history=history)
		
		return resp.response

• Document Reading: We use SimpleDirectoryReader to read documents from a file.

• Text Splitting: The SentenceSplitter splits the documents into chunks for efficient processing.

• Embedding Model: We use HuggingFaceEmbedding with the sentence-transformers/all-MiniLM-L6-v2 light weight model.

• Language Model: The Groq model is initialized with the llama3-70b-8192 model. (Why Gooq? because it uses LPUs which are insanely fast you should read more about)

• Service Context: Combines the embedding model, language model, and a system prompt. (I found this from source code, its not mentioned in the docs)

• Vector Index: Creates a vector index from the documents and persists it to disk.

• Query Engine: Loads the index and sets up the query engine to handle chat interactions.

query_engine = index.as_chat_engine(service_context=service_context, chat_mode='condense_plus_context')

Setting Up Flask Routes

@app.route('/chat', methods=['POST'])
def chat():
		data = request.json
		history_json = data.get('messages', [])
		system_prompt = {
		    'role': 'system',
		    'content': '<enter your system prompt here optional>'
		}
		history_json.insert(0, system_prompt)
		
		user_message = ''
		history = []
		
		if history_json:
		    user_message = history_json[-1]['content']
		    history = [convert_to_chat_message(msg['role'], msg['content']) for msg in history_json]
		
		bot_response = chat_with_rag(user_message, history=history)
		
		return {'response': bot_response}

• Chat Route: Handles POST requests with user messages, processes the chat history, and returns the bot's response. data.get.messages is an array of json objects that should come from your client or from your other server if you have a microservice arch. Each json should be in the form

{role:"user or assistant", content:"some sample text"}

Helper Function

def convert_to_chat_message(role: str, content: str) -> ChatMessage:
    """Converts a role and content into a ChatMessage object."""
    # Convert role string to MessageRole enum
    role_enum = MessageRole(role)
    # Create and return a ChatMessage object
    return ChatMessage(role=role_enum, content=content)

Running the Application

if __name__ == '__main__':
    app.run(debug=True)

Conclusion