Retrieval Augmented Generation (RAG) is a method to augment the relevance and transparency of Large Language Model (LLM) responses. In this approach, the LLM query retrieves relevant documents from a database and passes these into the LLM as additional context. RAG therefore helps improve the relevancy of responses by including pertinent information in the context, and also improves transparency by letting the LLM reference and cite source documents.
While RAG is an increasingly popular method, it requires textual documents as references and cannot directly use the wealth of audio data that many organizations have or that is available online, like meeting recordings, lectures, webinars, and more.
In this tutorial, we will demonstrate how to perform Retrieval Augmented Generation with audio data by leveraging AssemblyAI’s document loader for LangChain, a popular framework that provides building blocks for LLM-based applications, using Chroma as the vector database to store our document embeddings.
The source code for this tutorial can be found in this repo, and you can watch a video version of this tutorial on YouTube.
Getting started
To follow this tutorial, you’ll need an AssemblyAI API key. You can get one for free here if you don’t already have one. Additionally, we’ll be using GPT 3.5 for this tutorial, so you’ll need an OpenAI API key as well - sign up here if you don’t have one already.
Setting up the virtual environment
In a terminal, create a directory for this project and navigate into it:
Now, enter the following command to create a virtual environment called venv
Next, activate the environment. If you’re on MacOS/Linux, enter
If you are on Windows, enter
Next, install the libraries we’ll need for this tutorial:
Setting up the environment file
We’ll use python-dotenv to load environment variables for our project. In order to do so, we’ll need to store our environment variables in a file that the package can read. In your project directory, create a file called .env and paste your API keys as the values for the corresponding environment variables:
Note, it is extremely important to not share this file or check it into source control. Anybody who has access to these keys can use your respective accounts, so it is a good idea to create a .gitignore file to make sure that this does not accidentally happen. Additionally, we can exclude the virtual environment from source control. The .gitignore file should therefore contain the following lines:
Writing the application
Now we’re ready to write the application. Overall, the application will work as follows:
- First, we will load our documents using AssemblyAI’s document loader for LangChain
- Next, we will split these documents in order to have smaller chunks that can be retrieved by LangChain during RAG
- We will then embed these chunks with HuggingFace, yielding one vector for each chunk
- After that, we will store these embeddings in a Chroma vector database
- Finally, we will use LangChain’s built in QA chain to write a simple loop that lets us query OpenAI’s GPT 3.5. We will be shown the answer along with the texts retrieved for RAG when generating the answer
We can see this process pictorially as below:
Imports and environment variables
Open a new file in your project directory called main.py. First, we’ll add all imports required for this project. Additionally, we'll use the load_dotenv() function in order to load the contents of our .env file as environment variables.
Loading the documents
The AssemblyAIAudioTranscriptLoader allows us to load audio files, local or remote, into LangChain applications. In this example, we will use several audio files from LangChain’s webinar series on YouTube, but feel free to use any files you’d like. Note that remote files must be publicly accessible for the loader to work properly.
Since the AssemblyAIAudioTranscriptLoader can only accept one input at a time, we write a small function that wraps a for loop to load each audio file, printing the url each time a new file begins loading. We then simply pass in our list of URLs to generate a list of LangChain Documents:
Document splitting and metadata
Now, we need code that can split all of the Documents into chunks by using a RecursiveCharacterTextSplitter:
Next, set the metadata for each chunk to contain only its source (i.e. filepath/URL). The AssemblyAIAudioTranscriptLoader returns a large amount of metadata about the transcribed audio file, and some of the types are incompatible with the Chroma database that we will use later, so for the sake of simplicity we strip off everything extraneous that we don’t need for this tutorial.
Embed texts
Next up we create embeddings for all of our texts and load them into a Chroma vector database. For this, we write a make_embedder function that wraps the process for instantiating an embedding model from HuggingFace. Then we simply pass this object and our texts into the function to create a Chroma vector database.
Create the Q&A chain
Finally, we create the Q&A chain that ties all of these components together. Write a function make_qa_chain that returns a RetrievalQA chain with GPT-3.5 as the LLM and Chroma as the retriever vector database. The RetrievalQA chain will also return source documents used as references during RAG.
Then we write a loop to take in questions, pass them into the Q&A chain, and display the answers/sources:
You can find all of the source code for this project in this repository.
Running the application
To run the app, execute python main.py or python3 main.py from the project directory with your virtual environment activated. The transcription of the file (i.e. loading with the AssemblyAI loader) may take several minutes depending on the lengths of the files. To speed this up, you can consider extending the application to do this in parallel so each file does not have to be transcribed before starting the transcription of the next file.
Note
If you receive an error huggingface/tokenizers: The current process just got forked, you can add TOKENIZERS_PARALLELISM=false to your .env file. See here for additional details.
Simply enter a question into the terminal and a response will be generated, along with a list of sources (with location and source content).
Example
Final words
In this tutorial, we learned how to combine several tools to perform Retrieval Augmented Generation (RAG) with audio data. In particular, we used the LangChain framework to load audio files with AssemblyAI, embed the files with HuggingFace into a Chroma vector database, and then perform queries with GPT 3.5.
For more tutorials like this, check out the tutorials section of our blog. Alternatively, check out our YouTube channel for other learning resources on Machine Learning. Finally, feel free to follow us on Twitter in order to stay in the loop when we release new content.
