Processing Audio Files with LLMs using LeMUR

In this guide, we'll show you how to use AssemblyAI's LeMUR (Leveraging Large Language Models to Understand Recognized Speech) framework to process audio files with an LLM. You can use LeMUR to ask questions and generate answers, summarize, and extract action items from one or more transcripts processed by AssemblyAI.

Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for an account and get your API key from your dashboard.

LeMUR features are currently only available to paid users, at two pricing tiers: LeMUR and LeMUR Basic. Refer to for more detail.

Custom Summary

Generate use case-specific content based on external context, formatting guidelines, and accurate transcripts.

1. 1

   LeMUR works on AssemblyAI transcripts. If you haven't already created a transcript to use with LeMUR, start with our guide to Transcribing an audio file. You need the transcript id, and that transcript needs to be in a completed state.

   Import the necessary libraries for making an HTTP request and set up your API key.

2. 2

   Next, provide one or more transcript IDs and define your LeMUR request parameters for Custom Summary.

3. 3

   Now, construct and send the API request. Your LeMUR output is contained in the response key of the JSON API output. If there's an error, an error key is returned instead of the response key:

4. 4

   The Custom Summary endpoint returns the summary as a string:

Question & Answer

Ask questions about what was covered in a single transcript or across multiple transcripts.

1. 1

   LeMUR works on AssemblyAI transcripts. If you haven't already created a transcript to use with LeMUR, start with our guide to Transcribing an audio file. You need the transcript id, and that transcript needs to be in a completed state.

   Import the necessary libraries for making an HTTP request and set up your API key.

2. 2

   Next, define your LeMUR request parameters for Q&A.

3. 3

   Now, provide one or more transcript IDs, then construct and send the API request. Your LeMUR output is contained in the response key of the JSON API output. If there's an error, an error key is returned instead of the response key:

4. 4

   The Question & Answer feature returns a set of question-answer pairs:

Action Items

Generate a list of action items based on the transcript(s) for follow up.

1. 1

   LeMUR works on AssemblyAI transcripts. If you haven't already created a transcript to use with LeMUR, start with our guide to Transcribing an audio file. You need the transcript id, and that transcript needs to be in a completed state.

   Import the necessary libraries for making an HTTP request and set up your API key.

2. 2

   Next, provide one or more transcript IDs and define your LeMUR request parameters for Action Items.

3. 3

   Now, construct and send the API request. Your LeMUR output is contained in the response key of the JSON API output. If there's an error, an error key is returned instead of the response key:

4. 4

   The Action Items feature returns the folow up items generated from the transcript:

Custom Task

Use LeMUR to ask anything with Custom Task.

1. 1

   LeMUR works on AssemblyAI transcripts. If you haven't already created a transcript to use with LeMUR, start with our guide to Transcribing an audio file. You need the transcript id, and that transcript needs to be in a completed state.

   Import the necessary libraries for making an HTTP request and set up your API key.

2. 2

   Next, provide one or more transcript IDs and define your LeMUR request parameters for Custom Task.

3. 3

   Now, construct and send the API request. Your LeMUR output is contained in the response key of the JSON API output. If there's an error, an error key is returned instead of the response key:

4. 4

   Your Custom Task output is contained in the response key:

Delete data for a LeMUR request

Delete the data for a previously submitted LeMUR request. Response data from the LLM, as well as any context provided in the original request will be removed.

1. 1

   You can only delete successful LeMUR requests. If you haven't yet submitted a LeMUR request, see Custom task for an example. You can find the request_id in the response of a successful LeMUR request.

2. 2

   Now that we have a request_id, we can submit a deletion request.

3. 3

   The response contains information about the status of the deletion request.

Conclusion

Building Generative AI products centered around human speech is challenging because audio files present challenges for LLMs. LeMUR chains together prompts, connects multiple Large Language Models together, and overcomes the need for extensive set-up or implementing a vector database for long-term information storage. LeMUR makes it possible to process and get responses on multiple audio files at once with a single API request.

To learn more about LeMUR, refer to the AssemblyAI blog.

If you encounter any issues or have any questions, you can refer to our FAQ or reach out to our Support team.

For details on how to call the LeMUR API check out the LeMUR API Docs.