Help Build The Future Of Primary Care Delivery

Globally, more than 350,000 digital health apps are listed in the various app stores yet visits to a physician often feel like time travel back to the 70s.

Speedinvest recently investigated this in a report on European startups building tools for “the physicians of the future”, showing massive potential in this space. I’m working with a Berlin-based entrepreneur who is building an ML-based workflow optimization tool for medical practices. He has validated the need and found practices that are willing to co-create the product.
Now, he is looking looking for an engineer to join him as CTO / Co-Founder.
If you are looking to start a company and are passionate about the health care space, drop me a note at: gregor@thedecoding.net!

I am happy to make the connection.

​
​

How To Reduce The Cost Of Using LLM APIs by 98%

Cost is still a major factor when scaling services on top of LLM APIs.

Especially, when using LLMs on large collections of queries and text it can get very expensive. It is estimated that automating customer support for a small company can cost up to $21.000 a month in inference alone.

The inference costs differ from vendor to vendor and consists of three components:

1) a portion that is proportional to the length of the prompt

2) a portion that is proportional to the length of the generated answer

3) and in some cases a small fixed cost per query.

In a recent publication researchers at Stanford proposed three types of strategies that can help us to slash costs. The cool thing about it is that we can use these strategies in our projects independently of the prices dictated by the vendors!

Let’s jump in!

​

How To Adapt Our Prompts To Save Costs

Most approaches to prompt engineering typically focus only on increasing performance.

In general, prompts are optimized by providing more detailed explanations of the desired output alongside multiple in-context examples to steer the LLM. However, this has the tendency to result in longer and more involved prompts. Since the cost per query grows linearly with the number of tokens in our prompt this makes API requests more expensive.

The idea behind the first approach, called Query Adaption, is to create effective (often shorter) prompts in order to save costs.

This can be done in different ways. A good start is to reduce the number of few-shot examples in your prompt. We can experiment to find out what the smallest set of examples is that we have to include in the prompt to maintain performance. Then, we can remove the other examples.

So far so good!

Once we have a more concise prompt, there is still another problem. Every time a new query is processed, the same in-context examples and detailed explanations to steer the model are processed again and again.

The way to avoid this redundant prompt processing is by applying query concatenation.

In essence, this means that instead of asking one question in our lengthy prompt, we add multiple questions Q1, Q2, … in the same prompt. To get this to work, we might need to add a few tokens to the prompt that make it easier for us to separate the answers from the model output. However, the majority of our prompt is not repeatedly sent to the API as a result.

This allows us to process dozens of queries at once, making query concatenation a huge lever for cost savings while being relatively easy to implement.

That was an easy win! Let’s look at the second approach!

​

LLM Approximation

The idea here is to emulate the performance of a better, more expensive model.

In the paper, they suggest two approaches to achieve this. The first one is to create an additional caching infrastructure that alleviates the need to perform an expensive API request for every query. The second way is to create a smaller, more specialized model that mimics what the model behind the API does.

Let’s look at the caching approach!

The idea here is that every time we get an answer from the API, we store the query alongside the answer in a database. We then pre-compute embeddings for every stored query. For every new query that comes in, we do not send it off to our LLM vendor of choice. Instead, we perform a vectorized search over our cached query-response pairs.

If we find a question that we already answered in the past, we can simply return the cached answer without accruing any additional cost. This obviously works best if we repeatedly need to process similar requests and the answers to the questions are evergreen.

Now let’s move on to the second approach!

Don’t worry! The idea is not to spend hundreds of thousands of dollars to fine-tune an LLM. If the overall variety of expected questions and answers is not crazy huge - which for most businesses it is not - a BERT-sized model should probably do the job.

The process could look as follows: first, we collect a dataset of queries and answers that are generated with the help of an API. The second step is to fine-tune the smaller model on these samples. Third, use the fine-tuned model on new incoming queries.

To reduce the cost even further, It could be a good approach to implement the caching first before starting to train a model. This has the advantage of passively building up a dataset of query-answer pairs during live operation. Later we can still actively generate a dataset if we run into any data quality concerns such as some queries being underrepresented.

A pretty cool byproduct of using one of the LLM approximation approaches is that they can significantly reduce latency.

Now, let’s move on to the third and last strategy which has not only the potential to reduce costs but also improve performance.

​

LLM Cascade

More and more LLM APIs have become available and they all vary in cost and quality.

The idea behind what the authors call an LLM Cascade is to start with the cheap API and then successively call APIs of increasing quality and cost. Once an API returns a satisfying answer the process is stopped. Especially, for simpler queries this can significantly reduce the costs per query.

However, there is a catch!

How do we know if an answer is satisfying? The researchers suggest training a small regression model which scores the reliability of an answer. Once this reliability score passes a certain threshold the answer gets accepted.

One way to train such a model would obviously be to label the data ourselves.

Since every answer needs only a binary label (reliable vs. unreliable) it should be fairly inexpensive to build such a dataset. Better still we could acquire such a dataset semi-automatically by asking the user to give feedback on our answers.

If running the risk of serving bad answers to customers is out of the question for whatever reason, we could also use one of the stronger APIs (cough GPT **cough**) to label our responses.

In the paper, the authors conduct a case study of this approach using three popular LLM APIs. They successively called them and used a DistillBERT (very small) to perform scoring. They called this approach FrugalGPT and found that the approach could save up to 98.3% in costs on the benchmark while also improving performance.

How would this increase performance you ask?

Since there is always some heterogeneity in the model’s outputs a weaker model can actually sometimes produce a better answer than a more powerful one. In essence, calling multiple APIs gives more shots on goal. Given that our scoring model works well, this can result in better performance overall.

In summary, strategies such as the ones described above are great because they attack the problem of high inference costs from a different angle. They allow us to be more cost-effective without relying on the underlying models to get cheaper. As a result, it will become possible to use LLMs for solving even more problems!

What an exciting time to be alive!

As always, I really enjoyed making this for you and I sincerely hope you found value in it!

If you are not subscribed to the newsletter yet, click here to sign up! I send out a thoughtful 5-minute email every week to keep you in the loop about machine learning research and the data economy.

Thank you for reading and I see you next week ⭕!

113 Cherry St , #92768, Seattle WA
​Unsubscribe · Preferences​