Mind the Waste. The hidden costs of unwanted AI Blurp

Who is controlling your AI output and consequently… costs?’

I recently asked Perplexity to clarify an AI-related acronym. I was not surprised by the low confidence of the response – a lot of concepts that relate to AI are work-in-progress and are not universally accepted as consensus.

What got me interested though was the length of the response. In the context of AI we measure inputs and outputs in ‘token’. A ‘token’ is a basic unit of text, typically four characters. My request here generated a 680 tokens-of-speculation response.

And: adding more context to the prompt did still not yield the accurate explanation of the acronym I was looking for. My 24-tokens context investment yielded yet another 1.2k tokens of speculative response. An additional 15-tokens context prompt finally got me to the desired answer at the right level of confidence. But, again, the response generated 1.3k tokens.

To summarize, an aggregated 50 tokens input generated a total of 3270 output tokens at an estimated total cost of 0.07USD.

In this example we are talking pennies - literally. But assuming that I am not the only casual user of AI on this planet who is splashing hidden money on trivial topics this becomes a question of scale with a huge potential for waste reduction.

The cost of an output token tends to be significantly higher than the price of an input token. This makes sense considering the computational ‘effort’ required to create them. Coming back to my example, this means that around <99.9% of the (invisible) costs generated by my prompt were beyond my direct control.

In all fairness, I do have a lever here to control the response either by 1) explicitly restricting the max number of output tokens allowed or 2) by requesting a short response via single/few-shot prompting, 3) set user-profile level preferences on how to respond to vague and ambiguous prompts

However, a smarter approach would be if the default system level prompt template would gear automatically towards follow-up and clarifying questions as opposed to gearing towards instantly gratifying the user with verbose yet potentially inaccurate responses.

Only Pennies. Open AI claims that Chat GPT alone has in the region of 1.2B weekly users. Taken together with other similar GenAI tools, this likely translates to … a lot of users, a lot of weekly prompts, a lot of potential token waste. So: getting this right at scale may well translate into a few less datacentres being needed in the future. And then we are no longer talking pennies.

Prompt Details – Tool Used: Perplexity