Start Learning Contact Us
Generative AI and LLMs

Working with Model APIs

Lesson 5 of 10

A chat application is one way to use a model. To build your own product, you call the model through an API: your program sends a request over the internet and receives the model’s reply.

Every major provider offers an API and libraries for common programming languages. The details differ, but the concepts in this lesson apply to all of them. Use the provider’s current documentation for exact names.

The shape of a request

A request is a structured message, usually written as JSON.

{
  "model": "name-of-the-model",
  "max_tokens": 500,
  "temperature": 0.2,
  "system": "You are a support assistant for a parcel delivery company.",
  "messages": [
    {"role": "user", "content": "Where is my parcel?"},
    {"role": "assistant", "content": "I can help. What is your tracking number?"},
    {"role": "user", "content": "TR-20931"}
  ]
}
Field Purpose
model Which model to use
messages The conversation so far, oldest first
system Standing instructions
max_tokens The upper limit on the length of the reply
temperature How varied the output is

The shape of a response

The reply contains the generated text, a stop reason that says why generation ended, and a usage count of input and output tokens.

Check the stop reason. If it shows that the token limit was reached, the answer has been cut off.

The API has no memory

The service keeps nothing between requests. To continue a conversation, your code stores the messages and sends the full list every time.

conversation = []

def add(role, content):
    conversation.append({"role": role, "content": content})

add("user", "Where is my parcel?")
add("assistant", "I can help. What is your tracking number?")
add("user", "TR-20931")

for message in conversation:
    print(f'{message["role"]:9s} | {message["content"]}')
print("messages sent with the next request:", len(conversation))

Tokens and cost

Providers charge per token, with separate prices for input and output. Output tokens usually cost more.

def cost(input_tokens, output_tokens, price_in_per_million, price_out_per_million):
    return (input_tokens * price_in_per_million + output_tokens * price_out_per_million) / 1_000_000

# illustrative prices only; check your provider's current price list
price_in, price_out = 3.00, 15.00

one_request = cost(input_tokens=2_000, output_tokens=400, price_in_per_million=price_in, price_out_per_million=price_out)
print(f"one request: {one_request:.4f}")
print(f"10,000 requests a day for 30 days: {one_request * 10_000 * 30:,.2f}")

Because the whole conversation is sent each time, long conversations cost more with every turn.

Ways to reduce cost:

  • Use a smaller, cheaper model for simple tasks.
  • Keep prompts as short as the task allows.
  • Limit the output length.
  • Use prompt caching where the provider offers it, so that a long repeated prefix such as a system prompt is charged at a lower rate.
  • Use batch processing for work that does not need an immediate answer.

Streaming

By default you wait for the complete reply. With streaming, the service sends the text in pieces as it is generated, and your application shows it as it arrives. The total time is the same, but the user sees the first words almost at once.

Errors you must handle

Problem What to do
Rate limit reached Wait and retry, with increasing delays
Service overloaded or a timeout Retry a limited number of times
Invalid request Fix the request. Retrying will not help
Input longer than the context window Shorten or summarise the input
Output cut off at the token limit Raise the limit, or ask for a shorter answer
Reply declined by a safety filter Show the user a clear message 
import time

def call_with_retries(send, attempts=4):
    for attempt in range(1, attempts + 1):
        try:
            return send()
        except TimeoutError as error:
            if attempt == attempts:
                raise
            delay = 2 ** attempt
            print(f"attempt {attempt} failed ({error}); retrying in {delay}s")
            time.sleep(0)   # use time.sleep(delay) in real code

calls = {"n": 0}
def flaky():
    calls["n"] += 1
    if calls["n"] < 3:
        raise TimeoutError("service busy")
    return "reply text"

print(call_with_retries(flaky))

Doubling the delay after each failure is called exponential backoff.

Protect your API key

An API key identifies your account and is charged for every request made with it.

  • Keep it in an environment variable or a secrets manager, never in your code.
  • Never commit it to version control.
  • Never put it in a web page or mobile app. Call the model from your server.
  • Set spending limits, and replace a key immediately if it is exposed.

Choosing a model

Providers offer several models that trade capability against speed and price. Start with a capable model to find out whether the task can be done at all. Once it works, test whether a cheaper model does it well enough.

Practice

Using the cost function, compare two designs: one request with 6,000 input tokens, and three requests with 2,000 input tokens each, all with 400 output tokens. Which is cheaper?