Start Learning Contact Us
Generative AI and LLMs

Image, Audio and Multimodal Models

Lesson 7 of 10

Generative AI is not limited to text. The same ideas, large models trained on very large datasets and directed by prompts, apply to images, sound and video.

How image generation works: diffusion

Most current image generators are diffusion models. Training has two parts.

Forward process. Take a real image and add a little random noise, again and again, until only noise remains.

Reverse process. Train a neural network to undo one step: given a noisy image, predict the noise that was added.

To generate a new image, start from pure random noise and apply the trained network repeatedly, removing a little noise each time, until an image emerges.

import numpy as np

rng = np.random.default_rng(0)
image = np.array([0.0, 0.2, 0.9, 1.0, 0.9, 0.2, 0.0])   # a tiny one-row "image"

noisy = image.copy()
for step in range(1, 6):
    noisy = 0.8 * noisy + 0.6 * rng.normal(0, 0.3, image.shape)
    print(f"after noise step {step}: {noisy.round(2)}")

After a few steps the original pattern is hard to see. A diffusion model learns to run this process backwards.

Where the text prompt comes in

A text encoder turns the prompt into embeddings, and the denoising network is given those embeddings at every step. The network removes noise in a direction that matches the description. This is called conditioning.

Many systems run the diffusion process on a compressed representation of the image, which is much cheaper, and then decode the result to full size. This approach is known as latent diffusion.

Writing prompts for images

  • Describe the subject, the setting, the style and the composition.
  • Be specific about the things that matter: lighting, viewpoint, colours, medium.
  • Generate several variations and refine the one that is closest.
  • Text inside images and exact counts of objects are common weak points. Check them.
A product photograph of a ceramic coffee mug on a wooden table,
soft morning light from a window on the left, shallow depth of field,
neutral colours, viewed slightly from above.

Other image tasks

  • Image-to-image: start from an existing picture and transform it.
  • Inpainting: replace a selected region.
  • Outpainting: extend a picture beyond its borders.
  • Upscaling: increase resolution.

On your own hardware, the Hugging Face Diffusers library runs open image models.

# pip install diffusers transformers torch
from diffusers import AutoPipelineForText2Image
import torch

pipe = AutoPipelineForText2Image.from_pretrained("name-of-an-open-image-model", torch_dtype=torch.float16)
pipe = pipe.to("cuda")
image = pipe("a ceramic coffee mug on a wooden table, soft morning light").images[0]
image.save("mug.png")

Models that see

A vision-language model accepts images as well as text. The image is split into patches, each patch becomes a vector, and those vectors are processed alongside the text tokens.

Uses include:

  • describing photos and reading charts,
  • extracting data from scanned forms, receipts and screenshots,
  • answering questions about a diagram,
  • checking a design against written requirements.

Images are counted in tokens too, so large or numerous images add cost. Check anything the model reads from an image, especially small text and numbers.

Audio

  • Speech-to-text turns recordings into transcripts.
  • Text-to-speech produces natural-sounding voices from text.
  • Speech-to-speech models hold a spoken conversation directly.
  • Music and sound generation create audio from a description.

Combining these with a language model gives a voice assistant: transcribe the speech, generate a reply, speak it.

Video

Video models extend image generation with consistency over time. They are improving quickly, and are computationally expensive. Output is usually limited to short clips.

Responsibilities

Generated media raises particular issues.

  • Deepfakes. Realistic fake images, voices and video of real people can be used to deceive. Do not create them without consent, and be sceptical of media you cannot verify.
  • Copyright. The legal position on training data and on ownership of generated output differs between countries and is still developing. Check the licence terms of the tool you use and take advice for commercial work.
  • Disclosure. Label generated media as AI-generated where people could be misled. Some content tools add provenance information or watermarks to help.
  • Bias. Image models can reproduce stereotypes, for example in how they depict professions. Review what you publish.

Practice

Write an image prompt for a banner for a data course. Specify the subject, style, colours and composition in four sentences or fewer.