TL;DR:
- LLMs operate on “tokens,” discrete numerical vectors that can represent words, musical notes, or visual pixel patches with equal fluency.
- Milestone architectures like CLIP and modern LMMs project text, images, and audio into a unified vector space, allowing descriptive prompts to correlate directly with sensory data.
- When an AI creates an image or video, it interprets the prompt, predicts the corresponding visual tokens within its learned pattern space, and decodes those tokens into rendered pixels.
- A single visual input or output can consume thousands of tokens compared to roughly a hundred for a paragraph, making visual AI more resource-heavy and computationally demanding.
There’s something that confused many people (including yours truly) when they first learned about large language models. LLMs are literally called language models, so how do they generate all other kinds of information?
All that LLMs do, at a basic level, is predict the next word. Yep, that’s it; that’s the whole thing. It’s easy to see how this works for programming languages, but LLMs also generate pictures, videos, audio, and 3D objects. You can use LLMs to build websites and apps end-to-end, from code to content and multimedia. But why exactly is this possible?
Let’s answer this question by taking a quick look at what’s under the hood.
Large LEGO models: everything is tokens
The key point we should all understand is that the concept of “language” differs between humans and AIs. For an AI model, it’s all about tokens, and, practically speaking, they’re less like words or phrases and more like… LEGOs. Let’s briefly talk about why that’s the case.
When playing with LEGO, you can use the same kinds of building blocks to build a castle, a space shuttle, or a dinosaur. Tokens are a lot like that: they are AI’s building blocks that can represent anything. A token can be a whole word, like “cat,” or a part of a word, like “in-” or “-ing.” Now, here’s the important part: a token can also represent visual or audible information, a patch of color, or a musical note.
Let’s get a little nerdy. Each token has a mathematical representation of its meaning and context, which looks like a string of numbers. For example, the word “king” might become something like [0.2, 0.8, 0.1, …] with hundreds of numbers for various dimensions and relationships. With tokens encoded in this manner, AI can do math with meaning. It can literally calculate that “king” – “man” + “woman” ≈ “queen” because the numbers encode relationships, not just definitions.
Technically, the model doesn’t care what the token represents. What it cares about is how tokens relate to one another and the patterns that emerge from these relationships.
You can also think of it this way: everything on a computer is ones and zeros. Text, images, music, and videos are all binary at their core. LLMs operate one level above that. They’re pattern-matching machines that can learn relationships between any sequences of tokens.
Everything is language
One of the biggest recent breakthroughs in GenAI occurred when researchers realized that models can process images using language. More specifically, if you can describe images with words, you can teach an AI model to translate between the two.
A milestone model that emerged from this realization was CLIP, and it became a game-changer. Released in 2021, CLIP was trained on millions of images paired with natural-language captions. As a result, it could establish the connection between diverse data points, such as, for example:
- The pixels that make up a photo of a dog
- The word “dog”
- The sentence “a golden retriever playing in a park”
Whenever you ask an AI to “draw a sunset over mountains,” the model isn’t magically learning to paint. It’s doing something much weirder and more interesting:
- It understands your text description (i.e., the language model doing its thing).
- It knows what visual patterns are associated with those concepts (thanks to training on image-text pairs).
- It generates tokens that represent those visual patterns.
- Those tokens get decoded back into pixels.
From LLMs to LMMs: the multimodal revolution
Fast forward to 2026, and the AI landscape is becoming increasingly multimodal. Large multimodal models (LMMs) build on LLMs primarily by incorporating specialized encoders and layers that map non-text data (such as images or audio) into the same numerical “embedding space” that the LLM backbone understands.
LMMs don’t just work with one type of data. Instead, these models enable services like GPT, Gemini, etc., to:
- Read text and images in the same conversation
- Understand charts, diagrams, and screenshots
- Generate responses that reference visual information
Note that the AI is still fundamentally doing the same thing here: finding patterns in sequences. The only difference is that the sequences now include visual tokens and text tokens — all mixed together.
Why this matters
Understanding this helps explain both the power and limitations of AI:
The power: If you can represent something as a sequence of patterns, AI can potentially learn to work with it. Audio, video, 3D models, scientific data — it’s all fair game.
The limitations: AI models don’t truly understand images the way humans do — but they surely can simulate understanding via increasingly sophisticated pattern-matching. AI models know that the word “sunset” correlates with orange and purple colors and the landscape orientation, even though they don’t experience the beauty of a sunset.
One more thing to keep in mind is that working with images is way more resource-intensive than working with text. A single image might require thousands of tokens to represent, while a paragraph might only need a hundred. It’s no secret that training multimodal models requires enormous computing power and energy. AI-based image generation is also far less predictable than text generation. While this may be temporary — and reflective of AI’s ongoing evolution — some people theorize that AI models are actually about to reach their peak.
Conclusion
So if LLMs work with language, how can they generate images?
The short answer is that they don’t really work with language. They work with patterns.
Language just happens to be the easiest pattern to feed into AI. That said, believe me when I say these models train on ALL (and I mean it, ALL) publicly available images.
The moment you start thinking about text, images, and music as different ways of encoding patterns — different languages describing the same universe — that’s when AI stops seeming like magic and starts seeming like… well, something even weirder than magic.
Looking for production-ready AI expertise?
AgileEngine’s expertise spans the development of patented AI-first products that are top-rated by Gartner and G2. If you seek a consultation on AI adoption and need high-skilled experts to drive your initiatives, feel free to contact us — or explore more about our AI Studio and check our case studies.
Grow faster with dedicated teams of top 1% software experts, speeding up time-to-hire by 3x and project delivery by 2x.
Let’s talkFAQ
Generative visual tools like Midjourney and Stable Diffusion are primarily diffusion models rather than standalone LLMs, though they frequently use LLM text encoders (such as CLIP or T5) to parse prompt syntax. While LLM-style autoregressive models predict discrete tokens sequentially, diffusion engines generate media by iteratively removing Gaussian noise from a continuous latent space. Modern generative pipelines increasingly hybridize both approaches to combine the semantic reasoning of transformers with the spatial fidelity of diffusion architectures.
Audio waveforms are converted into tokens using neural audio codecs (such as EnCodec or SoundStream) that compress raw acoustic data or spectrograms into quantized numerical values. These discrete acoustic tokens encode pitch, timbre, dynamics, and temporal cadence without requiring symbolic musical notation or MIDI files. A multimodal transformer predicts these audio tokens sequentially, after which a neural decoder synthesizes the sequence back into playable audio.
Video generation introduces a time dimension that requires temporal consistency across dozens of frames per second, multiplying token volume and memory footprints. Without 3D spatiotemporal attention layers tracking object persistence and motion trajectories, sequential frame prediction results in severe visual flickering, ghosting, and anatomical warping. Modeling realistic physical interactions and camera dynamics across time demands orders of magnitude more compute than rendering a single static frame.
Vision tokenizers, such as VQ-VAEs (Vector Quantized Variational AutoEncoders) or Vision Transformers (ViT), slice an image into a fixed grid of pixel patches, typically 16×16 pixels wide. Each patch is projected into an embedding and mapped to the closest matching vector entry within a learned visual “codebook.” This conversion turns a high-dimensional continuous pixel array into a sequence of discrete token indices that standard transformer attention heads can compute identically to text.
