Tokens and Tokenization

Today, we're going to understand one of the most fundamental concepts in Large Language Models (LLMs): tokens.

By the end of this lesson, you'll understand:

• What tokens are and how tokenization works.

• Why one word doesn't always equal one token.

• How an LLM processes your prompt.

• How tokens affect context limits, response length, speed, and cost.

• How to estimate tokens when building AI applications in C#/.NET.

1. What is a token?

A token is a small unit of text that an AI model processes. Depending on the tokenizer, a token can represent a whole word, part of a word, punctuation, or another piece of text.

Think of tokens as building blocks of language.

For example, the sentence:

I love learning AI.

could be split into tokens like this:

In this example, there are five displayed pieces, but the actual token count depends on the tokenizer used by the model.

Remember: tokens are not the same thing as words, characters, or spaces.

2. How does tokenization work?

Tokenization is the process of converting text into tokens that a model can process.

Let's follow a simple example.

Let's clarify two terms:

• Token: the piece of text identified by the tokenizer.

• Token ID: the numeric identifier assigned to that token in the model's vocabulary.

The model doesn't simply read the original sentence as a human would. Text is tokenized, converted into numeric IDs and representations, and processed by the neural network.

When the model generates an answer, it produces tokens that are then converted back into readable text.

3. Why isn't one word always one token?

This is one of the most important concepts to understand.

An LLM tokenizer doesn't necessarily split text according to dictionary words. Many modern tokenizers use subword methods, allowing common text pieces to be represented efficiently.

Consider these illustrative examples:

Why does this happen?

The tokenizer has a vocabulary of known tokens. Common words or word pieces may be represented efficiently, while uncommon text can require multiple tokens.

The exact tokenization depends on the model and its tokenizer. Two different models can tokenize the same sentence differently.

4. How many tokens are in a sentence?

As a rough rule of thumb for ordinary English text, one token often corresponds to about ¾ of a word, or roughly four characters of text. This is only an estimate, not a rule.

Let's compare:

This is a simplified example, not the specification of any particular model. Actual limits and accounting depend on the model and API.

Important distinction: a context window is not the same as long-term memory.

• Context window: information available in the current model request or conversation context.

• Memory: information that may be retained or retrieved across interactions, depending on the product or application.

• Training data: data used to train the model; this is different from information supplied in your current prompt.

We'll study memory separately in a future lesson.

6. Why do tokens matter in real AI applications?

Tokens affect at least four important things.

1. Cost

Many LLM APIs charge based on input and output tokens, with different rates for different models and token categories. Larger prompts and longer answers can cost more.

2. Speed

More input tokens can increase processing time, and generating more output tokens generally takes longer.

3. Context limits

If the relevant input and requested output exceed the applicable limits, you may need to shorten, summarize, or split the content.

4. Output length

Output-token limits can restrict how long a generated answer can be. A low output limit may cause an answer to stop before completion.

Example: An AI feature in your .NET application

Imagine an endpoint that sends 20,000 tokens of source code to an LLM and asks it to review the code.

If the model's applicable input limit is smaller than the full request, you cannot simply send everything and expect it to work. You may need to select relevant files, summarize dependencies, or retrieve only the most relevant code.

This is one reason techniques such as RAG (Retrieval-Augmented Generation) are useful: instead of sending every document, an application retrieves relevant information for the current question.

Also, tokenization alone does not determine the quality of the answer. The relevance of the input, the model, the instructions, and other factors matter too.

7. How can you count tokens in practice?

You don't need to count tokens manually. Use a tokenizer compatible with the model you are working with.

For example, you can use the OpenAI Tokenizer to experiment with how text is divided into tokens.

Open the OpenAI Tokenizer.

Try entering these examples:

• Hello world

• unbelievable

• GetMaterialById()

• A paragraph in Hindi.

• A long JSON object.

Observe how the token count changes.

For your future C# applications, use the model provider's documented tokenizer or token-usage information rather than relying on a rough character-count estimate.

Revision

1. A token is a piece of text processed by an LLM.

2. Tokenization converts text into tokens; token IDs represent those tokens numerically.

3. A word can correspond to one token or multiple tokens.

4. Token counts affect context usage, API cost, and generation time.

5. A context window is not the same as persistent memory.

Next lesson: Temperature, Top-P, and Top-K. You'll learn how these settings influence token selection and why they matter when generating creative content versus reliable, structured responses.