Context and Memory in LLMs

Today we'll answer a very important question:

How can an AI remember what you said earlier, understand your current question, and use information from a long conversation?

Imagine you're building an AI assistant for your Odin drilling management application. You ask:

1. "Explain our demand dashboard."

2. "Now explain how it calculates inventory."

3. "Can you suggest ways to improve it?"

To answer the third question properly, the AI needs to understand what "it" refers to and may need information from the earlier discussion.

This brings us to two fundamental concepts: context and memory.

1. What is context?

Context is the information available to the model when it generates a response.

It may include:

• Your current question.

• Earlier messages in the conversation.

• Instructions about how the AI should respond.

• Relevant documents or data supplied by your application.

• Results returned by tools.

• Other information included in the current model request.

Think of context as the information on your desk while solving a problem.

If you provide the SQL query and schema, the AI has more relevant information to work with. If you provide only "Optimize this query" without the query itself, it may need to ask you for more information.

Key takeaway: context helps the model understand what you mean and what information it should use.

2. What is memory?

Memory is the ability of an AI application or system to retain or retrieve information for use in later interactions.

For example, imagine that an AI tutor learns these preferences during earlier conversations:

• You develop applications using C# and .NET.

• You prefer practical examples over purely theoretical explanations.

• You are learning AI step by step.

If the system retains those details and makes them available later, you might not need to repeat them every time.

Memory is not necessarily a capability stored inside the LLM itself. It can be implemented by the surrounding application, a product's memory feature, a database, or a combination of these.

Context vs memory

An important distinction: a conversation can provide context without creating persistent memory. Likewise, stored memory only helps if the relevant information is retrieved and made available when needed.

3. What is a context window?

In Lesson 2, we discussed tokens. Now let's connect them to context.

A context window is the maximum amount of tokenized information a model can process within the applicable request or conversation context.

That information can include input messages, instructions, tool results, and sometimes the space reserved for the generated answer, depending on the model and API.

Imagine a model with a hypothetical context limit of 20,000 tokens.

Why does this matter?

Suppose you have a very long conversation with an AI. Eventually, the entire conversation may no longer fit in the context window.

Depending on the application, it might:

• Include only recent messages.

• Summarize older messages.

• Retrieve selected information from storage.

• Remove less relevant information.

• Ask you to provide missing information again.

This is why a longer context window doesn't automatically mean unlimited memory.

4. What is conversation history?

Conversation history is the sequence of messages exchanged between the user and the assistant, potentially including tool calls and tool results.

For example:

User · Message 1

Explain dependency injection in C#.

Assistant · Message 1

Dependency injection allows a class to receive the services it needs instead of creating those services itself.

User · Message 2

Show me an example using an interface.

Assistant · Message 2

The assistant can use the earlier discussion to understand that you want a C# dependency-injection example.

Conversation history can help resolve references such as "it", "that method", or "show me another example."

However, the model can only use history that is available to it in the current context. A chat interface may retain a long conversation while sending only a selected or transformed portion to the model for a particular request.

5. Types of memory in AI applications

There is no single universal classification, but a useful engineering model divides memory into four categories.

These categories can overlap. For example, a project decision can be stored as persistent memory and also retrieved as part of a knowledge search.

A useful distinction: memory versus model training

If you tell an AI, "I prefer concise C# examples," that information may be saved and supplied in a later request.

It does not necessarily mean the underlying model has been retrained or that its parameters have changed.

Think of it this way:

• Training: changes the model's learned parameters.

• Memory: retains or retrieves information for future use.

• Context: supplies information for the current generation.

These are three different mechanisms.

6. How does RAG help an AI remember information?

RAG stands for Retrieval-Augmented Generation.

It is a technique in which an application retrieves relevant information from an external knowledge source and supplies it to an LLM to help generate an answer.

Suppose an application has thousands of pages of technical documentation. You ask:

What is the procedure for updating a material's category?

Instead of putting every document into every request, your application can retrieve the most relevant documentation and include it in the prompt.

1. User asks a question - "How do I update the material's category?"

2. Retrieval - Search relevant documentation, database records, or indexed content.

3. Augment the prompt - Provide the question and relevant retrieved information to the LLM.

4. Generate the answer - Explain the procedure using the supplied information.

RAG is useful because it can provide information that wasn't present in the model's original training data, including updated or private documentation that the application is authorized to access.

However, retrieval isn't perfect. The system can retrieve irrelevant or incomplete documents, and the model can still misinterpret the evidence. Good retrieval, access controls, source citations, and answer evaluation remain important.

7. How would you implement context and memory in C#/.NET?

Let's look at a simplified architecture for a production-style AI assistant.

In this architecture, the LLM does not need direct unrestricted access to your database. Your .NET backend controls which information is retrieved and what actions are permitted.

A few design principles worth remembering:

• Store conversation messages separately from durable user preferences and project facts.

• Retrieve only information relevant to the current request.

• Enforce user permissions before retrieving private data.

• Keep context within the model's token limits.

• Don't automatically save every statement as permanent memory.

• Give users appropriate control over retained information.

This architecture is a useful foundation for the AI agents we'll study later.

Lesson summary

You now understand the difference between four related concepts:

Concept Simple meaning

Context What information is available for the current response

Conversation history The messages exchanged during a conversation

Persistent memory Selected information retained for future interactions

RAG Retrieving relevant external information to support an answer

Next lesson: Prompt Engineering Fundamentals.

We'll learn how to write effective prompts, how zero-shot, one-shot, few-shot, and many-shot prompting differ, and how examples can guide an LLM's output. Then we'll move to chain-of-thought techniques and your RTCF prompt framework.