Master the art of implementing short-term and long-term memory systems for your LLM agents. This comprehensive guide covers best practices, implementation patterns, and optimization strategies for the MemU memory management system.
Short-term and Long-term Memories
Short-term Memory
Input/Output Structure
The short-term memory system follows a straightforward structure for managing conversation context:
- Input: System prompt + Chat history (context) + User query
- Output: LLM/agent response
{"role": "system", "content": "..."}, # system prompt
{"role": "user", "content": "..."}, # chat message history
{"role": "assistant", "content": "..."},
...
{"role": "assistant", "content": "..."}, # chat message history
{"role": "user", "content": "..."}, # current user query
]
When to Use Short-term Memory
- Within a single session
- When the user is actively communicating with the LLM agent
Short-term Memory Features
- Uncompressed format - Maintains full conversation detail
- Chronological order - Preserves the natural flow of conversation
Context Length and Performance
Current LLMs support over 2 million input tokens. In practice, contexts up to ~8,000 tokens don't significantly impact latency because the input KV cache is preserved during inference.
Cache Pricing Comparison
Understanding the cost difference between cache hits and misses is crucial for optimization:
| Input Type | Cost per 1K tokens | Relative Cost |
|---|---|---|
| Standard Input | $0.25 | 10× |
| Cached Input | $0.025 | 1× |
Long-term Memory
Input/Output Structure
Long-term memory uses a more complex structure that combines static and dynamic memory elements:
- Input: [System prompt + Static memory] + Chat history (short-term memory) + [User query + retrieved relevant memories]
- Output: LLM/agent response
{
"role": "system",
"content": "... \nUser's profile: xxx"
}, # system prompt + static memory (profile, etc.)
{
"role": "user",
"content": "..."
}, # chat message
{
"role": "assistant",
"content": "..."
},
...
{
"role": "assistant",
"content": "..."
}, # chat message
{
"role": "user",
"content": "... \n [related memory1]xxx \n [related memory2]xxx \n [related memory3]xxx"
} # user query + context-related memory
]
When Long-term Memory is Needed
- Across multiple sessions
- When user returns after hours/days/weeks
- For persistent user preferences and historical interactions
- When context exceeds practical token limits (>100k tokens, when affecting inference speed)
Long-term Memory Features
- Compressed/summarized memories for efficient retrieval
- Semantic indexing for intelligent retrieval
Best Practices
Conversation Length for Memorization
MemU charges for each API call used in memorization tasks. For the best user experience and minimizing costs, it's best to merge multiple chat messages (short-term memories) into a complete conversation before making an API call.
Memory Categories
Profile / System Category
- Carries each user's basic information
- Generally more static and concise
- We recommend including profile category memories in the system prompt to enable agents to form a complete understanding of each user
Custom/Cluster Categories
- Relatively more scattered
- Relevant categories depend on users' current messages
- Typically take more tokens
- We recommend appending memories from custom and cluster categories to users' queries to provide dynamic and tailored contexts for responding to users' messages
How to Craft LLM Messages
System Prompt Structure
[Original system prompt of the agent]
User's profile:
[Memory retrieved from the Profile Category]
'''
User Query Structure
[Original user query or message]
Relevant Memories:
[Memory retrieved from Custom/Cluster Categories]
'''
Memory Retrieval Guide: When to Use Which Method
Available Methods
1. retrieve_default_categories() ⭐⭐⭐ Most Basic & Highly Recommended
Description:
- Retrieves the default categories from both system-level (e.g., profile, events) and user-defined categories
- Returns the full contents in pre-configured memory categories
- Should be placed into the System prompt for initialization
When to Use: During system initialization or session startup (Only once)
Latency: ~50ms
2. retrieve_related_clustered_categories(category_query) ⭐⭐ Advanced Clustering
Description:
- Retrieves categories that have been automatically clustered based on semantic similarity
- Returns the full contents in auto-generated memory categories
- Should be placed into the System prompt for initialization or dynamically placed into user queries
When to Use:
- When you need to find categories semantically related to a specific topic (In system prompt)
- For advanced semantic search for user queries (append to user query)
Latency: ~200ms
3. retrieve_related_memory_items(query, [include_categories]) ⭐⭐ Context-Specific Retrieval
Description:
- Retrieves specific memory items related to the current context or query
- Returns actual memory content rather than just categories
When to Use: For answering specific queries about past events or information
Latency: ~200ms
Example Use Cases:
- "What did I discuss about project X last month?"
- Retrieving relevant memories for context-aware responses
- Building a timeline of specific events
Example Implementation
system_prompt ← system_prompt + memu_client.retrieve_default_categories()
chat_history ← []
# Chat loop
while True:
user_query ← get_input()
[optional] memories ← memu_client.retrieve_related_memory_items(user_query)
response ← llm(system_prompt + chat_history + user_query + memories)
chat_history.append({"user": user_query, "assistant": response})
memu_client.memorize(chat_history)
Cloud SDK
For more details on implementation, see the MemU SDK Guide (cloud version).
Ready to implement? Start with the basic retrieve_default_categories() method and gradually incorporate more advanced retrieval methods as your application grows.
Get Help & Connect
Join our community and explore the codebase to get help and connect with other developers building with MemU!
Additional Resources:
- Official Website: https://memu.pro
- API Documentation: https://api.memu.so/docs
- Email Support: contact@nevamind.ai