Prompting
How Daydreams structures prompts to guide LLM reasoning and actions.
What is a Prompt?
A prompt is the text you send to an AI model to tell it what to do. Think of it like giving instructions to a smart assistant.
Simple Prompts vs Agent Prompts
Simple Prompt (ChatGPT style)
User: What's the weather in New York?
Assistant: I don't have access to real-time weather data...Agent Prompt (what Daydreams creates)
You are an AI agent that can:
- Call weather APIs
- Send Discord messages
- Remember conversation history
Current situation:
- User asked: "What's the weather in New York?"
- Available actions: getWeather, sendMessage
- Chat context: user123 in #general channel
Please respond with:
<action_call name="getWeather">{"city": "New York"}</action_call>
<output type="discord:message">It's 72°F and sunny in New York!</output>The Problem: LLMs Need Structure
Without structure, LLMs can't:
- Know what tools they have available
- Remember previous conversations
- Follow consistent output formats
- Handle complex multi-step tasks
Example of what goes wrong:
User: "Check weather and send to Discord"
LLM: "I'll check the weather for you!"
// ❌ Doesn't actually call any APIs
// ❌ Doesn't know how to send to Discord
// ❌ Just generates textThe Solution: Structured Prompts
Daydreams automatically creates structured prompts that include:
- Available Tools - What the agent can do
- Current State - What's happening right now
- Response Format - How to respond properly
- Context Memory - What happened before
Available Actions:
- getWeather(city: string) - Gets current weather
- sendDiscord(message: string) - Sends Discord message
Current Context:
- User: user123
- Channel: #general
- Previous messages: [...]
New Input:
- "Check weather in Boston and send to Discord"
Respond with XML:
<action_call name="getWeather">{"city": "Boston"}</action_call>
<output type="discord:message">Weather in Boston: 65°F, cloudy</output>How Daydreams Builds Prompts
Every time your agent thinks, Daydreams automatically builds a prompt like this:
1. Instructions
You are an AI agent. Your job is to:
- Analyze new information
- Decide what actions to take
- Respond appropriately2. Available Tools
<available-actions>
<action name="getWeather">
<description>Gets current weather for a city</description>
<schema>{"type": "object", "properties": {"city": {"type": "string"}}}</schema>
</action>
</available-actions>
<available-outputs>
<output type="discord:message">
<description>Sends a message to Discord</description>
<schema>{"type": "string"}</schema>
</output>
</available-outputs>3. Current Context State
<contexts>
<context type="chat" key="user123">
Previous messages:
user123: Hi there!
agent: Hello! How can I help?
user123: What's the weather like?
</context>
</contexts>4. What Just Happened
<updates>
<input type="discord:message" timestamp="2024-01-15T10:30:00Z">
What's the weather in Boston?
</input>
</updates>5. Expected Response Format
Respond with:
<response>
<reasoning>Your thought process here</reasoning>
<action_call name="actionName">{"argument": "value"}</action_call>
<output type="outputType">Your response here</output>
</response>What the LLM Sees (Complete Example)
Here's what a complete prompt looks like:
You are an AI agent. Analyze the updates and decide what to do.
<available-actions>
<action name="getWeather">
<description>Gets current weather for a city</description>
<schema>{"type": "object", "properties": {"city": {"type": "string"}}}</schema>
</action>
</available-actions>
<available-outputs>
<output type="discord:message">
<description>Sends a message to Discord</description>
<schema>{"type": "string"}</schema>
</output>
</available-outputs>
<contexts>
<context type="chat" key="user123">
user123: Hi there!
agent: Hello! How can I help?
</context>
</contexts>
<working-memory>
<!-- Previous actions from this conversation -->
</working-memory>
<updates>
<input type="discord:message" timestamp="2024-01-15T10:30:00Z">
What's the weather in Boston?
</input>
</updates>
Respond with:
<response>
<reasoning>Your thought process</reasoning>
<action_call name="actionName">{"arg": "value"}</action_call>
<output type="outputType">Your message</output>
</response>LLM Response Example
The LLM responds with structured XML:
<response>
<reasoning>
The user is asking about weather in Boston. I should:
1. Call the getWeather action to get current conditions
2. Send the result to Discord
</reasoning>
<action_call name="getWeather">{"city": "Boston"}</action_call>
<output type="discord:message">Checking the weather in Boston for you!</output>
</response>Daydreams automatically:
- Parses the
<action_call>and runs the weather API - Parses the
<output>and sends the Discord message - Saves the
<reasoning>for debugging
Advanced Features
Template References
LLMs can reference previous action results within the same response:
<response>
<reasoning>I'll get weather, then send a detailed message</reasoning>
<action_call name="getWeather">{"city": "Boston"}</action_call>
<output type="discord:message">
Weather in Boston: {{calls[0].temperature}}°F, {{calls[0].condition}}
</output>
</response>The {{calls[0].temperature}} gets replaced with the actual weather data.
Multi-Context Prompts
When multiple contexts are active:
<contexts>
<context type="chat" key="user123">
Chat history with user123...
</context>
<context type="game" key="session456">
Current game state: level 5, health 80...
</context>
</contexts>Key Benefits
- Consistency - All agents use the same reliable prompt structure
- Clarity - LLMs always know what tools they have and how to use them
- Memory - Context and conversation history included automatically
- Debugging - You can see exactly what the LLM was told
- Extensibility - Easy to add new actions and outputs
Prompt Architecture
Daydreams uses a modular prompt system defined in packages/core/src/prompts/main.ts:
// Template sections with placeholders
export const templateSections = {
intro: `You are an expert AI assistant...`,
instructions: `Follow these steps to process...`,
content: `## CURRENT SITUATION\n{{currentTask}}...`,
response: `Structure your response: <response>...`,
footer: `Guiding Principles for Your Response...`
};
// Formatter processes data into prompt sections
export function formatPromptSections({
contexts, outputs, actions, workingMemory
}) {
return {
currentTask: xml("current-task", undefined, unprocessedLogs),
contextState: xml("context-state", undefined, contexts),
actions: xml("available-actions", undefined, actions),
// ... more sections
};
}
// Main prompt combines template + formatter
export const mainPrompt = {
template: promptTemplate,
sections: templateSections,
render: (data) => render(template, data),
formatter: formatPromptSections
};Context Customization
Contexts can customize their contribution to prompts:
const chatContext = context({
type: "chat",
schema: z.object({ userId: z.string() }),
// Custom instructions for this context type
instructions: `You are a helpful chat assistant. Be friendly and conversational.`,
// Custom rendering of context state in prompts
render: (state) => `
Chat Context: ${state.id}
User: ${state.args.userId}
Messages: ${state.memory.messages?.length || 0}
Last active: ${new Date(state.memory.lastActive || Date.now()).toLocaleString()}
`
});Key Takeaways
- Prompts are automatically generated - You don't write them manually
- Structure enables capabilities - Tools, memory, and context included automatically
- LLMs respond with XML - Parsed automatically into actions and outputs
- Templates enable complex flows - Reference previous results within responses
- Customizable per context - Add specific instructions and state rendering
The prompting system is what makes your agent intelligent - it provides the LLM with everything needed to understand the situation and respond appropriately.