Orchestration Memory System
Initially, SlateDB should be thought of as one layer of the memory system, not the whole memory system.
A serious long-term orchestration memory system probably needs at least four distinct capabilities: 1. Durable event/state storage 2. Semantic/vector retrieval 3. Structured metadata/querying 4. Memory consolidation and lifecycle management
SlateDB is potentially very strong for the first one. Its design is an object-storage-native embedded LSM tree: WAL, memtables, SSTables, manifests and compaction all ultimately live in object storage. That gives you durable, very large key/value storage without tying memory to the lifetime or disks of an individual agent host.
For an orchestration system, the architecture should be something like:
ORCHESTRATOR
│
┌─────────────┼──────────────┐
│ │ │
Agent A Agent B Agent C
│ │ │
└─────────────┼──────────────┘
│
Memory API
│
┌──────────────┼───────────────┐
│ │ │
▼ ▼ ▼
Event Store Semantic Store Metadata Store
SlateDB LanceDB/vector SQLite/Postgres
│ │ │
└──────────────┼───────────────┘
│
Memory Manager
│
summarize / merge / rank
forget / archive / promote
1. SlateDB as the durable memory substrate
Every significant orchestration should be stored in the following containers:
agent.started
agent.completed
prompt.submitted
model.response
tool.called
tool.result
observation.created
memory.created
memory.updated
evaluation.completed
decision.made
plan.created
plan.modified
For example, a SlateDB key might look like:
events/2026/08/13/agent-007/00000018493
With a value containing something like:
</> JSON
{
"type": "model.response",
"agent_id": "agent-007",
"model": "gpt-x",
"conversation_id": "c-88431",
"timestamp": "...",
"prompt_id": "p-1921",
"content": "...",
"tokens": 1854,
"latency_ms": 920,
"confidence": 0.82
}
That gives you something extremely important for an agent system: a durable historical record independent of whatever higher-level memory abstractions you invent later.
SlateDB also exposes change-data capture through its WAL (Write Ahead Log) reader, which is particularly attractive here. You could treat new writes as a durable stream and have downstream processes generate embeddings, summaries, evaluations and derived memories asynchronously from that stream.
2. Don't make embeddings your actual memory
This is an architectural mistake that should be avoided. A vector database should be an index into memory, rather than memory itself.
For example:
SlateDB
--------------------------------------------------
Memory ID M00018721
Complete original event
Complete model output
Metadata
Source
Timestamp
Relationships
Evaluation
etc.
│
│ derive embedding
▼
Vector Index
--------------------------------------------------
M00018721 → [0.142, -0.394, 0.821, ...]
Then a semantic search returns something like:
M00018721
M00003381
M00091822
...and you retrieve the authoritative records from the durable store.
That separation gives you a major long-term advantage: you can replace the embedding model and rebuild the vector index without losing memory.
LanceDB is one interesting local option because its OSS version can run embedded from a local directory, supports vector search and can generate embeddings at ingestion.
3. Keep several kinds of memory
Explicitly model memory categories instead of throwing everything into one vector namespace.
Do Something like:
Memory
│
├── Episodic
│ ├── conversations
│ ├── model calls
│ ├── agent actions
│ └── tool interactions
│
├── Semantic
│ ├── learned facts
│ ├── concepts
│ ├── user preferences
│ └── domain knowledge
│
├── Procedural
│ ├── successful workflows
│ ├── scripts
│ ├── prompts
│ └── agent strategies
│
├── Evaluative
│ ├── successes
│ ├── failures
│ ├── scores
│ └── human feedback
│
└── Working Memory
├── current task
├── active plan
├── temporary observations
└── agent scratch state
This is where your idea gets considerably more interesting than ordinary RAG.
You are no longer merely asking:
"What documents are semantically similar to this prompt?"
You can ask:
"What has this orchestration system learned from having performed similar tasks hundreds of times?"
That is much closer to experience.
4. Add an explicit memory consolidation process
This is probably the component that will differentiate a sophisticated system.
Raw agent activity might produce:
100,000 observations
↓
20,000 candidate memories
↓
5,000 useful episodic memories
↓
700 recurring patterns
↓
120 durable learned concepts
A consolidation service could periodically examine old memories and perform operations such as:
PROMOTE
"This fact has appeared repeatedly."
MERGE
"These 37 observations describe essentially the same thing."
SUMMARIZE
"Compress this week of agent activity into a durable episode."
DECAY
"This information hasn't been useful for six months."
INVALIDATE
"New evidence contradicts the old memory."
GENERALIZE
"Five successful solutions imply a reusable procedure."
That would prevent the common agent-memory problem where the database simply becomes an enormous pile of embeddings.
5. Track provenance aggressively
Every learned memory should retain its origin.
For example:
Memory: M92817
statement:
"Technique X usually works better than Y for this workload."
derived_from:
event-11827
event-11842
event-11891
event-12003
created_by:
memory-consolidator-v3
confidence:
0.91
first_observed:
2026-03-18
last_confirmed:
2026-08-12
contradictions:
event-11933
times_retrieved:
84
times_helpful:
72
That gives you something current LLM memory systems often lack: auditable learned experience.
- Store feedback as first-class data
This should be made central.
For every retrieved memory:
memory_id
retrieved_at
used_by_agent
task_id
retrieval_score
agent_selected
result_success
human_feedback
later_evaluation
You can then learn:
Memory M1832
retrieved 492 times
used 327 times
helpful 301 times
→ very valuable
versus:
Memory M7721
retrieved 193 times
used 4 times
helpful 1 time
→ probably noise
Over months, that creates a memory system shaped by operational experience rather than merely semantic similarity.
Where SlateDB becomes especially attractive:
SlateDB's object-store design would allow the raw historical layer to grow extremely large while keeping the compute layer relatively disposable. It also supports optional caching to mitigate object-store latency, and compaction can be separated from the writer process.
For a local system you could therefore have:
Agent Machines
│
▼
Memory Service
│
┌──────────────┼──────────────┐
▼ ▼ ▼
SlateDB LanceDB SQLite
│
▼
MinIO
│
▼
ZFS Pool
That is a particularly appealing architecture because MinIO + ZFS gives you an entirely local object-store-backed SlateDB deployment while preserving the architecture you'd use later with S3 or another cloud object store.
One design decision that should be made early:
Give every piece of information a permanent global identifier:
mem_01JZX4P9M7A...
evt_01JZX4PCD21...
run_01JZX4PE983...
agent_01JZX4PG12...
...and represent relationships explicitly:
event
├── caused_by → event
├── part_of → run
├── generated_by → agent
├── derived_memory → memory
└── references → memory
Once you do this, you're quietly beginning to build something closer to a temporal knowledge graph of agent experience and that is going to be ultimately more interesting than simply building a vector database.
The architecture that should be investigated
For building the first prototype, start with:
C# or Rust Orchestrator
│
▼
Memory Service
│
┌──────┼────────┐
▼ ▼ ▼
SlateDB LanceDB SQLite
│
▼
MinIO
│
▼
ZFS
Starting with five basic APIs:
Remember(event)
Recall(query, context)
Observe(result)
Evaluate(memory, outcome)
Consolidate()
Evolve the rest of the system around those APIs.
The particularly powerful idea is that the orchestration system itself can become progressively better at deciding what deserves to become memory. At that point, you're no longer merely persisting agent state, you're building an experiential storage layer for the entire orchestration environment.
This is a sufficiently interesting architecture that the next useful step would be to design the actual storage schema and key structure — including SlateDB keyspaces, event records, memory records, embeddings, provenance, feedback and consolidation records. That would reveal very quickly whether SlateDB should be the primary substrate or just one component.