How Vector Memory Tables Give AI Employees Long-Term Context

A major limitation of modern Large Language Models (LLMs) is their lack of long-term memory. A standard chat session begins with a clean slate. Once the user closes the browser or clears the session, all background context—such as preferred coding frameworks, business guidelines, writing styles, and project parameters—is lost. To function as a reliable AI employee, an assistant must retain context over days, weeks, and months, just like a human colleague.

remii solves this limitation using a real-time vector memory pipeline that extracts, indexes, queries, and secures contextual facts derived directly from user conversations.

### The Limitations of Context Windows

While LLM context windows have expanded dramatically (with some models supporting millions of tokens), simply stuffing the entire history of every conversation into the context window for every prompt is wildly inefficient. 1. **Latency**: Processing 100,000 tokens of chat history takes significantly longer than processing 2,000 tokens of relevant context. 2. **Cost**: API costs scale linearly (or worse) with token usage. 3. **Lost in the Middle**: Research shows that LLMs often struggle to retrieve specific facts buried deep within massive context windows.

Instead, remii uses a Retrieval-Augmented Generation (RAG) pattern, injecting only the exact memory fragments relevant to the current task.

### Real-Time Memory Extraction Pipeline

Instead of relying on users to write system instructions manually, remii runs an asynchronous background worker that parses chat sessions. When a conversation turn finishes, the background worker uses a lightweight LLM pipeline to determine if the user shared persistent context.

For example, if a user writes: *"Our staging server is located at stage.remii.space and we use the dark theme for the admin panel,"* the extraction model converts this unstructured text into structured facts: - **Fact 1**: The staging server URL is stage.remii.space. - **Fact 2**: The admin panel utilizes a dark theme UI.

Once a fact is verified, it is sent to an embedding model (like OpenAI's `text-embedding-3-small`) to generate a high-dimensional vector embedding (e.g., 1536 dimensions). This vector represents the semantic meaning of the text rather than just the exact keywords used.

### pgvector Storage and Cosine Similarity

We store these vector embeddings inside a dedicated PostgreSQL cluster utilizing the open-source `pgvector` extension. This allows us to perform fast, scalable vector similarity searches using standard SQL queries alongside our relational application data.

When the user asks a new question, such as *"Can you deploy the latest build to staging?"*, the system: 1. Generates an embedding for the user's prompt. 2. Queries the PostgreSQL database to find the closest matching vectors using the cosine distance operator (`<=>`). 3. Retrieves 'Fact 1' (The staging server URL is stage.remii.space).

To ensure high performance, we use **HNSW (Hierarchical Navigable Small World)** indexes on our vector columns. HNSW provides approximate nearest neighbor (ANN) search, ensuring that similarity search latency remains below 10ms even as the memory table scales to millions of records.

### Context Decay and Conflict Resolution

Human memory is dynamic; facts change over time. If a user previously stated they use React, but today they say they are migrating to Vue, the memory system must adapt. remii implements **Time-Weighted Context Scoring**. When querying memories, the system calculates a final relevance score based on: - Semantic similarity (cosine distance). - Recency (when the memory was created or last accessed). - Access frequency (how often this memory has been successfully utilized).

When contradictory facts are retrieved, the system uses an LLM-based conflict resolution step to prioritize the most recent and explicit instruction, ensuring the agent doesn't act on outdated information.

### Enforcing Row-Level Security (RLS)

Because vector databases contain private and confidential user data, isolation is critical. In a multi-tenant SaaS application, a bug in the application logic must never result in one user's memories leaking to another user.

We enforce strict PostgreSQL Row-Level Security (RLS) policies at the database level. Every query is scoped to the user's authenticated ID.

```sql CREATE POLICY user_isolation_policy ON vector_memories FOR ALL USING (user_id = auth.uid()); ```

An agent query can never view, match, or leak memory records belonging to a different tenant, even if the application code attempts to perform a global `SELECT` without a `WHERE` clause.

### Conclusion

By combining asynchronous extraction, HNSW-indexed pgvector storage, time-weighted retrieval, and robust RLS policies, remii gives AI agents persistent, secure, and highly relevant long-term memory. This architecture transforms AI tools from forgetful chatbots into capable, context-aware digital employees.