remii Watchdog Architecture: A Deep Dive

AI workflows are inherently unpredictable. When an agent is tasked with a complex goal—such as researching a company, scraping their website, summarizing the findings, and inserting the data into a CRM—it relies on a fragile chain of external dependencies.

If the target website takes too long to load, the scraping tool might hang. If the CRM API returns a 502 Bad Gateway error, the tool might throw an exception. To build a reliable AI employee that can operate 24/7 without human supervision, remii utilizes a highly resilient state machine architecture managed by continuous **Watchdog Reconcilers**.

### The Problem with Ephemeral Processes

In standard web applications, a user makes a request, the server processes it for a few milliseconds, and returns a response. If the server crashes during those milliseconds, the user simply refreshes the page.

Background AI agents cannot rely on the user to "refresh." If a background task is running for 45 minutes and the Node.js process crashes due to an Out-Of-Memory (OOM) error, that execution is lost forever. To solve this, remii abandons ephemeral in-memory processing in favor of durable state machines.

### Durable State Machines

Every agent workflow is broken down into a series of discrete state transitions logged in a PostgreSQL database. - **States**: `QUEUED`, `PLANNING`, `EXECUTING_TOOL`, `AWAITING_REPLY`, `COMPLETED`, `FAILED`. - **Idempotency**: Before an agent executes a tool (like sending an email), it commits the intent to the database. If the process crashes immediately after sending the email but before logging the success, the recovery system knows exactly where the loop stopped and can prevent sending duplicate emails.

### The Role of the Watchdog Reconciler

While the state machine provides the ledger of truth, we need an active mechanism to enforce progress. This is the Watchdog Reconciler.

The Watchdog is a separate, highly available background daemon. It does not run agent code; its only job is to monitor the state database and look for anomalies.

#### 1. Detecting Hung Processes Every 30 seconds, the Watchdog runs an audit query. It looks for tasks that are currently marked as `EXECUTING_TOOL` but haven't updated their heartbeat timestamp in over 5 minutes. If a web scraping sandbox encounters a massive infinite-scroll page and hangs, the Watchdog detects this latency anomaly.

#### 2. Process Termination Once a hung task is detected, the Watchdog assumes the execution container is compromised or frozen. It issues a forceful kill command to the specific Kubernetes pod or Docker container hosting that sandbox, ensuring that CPU and memory resources are instantly freed up and preventing the system from locking up.

#### 3. The Recovery Loop After terminating the frozen container, the Watchdog initiates a recovery loop: - It rolls the task state back from `EXECUTING_TOOL` to `QUEUED`. - It applies an exponential backoff penalty (e.g., waiting 60 seconds before allowing the task to be picked up again). - When a healthy worker node picks up the task, it resumes from the last successfully committed state, trying the tool call again (perhaps with a strict timeout parameter passed to the LLM).

### Preventing Infinite Loops

A common issue with autonomous agents is getting stuck in logical infinite loops. For example, the agent tries to query a database with a syntax error, receives an error, tries the exact same query again, receives an error, and repeats this forever, burning through API tokens.

The Watchdog tracks the number of sequential tool failures. If an agent fails the same tool call 3 times in a row, the Watchdog intervenes. It forcefully pauses the execution, changes the state to `NEEDS_HUMAN_INTERVENTION`, and sends a Slack alert to the user.

### Conclusion

By decoupling execution logic from state storage and employing active Watchdog Reconcilers, remii ensures that background workflows are virtually indestructible. Whether faced with network latency, API outages, or logical LLM loops, the system autonomously detects, recovers, and resumes operations, delivering true enterprise-grade reliability.