Two-Phase Constrained Decoding for Small Local Agents

Small local reasoning models like VibeThinker-3B are surprisingly capable at multi-step problem solving, yet notoriously fragile as tool-calling agents. Ask a 3B model to reason in free text and emit structured tool JSON inside the same generation—especially at non-zero temperature—and syntax breaks, argument keys drift, or the model abandons the schema mid-object.

While building VibeHarness, a dependency-free Ralph-loop coding agent backed by Ollama, the single biggest reliability jump came from splitting every agent turn into two distinct model calls instead of forcing one call to do both jobs.

Separating Thinking from Acting

Adapting the structural-tag pattern to Ollama’s HTTP API cleanly isolates free-form reasoning from schema-enforced execution:

 task + plain-English narrative of past actions
        |
        v
 +- phase 1: free reasoning -----+  /api/chat, stop at </think>
 |  <think> ... </think>         |  (streamed, then discarded)
 +-------------------------------+
        |
        v
 +- phase 2: constrained action -+  /api/generate (raw),
 |  [{"tool":"...","args":{}}]   |  format = tool array schema
 +-------------------------------+
        |
        v
 parse -> execute via ToolRegistry -> append observation
        |
        +------> repeat until `finish` or step budget

In Phase 1, the model thinks without schema constraints over /api/chat, stopping cleanly at </think>. Users watch the live stream in the terminal, and the harness writes the complete reasoning trace to .vibe/<stamp>.json and .vibe/<stamp>.md. Crucially, that raw <think> trace is dropped from the running context before the next turn so working memory stays focused on facts rather than earlier hypotheses.

In Phase 2, Ollama’s format JSON schema constraint takes over as a raw continuation on /api/generate. Because decode-time grammar masking only permits valid tokens for the active ToolRegistry schema, malformed JSON actions become structurally impossible—even if you raise --temp 1.0.

Single Source of Truth in ToolRegistry

Schema drift happens when prose tool instructions in the system prompt diverge from the JSON grammar enforced at decode time. VibeHarness avoids this by generating both from one Python definition:

# Every tool declares its parameters once inside ToolRegistry.
# System prompt documentation AND Ollama's decode-time `format`
# JSON schema are derived dynamically from the same Param specs.

TOOLS = [
    "list_directory",  # path?, recursive?
    "read_file",       # path
    "write_file",      # path, content, mode? (overwrite/append/prepend)
    "search",          # query, path?, target?, max_results?
    "manage_path",     # action (mkdir/delete/move), path, destination?
    "finish",          # summary
]

Keeping the surface small and widening capability through optional parameters (write_file.mode, browse.action) keeps grammar branching compact for small models. A confident turn can also batch multiple actions—such as writing a file and immediately reading it back—inside one validated array.

Plain-English Narrative Memory

Most coding agents feed turns back to the model as raw assistant/tool ChatML JSON. On 3B local weights, deeply nested JSON history quickly wastes context tokens and degrades instruction following.

NarrativeMemory replaces raw tool transcripts with a concise English chronicle ("First, you read pyproject.toml... Then, you wrote CHANGELOG.md..."), while archiving full structured arguments and reasoning traces to disk inside .vibe/.

Pluggable Toolsets Without Runtime Bloat

The core harness uses zero third-party runtime dependencies—just Python 3.10+ standard library talking to Ollama over HTTP. When a task needs a stateful browser, --toolset web,fs composes the Playwright CLI adapter alongside filesystem tools without altering the core loop:

# Filesystem only (zero external runtime dependencies)
vibe "Create notes.txt containing 'hello hello hello', then read it back to verify."

# Compose stateful browser + filesystem toolsets
vibe --toolset web,fs "Inspect the page links and write a Markdown summary."

Because RalphAgent depends only on LLMClient and Reporter interfaces, the entire multi-action loop runs offline under unittest with a deterministic fake client—and live integration suites verify real Ollama streaming and browser navigation whenever those services are running.