The 3-Layer Verification Engine

Autonomous AI models are fundamentally prone to Premature Victory. When an LLM completes an edit, its next generated tokens will almost always declare: "I have implemented all the requested features and verified that everything works perfectly!"

In reality, the model has no sensory apparatus inside its neural weights to know if the code compiles, let alone runs.

Imagine a student taking a math exam who writes down an answer, immediately closes the booklet, and declares with total confidence that they got a 100% score—without ever re-reading the question or checking their arithmetic.

The 3-Layer Verification Engine is the strict teacher standing over the desk: it refuses to accept the student's word. It runs 3 automated tests in sequence—syntax check, unit test, and behavioral probe—and only allows the agent to submit when all three pass with zero warnings.

1. The Interactive Verification Simulator

Test how the harness intercepts failures and forces agent self-correction across the 3 deterministic tiers:

Test Scenario

LAYER 1: STATIC✓ PASSED

TypeScript & Linter Audit

ruff check . && mypy --strict

Clean: All 24 Python modules passed ruff lint & mypy strict checks.

LAYER 2: UNIT✕ FAILED

Deterministic Test Suite

pytest tests/test_attention.py -v

FAILED tests/test_attention.py::test_qkv_projection_shape - AssertionError: assert (2, 1, 512) == (2, 8, 64). Dropped num_heads dimension.

LAYER 3: RUNTIMESKIPPED

End-to-End Behavioral Smoke

python -m tests.e2e.test_smoke

Skipped: Unit test assertions failed.

⚡ HARNESS FEEDBACK INTERCEPTEDTriggering Agent Self-Correction Turn

The model attempted to conclude the turn, but the harness blocked premature completion. The diagnostic output above was automatically injected into the next prompt turn, compelling the agent to resolve the regression.

The 3-Layer Verification Rule: A change is never declared complete until it survives Static Audit (Layer 1), Unit Assertion (Layer 2), and Behavioral Smoke Execution (Layer 3). When any layer fails, the harness catches the error and feeds actionable feedback into the agent's next turn.

Predict First

Why must a harness always execute Layer 1 (ruff check and mypy typechecking) before triggering Layer 2 (pytest unit tests)?

2. The Verification Hierarchy

Verification is structured into three progressive gates. Each layer is cheaper and faster than the next, creating an efficient filtering funnel:

        ┌──────────────────────────────────────────────┐
        │       LAYER 1: STATIC AUDIT (TYPECHECK)      │
        │ • Time: ~200ms                               │
        │ • Tools: ruff check ., mypy --strict         │
        │ • Catches: syntax errors, bad types, typos   │
        └──────────────────────┬───────────────────────┘
                               │ Passed
                               ▼
        ┌──────────────────────────────────────────────┐
        │      LAYER 2: DETERMINISTIC UNIT SUITES      │
        │ • Time: ~2s                                  │
        │ • Tools: pytest tests/unit/ -v               │
        │ • Catches: algorithmic bugs, math regressions│
        └──────────────────────┬───────────────────────┘
                               │ Passed
                               ▼
        ┌──────────────────────────────────────────────┐
        │      LAYER 3: BEHAVIORAL RUNTIME PROBES      │
        │ • Time: ~8s                                  │
        │ • Tools: python -m tests.smoke, playwright   │
        │ • Catches: runtime deadlocks, memory leaks   │
        └──────────────────────┬───────────────────────┘
                               │ Passed
                               ▼
               [ AUTHORIZATION TO COMMIT / DEPLOY ]

3. Feedback Loop Engineering: Structured Diagnostic Parsing

When a test fails, dumping 300 lines of raw terminal output clutters the agent's context and causes attention dispersion. The harness parses raw terminal output into a structured, actionable XML diagnostic envelope.

Run this live Python implementation to see how the harness converts noisy traceback logs into high-signal feedback:

Run it yourself

import re

raw_pytest_traceback = """

============================= test session starts ==============================

collecting ... collected 1 item

tests/unit/test_attention.py::test_qkv_projection FAILED [100%]

=================================== FAILURES ===================================

_____________________________ test_qkv_projection ______________________________

def test_qkv_projection():

mha = MultiHeadAttention(d_model=512, n_heads=8)

x = torch.randn(2, 10, 512)

> q, k, v = mha.forward(x)

E ValueError: too many values to unpack (expected 3, got 2)

tests/unit/test_attention.py:18: ValueError

=========================== 1 failed in 0.42s ===========================

"""

def parse_diagnostic_envelope(raw_output: str) -> str:

# Extract failing test and root cause using regex

test_match = re.search(r"::(test_\w+)\s+FAILED", raw_output)

error_match = re.search(r"E\s+([A-Za-z]+Error:.*)", raw_output)

location_match = re.search(r"([a-zA-Z0-9_/.]+\.py:\d+):\s+[A-Za-z]+Error", raw_output)

test_name = test_match.group(1) if test_match else "unknown_test"

error_msg = error_match.group(1) if error_match else "Test execution failed"

location = location_match.group(1) if location_match else "unknown location"

return f"""<VERIFICATION_FAILURE layer="2" tool="pytest">

<target_test>{test_name}</target_test>

<location>{location}</location>

<error_diagnostic>{error_msg}</error_diagnostic>

Do not declare victory. Inspect the function signature at {location}.

Ensure the return statement matches the test expectation. Re-run pytest to verify.

Verification Invariant

Never permit an agent to mark its own work as complete. Verification must always be performed by an independent deterministic process. If the 3-layer verifier fails, the agent must be re-prompted with the structured diagnostic envelope until all tests pass.

Next in Curriculum

Learn how to persist multi-session progress and compress tokens in State, Continuity & Memory Compaction.