Part 4: Real-Time Pipeline Diagnostics

How Calibrated Classification Probabilities Eliminate Build Retries and Alert Fatigue

By Jason Vertrees
Part 4 of the Machine-Native AI Series

The CI/CD Dilemma: Blind Retries vs. Hallucinated Root Causes

In high-velocity engineering organizations and autonomous coding pipelines, continuous integration is where velocity goes to die.

When a test fails in an automated build, teams are universally caught between two flawed operational defaults:

1. The Blind Retry Antipattern: Many teams configure their test runners to rerun every failed test three times. If it passes on run two, it's marked green. This hides real concurrency bugs, inflates cloud compute bills, and adds 15 to 30 minutes of delay to every merge.

2. The Conversational LLM Antipattern: Some teams pipe stack traces into a foundation LLM asking for an explanation. But foundation models are slow (adding seconds to pipeline execution), expensive at volume, and prone to hallucinating plausible-sounding code bugs when the real issue was a transient 504 gateway timeout from a mock service.

The pipeline doesn't need an LLM to write a speculative essay about why code failed.

The pipeline needs a fast, sound judgment call returning calibrated classification probabilities: Is this specific failure a transient network/timing flake, a real code regression, or an infrastructure outage?

The Architecture: Probabilistic CI/CD Triage

To solve this, we built an automated triage router powered by TypeSafe AI's Jev model.

The Triage Schema

In a single sub-second evaluation, the CI router passes the stack trace, error log, and recent commit diff to Jev:

from typesafe_sdk import Choice, Noul, Score

CI_QUESTIONS = {
    # 1. Calibrated probability of non-deterministic failure
    "is_transient_flake": Noul(
        instructions=(
            "Is this failure caused by a transient, non-deterministic environmental condition "
            "(such as a socket timeout, race condition, or external resource lock) rather than a code bug?"
        )
    ),
    # 2. Discrete root cause categorization
    "root_cause_category": Choice(
        instructions="Classify the root cause of the build failure.",
        criteria={
            "TRANSIENT_FLAKE": "Network jitter, async timing issue, port collision, or transient timeout",
            "CODE_REGRESSION": "Syntax error, assertion failure, missing attribute, or breaking API contract",
            "ENVIRONMENT_OR_INFRA": "Database connection refused, Redis down, out of disk, or runner OOM",
            "TEST_HARNESS_DEFECT": "Malformed fixture, invalid test setup, or broken mock configuration",
        },
    ),
    # 3. Evidence certainty score
    "triage_confidence": Score(
        instructions="Rate the certainty of the root cause classification based on the error trace from 0 to 3.",
        criteria=[
            "Level 0: Ambiguous or generic exit code with missing stack trace",
            "Level 1: Partial trace, root cause suspected but unconfirmed",
            "Level 2: Clear stack trace pointing to a specific failure mechanism",
            "Level 3: Conclusive failure trace with explicit exception and line number",
        ],
    ),
}

Deterministic Routing Logic

Your CI/CD workflow script consumes these probabilities to make immediate routing decisions:

def triage_build_failure(failure_payload: dict) -> TriageDecision:
    with client:
        res = client.system_one(state=failure_payload, questions=CI_QUESTIONS)

    p_flake = res.nouls["is_transient_flake"].noul
    root_cause = res.choices["root_cause_category"].choice
    confidence = res.scores["triage_confidence"].score

    # 1. Conclusive transient flake -> Automatically rerun ONLY this test
    if p_flake >= 0.85 and root_cause == "TRANSIENT_FLAKE":
        return TriageDecision(
            action="AUTOMATIC_PIPELINE_RERUN_TRIGGERED",
            reason=f"High-confidence transient flake (p={p_flake:.2f}). Rerunning single test target.",
        )

    # 2. Genuine code regression -> Fast-fail the pipeline immediately
    if root_cause == "CODE_REGRESSION":
        return TriageDecision(
            action="HARD_FAIL_PIPELINE",
            reason="Confirmed code regression in commit. Blocking merge without retries.",
        )

    # 3. Infrastructure outage -> Alert DevOps on-call with attribution
    if root_cause == "ENVIRONMENT_OR_INFRA":
        return TriageDecision(
            action="ESCALATE_TO_DEVOPS_ONCALL",
            reason="External service or infrastructure unavailable (e.g. Postgres connection refused).",
        )

    # 4. Harness defect or low confidence -> Route to test maintainers
    return TriageDecision(action="NOTIFY_TEST_MAINTAINERS")

Empirical Triage Results Across Real Stack Traces

When tested against enterprise error logs, Jev's calibrated probabilities produce immediate, unambiguous classifications:

Stack Trace Sample	Calibrated Sensor Reading	Deterministic CI Action
httpx.ConnectTimeout: timed out waiting for mock-oauth:8080	p_flake = 0.94
cause = TRANSIENT_FLAKE
conf = Level 3	AUTOMATIC_PIPELINE_RERUN_TRIGGERED
(Reruns test immediately; no humans paged)
TypeError: AuthToken.__init__() missing 1 required positional argument: 'user_id'	p_flake = 0.02
cause = CODE_REGRESSION
conf = Level 3	HARD_FAIL_PIPELINE
(Aborts build in 50ms; zero wasted retries)
psycopg2.OperationalError: could not connect to server: Connection refused (port 5432)	p_flake = 0.12
cause = ENVIRONMENT_OR_INFRA
conf = Level 3	ESCALATE_TO_DEVOPS_ONCALL
(Pages database team with exact server trace)

Key Takeaway for Engineering Teams

Structured output was solved two years ago. The real engineering breakthrough is calibrated judgment velocity.

By introducing rapid, sound classification probabilities into your automated pipelines:

• You stop burning thousands of dollars on blind 3x test reruns.

• You stop waking up on-call engineers for non-deterministic socket flakes.

• You give your build systems the common sense needed to triage failures in milliseconds.

Series Conclusion

Throughout this 4-part series, we explored what changes when software can access System 1 decision models:

• In Part 1, we probed the boundaries: fast perceptual judgment vs. code computation and generative reasoning.

• In Part 2, we protected ingress with calibrated probability thresholds.

• In Part 3, we governed autonomous agent tool loops via code-driven state machines.

• In Part 4, we automated real-time failure diagnostics in CI/CD pipelines.

By letting code own the decision logic and using calibrated System 1 models as rapid sensory primitives, our software becomes faster, safer, and mathematically predictable.

---

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.