High Confidence Without a Critic Artifact

I gave a report generator one synthetic critical finding and no critic output. It emitted the finding with Confidence: High. A malformed critic file, a valid file containing the wrong finding ID, and an empty decision object reached the same fallback.

The finding did not become harmless: its report status was Non-compliant. The problem is different. An unreviewed worker claim acquired the same action used for a confirmed claim. Missing review evidence and intentional review bypass share one branch in the report builder.

This is a narrow artifact-boundary replay of the Play Policy Insights component in Android's agent skills collection, pinned to revision 5bda39f154fb859a08d45e2392f3568e049f83fb. I ran the unchanged Python modules on Linux with Python 3.12.14, using synthetic findings. I did not run an agent, inspect an Android application, contact the store, or validate any actual policy violation.

Review Happens Through Files

This component has more substance than a prompt that asks a model to write a compliance report. Its scanner collects source signals, backed by a structured pattern configuration. The orchestrator prepares work, consolidates worker findings, assigns string finding IDs, and partitions critic work. A separate scraper handles store metadata. A template engine and report generator produce the final artifacts.

The examined component contains five Python scripts and JSON/Markdown resources. Its report path imports standard-library modules and two sibling modules; no package installation was needed for this replay. The inspected component subtree contains no dedicated test directory or Python dependency manifest. That observation concerns this component, not the wider collection's testing practices.

The orchestrator deliberately excludes SUGGESTION findings from critic evaluation. Other severities enter critic chunks keyed by finding ID. The critic instructions define three verdicts:

Action	Intended meaning
VERIFIED	Source evidence confirms the finding
MANUAL_REVIEW	Automatic verification cannot resolve it
PRUNED	The finding lacks support or is a false positive

The documented workflow requires each critic output file to exist and contain valid JSON before proceeding. It also describes retries when that condition fails. This is a meaningful guard. The replay bypasses that workflow to examine what the final consumer enforces on its own.

The Join Loses the Reason for Absence

In aggregate_findings, the report builder loads aggregated_findings.json and merges critic_output_*.json into an ID-to-decision mapping. Its JSON loader warns on malformed input and returns no value; the caller substitutes an empty mapping.

The decisive branch is:

decision = critic_decisions.get(str(fid))
if not decision:
    decision = {
        "action": "VERIFIED",
        "confidence": "High",
        "critic_justification": (
            "Automatically verified (bypassed critic review)."
        ),
    }

The source comment explains the intended use: findings that bypassed the critic, such as suggestions. The condition does not check severity or an explicit bypass marker. It checks whether the lookup result is truthy. A missing decision and {} therefore have the same meaning as a deliberate bypass.

Later, the builder appends the justification and confidence to the finding's evidence text. Severity still determines the overall compliance status. That separation explains how a critical claim can remain critical while receiving unsupported confidence about its verification.

Nine Controlled Cases and a Real CLI Invocation

The replay probe verifies SHA-256 hashes for the three imported modules, the report template, and the policy resource before executing them. It creates temporary fixture directories and asserts the outputs. Every finding is explicitly synthetic and contains no data-transfer fields, keeping the test on the ordinary finding path rather than the separate Data Safety comparison path.

With the pinned component available at /tmp/android-report-note/component, I ran:

python --version
python public/blog/high-confidence-without-critic/probe.py /tmp/android-report-note/component

Both commands exited zero. These are the observed results; the status strings describe the fixtures, not a real app:

Critic artifact	Worker severity	Overall status	Risks	Manual items	High-confidence bypass text
No file	CRITICAL	Non-compliant	1	0	Yes
Malformed JSON	CRITICAL	Non-compliant	1	0	Yes
Valid JSON, wrong ID	CRITICAL	Non-compliant	1	0	Yes
Matching ID, empty object	CRITICAL	Non-compliant	1	0	Yes
No file	SUGGESTION	Compliant	1	0	Yes
No file	IMPORTANT	Needs review	1	0	Yes
Explicit VERIFIED, Low confidence	CRITICAL	Non-compliant	1	0	No
Explicit MANUAL_REVIEW	CRITICAL	Needs review	0	1	No
Explicit PRUNED	CRITICAL	Compliant	0	0	No

For the missing-ID case, the critic fixture was {"2":{"action":"PRUNED"}} while the finding ID was "1". The file is syntactically valid. This case matters because checking JSON syntax alone does not establish decision coverage. The malformed-file case also asserted that the loader emitted a warning; the warning did not stop aggregation.

The probe then launched the actual CLI in a subprocess, equivalent to:

python /tmp/android-report-note/component/scripts/generate_report.py <fixture-directory>

It supplied nonempty synthetic store metadata, so this path did not invoke the network scraper. The unmodified generator wrote both report files. The probe checked the fallback phrase in compliance_report.json and the confidence text in compliance_report.md.

The final terminal lines were:

generator-cli: exit=0; JSON fallback_high=True; Markdown fallback_high=True
PASS: 9 aggregate cases and 1 generator CLI case

That last check establishes that the fallback survives rendering. It does not establish agent reliability, policy accuracy, or how often an incomplete artifact reaches this command during normal use.

Coverage Belongs at the Consumer Boundary Too

The workflow's required file checks reduce the chance of absent or broken critic artifacts reaching the generator. They do not make a valid object with the wrong IDs complete. The report builder has the worker IDs and the decision mapping together, making it the natural place to check their relationship.

A stricter contract would distinguish reviewed, bypassed_by_policy, and missing_review before assigning a verdict. Suggestions could carry an explicit bypass reason. Findings requiring review could remain unresolved, or report generation could fail, when their decisions are absent. An empty decision would fail schema validation instead of activating a high-confidence default.

Those are proposed changes, not behavior implemented or tested here. The observed boundary is smaller and concrete: the current generator uses absence as approval, across all three tested severities, and writes that confidence into both output formats. In a file-mediated agent pipeline, successful JSON parsing is only the start of evidence validation.