cchu@nycu:~/blog$ cat ai-chain-of-custody-july-2026.md

2026.07.2812 min[ai-agents][model-serving][agent-security][data-provenance]

AI Needs Chain of Custody

A field note on data provenance, agent trust metadata, Kimi K3 serving, security harnesses, design specs, video tools, token ledgers, and risky credential gateways.

AI work has a custody problem.

Reports about model-data buyers cutting physical books for scanning, a public audit of the Bun Rust rewrite claims, Google's Open Knowledge Format v0.2, vLLM's Kimi K3 launch notes, NVIDIA's Open Secure AI Alliance, and a wave of agent tools all point at the same missing layer.

You need to know who touched the work.

That sounds bureaucratic until you follow one AI task from source to output:

source material
  -> knowledge bundle
  -> model route
  -> cache and context
  -> tool call
  -> generated artifact
  -> review
  -> reuse

Each step changes the trust boundary. A person, model, script, gateway, cache, tool, or agent can alter the result. If the system keeps no custody record, you cannot tell whether an answer came from a verified policy, a stale scrape, a borrowed account, a cached prefix, a video frame, a hidden tool call, or a human-reviewed source.

The useful AI products in this scan gave the human a record before asking for more autonomy.

AI chain of custody map

A useful agent system needs custody records across sources, generated knowledge, serving, tool calls, credentials, artifacts, and review.

Books Turn Data Provenance Into a Product Issue

The book-scanning story made the data problem concrete. Reports described AI data buyers acquiring old and out-of-print books, cutting bindings, scanning pages, and discarding originals. The legal debate will take its own path. The product lesson is narrower: clean data has a provenance cost.

Web data became polluted by generated text. Teams that need higher-signal corpora will chase sources with older custody chains: printed books, licensed archives, internal documents, verified databases, private code, meeting transcripts, and workflow logs. Each source needs answers before a model trains on it or an agent cites it:

Custody question	Record needed
Who acquired the source?	Buyer, license, or owner record.
What was transformed?	Scan method, OCR output, page range, and loss record.
What survived?	Original location, copy quality, checksum, and retention policy.
Who may use it?	License terms, jurisdiction, consent, and allowed model use.
Which output used it?	Dataset version, retrieval citation, or training run link.

That table applies outside books. A company wiki, a design system, a finance definition, and a support transcript all need the same shape of custody once agents read and rewrite them.

The Bun Rust rewrite audit raised a second custody issue: cost and labor. One observer challenged the public narrative around an AI-assisted rewrite by looking at release tags, pull requests, CI work, API cost, and human involvement. You do not need to accept every inference to see the value of the method. Claims about AI labor need receipts that include model spend, CI spend, human review, failed attempts, release status, and maintenance debt.

If a team says "the agent did the rewrite," the custody record should show:

which agent produced which pull requests
which humans reviewed them
which tests and CI jobs ran
which costs sat outside model tokens
which release carried the work
which defects arrived after merge

AI cost claims without custody records collapse into marketing.

Trust Metadata Belongs Next to Knowledge

Google's OKF v0.2 is the cleanest format-level signal from the day.

OKF treats agent-readable knowledge as markdown with YAML frontmatter. The v0.2 addition gives each concept frontmatter fields for provenance, trust, freshness, lifecycle, and attestation. The important move: an agent can decide whether to read or use a concept before it spends tokens on the body.

That gives you filters such as:

generated:
  by: reference_agent/gemini-2.5-pro
  at: 2026-06-30T14:00:00Z
verified:
  - by: human:finance-owner@example.com
    at: 2026-07-01T09:00:00Z
status: stable
stale_after: 2026-12-31
sources:
  - id: revenue-policy
    resource: policies/revenue-recognition.md

The format names the custody gap. Once agents generate thousands of facts, tables, metrics, runbooks, and definitions, another agent needs to know which concepts came from a trusted source, which concepts a human verified, which ones expired, and which computations ran through a sanctioned path.

Attested computation gives the format teeth. A knowledge bundle can define the computation, the executor, and the receipt fields. A deterministic attester can inspect the receipt and reject a value if the agent improvised SQL, changed a table, or displayed a number that does not match the run.

That pattern should spread.

Agent output	Custody field to add
Finance metric	Source policy, approved SQL, job id, attester verdict.
Customer-support answer	Source article version, retrieval span, escalation policy.
Design change	Design token version, lint result, screenshot, reviewer.
Code patch	Issue, diff, tool log, tests, review status.
Meeting summary	Consent, speaker segments, transcript confidence, retention.

Knowledge formats should make trust cheap to check.

Large Models Push Custody Into Serving

Kimi K3 on vLLM looked like a model-serving story at first: a 2.8T-parameter multimodal Mixture-of-Experts model, 1M-token context, Kimi Delta Attention, Attention Residuals, DSpark speculative decoding, hybrid prefix caching, tool calling, structured outputs, NVIDIA and AMD launch paths, and 370 tokens per second per user on a 16 GPU GB300 setup in vLLM's benchmark.

The agent lesson sits in the cache.

Long-context agents replay prompts, tool specs, repo snapshots, conversation state, and work logs. Serving engines now decide which prefixes deserve cache retention, which recurrent states survive, which prompt boundaries get checkpoints, and which shared prefixes get promoted after repeat hits. Those decisions affect cost, latency, privacy, and answer behavior.

That makes serving a custody layer.

Serving decision	Why the user needs a record
Model route	Determines capability, provider exposure, and cost.
Prefix cache hit	Determines which prior context shaped the response.
Retention policy	Determines what gets reused and for how long.
Tool parser	Determines whether an action payload passed schema checks.
Structured output constraint	Determines whether the model could leave the contract.
Fallback route	Determines whether a weaker or different model handled the task.

vLLM's Kimi K3 notes also contain a useful caveat for agent builders: tool calling must be validated on your own traffic. The serving layer can expose a tool parser, but production agents still need schema checks, retries, and fallbacks when tool calls parse empty or malformed.

Large models no longer sit behind a stateless /chat endpoint. The serving engine now participates in memory, policy, and audit.

Security Agents Need Proof, Scope, and Identity

NVIDIA's Open Secure AI Alliance announcement argued for open models, open harnesses, and shared security tooling. The useful part was the agent-stack framing: safety and security depend on identity, permissions, harnesses, guardrails, logs, evaluation, and model weights together.

That matched the better security projects in the scan.

Strix, for example, pitches autonomous AI penetration-testing agents that run code, validate vulnerabilities, produce proof-of-concept exploits, and generate remediation guidance. That category has real value if the system keeps tight scope. A security agent should record target authorization, sandbox limits, exploit steps, proof artifacts, data touched, fix diff, and reviewer sign-off.

Open Code Review makes a similar point from the code-review side. Its stronger claim is not "LLM reviews code." Its stronger claim is that deterministic file selection, rule matching, bundling, positioning, and reflection constrain the model. The agent works inside a review process.

The security pattern is simple:

authorize scope
  -> run bounded agent
  -> capture proof
  -> patch or escalate
  -> preserve evidence
  -> review before promotion

Agent security tools should make the proof boring. That includes the proof that the agent stayed inside the scope.

Projects Worth Time

The useful projects fell into custody categories. Some help you preserve evidence. Some help you package behavior. Some create new risk at the credential layer.

Agent project custody triage

My triage after opening repos and reading docs. High value means the project attacks a real bottleneck. A production workflow still needs a local trial.

Project	Value	Caveat
DESIGN.md	Turns visual identity into tokens plus rationale that agents can read, lint, diff, and export. This gives design work a versioned custody file.	A file format cannot create taste. Teams still need product direction and visual review.
Impeccable	Packages design guidance, commands, and deterministic detectors for agent-built frontends. It gives design review a harness.	Treat detector output as evidence before final approval.
Tokscale	Reads local session stores across coding agents and gives teams a token ledger. This turns spend into a traceable work artifact.	The tool reads sensitive local agent data. Review retention and sharing defaults.
Wattage	Profiles token spend from traces and can fail CI on cost regressions. It moves budget review closer to the change.	It needs real trace coverage and up-to-date pricing.
Strix	Agentic pentesting with exploit validation points at a real security workflow.	Run inside explicit scope. A tool that proves exploits needs strong boundaries.
Open Code Review	Hybrid deterministic plus LLM review is the right shape for line-level feedback.	Validate precision and recall on your codebase before replacing review rituals.
Claude Video `/watch`	Video becomes a first-class agent input: captions, frames, timestamps, and focused questions.	Video carries privacy, copyright, and consent risk. Keep frame and transcript receipts.
last30days-skill	A research skill that joins social, video, forum, market, and web sources can produce useful recency briefs.	Engagement is not truth. Treat source scoring and auth cookies as custody issues.
AG Kit	Project-level rules, skills, memory, MCP guidance, validation hooks, and content inventories show a mature workspace-contract direction.	It binds to Antigravity as the production runtime. Portability needs testing.
OpenWork	Shared MCP capabilities, skills, plugins, and org access controls show where agent workspaces are headed.	Remote capability execution needs audit logs and least-privilege access.
Crush	Terminal coding agent with multi-model, session, LSP, and MCP support. Strong developer-surface signal.	Session custody matters once multiple providers and local tools enter one shell.
DocsGPT	Enterprise search plus agents, tool connections, SSO, RBAC, logs, and BYOM fit the private-knowledge custody problem.	Connector quality and permission mapping decide whether it is safe.
AIRI	Local companion and voice/game surfaces show embodied agents moving beyond chat.	Personal context, voice, and identity need stricter consent and deletion controls.
career-ops	Structured local job-search workflow with human-in-the-loop gates. It treats agent output as a draft and tracking system.	It touches sensitive personal data. Store less than the workflow tempts you to store.
Agent-teams dashboards	Kanban, logs, costs, providers, and multi-agent teams match the future work surface.	More agents multiply review load. Avoid "AI company" theater without ownership records.

The weaker category was also clear: credential and quota gateways.

OpenAI OAuth, FreeLLMAPI, Sub2API, Cursor Bridge, and similar projects all respond to a real pressure: AI usage costs too much, providers fragment access, and developers want one endpoint. The risk sits in the solution. If a project turns a personal subscription, OAuth session, free tier, relay account, or pooled provider key into an API surface, the custody problem becomes sharp.

Ask these questions before using one:

Gateway question	Required answer
Which account owns the request?	Per-user identity, not a shared mystery token.
Which provider saw the data?	Provider log for each request.
Which terms apply?	Explicit policy review and allowed use.
Which credentials persist?	Storage path, encryption, rotation, and revocation.
Which model answered?	Model id, fallback route, and response metadata.
Which data may cross the gateway?	Classification rules and redaction.

Cheap token routes can erase custody.

Skills Package Behavior

The skill repos were more interesting than their marketing copy.

DESIGN.md, Impeccable, Marketing Skills, last30days, career-ops, AG Kit, and follow-builders all treat instructions as artifacts that agents can install, route, edit, and reuse. That makes skills a distribution format for behavior.

A skill can encode:

design taste
research sources
marketing process
job-search rubric
video analysis workflow
security guardrails
memory rules
tool permissions

That power needs the same custody records as code:

Skill artifact	Record needed
Source material	License, author, date, and allowed reuse.
Generated instruction	Generator, reviewer, diff, and test prompt.
Selection event	Why the agent loaded the skill.
Tool access	Which commands, files, browsers, and accounts it can touch.
Behavior update	Version, changelog, rollback path.
Evaluation	Before/after tasks and failure cases.

Skill libraries will break down if teams treat markdown instructions as harmless notes. A skill changes behavior. Behavior needs ownership.

Video and Voice Need Stronger Receipts Than Text

Claude Video, Meetily, DocsGPT speech workflows, and AIRI all point toward the same input shift. Agents are learning to watch, listen, transcribe, summarize, and act from richer context.

That creates better products. A bug report video can show the frame where a UI breaks. A meeting transcript can preserve the decision and owner. A companion app can keep a long-lived personal context.

The custody bar rises with media:

Media input	Extra receipt
Screen recording	Frame timestamps, cropped regions, and redaction record.
Meeting audio	Consent, speaker labels, transcript confidence, retention.
Video URL	Download scope, captions source, frame budget, copyright status.
Voice agent	Wake state, command boundary, synthetic output marker.
Personal companion	Memory owner, deletion path, export path, and model route.

Text agents can already leak secrets. Media agents can leak rooms, faces, voices, screens, and habits.

What I Would Try

I would run six small trials rather than chase every new agent repo.

Write a custody checklist for agent work. Include source, model route, credentials, tool authority, cache, artifact, review, and reuse.
Add trust metadata to one knowledge corpus. Use OKF-style frontmatter for generated, verified, sources, status, and stale dates.
Measure one coding-agent week. Use Tokscale, Wattage, or a local script to map token spend to tasks, models, retries, and failed attempts.
Version design behavior. Add a DESIGN.md file, run a linter, and attach screenshots to design changes.
Test a security agent in a narrow sandbox. Give Strix or a similar tool one scoped target, then review proof artifacts and boundaries.
Ban mystery gateways for sensitive work. If a gateway hides provider, account, credential storage, or terms, keep private code and customer data away from it.

The practical goal is recoverable delegation, with automation scoped by records.

Closing

AI systems keep crossing custody boundaries:

data becomes training material
documents become agent knowledge
prompts become long-lived cache entries
credentials become gateways
videos become evidence
skills become behavior packages
model routes become policy decisions
tool calls become business actions

You can make that work if each boundary leaves a record.

The next useful layer in AI infrastructure is a custody log that a person can inspect under pressure:

Who supplied the source?
Who generated the knowledge?
Who verified it?
Which model and route ran?
Which credential allowed the action?
Which cache or memory shaped the answer?
Which tool changed the world?
Which test or reviewer accepted the result?

That is the difference between a demo and a system you can trust.