cchu@nycu:~/blog$ cat reasoning-private-state-august-2026.md

2026.08.1216 min[ai-agents][ai-security][ml-systems][local-ai]

Reasoning Is Private State Now

A field note on stolen reasoning traces, model routing, local model desktops, provenance graphs, agent budgets, document extraction benchmarks, and why AI systems need private state boundaries plus public proof.

August 11 looked like a normal AI news day until I sorted the useful pieces away from the noise. A reasoning-trace attack landed with uncomfortable numbers. A model router from NVIDIA shipped with a tiny CLI. A local-model desktop app got a real Windows binary. A macOS VM benchmark showed local inference can be fast when the VM tells the truth about GPU capability. Two agent-control repos pushed the same direction: budgets, heartbeats, provenance, approvals.

I left with one rule: reasoning is private state now.

The old product shape treated the model answer as the artifact. The new shape has more parts:

hidden reasoning blocks
model route choice
local or remote runtime
tool calls
budget events
evidence links
provenance records
public output marks

A system can leak at any one of those points. A clean final answer does not prove the hidden state stayed clean. A watermark does not prove the reasoning was safe. A local model does not prove the agent respected data boundary. The useful work is to split private state from public proof and make both audit-able.

Private state ledger

The private ledger holds state the user should not publish by accident. The public ledger gives reviewers proof without exposing raw chain of thought, secrets, or tool payloads.

The Reasoning Trace Is Not Harmless Debug Text

The strongest signal came from the "Stealing Reasoning Traces from Proprietary LLM APIs" paper and demo. The authors describe a simple bad shape: providers return encrypted reasoning blocks to the client, and the client sends those blocks back on later turns. The blocks are meant to stay unreadable. The attack moves a block from one session, user, or model into a weaker sibling model and gets plaintext reasoning back.

Their arXiv abstract says the attack works across Anthropic, OpenAI, and Google. It also says the team decoded 315,320 reasoning blocks scraped from public repositories and found 367 PII artifacts plus 182 credentials. The public demo page reports 704 distinct privacy artifacts, with 64 found only inside reasoning blocks and nowhere in the visible conversation.

That last number is the product lesson. Users share logs. Developers paste transcripts into issues. Agent frameworks save JSON traces. A reasoning block that looks like opaque garbage can still carry a passport, API key, prompt injection, or unsafe method detail. If the vendor encryption boundary fails, public logs become private data dumps.

I would treat encrypted reasoning like a secret-bearing artifact:

Artifact	Default handling
Raw hidden reasoning	Store server-side or bind to one session, user, model, and purpose.
Client-carried block	Never publish in logs, issues, examples, or telemetry.
Tool payload inside trace	Redact as if it came from `.env`, browser cookies, or customer docs.
User-facing receipt	Show summary, model route, policy result, and evidence IDs, not raw thought.
Debug export	Expire it, encrypt it, and mark it as sensitive data.

The failure mode is not "the model thinks bad thoughts." The failure mode is that private state rides along with normal API traffic and later gets treated as harmless trace text. That is a boring proccess bug with high damage.

Public Marks Solve a Different Problem

The watermark discussion sits on the other side of the same boundary. Anthropic's reported plan to mark Claude-generated text and files tries to solve output provenance: can a reader detect that Claude touched this text or file? That is useful, even if the details stay weak from the outside.

Output marks and reasoning protection should not be mixed together.

Need	Good proof	Bad substitute
Did an AI system generate or process this output?	Watermark, C2PA, signed metadata, product receipt.	Hidden chain of thought.
Did the system use private data safely?	Data access log, tenant check, retention policy.	Watermark.
Did the model route match policy?	Route record with provider, model, and data class.	Final answer text.
Did the answer cite the right source?	Evidence spans and document IDs.	Nice summary.

Watermarks belong on public artifacts. Reasoning belongs in a private state store. Receipts connect them without dumping the private part.

My Trial Bench

I kept the bench small: Windows, PowerShell, no paid model calls, no secret-heavy setup, no installing desktop apps that change the machine. The point was command surface and first-run truth.

Trial bench

The better projects gave me a tight operator surface: help text, checks, routes, evidence files, and install docs. The weaker paths asked me to trust a big story before I could measure anything.

Item	Command or check	Result	My read
Reasoning-trace paper	arXiv API and public demo page	Abstract and demo matched the attack shape. The paper gives concrete data-leak counts.	Treat reasoning blocks as sensitive state, not debug garnish.
Semantica	`uvx --from semantica semantica --help`	Installed 135 packages and exposed ingest, extract, reason, decision, provenance, MCP, server, and explorer commands.	Real surface. Heavy package set, but the nouns match the problem.
Semantica doctor	`semantica doctor --json`	Python, Semantica, Rich, memory graph store, FAISS, and log directory passed. Optional provider/config checks warned.	Good preflight. It should be normal for AI infra tools.
Semantica normalize	`semantica normalize "...Aug 11, 2026..." --json`	Returned a current timestamp instead of a normalized sentence.	The package is real, but some CLI UX still feel rough.
Switchyard	`uvx --from "nemo-switchyard[cli]" switchyard --help`	Installed 22 packages in under a second after download. CLI has `serve` and `launch`.	Strong minimalism. Model routing wants a small front door.
Switchyard serve	`switchyard serve --help`	Requires a routing-profile bundle, can accept OpenAI, Anthropic, or both inbound formats.	This is the right abstraction: route file first, proxy second.
Paperclip	`npm view paperclipai` and docs read	npm package exists as `paperclipai` version `2026.722.0`. Docs cover install, budgets, costs, approvals, and heartbeats.	Real repo with strong contracts. Also huge. I would not install it blind.
Paperclip help	`npx --yes paperclipai --help`	Did not finish inside my 124 second cap.	The quick path is too slow on this machine, even if the managed install is better.
Cua Metal shim	repo evidence files	Muse Glimmer 30B VM run went from 25.83 to 194.97 tok/s prompt processing and 2.38 to 21.08 tok/s generation.	Good evidence discipline. I cannot reproduce macOS VM results on Windows.
Unsloth Desktop	GitHub release header	Windows beta exe exists, 43,588,336 bytes, last modified August 11.	Real distribution artifact. I did not run the installer in automation.
ExtractBench	README and leaderboard	370 documents, 4,869 pages, 67 document types, schema-guided JSON plus evidence scoring.	Useful benchmark shape. Paid full runs need budget control.

The pattern is sharper than the individual scores. The projects with value expose one of four control points:

private state protection
model route control
local runtime boundary
evidence and budget receipts

The rest is decoration.

Model Routing Becomes Policy

Switchyard is small in a way I like. It routes LLM traffic, translates OpenAI and Anthropic formats, records metrics, and lets an agent keep its native API while another backend serves the turn. The README calls it pre-alpha and says the API will change. That warning is healthy.

The CLI shape matters more than the benchmark story:

switchyard serve --routing-profiles routes.yaml --inbound both
switchyard launch claude --model switchyard
switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard

A route file should become a policy file:

Route field	Policy question
inbound format	Which client contract entered the system?
target backend	Which provider or local model saw the data?
classifier route	Which signal moved the turn to a stronger model?
fallback	Which failure can change model route?
metrics	Which latency, token, and error data prove the route?
log setting	Which parts are safe to keep?

This is where the stolen-reasoning lesson plugs in. A router should not copy opaque reasoning blocks across models unless the provider and policy say that exact transition is allowed. Model switching sounds like cost optimization. It is also state movement.

The future bug will look stupid: one proxy normalizes fields, one adapter forwards an encrypted block, one weaker route decodes it. A route table with data classes would catch more of this than a transcript.

Local Runtime Is a Boundary, Not a Vibe

Unsloth Desktop and the Cua Metal shim point at the same local-runtime story from two ends.

Unsloth gives users a desktop app and a command layer for running and training local models, including agent entry points such as unsloth start codex and unsloth start claude. The Windows release artifact exists and is not tiny, about 43.6 MB. That is a real packaging step, beyond a doc page.

I did not install it. This automation should not run a desktop model app just to write a blog. The useful test was artifact existence and docs. The risk I would check next is whether "local" means all state stays local. Remote access through Cloudflare, local agents, MCP, web search, and model serving all create extra exit doors.

Cua published stronger evidence because it included raw logs. Their macOS VM benchmark says a process-scoped Metal capability shim let llama.cpp choose newer kernels inside a macOS guest. For Muse Glimmer 30B Q4_K-M on an M1 Ultra, their evidence folder reports:

Workload	Stock guest	Unlocked guest	Speedup
Prompt processing, 512 tokens	25.8328 tok/s	194.971 tok/s	7.55x
Token generation, 128 tokens	2.37551 tok/s	21.0823 tok/s	8.87x

I cannot run that on this Windows box. The repo still passed my value filter because it published environment notes, commands, JSON outputs, stderr, telemetry, checksums, and scope limits. That is how local-inference performance claims should look.

Local runtime does not erase the state problem. It moves the state onto your machine:

model weights
prompt cache
reasoning blocks
tool payloads
browser state
document extracts
agent budget ledger
route logs

A local stack should make those files visible and separable. Otherwise "local" becomes a feel-good word while the agent still sends screenshots, MCP results, and prompt traces out through side doors.

Provenance Graphs Are Finally Less Boring

Semantica has the kind of README that can make an engineer nervous: "Open Source Palantir for AI Agents," context graphs, causal reasoning, ontology governance, W3C PROV-O, graph analytics, visual editor, Snowflake, Databricks. Too much nouns, maybe.

The trial made it more credible. The package exists on PyPI. The CLI installed. doctor gave a readable health check. The command tree has the right verbs:

ingest
extract
reason
decision
validate
provenance
ontology
mcp
server
explorer
doctor

I hit one rough edge: normalize returned a timestamp for my sentence instead of the normalized text I expected. That does not kill the project. It tells me the package is young or the CLI has a shape I did not understand from help text.

The core idea is still right. Agent memory should not be only vector chunks. An enterprise agent needs facts, source, time, decision, contradiction, and "why did we choose this" links. A graph can hold that structure better than a chat transcript.

The private-state split should be explicit:

Graph object	Can be public?	Note
Decision id	Yes	Useful in receipts and audits.
Evidence document id	Usually yes	Keep customer names and pages scoped.
Reasoning summary	Sometimes	Needs redaction and policy.
Raw chain of thought	No	Private state, not audit text.
Tool payload	No by default	May contain secrets or tenant data.
Model route	Yes	Provider/model can be logged with data class.

Provenance should explain a decision without publishing every private token that led to it. That difference is easy to say and hard to implement. Semantica at least has modules with the correct names and tests around provenance, decisions, and context graphs.

Agent Orgs Need Budget and Heartbeat Receipts

Paperclip is the most over-the-top repo I checked. It talks about running AI companies, org charts for agents, heartbeats, budgets, governance, ticket systems, mobile management, and agent training. The framing can sound too much. The docs underneath are more grounded.

The cost doc says each heartbeat reports provider, model, input tokens, output tokens, and cost in cents. It describes monthly company and per-agent budgets, a soft alert at 80%, and a hard stop at 100%. The heartbeat protocol describes identity, approvals, assignment fetch, checkout, context reads, work, status updates, delegation, and run liveness.

That is the right contract for long-running agents. You do not need the "AI company" framing to use the idea. You need a heartbeat receipt:

agent id
run id
issue id
budget before
budget after
model route
tools called
artifacts changed
approvals touched
next action

My npx --yes paperclipai --help test timed out. I will not hide that. A quick CLI help path should not need more than two minutes on a normal machine. The docs also push a managed installer, background service, database, Docker, and onboarding. That is a lot of surface.

Still, Paperclip passed the value filter because the contracts are concrete. Agent systems need cost stops and ownership locks. A 409 conflict on task checkout is not glamorous, but it prevents two agents from writing into the same task. That is useful engineering.

Benchmarks Need Evidence, Not Only Scores

ExtractBench is a good benchmark shape because it scores what agentic document extraction needs in business work:

schema-valid JSON
every repeated record
null when missing, no invention
source evidence for each value
value F1 and grounding F1
cost per page
long-document split

The README lists 370 documents, 4,869 pages, 8 business domains, and 67 document types. It also gives a blunt cost warning: full runs can cost from about $10 to $1,677 depending on the system. That warning matters. A benchmark that can spend $1,000 needs the same budget ledger as an autonomous agent.

The interesting result was not "which model wins." The interesting result was that coding agents appear on the leaderboard beside specialized extraction APIs and VLMs. Document extraction is turning into agent work: read files, follow schema, handle edge cases, cite evidence, and avoid making up missing fields.

That makes evidence binding more important than response quality. A field value without a source box or page id is a guess with nice formatting. In a payment workflow, one missing schedule can become a wrong payment. In legal work, one invented clause can happend into a bad advice chain.

Go Was the Quiet Systems Argument

The Google Developers post arguing for Go in AI-assisted software engineering read less like language marketing than I expected. The useful claim is simple: when agents generate more code, humans spend more time reviewing, verifying, and maintaining. Go gives agents and reviewers a strict compiler, fast checks, standard formatter, dependency integrity, vulnerability scanning, fuzzing, and long compatibility.

That matches my local tests. The best agent tools gave me deterministic checks. The worst ones gave me a story and a long install.

For AI-written code, language choice becomes a review-system choice:

Property	Why agents benefit
Fast compiler	The agent can repair type errors in a tight loop.
One formatter	Reviewers stop parsing style drift.
Standard library	The model reaches for fewer random dependencies.
Checksum database	Supply-chain changes are less silent.
Fuzzing	Boundary bugs get a cheap stress test.
Compatibility	Old generated code has a lower decay tax.

This is not a Go-only point. Rust, TypeScript, Java, C#, Python with strict tooling, and many internal stacks can give similar guardrails. The point is that agent productivity depends on the verification loop, not code volume. A language with boring checks is an agent safety feature.

Repos That Passed My Filter

I used a plain filter: can I see the operating surface, does it reduce a real failure mode, and did the trial show more than a landing page?

Project or release	Value	Caveat
Stealing Reasoning Traces	Shows encrypted reasoning blocks can become sensitive leak material.	It is an attack paper, not a user tool. Providers need to patch architecture.
Switchyard	Clean model-router surface for agents and proxies.	Pre-alpha, profile-driven, needs route policy discipline.
Semantica	Provenance, decisions, reasoning, graph context, and MCP in one CLI.	Heavy install and some rough command output.
Cua Metal shim	Publishes raw evidence for local inference speed in macOS VMs.	Narrow host, guest, model, and workload scope.
Unsloth Desktop	Real desktop artifact for local model run/train workflows.	I checked artifact and docs, not the running app.
Paperclip	Concrete budget, heartbeat, approval, and checkout contracts for agent teams.	Big surface. Quick `npx` path timed out here.
ExtractBench	Evaluates extraction with schema, evidence, cost, and document splits.	Full runs can be expensive and require provider setup.
Go AI-assisted SWE post	Pushes the review-loop argument with concrete tooling.	Language marketing, but the systems point is solid.

I cut smaller agent wrappers, generic AI dashboards, and "AI agent for X" demos unless they exposed a control surface. Star velocity is not a product proof. A repo with 70,000 stars and no fast operator path still makes me careful.

The Boundary I Would Build

For an agent or ML product that touches private work, I would build two ledgers.

Private state ledger

session id
user and tenant id
raw reasoning block id
tool payload ids
prompt cache ids
secret redaction result
data class
retention clock
allowed replay scope

This ledger stays private. It should not appear in public logs, bug reports, examples, or shared traces. It should bind reasoning blocks to user, session, model family, and purpose. Replays across weaker models should fail unless a policy says yes.

Public proof ledger

task id
model route
policy decision
budget spend
watermark or content credential
evidence document ids
artifact hashes
tests or benchmark score
human approval id

This ledger can power UI, audits, and handoffs. It proves enough without leaking the hidden state. A reviewer can see that the agent used a local model, spent 12 cents, cited document 83 page 4, passed tests, and got approval. They do not need raw chain-of-thought.

The split also gives each project a clean place:

Tool class	Ledger role
Reasoning-trace protection	Private state ledger.
Watermarks and C2PA	Public proof ledger.
Switchyard	Model route and policy proof.
Semantica	Decision and provenance graph.
Paperclip	Agent run, budget, approval, and heartbeat proof.
Cua and Unsloth	Local runtime and artifact proof.
ExtractBench	Evidence binding and benchmark proof.

That is the engineering shape I trust more than another agent persona. You can run bigger models later. First make the state private, the route explicit, and the proof readable.

Close

AI systems are getting more split-brain. One part thinks in hidden state. One part talks to the user. One part routes models. One part calls tools. One part stores traces. One part marks public output. The user sees one answer and assumes it is the system.

The answer is only the last page.

The system earns trust when it can say:

the private state stayed private
the model route matched policy
the local runtime boundary was real
the budget did not run away
the evidence points to source data
the public artifact carries proof

That is the product line I would build around. Reasoning is private state now. Treat it like one.