cchu@nycu:~/blog$ cat agent-working-surfaces-july-2026.md

2026.07.3014 min[ai-agents][agent-ui][voice-ai][model-serving]

Agents Need Working Surfaces

A field note on voice agents, robot control, cheaper model routes, agent governance, MCP tools, executable skills, offline knowledge, and the interfaces agents need before they touch real work.

The useful signal today came from the surface around the model.

Agents are leaving chat boxes and moving into working surfaces: voice loops, robot bodies, DevOps boards, email clients, video studios, knowledge servers, frontends, team chats, and local terminals. A text box cannot carry all that authority. The product needs ports, policy, and receipts close to the action.

I would describe the shift like this:

human intent
  -> context port
  -> model route
  -> tool or media surface
  -> action boundary
  -> receipt

That sounds plain. It matters when the agent can hear you, move a robot arm, open a pull request, spawn a coding CLI from a Lark thread, or use an Azure DevOps token.

Agent working surface map

The agent surface now sits between human intent and a real system. The best projects make the boundary visible before the action happens.

Local Checks

I did the tests that can fit this laptop. I did not run GPU inference, robot policies, or long audio generation. Those need different hardware and more time.

Item	Command or check	Result	My read
`speech-to-speech`	`python -m pip index versions speech-to-speech`	Latest package resolved as `0.2.11`.	Good install surface. The repo also documents an OpenAI Realtime-compatible WebSocket server.
VibeVoice	`python -m pip index versions vibevoice`	PyPI showed `0.0.1`.	The real value sits in the repo, model cards, and CPU ASR path, not in a polished pip story.
SGLang	`python -m pip index versions sglang`	Latest package resolved as `0.5.10.post1`.	Mature serving surface. I would still benchmark on my exact model and GPU.
TokenSpeed	`python -m pip index versions tokenspeed`	Package resolved as `0.1.0`.	Early, interesting, and hardware bound. Treat as lab infra for now.
AG-UI	`npm view @ag-ui/client ...`	Client package resolved as `0.0.57`.	Real SDK surface for frontend-agent events.
Composio	`npm view @composio/core ...` and `pip index versions composio-core`	npm package `0.14.1`; Python package `0.7.21`.	Tool/auth layer has enough packaging to deserve a trial.
Azure DevOps MCP	`npx --yes @azure-devops/mcp --help`	Help ran and showed organization, domain, and auth flags.	Good sign. The first real test should use read-only scope.
`book-to-skill`	`python -m pip index versions book-to-skill`	No PyPI distribution found.	Strong idea, weaker install story. I would test from repo before trusting a workflow.
`botmux`	`npx --yes botmux --help`	CLI ran, showed setup, daemon, sessions, dashboard, schedule, plugin, and send commands.	Real tool. It also touches chat credentials and local CLI sessions, so policy comes first.
`jcode` / OpenWork	npm package-name check	Same names pointed at unrelated packages.	Judge the repos, not the package names. Install path needs manual confirmation.

That table kept a few shiny repos out of the main argument. Star velocity gets attention. A package surface, docs, auth model, and failure path decide whether the project is useful.

Voice Is Now a Work Port

speech-to-speech has the cleanest shape among the voice projects. It chains VAD, STT, LLM, and TTS, then exposes the loop through an OpenAI Realtime-compatible WebSocket API. The README says each component can swap out, and the LLM slot can point at a hosted provider, HF Inference Providers, vLLM, or llama.cpp. That is the right architecture. Voice agent work should not bind the microphone, transcription, model, and speech output into one black box.

The production claim also matters: the repo says this pipeline runs as the conversation backend for thousands of Reachy Mini robots. I did not verify the fleet, but the design tells me the authors care about latency, endpoint switching, and local fallback. Those are real constraints.

VibeVoice shows a different side of the same problem. The repo covers ASR and TTS models, including long-form speech recognition with speaker, timestamp, and content output. Its July update describes a CPU inference engine for VibeVoice-ASR-BitNet, compressed from 4.62 GB to 1.58 GB with real-time inference on three or more CPU threads. That is a useful product direction: speech recognition should not always require a datacenter GPU.

The uncomfortable part made the project more credible to me. The README says Microsoft removed the VibeVoice-TTS code after misuse cases. That is not a fun sentence to write in a README. It also names the core issue: voice is identity material. A voice model needs consent, watermarking or output marks, retention policy, and a clear delete path. If a team only talks about voice quality, it is missing half the product.

Gemini Robotics 2 pushes the same idea into bodies. Whole-body control, fine dexterity, and multi-robot teamwork turn model output into motion. Once an agent can move limbs, every "small" instruction becomes a safety question:

Surface	Boundary I would require first
microphone	consent state, wake state, local mute, raw audio retention
transcript	speaker confidence, timestamp links, edit history
voice output	synthetic marker, allowed voices, abuse report path
robot hand	workspace limit, force limit, object class, emergency stop
robot team	task owner, coordination log, collision and rollback plan

Voice and robotics make the agent feel natural. That is why the boundary needs to be boring.

Cheaper Models Move the Bottleneck

OpenAI cut GPT-5.6 Luna pricing by 80 percent and Terra pricing by 20 percent. Kimi Code added a Kimi K3-256k route that sits below the full 1M-context K3 model in quota cost. Those moves change agent economics. More teams can afford long-running workflows, retries, background summarization, and cheap planning calls.

Cheaper calls also make messy routing easier to hide.

An agent stack now needs to record:

Routing field	Why it matters
model id	A cheap planning model and a frontier edit model do different work.
context size	Long context can hide stale state and private material.
cache hit	Cache can cut cost, but it also reuses prior context.
tool parser	A malformed tool call should fail in schema, not in production.
fallback	A silent downgrade can change quality and policy exposure.
cost	Cheap per-token price still becomes big money at agent scale.

SGLang and TokenSpeed sit under that routing problem. SGLang already looks mature: broad model support, RadixAttention, continuous batching, structured outputs, quantization, multiple hardware backends, and a long release history. TokenSpeed looks younger and narrower, but the thesis is right: agentic workloads stress inference differently from single chat turns. Long context, repeated prefixes, tool calls, and structured output need serving-level support.

I would not adopt either because of a benchmark number alone. I would run one boring workload:

same agent task
same tool schemas
same 20 request replay
measure p50, p95, tool-call parse failure, cache hit, and dollars

If the serving layer cannot return those fields, it does not yet fit agent work.

The publishing signal points in the same direction. A bibliometric review of AI unicorns found a thin public research record across hundreds of companies, with many firms publishing no papers or preprints at all. I do not read that as a moral panic. I read it as a product evidence problem. If labs and startups publish less, buyers will lean harder on model cards, eval traces, repo docs, changelogs, package metadata, and independent smoke tests.

A paper used to be one public receipt. Agent products now need many smaller receipts:

Evidence	Useful question
model card	Which data, evals, and safety notes exist?
package metadata	Can I install it cleanly and pin a version?
trace log	Which model, prompt, cache, and tool route ran?
benchmark replay	Does the claim hold on my workload?
incident report	Can the team explain failure without marketing fog?

Without those receipts, a cheaper or more capable model only gives you more faster uncertainty.

Governance Belongs in the Path

The frontier-lab agent intrusion timeline was the hardest security reading in the batch. The lesson is direct: an autonomous agent can chain actions faster than a human reviewer can read them. It can probe, pivot, retry, and hide inside normal-looking service traffic. The fix does not start with nicer prompts. It starts with policy and evidence in the path.

Microsoft's Agent Governance Toolkit targets that exact layer: policy enforcement, zero-trust identity, execution sandboxing, reliability engineering, and OWASP Agentic Top 10 coverage. I have not deployed it. The value is in the product shape. It treats agent actions as policy decisions, not as chat outputs.

Azure DevOps MCP makes the same issue concrete. The local CLI help showed organization, domain selection, and authentication modes. That is what I want to see first. A DevOps MCP server should not expose builds, repos, work items, wikis, and pull requests with one vague permission. It should let the operator choose domains, then log which tool ran.

Composio and AG-UI cover two other parts of the path. Composio handles toolkits, tool search, context management, authentication, and a sandboxed workbench. AG-UI standardizes how an agent backend connects to a user-facing app through events, state, and interaction. One sits near tool authority; the other sits near the user. Both need receipts.

For any agent surface with real authority, I would require these records:

Record	Minimum content
identity	user, agent, model route, organization
scope	allowed domains, repos, files, tools, network
policy decision	allow, block, ask-human, reason
action payload	normalized tool name and validated args
output	artifact, diff, message, audio, video, ticket, PR
reviewer	automatic check, human approval, or explicit no-review label

That is not enterprise theater. It is how you debug a bad Tuesday.

Knowledge Is Turning Into Executable Skill

The content-to-skill projects were more interesting than I expected.

book-to-skill turns technical books or document folders into agent skills. cangjie-skill distills books, long videos, and podcasts into executable agent skills. OpenMontage ships an agentic video production system with pipelines, tools, and hundreds of production-knowledge files. Project NOMAD packages offline Wikipedia, books, courses, maps, local AI, Qdrant, Kiwix, Kolibri, CyberChef, and Docker apps into a hardware-owned knowledge server. DocsGPT brings private document search, agents, multi-model support, and API connectivity into one platform.

These projects all treat knowledge as operational material. The agent does not only "know" a book. It loads a skill, follows a process, and produces work.

That pattern has value, but it creates a new maintenance problem:

Knowledge artifact	Failure mode
book skill	The agent follows an old edition or a bad extraction.
video skill	Missing context becomes confident workflow advice.
offline server	Local AI answers from stale content without warning.
enterprise docs	Permissions leak through search or summarization.
video pipeline	A skill file hides vendor assumptions, license terms, or unsafe defaults.

My quick check made book-to-skill less plug-and-play than the star count suggests. I could not find a PyPI package under that name. That does not make the repo bad. It means the install story needs repo-level testing, and I would not put it into a daily agent workflow before one full conversion run.

OpenMontage has a strong idea and a heavy surface. I like the production-knowledge approach more than a generic "make video" agent. But a video agent touches scripts, narration, music, images, licenses, timing, and final rendering. That is many places to be wrong. I would start by using its files as a reference library, then run a tiny 15-second test with local assets and no private material.

Project NOMAD is the rare non-enterprise project that still feels strategic. Offline knowledge plus local AI is not only disaster-computer branding. It is also a privacy and sovereignty pattern. If your agent depends on external search, SaaS auth, and cloud embeddings, you do not own the working surface. NOMAD points in the other direction.

Team Interfaces Need Ownership

OpenWork, botmux, jcode, AG-UI, Azure DevOps MCP, Inbox Zero, and Tolaria all point at one practical shift: agents need to live where work already happens.

OpenWork frames shared skills, MCPs, connected services, and team capabilities as a cross-tool workspace. botmux bridges Feishu or Lark to local coding CLIs, with one chat topic spawning one live-streaming CLI session. jcode argues for a RAM-efficient coding harness. Inbox Zero turns email into an AI assistant surface. Tolaria manages markdown knowledge bases. AG-UI connects agents into frontends instead of leaving them in terminal land.

I like the direction. I also think this category will create the most local mess.

When an agent enters team chat or email, it inherits social authority. A message in a thread can summon a process on a developer machine. A reply can look like a teammate. A scheduled task can run while the owner sleeps. A dashboard can make half-finished output feel official.

The first setup screen should ask less about model provider and more about ownership:

Which workspace can this agent read?
Which CLI can it start?
Which account sends messages?
Which actions need confirmation?
Who can see terminal output?
How do I kill the session?
Where do logs and tokens stay?

botmux impressed me because the CLI help exposed sessions, delete, suspend, dashboard, schedule, send, plugins, and device controls. It also made me cautious. A tool that can connect chat to a local coding CLI is powerful enough to need boring controls from day one.

Project Triage

Agent project triage

I scored projects by workflow value, install surface, authority risk, and how much real testing they need before daily use.

Project	My value read	First real test
speech-to-speech	High value. Realtime-compatible local voice-agent stack with swappable components.	Run one local loop with a small LLM endpoint, measure turn latency and transcript errors.
SGLang	High value. Mature serving layer with agent-relevant cache, batching, and structured output features.	Replay one agent workload and compare parse failures, p95, cache hit, and cost.
Agent Governance Toolkit	High value. Puts policy and identity close to agent actions.	Wrap one risky tool with allow/block/ask rules and inspect the log.
Azure DevOps MCP	High value for teams already in ADO. CLI help and package surface are real.	Start with read-only domains and one non-sensitive project.
AG-UI	High value if you build agent frontends. Event protocol is better than ad hoc streaming.	Wire one small stateful UI and test cancel, human input, and replay.
Composio	Useful tool/auth layer with real packages.	Connect one low-risk SaaS tool and inspect auth storage plus action logs.
VibeVoice	Useful and sensitive. ASR direction is strong; TTS misuse history deserves attention.	Test ASR on consented long audio; skip cloned voices until policy exists.
TokenSpeed	Promising serving research. Early package and hardware dependency.	Lab benchmark only, no production bet yet.
book-to-skill / cangjie-skill	Strong idea. Turns durable material into loaded agent behavior.	Convert one small public PDF and inspect every generated instruction.
OpenMontage	Interesting production-knowledge package. Heavy workflow and license surface.	Render a 15-second public-assets clip and trace each generated file.
Project NOMAD	Strategic offline knowledge direction. Hardware and content setup are non-trivial.	Install minimal stack, load one ZIM, run one local RAG query.
OpenWork	Good control-plane concept for shared skills and MCPs.	Confirm license, install path, and org-level access model before team use.
botmux	Real team-chat bridge. The CLI works and exposes many controls.	Create one disposable Lark app, one dummy repo, and test session kill.
jcode	Worth watching. The RAM-efficiency claim fits multi-session coding work.	Verify installer and benchmark a simple edit loop against existing agent CLI.

Working Surface Rules

After reading and testing, I would set five rules for any agent project in this group.

Use cheap models, but record the route. A cheaper model call is good. A hidden downgrade is bad.
Treat voice as identity data. Raw audio, speaker labels, and generated speech need consent and deletion, not only quality scores.
Put policy before tools. MCP, DevOps, SaaS actions, browsers, chat bridges, and shell sessions need allowlists before first real use.
Keep skills inspectable. A generated skill should show source, extraction date, scope, and owner. The agent should not load mystery doctrine.
Make the kill switch visible. A user should know how to stop a session, revoke a token, delete logs, and mark an output untrusted.

Those rules are small enough to actually use. They also catch most of the bad cases I saw in these projects.

Close

The next agent product will not win because it has the most dramatic chat demo. It will win because it gives the agent a working surface that people can operate under pressure.

That surface needs five answers:

Who asked?
What context entered?
Which model and route ran?
Which tool or media surface acted?
Which receipt proves the result?

Voice agents, robot policies, DevOps MCP servers, executable skills, local knowledge servers, and team-chat coding bridges all land on the same point. The agent can do more work now. The interface has to carry more responsibility.