Product Jobs for Small Models

Gemma 4 12B and Max Leiter's "They're made out of weights" arrived within the same week. One was a product and model announcement. The other was a short piece that strips away some mysticism around LLMs by reminding readers that the system is weights doing computation, not a little database or mind hiding inside the box.

That pairing makes a useful product point. Teams should assign a small model to a product job with explicit latency, cost, privacy, and failure constraints.

Large models dominate open-ended evaluation because open-ended evaluation rewards broad competence. Products rarely need broad competence at every step. Products need bounded operations that run many times, under constraints, with measurable outputs. A small model that performs one operation well can be more valuable than a larger model used as a vague assistant.

The Shift From Model Score to Model Job

A model benchmark tells you something about capability. Architecture work starts when the team decides where the model belongs.

For product work, a model job should name:

Field	Example
Input	Screenshot crop plus DOM text
Output	JSON with selected UI element and confidence
Latency budget	Under 700 ms for interactive use
Error tolerance	Wrong action must be caught by a verifier
Privacy boundary	Runs locally or inside tenant region
Fallback	Escalate to a larger model or ask the user
Evaluation set	500 real UI states with labels and traps

This framing exposes the value of small models. They can run closer to the user, cost less per call, support more retries, and handle background work without turning every feature into a cloud API dependency.

Why Encoder-Free Multimodality Matters

Google described Gemma 4 12B as a unified, encoder-free multimodal model. Traditional multimodal systems often use separate encoders to translate image or audio input into representations the language model can consume. That split can add memory, latency, and implementation complexity. A model that ingests modalities through a more unified path changes the deployment discussion, especially for local or workstation workloads.

Product teams should care about this for boring reasons. Boring reasons decide whether a model ships.

If a model needs a separate vision stack, a separate audio stack, custom preprocessing, and a large memory budget, teams will use it for explicit "analyze this" buttons. If the model fits into a laptop-class environment and can process mixed context with fewer moving pieces, teams can place it inside ordinary workflows:

• Inspect the current app screen before a local agent takes the next step.
• Read a diagram in a local engineering notebook.
• Summarize a short screen recording transcript plus a few frames.
• Extract structured fields from a scanned form without sending it to a third-party endpoint.
• Help an IDE assistant reason over a screenshot of a failing UI test.

The technical question becomes less glamorous: how often can the product afford perception?

Small Models Are Good System Components

Small models shine when the system limits the problem. A local model can classify the first message in a support case, redact sensitive spans, detect whether screenshots contain credentials, or create a search query. A coding model can propose test names, explain a compiler error, or summarize a diff for review.

Those jobs are system components.

The architecture often looks like this:

1. Deterministic code gathers context and strips irrelevant data.
2. A small model handles the first-pass interpretation.
3. The system checks the output against a schema or verifier.
4. A larger model handles uncertain or high-value cases.
5. The product records both the decision and the evidence.

This design gives the small model a narrow surface. It also lets the team measure the job directly. The model either extracted the right field, selected the right UI control, flagged the right sensitive span, or produced a useful query. The evaluation should expose where it fails.

The Failure Modes Change

Small models have constraints that product teams must respect.

They may hallucinate less eloquently than large models, but they still hallucinate. They may miss long-range context. They may follow formatting instructions less reliably. They may struggle with ambiguous visual details, dense tables, or multi-step reasoning. Local deployment can also create version drift: one user runs the 4-bit quantized model, another runs a newer checkpoint, and a third runs on a throttled laptop.

Treating the model as a component means designing around those failures:

• Use bounded prompts with a single task.
• Require structured output for downstream actions.
• Keep deterministic parsers in the path for data formats that work without model judgment.
• Add confidence thresholds that route uncertain cases upward.
• Log model version, quantization, hardware class, and prompt version.
• Build evals from the product's own inputs, not generic benchmark tasks.

The last point matters. A small model that scores well on general tests may still fail on your support screenshots, CAD notes, warehouse labels, or internal dashboards. A weaker public model can outperform a stronger one in a narrow domain if the task is well framed and the evaluation set matches production.

Local Deployment Changes Product Behavior

Local models change how often users ask for help. A remote model call feels like a resource. A local model call feels like part of the tool.

That difference changes interface design. You can run a local classifier as the user types. You can inspect a screenshot after each UI test failure. You can suggest a commit message without sending the diff outside the machine. You can precompute embeddings or summaries while the laptop is idle. You can let users turn the model off without breaking the product's core path.

Local deployment also gives enterprise buyers a different governance story. Security teams may still ask hard questions about training data, model license, logging, and output handling, but the data boundary becomes easier to explain. Source code, customer data, or regulated documents can stay inside the user's device or tenant-controlled environment.

A Selection Framework

I would evaluate a small model for a product job with six tests:

Test	Question
Job fit	Can one sentence describe the model's task?
Input control	Can the system trim context before the model sees it?
Output control	Can the model return a schema a verifier can check?
Latency	Does the model fit inside the user's interaction loop?
Escalation	Can uncertain cases route to a larger model or human?
Governance	Can the team explain where data goes and what gets logged?

If the answer is no across that table, a larger model will not fix the product design. It will hide it for a while.

Model Routing as Product Infrastructure

Once a product has more than one model, routing becomes infrastructure. The route should not live inside a pile of prompt conditionals. The product needs a policy layer that decides which model handles which task, which context it receives, and how the system escalates.

A simple routing table can start like this:

Task	First Model	Escalation Trigger	Escalation Target
Classify support message	Local small model	Low confidence or policy keyword	Cloud model plus policy checker
Summarize local notes	Local small model	Long context or missing citations	Larger local or tenant model
Inspect UI screenshot	Small multimodal model	Ambiguous element or destructive action	Stronger multimodal model
Draft code test	Local code model	Failing generated test or broad file edit	Cloud code model
Extract fields from PDF	Local OCR plus small model	Schema invalid or low evidence	Human review queue

Routing makes small models safer because the product can admit uncertainty. The system does not need the small model to handle everything. It needs the small model to handle the common cases cheaply and escalate the rest with useful context.

The router should log decisions. For each task, record task type, selected model, model version, context size, confidence, validator result, escalation path, latency, and cost. Those logs become the basis for product tuning. If a local model escalates 70 percent of screenshot tasks, the task is too broad or the model is wrong. If it handles 95 percent with low correction rate, it deserves more product surface.

A Concrete Eval Set

Teams often postpone evals because "model quality" feels too broad. A model job makes evals manageable.

For a screenshot-inspection model inside a browser agent, I would build an eval set with:

• 100 normal pages with labeled primary actions.
• 50 modal dialogs with close, cancel, and destructive buttons.
• 50 disabled controls.
• 50 pages with duplicate labels.
• 50 low-contrast or visually crowded states.
• 50 cases where the right answer is "ask the user."
• 50 adversarial states, including fake buttons and misleading banners.

Each example should include the screenshot, optional DOM text, expected structured output, and an explanation of the acceptable evidence. The model should return a bounded schema:

{
  "action": "click" ,
  "target": "Save changes button in footer",
  "confidence": 0.82,
  "evidence": ["visible label", "enabled button", "near form footer"],
  "requires_confirmation": false
}

The evaluator should score schema validity, target correctness, evidence quality, refusal quality, and latency. The team should inspect failures manually. A small model that fails on duplicate labels needs different context or a narrower job. A model that clicks destructive buttons without confirmation needs a policy gate, not another prompt adjective.

The Economics

Small models change the unit economics of AI features. A feature that calls a large remote model on every keystroke will either become expensive or get throttled. A feature that calls a local model for cheap classification and escalates one in ten cases can feel instant without draining budget.

Cost also affects design courage. When each call is cheap, product teams can add verification passes. A local model can generate a draft and then another local model can check whether the draft contains unsupported claims. A small multimodal model can inspect a UI state before and after an action. The system can afford to observe more often.

Small models need jobs, evals, boundaries, and routing. The useful ones will sit in the seams of products, doing repeatable work close to the user. That role will outlast leaderboard screenshots.

Sources

• Google: Introducing Gemma 4 12B
• Max Leiter: They're made out of weights