An Exact Match Between C++ and C

Fix C++ parser and Fix C parser produced the same result from claude-mem's title classifier: tier: "exact", method: "exact", score: 1. The IDF callback was never called. Both titles had already become fix c parser.

The rare-token veto works later in the pipeline. It can reject a near-duplicate pair despite a high cosine score, but it cannot recover a distinction removed by the exact-match normalizer. Tightening fuzzy thresholds leaves that earlier identity decision unchanged.

I examined revision d8bf121ae98063e5e3158befff33a9d635c0246b, compiled five unchanged TypeScript modules using TypeScript 5.9.3, and ran them with Node 24.19.0 on Linux. Fourteen custom scenarios and eight published golden pairs passed. The custom inputs and IDF weights were synthetic. No observation database, worker, embedding service, MCP server, or model was executed.

Two Text Transforms Make Different Promises

claude-mem captures and stores observations for later agent context. Its inspected package configuration declares version 15.9.0, ESM modules, Node ^20.19.0 || >=22.12.0, and Bun >=1.1.31. This replay compiled a pure classification slice; it did not install that application or run its Bun test suite.

The relevant modules are small and separate from storage:

Module	Role
normalize.ts	Normalized-title identity and whitespace tokenization
nearDuplicate.ts	Orders the exact and candidate decisions
tfidfCosine.ts	Cosine over distinct tokens weighted by IDF
idfVeto.ts	Rejects disagreement on a sufficiently rare token
idf.ts	Smoothed IDF and a reusable lookup function

normalizeTitle() applies NFC normalization, lowercases, replaces characters outside Unicode letters, numbers, marks, and whitespace with spaces, then collapses whitespace. It returns an empty normal form unless at least one letter or number remains.

tokenizeWs() instead lowercases and splits on whitespace. It preserves identifiers such as download.rs and rdlp-api as individual tokens. That matters for IDF: splitting a rare compound identifier into common fragments would weaken its discriminating value.

Both transforms have a reason to exist. They simply do not define the same equivalence relation.

Exact Returns Before the Identifier Check

classifyPair() first checks whether the first title has a nonempty normal form equal to the second title's normal form. If so, it returns exact immediately. It does not compute shared tokens, cosine, or a veto.

The synthetic collision checks returned:

Title A	Title B	Shared normal form	Classifier tier
Fix C++ parser	Fix C parser	fix c parser	exact
Fix C# parser	Fix C++ parser	fix c parser	exact
Fix download.rs parser	Fix download rs parser	fix download rs parser	exact

For the filename pair, calling the token veto directly with synthetic high IDF weights returned true. Calling the complete classifier still returned exact. The veto was not malfunctioning; that branch never ran.

I also repeated the C++/C pair with three fuzzy configurations. These included a cosine threshold of 1.1, a veto threshold of 0, and a distinct-shared-token minimum of 999. Every configuration returned exact without invoking IDF. The out-of-range cosine value was a deliberate branch-order fixture, not a recommended application setting.

These examples test a semantic limit of punctuation-insensitive identity. Hyphenation changes may be harmless formatting; C, C++, and C# name different languages. The normalizer applies the same rule to all of them.

The Veto Still Protects the Candidate Path

For non-exact titles, the classifier requires two distinct shared whitespace tokens by default. It then computes cosine using a set of tokens with one IDF weight per token, rather than increasing weight for repeated words.

The veto examines the symmetric difference of those token sets. Any differing token with idf > vetoThetaIdf prevents a candidate classification, even when cosine exceeds its threshold.

To isolate that behavior, I constructed pairs sharing 25 synthetic tokens, each with weight 1. The differing identifier tokens had weight 2. With cosine threshold 0.8 and veto threshold 1.5, the measured result was:

shared tokens: w0 through w24
different tokens: rdlp-api / rdlp-plugin
cosine: 25 / 29 = 0.862069...
veto: true
tier: none

Replacing those identifiers with code and security, both assigned weight 1, yielded cosine 25/26, approximately 0.961538. The veto did not fire, and the classifier returned candidate.

That is a useful conservative distinction. The lexical system can nominate a pair for review without asserting that its observations are identical. A common word can still carry the entire semantic difference, so a high score alone is insufficient evidence for silent consolidation.

The shared-token prefilter has its own purpose. A pair with just one shared rare token had cosine above 0.99 in another synthetic fixture. The default minimum rejected it with score 0, before cosine was used for classification. Relaxing the minimum to one produced a candidate. Repeating the shared token did not satisfy the default distinct-token count.

What the Storage Code Adds

Storage inspection makes the exact normal form consequential. computeTitleNormKey() hashes the project, normalized platform source, main/subagent scope, and normalized title into an indexed key. An empty normal form produces a null key.

In the inspected SessionStore insertion path, enabled deduplication first handles identical-content retries. Unless skipTitleDedup is set, it then looks up a canonical row by the title key. On a hit, it increments occurrence_count, reinforces the existing observation, returns its ID, and continues before inserting the new observation's fields.

These are code-inspection findings. The replay did not perform that transaction or measure lost observations in a real database. It demonstrates the classifier collision; the storage code shows that the same normal form participates in row reuse within the relevant scope. A stronger hash cannot restore punctuation removed before hashing.

Preserve Marks Without Equating Empty Titles

The replay also checked the normalizer's useful safeguards. Composed and decomposed spellings of an accented title reached exact through NFC normalization. A meaningful vowel mark distinguished two Indic titles. A standalone combining mark normalized to empty and did not become an exact match, even against itself.

Null, punctuation-only, and emoji-only pairs also avoided exact. That nonempty guard prevents unrelated symbolic titles from becoming identical merely because normalization erased both. It does not address collisions between nonempty technical titles such as the language names above.

The upstream Unicode tests document the mark-preservation cases. I inspected those tests and executed equivalent inputs in the linked probe; I did not run them through Bun.

Reproduce the Classifier Slice

The probe checks digests for the five source modules, their emitted JavaScript, and the published golden fixture. The emitted digests correspond to TypeScript 5.9.3. The source root used the pinned repository's package.json, preserving its ESM setting.

The actual scoped compilation used these flags, with absolute paths to the five named source files:

tsc --target ES2022 --module NodeNext --moduleResolution NodeNext \
  --skipLibCheck --typeRoots /workspace/cc-portfolio/node_modules/@types \
  --outDir /tmp/mem-note-nov5/dist \
  src/services/dedup/normalize.ts src/services/dedup/nearDuplicate.ts \
  src/services/dedup/idfVeto.ts src/services/dedup/tfidfCosine.ts \
  src/services/dedup/idf.ts

Compilation exited 0. The actual cloud replay command was:

node public/blog/punctuation-before-dedup-veto/classifier-probe.mjs \
  /tmp/mem-note-nov5 /tmp/mem-note-nov5/dist

It exited 0. Selected output:

cplusplus_vs_c: exact; normal="fix c parser"; IDF calls=0
filename_vs_words: exact; direct token veto=true
strict_fuzzy_settings: exact under all 3 configurations; IDF calls=0
rare_identifier: cosine=0.862069 above 0.8; veto=true; tier=none
common_discriminator: cosine=0.961538; veto=false; tier=candidate
PASS: 14 custom scenarios and 8 upstream golden pairs; pure classifier only; no database or model

The eight pairs came from the published golden fixture. The probe rebuilt its document-frequency model and checked its expected tiers using Node assertions. This validates those fixture decisions, not the broader real-database evaluation described in source comments.

The practical boundary is the first return. Fuzzy scoring can be cautious about identifier disagreement while exact normalization remains permissive about it. Reviewing only cosine thresholds would miss the rule that already decided identity.