Two Holders After Deleting the Lease File

I deleted a lease file while a test process held its lock. A second process then acquired a lease for the same database path. Both processes remained alive, each holding a different lease-file inode. A third contender was refused by the second holder.

This was a controlled test of Open Duet's standalone lease module in a temporary directory. No agent, model, or backend server was started. The result explains an operational detail that is easy to miss: preserving a live lock file's identity is part of the single-backend guarantee. Its continued presence after a crash is normal.

I examined revision cca66e7a5028b23f352a24671e749afb6933da76 and ran the unchanged module using Python 3.12.14 on Linux. The complete backend declares Python >=3.14; this isolated standard-library test does not validate that application's installation or supported runtime. Windows locking and network filesystems were outside the experiment.

Backend Ownership Covers More Than Database Writes

Open Duet is a local agent workspace with conversations, approvals, engine adapters, routines, and optional isolated computers. Its architecture puts those responsibilities behind a FastAPI backend, with SQLite or PostgreSQL for persistence. The implementation includes database migrations, engine process management, scheduling, and dedicated process-level reliability tests.

SQLite can arbitrate database transactions. It cannot decide which application process should recover interrupted routines, sweep stale turns, or manage the same agent workspace directories. Those are backend ownership decisions. Two independent startup and recovery loops could interfere even if their SQL writes remain individually valid.

The application lifespan acquires the SQLite lease before entering _serving(). That serving function starts recovery and background work. The lease is released in a finally block after serving ends. This ordering is source inspection; I did not execute FastAPI startup.

The lease module resolves the configured database path and appends .backend-lease to its filename. It opens that sidecar with O_RDWR | O_CREAT, takes a nonblocking exclusive lock, and writes holder metadata only after acquisition succeeds.

On the tested Linux path, the authority is flock() on the open file descriptor. The JSON contains a PID, host, start time, and database path for diagnostics. A PID in that record does not establish whether a lock remains held.

Processes, Signals, and Actual Files

The replay probe checks the module's SHA-256 digest before loading it. It creates temporary SQLite fixture files and launches fresh Python workers that call the actual acquire() function. Holder workers wait on standard input, keeping their lease descriptor open until explicitly released or signaled.

Obtain backend/app/core/data_lease.py from the pinned revision, then point the probe at that file. From the portfolio checkout, I ran:

python --version
python public/blog/lease-file-two-holders/lease-probe.py /tmp/open-duet-note25/data_lease.py

Both commands exited zero. The worker's exit code 7 is a convention defined by this probe for DataDirInUse; it is not the backend application's startup exit code.

Case	Observed result
Another process acquires the same database path	Refused; diagnostic holder PID matched the live process
Contender uses a symlink to the database	Refused; path resolution selected the same lease
Contender uses a different database in the same directory	Acquired successfully
Holder JSON corrupted without replacing the inode	Still refused; holder details were empty
Holder terminated with SIGTERM	Process exited -15; old metadata remained; next acquisition succeeded
Holder killed with SIGKILL	Process exited -9; old metadata remained; next acquisition succeeded
Same process acquires twice	Both calls returned the same lease; only the final release freed it
Live lease file unlinked before another acquisition	Two live holders acquired distinct lease inodes

Selected stdout:

same database: contender exit=7; live holder PID reported
symlink spelling: contender exit=7; same resolved lease
corrupt holder metadata: contender exit=7; holder details empty
SIGTERM: holder exit=-15; old metadata remains; next acquire exit=0
SIGKILL: holder exit=-9; old metadata remains; next acquire exit=0
same-process sharing: first release keeps lock; last release frees it
unlink live lease: two live holders; distinct lease inodes; third contender exit=7
split-lock cleanup: stopping old holder leaves new lock held; stopping both frees it

The crash cases needed no file cleanup. Once the holder exited, a fresh worker acquired the existing sidecar and replaced its stale diagnostic record. Corrupting that record while its holder was alive also did not grant permission. Lock acquisition and metadata parsing are separate steps, which is the useful property here.

The upstream lease tests cover contention, signals, and shared acquisition. I inspected those tests; the reported output comes from the separate probe, not a run of the application's pytest suite.

Unlink Changes the Object Being Locked

The destructive case ran only against a temporary fixture. Its sequence was:

1. Holder A acquired the sidecar and kept its descriptor open.
2. The probe removed the sidecar's directory entry.
3. Holder B called acquire() for the same database path. O_CREAT created a new sidecar inode, which B locked successfully.
4. The probe confirmed both holders were still running and their lease descriptors referred to different inodes.
5. A third worker was refused and reported B as the holder. Ending A left B's lock held; ending B finally allowed another acquisition.

The database path never changed. The sidecar's filesystem identity did. The old lock remained associated with A's open file while later callers opened the replacement file. Checking the path or reading its JSON could see only the replacement.

This is an advisory process guard for cooperating launchers. Deleting or replacing its live sidecar violates the shared-object assumption. A cleanup script should leave the file in place and let a new acquisition determine ownership. A leftover PID record alone provides no reason to remove it.

Inside one process, the module also keeps a map of held leases and a reference count. That supports layered startup: two acquisitions share a lock, and one release does not make it available to another process. This local count and the kernel lock solve different parts of ownership. The first prevents premature release by cooperating callers; the second arbitrates between processes opening the same sidecar.

PostgreSQL Uses a Different Ownership Boundary

The scheduler lock permits multiple PostgreSQL backends while selecting one routine scheduler through a session-level advisory lock on a retained connection. A standby backend retries acquisition, and a holder checks that its connection is still alive. I inspected this implementation and its tests; I did not connect to PostgreSQL or verify failover.

That distinction keeps the claim specific. The SQLite lease arbitrates backend ownership through one persistent sidecar on the tested local filesystem. The PostgreSQL mechanism arbitrates scheduler ownership through a database session. Neither the successful crash tests nor the two-inode reproduction establishes end-to-end routine delivery, exactly-once execution, or safety on a shared network filesystem. They establish where the local process guard holds, and which filesystem operation splits it.