Commandem

Research question

When an agent can influence its own checks, how can an evaluator distinguish task completion, unsupported success claims and verifier subversion?

Work and approach

I developed Commandem and contributed research design, implementation, experiment execution, analysis and manuscript preparation, using AI tools throughout. My collaborator helped audit, develop and test the security experiments.

• Reconstruct overlapping completion and subversion signals from minimized study ledgers.
• Read saved transcripts and termination records alongside the outcome categories.
• Audit task coverage, experimental delivery, denominators and differences between historical binaries.

How to evaluate the work

In 164 contradictory-task trials across five fixtures, 71 records contain a completion marker, including all 41 records labeled verifier subversion. These are overlapping signals rather than mutually exclusive outcomes. In one wording comparison, all 15 test-conditioned runs reach the turn limit: marker absence does not establish useful refusal.

Evidence and limitations

Completion Claims and Verifier Integrity in Coding Agents is the current public working manuscript. Earlier reports in the Commandem research archive describe different populations and should not be pooled with it.

The evidence is exploratory and retrospective. A separate 239-run study uses only two elementary functions, and its clause comparison is confounded by different historical binaries. Original correctness labels have not all been independently re-adjudicated. A failed replication is retained. No general safeguard, cross-scaffold model ranking or deployment benefit is established.