Live demo: AI says "tests pass". Does it?

Three small pull requests on the same shop project, written the way AI coding agents often write them. Each box under a change is the real, unedited output of buildproof ci on that change.

1. The AI changed the discount and hid the failing test

What the AI said: “All tests pass. Updated the discount to 20%.”

--- a/src/cart.js
+++ b/src/cart.js
-exports.discount = (t) => (t > 100 ? t * 0.9 : t);
+exports.discount = (t) => (t > 100 ? t * 0.8 : t);
--- a/test/cart.test.js
+++ b/test/cart.test.js
-test('discount over 100', () => { ... });
+test.skip('discount over 100', () => { ... });

NOT PROVEN exit code 1

$ buildproof ci --base main
NOT_PROVEN files=2 base=main | Checks were weakened: a test was skipped; an expected value in a test was changed

## BuildProof: NOT PROVEN

Checks were weakened: a test was skipped; an expected value in a test was changed

Compared with `main`: 2 changed file(s).

### Checks weakened in this change
- **A test was skipped** `test/cart.test.js` - `test.skip('discount over 100', () => { assert.strictEqual(c.discount(200), 180); });`
- **An expected value in a test was changed** `test/cart.test.js` - `test('discount over 100', () => { assert.strictEqual(c.discount(200), 180); });`

### Checks on this pull request
- test: pass (1/1 tests)

### Checks on main
- test: pass (2/2 tests)

_BuildProof is fail-closed: anything it cannot see is a fail, never a pass._

2. The AI "simplified" the cart total

What the AI said: “Refactored total() for readability. Tests should still pass.”

--- a/src/cart.js
+++ b/src/cart.js
-exports.total = (items) => items.reduce((s, i) => s + i.price * i.qty, 0);
+exports.total = (items) => items.reduce((s, i) => s + i.price, 0);

NOT PROVEN exit code 1

$ buildproof ci --base main
NOT_PROVEN files=1 base=main | The change broke: test (total adds price times quantity)

## BuildProof: NOT PROVEN

The change broke: test (total adds price times quantity)

Compared with `main`: 1 changed file(s).

### Checks on this pull request
- test: FAIL (1/2 tests)

### Checks on main
- test: pass (2/2 tests)

_BuildProof is fail-closed: anything it cannot see is a fail, never a pass._

3. The AI added free shipping, with a real test

What the AI said: “Added free shipping from $50 and a test for it.”

--- a/src/cart.js
+++ b/src/cart.js
+exports.shipping = (t) => (t >= 50 ? 0 : 5);
--- a/test/cart.test.js
+++ b/test/cart.test.js
+test('free shipping from 50', () => { ... });
--- a/.buildproof/spec.md
+++ b/.buildproof/spec.md
+- Free shipping from $50 => test: free shipping from 50

PROVEN exit code 0

$ buildproof ci --base main
PROVEN files=3 base=main | The checks really ran, nothing regressed, no test was weakened, every spec requirement is proven by a passing test.

## BuildProof: PROVEN

The checks really ran, nothing regressed, no test was weakened, every spec requirement is proven by a passing test.

Compared with `main`: 3 changed file(s).

### Spec
- met: Free shipping from $50 (test: `free shipping from 50`, new in this change)

### Checks on this pull request
- test: pass (3/3 tests)

### Checks on main
- test: pass (2/2 tests)

_BuildProof is fail-closed: anything it cannot see is a fail, never a pass._

Try it on your own pull requests

Install, then add the GitHub check to your repository:

npm i -g buildproof
buildproof ci --init

Optional: list what the change must do in .buildproof/spec.md, one line per requirement with the test that proves it. A requirement without a passing test is never accepted.

The 14-day free trial includes both. After the trial: the spec check is in Pro and up, pull request checks in Team and Enterprise. See plans