Ship Check: one honest verdict before you deploy

1. A chatbot where three small problems add up to one big one

A Next.js chatbot that deploys to Vercel. A client component calls OpenAI directly with a NEXT_PUBLIC_ key, and the server route /api/chat calls gpt-4o with no sign-in, no rate limit and no output limit. Each finding alone is a to-do. Together they mean anyone can run up the OpenAI bill.

"use client";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
  dangerouslyAllowBrowser: true,
});

export async function POST(req: Request) {
  const { messages } = await req.json();
  const completion = await openai.chat.completions.create({
    model: "gpt-4o",
    messages,
  });

NEXT_PUBLIC_OPENAI_API_KEY=sk-pro…Z7LJ
OPENAI_API_KEY=sk-pro…Z7LJ

**NOT YET: 0 will break, 1 leak, 3 will cost you (2 critical cross-checks)**

Do these first:
1. Rotate the leaked key (#1) at the provider; removing it from code does not un-leak it.
2. Set the provider spending cap, then add sign-in, rate limits and output limits (#1, #2, #3, #4).

Repo: chatbot-vercel · 4 of 4 skills ran · 0.9s · history: 1 commit scanned · commit 4595d438d6
…
## Cross-checks (what no single skill sees)
1. **CRITICAL CC1: Anyone can run up your OpenAI bill** (items #1, #2, #3, #4)
   - Why: The OpenAI key is public (or will be once this ships), and the code shows this account makes uncapped or public OpenAI calls, so nothing in the code or the account stops someone else's usage from becoming your bill.
   - Fix: Rotate the OpenAI key now (revoke and reissue); deleting it from the code does not un-leak it.
   - Fix: Set the provider's spending cap or kill switch today: Hard cap: Yes: a monthly hard spend limit on the organization or a project makes affected requests fail with 429 (checked 2026-10-13).
   - Fix: Then fix the uncapped calls listed (output limits, sign-in, rate limits).
2. **CRITICAL CC3: Open paid endpoint: POST /api/chat** (items #2)
   - Why: No sign-in and no rate limit in front of a paid call means one script can call it as fast as the network allows, and every call is billed to you.
   - Fix: Require a signed-in user before the paid call, or put a per-IP rate limit and a bot check on it.
   - Fix: Add a small output limit (max_tokens) and a daily quota per user.
3. **HIGH CC2: Secret in browser code, and this project deploys to vercel: treat it as exposed** (items #1)
   - Why: Key Leaked marks a browser-prefixed secret 'at risk' when it sees no build output and no linked host. The host (vercel, from broke-after-deploy) means any build made with this value set bundled it into public JavaScript.
   - Fix: Ask: was the site deployed with this value set? If yes (or unsure), rotate the key first.
   - Fix: Remove the browser prefix and read it only in server code.
…
## Will leak (1)
### #1 OpenAI project API key (sk-pro...Z7LJ) **CRITICAL**
Found by: key-leaked at risk #6,7 + bill-spiked #16 · cross-checks CC2, CC1
- **Status:** Treat as exposed: Key Leaked rated it at risk, but this project deploys to vercel (broke-after-deploy), so any deploy made with this value set put it in public JavaScript.
- **Finding:** OpenAI project API key in .env.local and 1 more place: behind a browser-exposed env prefix, so the build bundles it into client JavaScript; used from browser code to call a paid API.
…
## Will cost you (3)
### #2 Public endpoint calls OpenAI with no sign-in or rate limit **CRITICAL**
Found by: bill-spiked #4,5 · cross-checks CC1, CC3
- **Finding:** app/api/chat/route.ts:6 (POST /api/chat) calls an LLM (OpenAI gpt-4o at app/api/chat/route.ts:8) and anyone on the internet can call it: there is no authentication check and no rate limit.
…
### #3 No output limit on gpt-4o call **CRITICAL**
Found by: bill-spiked #1 · cross-checks CC1
- **Finding:** app/api/chat/route.ts:8 calls gpt-4o with no output limit (no max_tokens or equivalent), so each reply can run to the model's maximum. Reached by anyone on the internet via POST /api/chat (app/api/chat/route.ts:6).
…
### #4 Whole conversation history sent on every call **HIGH**
Found by: bill-spiked #7 · cross-checks CC1
- **Finding:** app/api/chat/route.ts:8 sends the full message list the client sends (req.json().messages) to gpt-4o on every call, with no window or truncation. Each new turn re-bills every earlier turn, so cost grows with conversation length, and a client can send any size it likes. Reached by anyone on the internet via POST /api/chat (app/api/chat/route.ts:6).
…

components/Assistant.tsx
-import OpenAI from "openai";
-const openai = new OpenAI({
-  apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
-  dangerouslyAllowBrowser: true,
-});
…
-    const r = await openai.chat.completions.create({ model: "gpt-4o-mini", max_tokens: 300, messages: [{ role: "user", content: q }] });
-    setAnswer(r.choices[0].message.content ?? "");
+    const r = await fetch("/api/chat", { method: "POST", body: JSON.stringify({ messages: [{ role: "user", content: q }] }) });
+    const data = await r.json();
+    setAnswer(data.reply?.content ?? "");

app/api/chat/route.ts
+import { auth } from "@/auth";
+import { ratelimit } from "@/lib/ratelimit";
…
 export async function POST(req: Request) {
+  const session = await auth();
+  if (!session?.user) return NextResponse.json({ error: "sign in" }, { status: 401 });
+  const { success } = await ratelimit.limit(session.user.id);
+  if (!success) return NextResponse.json({ error: "slow down" }, { status: 429 });
   const { messages } = await req.json();
   const completion = await openai.chat.completions.create({
     model: "gpt-4o",
-    messages,
+    max_tokens: 800,
+    messages: messages.slice(-20),
   });

.env.local, .env.example
-NEXT_PUBLIC_OPENAI_API_KEY=…

2. Nothing leaks, nothing costs, and it still says NOT YET

A small Node notes server. No keys, no paid calls, no deploy problems. But one test fails: the slug function keeps punctuation. The scanners find nothing, and Ship Check still blocks, because a failing test is direct evidence the code does not work (cross-check CC5).

function slugify(title) {
  return title.toLowerCase().replace(/ /g, "-");
}

test("drops punctuation", () => {
  assert.strictEqual(slugify("Hello, World!"), "hello-world");
});

**NOT YET: 0 will break, 0 leak, 0 will cost you, 1 failing check**

Do these first:
1. Make the failing check pass (#1); fix the code, not the test.

Repo: notes-failing · 4 of 4 skills ran · 0.8s · history: 1 commit scanned · commit 7e7acb505f
…
## Cross-checks (what no single skill sees)
1. **BLOCKER CC5: 1 check failing: NOT YET regardless of everything else** (items #1)
   - Why: A failing test, type check or build means what you ship is not proven to work, even if nothing else was found.
   - Fix: Fix the code until the checks pass, then run Ship Check again.

## Not proven (1)
### #1 Tests fail: `npm run test` **BLOCKER**
Found by: prove-it-works FAIL · cross-checks CC5
- **Finding:** `npm run test` exited with code 1.
- **Evidence:**
  - `$ npm run test (exit 1, 0.34s)`
…
  - `  error: |-`
  - `  code: 'ERR_ASSERTION'`
  - `  name: 'AssertionError'`
  - `  expected: 'hello-world'`
- **Impact:** What ships is not proven to work; a failing test usually means a broken feature or a failed deploy.
- **Fix:**
  1. Fix the code until `npm run test` passes, then run Ship Check again. Do not skip, loosen or delete tests to get to green.
…
## Skills run
| Skill | Folder used | Status | Its verdict | Time |
| --- | --- | --- | --- | --- |
| broke-after-deploy | broke-after-deploy (schema 2) | ok | No known deploy breakers found. | 0.3s |
| key-leaked | key-leaked (schema 2) | ok | No exposed secrets found | 0.4s |
| bill-spiked | bill-spiked (schema 2) | ok | No uncapped spend found in the code. | 0.5s |
| prove-it-works | prove-it-works (schema 2) | ok | NOT DONE: 1 failing check, 0 problems in the change, no change to inspect | 0.8s |

Checks run by prove-it-works: `npm run test` FAIL

Notes from the skills (information, not gaps):
- broke-after-deploy: Host not detected, so host-specific problems (serverless file writes, SQLite, ports) are reported as 'check' rather than 'will break'. Pass --target with your host for a definite answer.
…

 function slugify(title) {
-  return title.toLowerCase().replace(/ /g, "-");
+  return title
+    .toLowerCase()
+    .replace(/[^a-z0-9\s-]/g, "")
+    .trim()
+    .replace(/\s+/g, "-");
 }

3. A clean shop: UNVERIFIED first, SAFE TO SHIP only after the tests run

An Express shop on Render with one AI feature done right: the route needs a signed-in user, has a rate limit, and caps output at 150 tokens. It also has two look-alikes that should not be flagged: a Stripe publishable key in browser code (public by design) and the OpenAI key in a gitignored .env that was never committed and is read only by server code.

// src/server.js
const limiter = rateLimit({ windowMs: 60 * 1000, max: 10 });
app.post("/api/describe", requireUser, limiter, describe);

// src/describe.js
  model: "gpt-4o-mini",
  max_tokens: 150,

// public/checkout.js
// Stripe publishable key: public by design.
const stripe = Stripe("pk_live_TCtk…NMg7");

**UNVERIFIED: tests and build not run (--no-tests); nothing will break, leak or cost you in what was checked**

Do these first:
1. Set the provider spending cap, then add sign-in, rate limits and output limits.

Repo: clean-shop · 3 of 4 skills ran · 0.5s · history: 1 commit scanned · commit 6e1aeb0d46
…
## Not verified
- tests and build not run (--no-tests)
…
## Skills run
| Skill | Folder used | Status | Its verdict | Time |
| --- | --- | --- | --- | --- |
| broke-after-deploy | broke-after-deploy (schema 2) | ok | No known deploy breakers found. | 0.3s |
| key-leaked | key-leaked (schema 2) | ok | No exposed secrets found | 0.4s |
| bill-spiked | bill-spiked (schema 2) | ok | No uncapped spend found in the code. | 0.5s |
| prove-it-works | prove-it-works | skipped |  | - |
…

**SAFE TO SHIP**

Do these first:
1. Set the provider spending cap, then add sign-in, rate limits and output limits.

Repo: clean-shop · 4 of 4 skills ran · 0.9s · history: 1 commit scanned · commit 6e1aeb0d46
…
## Skills run
| Skill | Folder used | Status | Its verdict | Time |
| --- | --- | --- | --- | --- |
| broke-after-deploy | broke-after-deploy (schema 2) | ok | No known deploy breakers found. | 0.4s |
| key-leaked | key-leaked (schema 2) | ok | No exposed secrets found | 0.6s |
| bill-spiked | bill-spiked (schema 2) | ok | No uncapped spend found in the code. | 0.6s |
| prove-it-works | prove-it-works (schema 2) | ok | CHECKS PASS (not fully verified): all 1 check passed, no problems in the change; there is no change to inspect | 0.9s |

Checks run by prove-it-works: `npm run test` PASS
…

       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+      - run: npm ci
-      - run: python3 .claude/skills/ship-check/scripts/ship_check.py . --ci --no-tests
+      - run: python3 .claude/skills/ship-check/scripts/ship_check.py . --ci --fail-on leak,failing