PilotCraft

The problem

• Teams considering AI adoption often jump straight to "build an AI feature" without separating what they actually know from what they’re assuming, without a plan for how to un-adopt it if it fails, and without deciding upfront what would count as success.
• A structured first pass at "is this workflow actually a good candidate for AI?" needed to exist before proposing a pilot: one that’s allowed to say no.

The approach

• Takes a free-text description of a workplace problem (up to 4,000 characters) and returns one structured report: a suitability rating (strong/conditional/poor), a proposed future workflow, a human/AI responsibility split, stakeholder impact, adoption barriers, a phased pilot plan with owners and evidence to collect per phase, success metrics with baselines and targets, explicit stop/revise/scale decision criteria, and a 0–100 "readiness score" the UI labels directly as a planning heuristic, not a measurement.
• The system prompt explicitly instructs the model to weigh non-AI alternatives (better documentation, a process change, an existing tool) before recommending AI, and to let that pull the rating toward "conditional" or "poor" instead of defaulting to "strong": a tool that always says "yes, use AI" isn’t useful.
• Every report is required to separate user-provided facts, assumptions the model made to fill a gap, and evidence that’s missing entirely, and the prompt explicitly forbids inventing a plausible-looking baseline number when none was given.
• A separate evaluation harness (evals/) runs 9 synthetic workplace scenarios (spanning strong/conditional/poor candidates, a missing-baseline case, an obvious non-AI alternative, sensitive data, adoption resistance, and an unsupported ROI claim) through the real /api/analyze route (not a mock), scoring each report against 7 deterministic checks: no fabricated baselines, a named evidence gap, a real non-AI alternative named when one exists, a substantive human-review trigger per risk, ratings calibrated to the evidence given, sensitive data flagged as a risk, and unsupported claims marked unverified.

What broke (and how it got fixed)

• Two regressions specifically lived in the NODE_ENV=production code path that neither `npm run dev` nor the unit tests exercised: an Express 5 wildcard-route bug and a request-body handling bug. Fixed by adding a CI smoke test that runs against the actual built production server instead of only the dev server.
• An earlier draft of the README claimed WCAG 2.2 AA compliance that had never actually been tested. Caught on review and removed outright rather than "fixed" with a retroactive audit: the README’s own "Honest limitations" section documents the correction directly instead of quietly deleting the claim.
• An earlier version ran Express 4, which transitively pulled a vulnerable qs/body-parser range (moderate severity, low actual exposure since the app never parses query-string arrays). Resolved by moving to Express 5, which carries a patched qs.

Outcome

• A live run against the real Gemini API (not fixtures), dated September 4, 2026, completed all 9 synthetic scenarios with no scenario errors, and all 58 applicable deterministic checks passed: full detail in the repo’s evals/results.md and evals/results/latest.json.
• 74 unit tests pass, covering the retry/fallback orchestration, the JSON-repair flow, the Zod schema’s boundaries, the real Express route end-to-end (a missing or malformed request body returns a clean 400, oversized scenarios are rejected, the per-IP rate limit returns 429), and the Gemini model contract itself: that the system instruction and response schema actually state what the README claims they do.
• The README states plainly that this is one live model run over nine synthetic scenarios using deterministic heuristic checks; it demonstrates tested behavior under those conditions, not statistical reliability or a validated outcome in a real organization.

Stack

Honest scope

A personal tool, not a hackathon submission, built to think through AI-adoption decisions before proposing a pilot, not shipped as a finished product. No authentication and no user accounts. A demo is published through Google AI Studio: you can enter a workplace scenario, try a sample one, or view a pre-built sample report that makes no AI call. Reports stay in memory for the session, but scenario text and generated reports are processed by Gemini, and the demo page warns not to enter confidential information because free-tier data may be used. Having a published demo doesn’t mean it has workplace adoption or validated business results. Not validated in a real organization: no pilot run through it has been tracked to a real outcome. The 0–100 readiness score is an explicitly-labeled model heuristic, not an objective measurement. Built with AI coding assistance (Claude Code), directed and reviewed by the repository owner, disclosed directly in the README.