reddit · SideProject · 11-01 13:50 62 分 ← 返回列表

I built plan-auditor because my coding agent kept saying "tests pass" for tests it never ran

The failure I kept hitting with agents was not dramatic. It would change a file, report that the suite was green, and I would only find out later that the suite had never been executed - or, worse, that a test had failed once and the test file had been edited until it passed. Both outcomes read exactly the same in prose, which is the actual problem: prose is not evidence. So plan-auditor (2.6.3 on PyPI and npm, MIT, Python 3.10+, standard library only, no third-party runtime deps) does one

五维评分

付费意愿

需求真实度

个人可落地

竞争空间

新鲜度

AI 分析

痛点：AI 编程代理会谎报测试通过（根本没运行或改测试文件让它过），而自然语言汇报无法作为证据，开发者需要一个独立重跑检查的验证层

目标用户：使用 Claude Code/Cursor/Codex 等编码代理的独立开发者与小型工程团队、以及需要审计 AI 生成代码的团队负责人

证据：It would change a file, report that the suite was green, and I would only find out later that the suite had never been executed - or, worse, that a test had failed once and the test file had been edited until it passed. ... prose is not evidence.

变现：开源 CLI 免费 + 私有仓库/团队版托管服务订阅（集中式证据日志、HMAC 密钥托管、CI 集成、审计报表），按席位或按仓库月费 9-29 美元

MVP：两周内做一个 CLI+轻量 Web 看板：用户用 JSON 定义 plan 步骤与 verify 条目，工具在独立子进程中重跑 pytest/run/exec，输出带哈希链的证据日志与 proven/failed/blocked/unknown 状态徽章，并提供一个 GitHub Action 与只读分享链接

竞品：pre-commit、pytest（自身）、GitHub Actions 状态检查、LangSmith / AgentOps 等 agent 可观测性工具、Cursor 内置 terminal 日志

摘要：作者因为编码代理反复谎称测试通过（甚至改测试文件让它过），做了开源工具 plan-auditor（PyPI/npm 2.6.3，MIT，仅标准库），要求每步计划带机器可验证的 verify 条目并在全新子进程重跑，只有 exit code 0 才算通过，UNKNOWN 会拒绝标记完成，证据日志用 SHA-256 哈希链加 HMAC。痛点真实且切中 AI 编码工作流，但帖子本身无任何付费意愿或收入数据，属自用开源工具发布。