Built a scraper that fixes its own broken selectors when a site redesigns

reddit SideProject · 原文链接 ↗ · 56 分

Hey r/SideProject , I kept hitting the same wall on past scraping projects: everything works fine until the target site redesigns, a CSS selector silently stops matching, and you don't find out until the data looks wrong days later. So I built SelectorLock — a scraping API where you bring your own proxy, give it a URL and a schema for what you want, and if a selector breaks, it tries to repair itself automatically using an LLM (cheap model first, only escalates if that fails) instead of you having to notice and fix it by hand. Once it finds the right elements again, it remembers the fi

五维评分

AI 分析

痛点：网站改版后 CSS selector 静默失效，爬虫数据出错数天后才被发现，需要人工修复。

目标用户：做网页抓取、数据监控、竞品采集的独立开发者、数据工程师和小团队。

证据：if a selector breaks, it tries to repair itself automatically using an LLM (cheap model first, only escalates if that fails) instead of you having to notice and fix it by hand. ... Still early days — free tier if anyone wants to poke at it

变现：按抓取请求量、自动修复次数或监控页面数做 API SaaS 订阅；免费层引流，付费层提供更高频率、私有代理、告警和团队协作。

MVP：做一个面向 Playwright/Puppeteer 的轻量监控与自愈 SDK：用户配置 URL 和字段 schema，定时抓取，selector 失败时调用低成本 LLM 生成候选选择器并回写，提供失败告警和修复历史。

竞品：ScrapingBee、Bright Data、Apify、Zyte、Kadoa、Nimble、ScrapeGraphAI

摘要：发帖人做了一个名为 SelectorLock 的抓取 API，能在 CSS selector 失效时用 LLM 自动修复并记住修复结果，目前提供免费层，仍处于早期。痛点真实且常见，但帖子中没有付费意愿或收入数据，且抓取 API 和自愈型爬虫已有不少竞品。