📡 Idea Radar 每天 8:00 / 18:00 推送 · 18:00 生成精选
hn · Ask HN · 55 分 ← 返回列表

Ask HN: The amount of AI bot traffic is out control?

原文链接 ↗ ·flyingcoder
We host a web app which has around 50,000 statically generated public pages, and the amount of bot traffic is insane. Facebook’s crawler requested the same pages over 3 million times in 2 days, and we’ve used Vercel and Cloudflare to block as much as possible but it’s not working. The AI scrapers are routing requests through residential proxies, and even after putting our entire app behind an Auth gate with cloudflare turnstile they’re creating accounts to get access to data that’s meant for hum

五维评分

付费意愿
68
需求真实度
55
个人可落地
55
竞争空间
20
新鲜度
88

AI 分析

痛点:AI爬虫和机器人流量绕过Cloudflare/Vercel及Turnstile,导致Web应用托管成本飙升、账号滥用和内容被非人类抓取。

目标用户:拥有大量公开页面或内容的Web应用运营者、SaaS团队、媒体/出版站点、独立开发者。

证据:Facebook’s crawler requested the same pages over 3 million times in 2 days, and we’ve used Vercel and Cloudflare to block as much as possible but it’s not working. ... our hosting costs are up like crazy ... even after putting our entire app behind an Auth gate with cloudflare turnstile they’re creating accounts to get access to data that’s meant for humans to read / use.

变现:按请求量或站点规模订阅的机器人流量识别与阻断SaaS;Cloudflare Worker/Edge Middleware插件;托管成本异常报警与自动限流服务。

MVP:2周内做一个Cloudflare Worker或日志分析工具:解析访问日志,识别异常高频AI爬虫、住宅代理IP段和单页重复请求,自动生成限流规则和托管成本估算面板,先面向小型SaaS和内容站。

竞品:Cloudflare Bot Management、DataDome、Akamai Bot Manager、AWS WAF Bot Control、hCaptcha、reCAPTCHA、Vercel BotID

摘要:发帖人运营约5万个静态页面的Web应用,Facebook爬虫2天内重复请求超300万次,AI爬虫通过住宅代理绕过Cloudflare/Vercel和Turnstile,甚至注册账号抓取人类数据,托管成本大幅上升。帖主已在尝试hCaptcha等方案,但热度仅5分2评论,尚未出现其他用户附和或明确付费提议。