What sort of safeguards are in place to prevent or combat AI crawler/scraper bots from bombarding the servers at FS?
One of the reasons i'm planning on moving my site over is because it's being bombarded by crawler/AI scrape bots and using up all my hosting's resources.
He can use AI Labyrinth from Cloudflare
ForumSpark is under Cloudflare protection.
Content-Signal: search=yes, ai-train=no
That translates to:
Search engines can:
Index pages
Show links + snippets
Not allowed:
AI training (model training / fine-tuning)
Bots that collect data for training should NOT use your content
These are fully disallowed (cannot crawl anything)
Amazonbot
Applebot-Extended
Bytespider (TikTok)
CCBot
ClaudeBot
CloudflareBrowserRenderingCrawler
Google-Extended (Google AI training bot)
GPTBot (OpenAI crawler)
meta-externalagent (Meta/Facebook AI)
PetalBot (Huawei)
AspiegelBot
AhrefsBot
SemrushBot / SemRush
DotBot
MauiBot
MJ12bot
ClaudeBot
All other bots not listed above, including:
Googlebot (normal search), Bingbot, DuckDuckBot, etc. can crawl everything and index content.
You can view that using your links and adding robots.txt after the link. I do want to note, I continuously monitor this, have specific notifications going in case something is being flooded, or we are under DDoS attack, and this is also being steered when needed.
ForumSpark Support