Fable vs 10 LLMs: refactoring a LangGraph god node

AI crawlers are quietly scraping your website right now, and you have a decision to make about whether to let them.

The Quiet Land Grab Happening On Your Website

If you run a website, bots from OpenAI, Google, Anthropic, and a growing list of AI companies are almost certainly visiting it regularly. Not to send you traffic. Not to rank you in search results. To harvest your content and use it to train models or power AI-generated answers that may never link back to you.

The question that's split the SEO and web publishing world recently is simple: should you block them, or should you try to understand what you're actually getting from them first?

It sounds like a straightforward call. Block the leeches, protect your content. But the reality is messier than that, and the answer genuinely depends on what kind of business you run.

Two Very Different Kinds of AI Traffic

Not all AI crawlers are the same, and that distinction matters more than most people realise.

Some crawlers are training crawlers. They visit your site to collect data that feeds into building or improving AI models. You get nothing in return. No referral traffic, no citations, no visibility. Your content goes in, and the model gets smarter, full stop.

Others are retrieval crawlers. These are sent out by AI tools that answer user questions in real time, pulling from live websites to construct responses. Think of tools that generate answers with citations, or AI assistants that say "according to [your site]..." There's at least an argument that being included here gives you some form of reach.

The problem is that most website owners are currently blocking both types with the same blunt instrument, or allowing both without realising there's a difference. Neither approach is particularly strategic.

What This Means If You Run a Small Business

Here is where it gets genuinely relevant to you, not just to publishers running editorial empires.

If your website generates leads, sells services, or ranks in search, your content is your competitive asset. You wrote it, paid someone to write it, or spent years developing the expertise behind it. Handing that over to an AI that will synthesise it into a generic answer, without attribution, without a click, is a legitimate business concern.

At the same time, blocking every crawler indiscriminately might mean you miss out on the emerging wave of AI-powered discovery. As more people use AI tools to find services, suppliers, and answers, being invisible to those systems could mean being invisible to potential customers.

We are still in the very early stages of understanding how AI referral traffic actually converts. But the businesses that are paying attention now will be better placed to make informed decisions in six months when the data is clearer.

The smart play right now is not to panic in either direction. It is to get informed before you act.

What To Do About It

1. 1.Audit your current crawler permissions. Check your robots.txt file (the document that tells bots what they can and cannot access on your site). Most web platforms let you view and edit this. See what you are currently allowing or blocking.

1. 1.Identify which AI crawlers are visiting you. Tools like Cloudflare's analytics, server logs, or dedicated crawler detection plugins can show you which bots are turning up and how often. Know your visitors before you evict them.

1. 1.Separate retrieval crawlers from training crawlers. GPTBot (OpenAI's training crawler) and SearchGPT's retrieval crawler are different bots with different purposes. You can block one without blocking the other. Look up the current list of named AI crawlers and make individual calls.

1. 1.Monitor any referral traffic from AI sources. Check your analytics for traffic sources that reference AI tools or assistants. Even small numbers are worth tracking as a baseline for future comparison.

1. 1.Reassess in 90 days. This space is moving fast. Set a calendar reminder to revisit your crawler policy quarterly rather than setting it once and forgetting it.