What is AI site indexing?
AI tools like Claude, ChatGPT, and Gemini use automated crawlers to learn from publicly available content on the web. When someone asks an AI chat tool a question related to your cause, issue, or organization, enabling AI crawlers to reference your content allows your information to inform the response.
If your organization is focused on public awareness, advocacy, or recruitment, allowing AI tools to index your publicly available content can extend your reach beyond traditional search results.
Enabling or disabling AI crawlers
By default, your site allows AI crawlers. To revoke this permission, go to Site settings > Basic and uncheck "Allow AI crawlers".
You can update this setting at any time.
What content can AI crawlers access?
This setting only controls access to your public pages by referencing your robots.txt file. Content on password-protected pages is never accessible to crawlers, regardless of how this setting is configured.
Disabling AI crawlers signals to agents that they do not have your consent to use content on your pages. However, some bots may ignore this configuration and crawl your public pages anyway. Password protecting a page is the only way to ensure it cannot be crawled.
📌 Note: An "unlisted" page is still publicly available, even though it isn't included in your website's navigation. To prevent a page from being crawled, set the page status to "Hidden" or limit the "Who can view this page?" visibility.
This setting works independently from your search engine indexing preferences. You can allow AI crawlers while keeping your site hidden from search engines, or vice versa.