Skip to main content
Use Crawler Logs to inspect recognised AI crawler requests captured by your connected edge or CDN log integration. Review the crawler, requested page, response status, and activity over time.

​
Connect request data

Open Crawler Logs and follow the available integration setup for the website. Log visibility begins with the data that the integration supplies; an empty view can mean there are no matching requests or no connected data for the selected period. Confirm the website and integration before reviewing the activity. Different environments or subdomains can have different traffic and crawling rules.

​
Investigate access problems

Filter by crawler, path, response status, or event type. Repeated unsuccessful responses on important pages are worth investigating alongside your site’s robots rules, authentication, firewall, and server behaviour. A successful request means the crawler received a response. It does not guarantee indexing, a citation, a recommendation, or a change in an AI answer.

​
Keep crawler activity separate from people

Bot requests are not customer visits. Use AI Referrals for attributed human traffic and Sources for citations captured in monitored answers. These three views describe different stages of discovery.