Get inside the head of a crawler

Every request to a server, from a person or a bot, leaves a line in the access log. Filip Podstavec is an SEO consultant whose clients include Heureka.cz. He argues that log files are the most accurate and most underrated source of data on how Googlebot moves through a website. In this talk he turns the audience into "Googlebot psychologists" and shows how to clear the road for crawlers.

You'll learn what a single log line tells you, which tool to use depending on the file size, and which questions logs answer: which bots crawl your site, where they go most often, where they hit errors and who is only pretending to be Googlebot. The second half is about fixes. Filip covers the 503 status code for downtime, pagination for both Google and Seznam, filter combinations in e-shops, the difference between disallow and noindex, the If-Modified-Since header and a clean sitemap.