# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /cameo/ Disallow: /cgi-bin/ Disallow: /partner/ Disallow: /preview/ Disallow: /sponsored/ Disallow: /topics