Sitemap: https://www.johncharles.net/sitemap.xml User-agent:* Disallow: /website/modules/blog/blog_datalist.ajax.php$ Disallow: /ajax/ Disallow: /admin/ Disallow: /cms/ Disallow: /callback-request Disallow: /request-a-call Disallow: /general-enquiry Disallow: /px-enquiry Disallow: /value-my-px Disallow: /cdn-cgi/l/email-protection Disallow: /*?print # blocking bad bots # http://www.opensiteexplorer.org/dotbot User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: rogerbot Disallow: / # http://ahrefs.com/robot/ user-agent: AhrefsBot disallow: / # http://go.mail.ru/help/robots User-agent: Mail.Ru Disallow: / # http://mj12bot.com/ User-agent: MJ12bot Disallow: / #Baiduspider User-agent: Baiduspider Disallow: / #Yandex User-agent: Yandex Disallow: /