## Robots.txt ns.nl ## Googlebot understands the 'Allow' directive (so far one-of-the-only) User-agent: Googlebot Disallow: Disallow: /servlet/ Disallow: /ns2007/ Disallow: /ns2009/ Disallow: /zva/VraagEnAntwoordResult Allow: /zva/VraagEnAntwoord Disallow: /zva/ ## Does MJ12 follow * for user-agent? I don't think so User-agent: MJ12bot Disallow: Disallow: /ns2007/ Disallow: /ns2009/ Disallow: /zva/ User-agent: * Disallow: Disallow: /ns2007/ Disallow: /ns2009/ Disallow: /zva/ ## Disallow 'aggressive' Russian indexer #User-agent: Yandex #Disallow: /