User-agent: * Disallow: /media/pdf/ Disallow: /media/banners/ Disallow: /media/email/ Disallow: /media/logos/ Disallow: /media/logoTaxes/ Disallow: /media/miscellaneous/ Disallow: /generator/ Disallow: /ajax/ Disallow: /scripts/ Disallow: /styles/ Disallow: /lib/ # jmoe 20090828: Cuil is not cool: it spiders things that look like directories, but aren't, generating tons of 404 messages. Nobody uses it, anyway. User-agent: twiceler Disallow: /