WordPress responds to requests to a robots.txt
with dynamic content if such a file does not exist. That’s one way how the settings from wp-admin/options-privacy.php
are used.
I recommend to create a static robots.txt
, just to make sure no plugin is getting in your way.
Sample robots.txt
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: */trackback
Disallow: */comments
User-agent: backlink-check.de
Disallow: /
# Prefetches everything. Mwaaah!
User-agent: Fasterfox
Disallow: /
# adjust the path
Sitemap: http://example.com/sitemap.xml
Your .htaccess
looks indeed … strange. You need RewriteEngine On
just once. And a memory limit of 32MB is very low. You cannot even run translation with such a low value.
You should limit the request methods to HEAD
, GET
and POST
.
Sample .htaccess
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
# FeedBurner
RewriteCond %{HTTP_USER_AGENT} !FeedBurner [NC]
RewriteCond %{HTTP_USER_AGENT} !FeedValidator [NC]
RewriteRule ^feed/?([_0-9a-z-]+)?/?$ http://feeds.feedburner.com/anonymousblog [R=302,NC,L]
RewriteRule ^comments/feed/?([_0-9a-z-]+)?/?$ http://feeds.feedburner.com/anonymous_comments [R=302,NC,L]
# WordPress
# Existing file
RewriteCond %{REQUEST_FILENAME} !-f
# Existing directory
RewriteCond %{REQUEST_FILENAME} !-d
# Symbolic link
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule ^ index.php [L]
</IfModule>
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*
<LimitExcept HEAD GET POST>
order deny,allow
deny from all
</LimitExcept>
php_value memory_limit 128M