WordPress 410 error, 410 (gone), 410 gone, and variations; plus: WP Super Cache, Hyper Cache, DB Cache Reloaded, User-Agent Blacklist, .htaccess, and more

(Updated via comment)

I will try to make a long story shorter. I had WP Super Cache loaded on WordPress. I'd been using WP Super Cache for years (?). For some reason, it was giving me a 410 error (See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html) when I used the cache-testing feature in the plugin's setting pages. After reading the various forums and such on the Internet, I decided to uninstall it, which required FTPing in and deleting the plugin there (/wordpress/wp-content/plugins/wp-super-cache) because WP Super Cache was returning an error when I used the standard WordPress delete-plugin link on the plugin page, which link is offered up on deactivated plugins. I also deleted the cache subdirectory (/wordpress/wp-content/cache).

/wordpress/wp-content/advanced-cache.php was installed by WP Super Cache. You can leave it or delete it. I deleted it. Hyper Cache, which I will discuss in a bit below, put in its own version.

After manually deleting the plugin, I used the Clean Options plugin to remove the two remaining WP Super Cache options.

I still wanted site caching, so I tried Hyper Cache and DB Cache Reloaded together, which I'm currently using after finally solving the underlying conflict that was causing WP Super Cache to give me the 410 error.

Not typical for WordPress bloggers I'm told, I wanted my rss feed cached as well. I'm on an economy server, which works fine but can be a bit overloaded at times, even though I've migrated to the grid hosting. I'm perfectly willing manually to clean the cache each time I post new blog posts. If I had other authors and such, I'd likely move to a more powerful server-arrangement and set things to automatic.

Well, once I had Hyper Cache going with the setting set to cache the feed, I checked the source code of the feed and didn't see anything to indicate that it was caching. Therefore, I tried yet another plugin, Static Feed, which I may yet use since I still don't see the caching indication on the feed's source code with Hyper Cache.

With Static Feed running, Static Feed supplied the validation link to http://feedvalidator.org. I'd used that many a time and clicked the link just to see how things were going. Yikes! It was telling me that it was being fed the 410 error, which tells services and robots and search engines (apparently not everyone of them but most) that they should remove the URL from their indexes). That was not a message I wanted to be broadcasting. Who knows which "good" bot has taken that error message and complied by deleting my feed as if it's dead, which it was never since I was able to use various readers to verify that it was in fact being served, to some at least? That wasn't the worst of it, as you'll see below.

As you can imagine, I hunted everywhere for answers. Finally, I decided to save my .htaccess file in my root http directory and then began eliminating various blocks one-by-one to check for a culprit. I did this after having deactivated all my plugins and such too, to no avail obviously. Well, I got lucky, as they say. The first block I eliminated stopped blocking the rss validation robot and some other entities I had pinged, along with some test sites, such as http://ismyblogworking.com.

That's what I meant above by the feed being blocked wasn't the worst of it. The 410 error message was being served concerning my whole domain.

I'm not sure if that was the cause, but Google went from having some 5,000 posts indexed down to about only 300. Google Blog Search had also stopped updating with new posts. Of course, one is very hard pressed to contact Google about why Google does anything in particular, so who can know without ESP/the Holy Spirit.

I'm not confident that Hyper Cache is caching the feed yet because http://ismyblogworking.com returns: "Your feed seems slow. It took 1.43 seconds to generate the feed." At least now, I can focus in on that more minor issue.

So, what was the "offending" block in the .htaccess file? It was the following:

[more regular text after this block quote, so scroll down beyond it]

# 2010 User-Agent Blacklist
# @ http://perishablepress.com/press/2010/08/09/2010-user-agent-blacklist/

RewriteCond %{HTTP_HOST} !^(127\.0\.0\.0|localhost) [NC]
RewriteCond %{HTTP_USER_AGENT} .*(Firs|exac|Cloak|Detect|uchoo|beaut|ASPSeek|swish|ICS\)|MSIE\ 6\.0\;\ Windows\ NT\;\ DigExt\)|pt\-BR\;\ rv\:1\.9\.0\.3\)\ Firefox\/3\.0|pt\-BR\;\ rv\:1\.9\.0\.18\)\ Firefox\/3\.0|\!susie|\$x0e|\%0a|\%0d|\@\$x|\_irc|\_works|\+select\+|\+union\+|\<\?|1\,\1\,1\,|3gse|4all|4anything|5\.1\;\ xv6875\)|59\.64\.153\.|85\.17\.|88\.0\.106\.|98|a\_browser|a1\ site|abac|abach|abby|aberja|abilon|abont|abot|accept|access|accoo|accoon|aceftp|acme|active|address|adopt|adress|advisor|agent|ahead|aihit|aipbot|alarm|albert|alek|alexa\ toolbar\;\ \(r1\ 1\.5\)|alltop|alma|alot|alpha|america\ online\ browser\ 1\.1|amfi|amfibi|anal|andit|anon|ansearch|answer|answerbus|answerchase|antivirx|apollo|appie|arach|archive|arian|aboutoil|asps|aster|atari|atlocal|atom|atrax|atrop|attrib|autoh|autohot|av\ fetch|avsearch|axod|axon|baboom|baby|back|baid|bali|bandit|barry|basichttp|batch|bdfetch|beat|become|bee|beij|betabot|biglotron|bilgi|bison|bitacle|bitly|blaiz|blitz|blogl|blogscope|blogzice|bloob|blow|bord|boi|bond|boris|bost|bot\.ara|botje|botw|bpimage|brand|brok|broth|browseabit|browsex|bruin|bsalsa|bsdseek|built|bulls|bumble|bunny|busca|busi|buy|bwh3|cafek|cafi|camel|cand|captu|casper|catch|ccbot|ccubee|cd34|ceg|cfnetwork|cgichk|cha0s|chang|chaos|char|char\(|chase\ x|check\_http|checker|checkonly|chek|chill|chttpclient|cipinet|cisco|cita|citeseer|clam|claria|claw|clush|coast|code\.com|cogent|coldfusion|coll|collect|comb|combine|commentreader|common|compan|compatible\-|conc|conduc|contact|control|contype|conv|cool|copi|copy|coral|corn|cosmos|costa|cowbot|cr4nk|craft|cralwer|crank|crap|crawler0|crazy|cres|cs\-cz|cshttp|cuill|CURI|curl|curry|custo|cute|cyber|cz3|czx|daily|dalvik|daobot|dark|darwin|data|daten|dcbot|dcs|dds\ explorer|deep|deps|detect|dex|diam|diibot|dillo|ding|disc|disp|ditto|dlc|doco|dotbot|drag|drec|dsdl|dsok|dts|duck|dumb|eag|earn|earthcom|easydl|ebin|echo|edco|egoto|elnsb5|email|emer|empas|encyclo|enfi|enhan|enterprise\_search|envolk|erck|erocr|eventax|evere|evil|ewh|exploit|expre|extra|eyen|fang|fast|fastbug|faxo|fdse|feed24|feeddisc|feedhub|fetch|filan|fileboo|fimap|find|firebat|firedownload\/1\.2pre\ firefox\/3\.6|firefox\/0|firefox\/1|firefox\/2|firs|flam|flash|flexum|flip|fly|focus|fooky|forum|forv|fost|foto|foun|fount|foxy\/1\;|free|friend|frontpage|fuck|fuer|futile|fyber|gais|galbot|gbpl|gecko\/2001|gecko\/2002|gecko\/2006|gecko\/2009042316|gener|geni|geo|geona|geth|getr|getw|ggl|gira|gluc|gnome|go\!zilla|goforit|goldfire|gonzo|google\ wireless|googlebot\-image|gosearch|got\-it|gozilla|grab|graf|greg|grub|grup|gsa\-cra|gsearch|gt\:\:www|guidebot|guruji|gyps|haha|hailo|harv|hash|hatena|hax|head|helm|herit|heritrix|hgre|hippo|hloader|hmse|hmview|holm|holy|hotbar\ 4\.4\.5\.0|hpprint|httpclient|httpconnect|httplib|human|huron|hverify|hybrid|hyper|iaskspi|ibm\ evv|iccra|ichiro|icopy|ida|ie\/5\.0|ieauto|iempt|iexplore\.exe|ilium|ilse|iltrov|indexer|indy|ineturl|infonav|innerpr|inspect|insuran|intellig|interget|internet\_explorer|internet\x|intraf|ip2|ipsel|irlbot|isc\_sys|isilo|isrccrawler|isspi|jady|jaka|jam|jenn|jet|jiro|jobo|joc|jupit|just|jyx|jyxo|kash|kazo|kbee|kenjin|kernel|keywo|kfsw|kkma|kmc|know|kosmix|krae|krug|ksibot|ktxn|kum|labs|lanshan|lapo|larbin|leech|lets|lexi|lexxe|libby|libcrawl|libcurl|libfetch|libweb|libwww|light|linc|lingue|linkcheck|linklint|linkman|lint|list|litefeeds|livedoor|livejournal|liveup|lmq|locu|london|lone|loop|lork|lth\_|lwp|mac\_f|magi|magp|mail\.ru|main|majest|mam|mama|mana|marketwire|masc|mass|mata|mvi|mcbot|mecha|mechanize|mediapartners|metadata|metalogger|metaspin|metauri|mete|mib\/2\.2|microsoft\.url|microsoft\_internet\_explorer|mido|miggi|miix|mindjet|mindman|mips|mira|mire|miss|mist|mizz|mj12|mlbot|mlm|mnog|moge|moje|mooz|more|mouse|mozdex) [NC]
RewriteRule ^.*$ - [G]
RewriteCond %{HTTP_HOST} !^(127\.0\.0\.0|localhost) [NC]
RewriteCond %{HTTP_USER_AGENT} .*(Windows\ NT\ 6\.1\;\ tr\;\ rv\:1\.9\.2\.6\)|mozilla\/0|mozilla\/1|mozilla\/2|mozilla\/3|mozilla\/4\.61\ \[en\]|mozilla\/firefox|mpf|msie\ 1|msie\ 2|msie\ 3|msie\ 4|msie\ 5|msie\ 6\.0\-|msie\ 6\.0b|msie\ 7\.0a1\;|msie\ 7\.0b\;|msie6xpv1|msiecrawler|msnbot\-media|msnbot\-products|msnptc|msproxy|msrbot|musc|mvac|mwm|my\_age|myapp|mydog|myeng|myie2|mysearch|myurl|nag|name|naver|navr|near|netants|netcach|netcrawl|netfront|netinfo|netmech|netsp|netx|netz|neural|neut|newsbreak|newsgatorinbox|newsrob|newt|next|ng\-s|ng\/2|nice|nikto|nimb|ninja|ninte|nog|noko|nomad|norb|note|npbot|nuse|nutch|nutex|nwsp|obje|ocel|octo|odi3|oegp|offby|offline|omea|omg|omhttp|onfo|onyx|openf|openssl|openu|opera\ 2|opera\ 3|opera\ 4|opera\ 5|opera\ 6|opera\ 7|orac|orbit|oreg|osis|our|outf|owl|p3p\_|page2rss|pagefet|pansci|parser|patw|pavu|pb2pb|pcbrow|pear|peer|pepe|perfect|perl|petit|phoenix\/0\.|php|phras|picalo|piff|pig|pingd|pipe|pirs|plag|planet|plant|platform|playstation|plesk|pluck|plukkie|poe\-com|poirot|pomp|post|postrank|powerset|preload|press|privoxy|probe|program\_shareware|protect|protocol|prowl|proxie|proxy|psbot|pubsub|puf|pulse|punit|purebot|purity|pyq|pyth|query|quest|qweer|radian|rambler|ramp|rapid|rawdog|rawgrunt|reap|reeder|refresh|reget|relevare|repo|requ|request|rese|retrieve|rip|rix|rma|roboz|rocket|rogue|rpt\-http|rsscache|ruby|ruff|rufus|rv\:0\.9\.7\)|salt|sample|sauger|savvy|sbcyds|sbider|sblog|sbp|scagent|scanner|scej\_|sched|schizo|schlong|schmo|scorp|scott|scout|scrawl|screen|screenshot|script|seamonkey\/1\.5a|search17|searchbot|searchme|sega|semto|sensis|seop|seopro|sept|sezn|seznam|share|sharp|shaz|shell|shelo|sherl|shim|shopwiki|silurian|simple|simplepie|siph|sitekiosk|sitescan|sitevigil|sitex|skam|skimp|sledink|sleip|slide|sly|smag|smurf|snag|snapbot|snapshot|snif|snip|snoop|sock|socsci|sogou|sohu|solr|some|soso|spad|span|spbot|speed|sphere|spin|sproose|spurl|sputnik|spyder|squi|sqwid|sqworm|ssm\_ag|stack|stamp|statbot|state|steel|stilo|strateg|stress|strip|style|subot|such|suck|sume|sunos\ 5\.7|sunrise|superbot|superbro|supervi|surf4me|surfbot|survey|susi|suza|suzu|sweep|sygol|synapse|sync2it|systems|szukacz|tagger|tagoo|tagyu|take|talkro|tamu|tandem|tarantula|tbot|tcf|tcs\/1|teamsoft|tecomi|teesoft|teleport|telesoft|tencent|terrawiz|test|texnut|thomas|tiehttp|timebot|timely|tipp|tiscali|titan|tmcrawler|tmhtload|tocrawl|todobr|tongco|toolbar\;\ \(r1|topic|topyx|torrent|track|translate|traveler|treeview|tricus|trivia|trivial|true|tunnel|turing|turnitin|tutorgig|twat|tweak|twice|tygo|ubee|ultraseek|unavail|unf|universal|unknown|upg1|uptime|urlbase|urllib|urly|user\-agent\:|useragent|usyd|vagabo|valet|vamp|vci|veri\~li|verif|versus|via|virtual|visual|void|voyager|vsyn|w0000t|w3search|walhello|walker|wand|waol|watch|wavefire|wbdbot|weather|web\.ima|web2mal|webarchive|webbot|webcat|webcor|webcorp|webcrawl|webdat|webdup|webgo|webind|webis|webitpr|weblea|webmin|webmoney|webp|webql|webrobot|webster|websurf|webtre|webvac|webzip|wells|wep\_s|wget|whiz|widow|win67|windows\-rss|windows\ 2000|windows\ 3|windows\ 95|windows\ 98|windows\ ce|windows\ me|winht|winodws|wish|wizz|wordp|worio|works|world|worth|wwwc|wwwo|wwwster|xaldon|xbot|xenu|xirq|y\!tunnel|yacy|yahoo\-mmaudvid|yahooseeker|yahooysmcm|yamm|yand|yandex|yang|yoono|yori|yotta|yplus\ |ytunnel|zade|zagre|zeal|zebot|zerx|zeus|zhuaxia|zipcode|zixy|zmao) [NC]
RewriteRule ^.*$ - [G]

When I had first added that block, the author of which by the way gives strong warning that it isn't something you set and forget, it had worked fine with the rest of my configuration. I am not disparaging http://perishablepress.com here at all. I like that site very much and will continue to consult there as time permits.

I hope this helps some people out there who are struggling with this 410 issue. Most of the sites on the subject are concerned with advantageous situations using 410.

As for WP Super Cache, I liked it; but even after getting rid of the 410 problem, it still returned info that time stamps don't match. I just thought it was time to try a different plugin or two or three or even four to see if I might get even better results. I did try the W3 Total Cache plugin by the way and found it serviceable but lacking WP Super Cache's ability to handle blogs with static front pages such as mine. That feature may call me back to WP Super Cache yet. There were a few other issues with W3 Total Cache that I don't recall right now off the top of my head. I tried it several months ago and not with this recent go-around.

So, I believe I will reinstall Static Feed again to see what it does now that the 410 issue is cleared away. I'll also give AskApache Crazy Cache another try too for the same reason. That one hasn't been updated in quite awhile though, so I'm not going to hold my breath.

http://www.askapache.com/ is a good source for .htaccess info, but be forewarned: "A little knowledge is a dangerous thing." If you are planning to mess around with your .htaccess file(s), you need to have a good memory and be sure to backup your file before you change it. Nose around too after making changes. Check your site when logged out. Check it with other browsers. Check it with a browser where the browser cache is off or just cleaned. Click around and use the testing sites. They aren't foolproof though. Some of them will attempt to steer you into things that might break plugins you may be using, so keep your wits about you.

If you don't cope well with this stuff, then don't attempt these things. Be satisfied with WordPress.com or some other platform where you don't configured things in the backroom.

Okay, I just re-tried AskApache Crazy Cache. It doesn't appear to be working for me. I'm currently on WordPress 3.0.1. Listing the entire cache via the plugin's settings page renders empty results, and FTPing into /wordpress/wp-content/plugins/hyper-cache/cache/ and refreshing, while AskApache Crazy Cache says it's working building the cache, shows no new cache results. Therefore, I'm about to deactivate and perhaps delete it. Leaving it undeleted though will alert me to any plugin updates. I do that sometimes but the plugin directory can swell and the database can slowdown — use with that in mind.

As for Static Feed, I just tried it again and no luck, so to speak.

So, I'm not sure about Hyper Cache and the caching of feeds. The feed source-code still doesn't indicate anything.

I reset "Expire a cached query after" on the DB Cache Reloaded - Options page to 525600 [that's about a year]. Remember, I want to manually clear the cache when I post. So, the two caching plugins jibe for a year in terms of their expiries. I have Hyper Cache set to 0 [that's zero], so it will never automatically expire. I'll have to wait a bit to see what they do.

I manually ping and manually rebuilt my sitemap too, all for a number of reasons I won't go into here; but suffice it to say, I've discovered that this is the best way for me considering my particular overall configuration and the methods I use (rss for one) to post and such.

Well after a bit, the DB Cache Reloaded cache subdirectories are empty again. I just manually hit the clean buttons in both plugins' settings pages. Now it's more wait to see.

I'm sure there are many aspects of these two plugins that I'm not comprehending, not for lack of trying though. Not all plugins work as advertised, in spite of the best efforts of their creators.

I Googled: "How can you tell Hyper Cache is caching the feed?" (without the quotation marks), but I didn't find anything. If the feed serving time bogs way down though, I'll get into it even if I have to ask at the plugin-creator's site, which I'm sure he wouldn't mind a bit.

If this post helped you, let me know with a comment. Use your real name please (first and last in title case — first letter caps). I won't approve comments that don't conform to this. I'm trying to promote real (less contrived) and non-paranoid behavior, within reason.

Thanks, and Peace,

Tom

  • Subscribe
  • Tom Usher

    About Tom Usher

    Employment: 2008 - present, website developer and writer. 2015 - present, insurance broker. Education: Arizona State University, Bachelor of Science in Political Science. City University of Seattle, graduate studies in Public Administration. Volunteerism: 2007 - present, president of the Real Liberal Christian Church and Christian Commons Project.
    This entry was posted in Uncategorized. Bookmark the permalink.