WordPress 410 error, 410 (gone), 410 gone, and variations; plus: WP Super Cache, Hyper Cache, DB Cache Reloaded, User-Agent Blacklist, .htaccess, and more

(Updated via comment)

I will try to make a long story shorter. I had WP Super Cache loaded on WordPress. I'd been using WP Super Cache for years (?). For some reason, it was giving me a 410 error (See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html) when I used the cache-testing feature in the plugin's setting pages. After reading the various forums and such on the Internet, I decided to uninstall it, which required FTPing in and deleting the plugin there (/wordpress/wp-content/plugins/wp-super-cache) because WP Super Cache was returning an error when I used the standard WordPress delete-plugin link on the plugin page, which link is offered up on deactivated plugins. I also deleted the cache subdirectory (/wordpress/wp-content/cache).

/wordpress/wp-content/advanced-cache.php was installed by WP Super Cache. You can leave it or delete it. I deleted it. Hyper Cache, which I will discuss in a bit below, put in its own version.

After manually deleting the plugin, I used the Clean Options plugin to remove the two remaining WP Super Cache options.

I still wanted site caching, so I tried Hyper Cache and DB Cache Reloaded together, which I'm currently using after finally solving the underlying conflict that was causing WP Super Cache to give me the 410 error.

Not typical for WordPress bloggers I'm told, I wanted my rss feed cached as well. I'm on an economy server, which works fine but can be a bit overloaded at times, even though I've migrated to the grid hosting. I'm perfectly willing manually to clean the cache each time I post new blog posts. If I had other authors and such, I'd likely move to a more powerful server-arrangement and set things to automatic.

Well, once I had Hyper Cache going with the setting set to cache the feed, I checked the source code of the feed and didn't see anything to indicate that it was caching. Therefore, I tried yet another plugin, Static Feed, which I may yet use since I still don't see the caching indication on the feed's source code with Hyper Cache.

With Static Feed running, Static Feed supplied the validation link to http://feedvalidator.org. I'd used that many a time and clicked the link just to see how things were going. Yikes! It was telling me that it was being fed the 410 error, which tells services and robots and search engines (apparently not everyone of them but most) that they should remove the URL from their indexes). That was not a message I wanted to be broadcasting. Who knows which "good" bot has taken that error message and complied by deleting my feed as if it's dead, which it was never since I was able to use various readers to verify that it was in fact being served, to some at least? That wasn't the worst of it, as you'll see below.

As you can imagine, I hunted everywhere for answers. Finally, I decided to save my .htaccess file in my root http directory and then began eliminating various blocks one-by-one to check for a culprit. I did this after having deactivated all my plugins and such too, to no avail obviously. Well, I got lucky, as they say. The first block I eliminated stopped blocking the rss validation robot and some other entities I had pinged, along with some test sites, such as http://ismyblogworking.com.

That's what I meant above by the feed being blocked wasn't the worst of it. The 410 error message was being served concerning my whole domain.

I'm not sure if that was the cause, but Google went from having some 5,000 posts indexed down to about only 300. Google Blog Search had also stopped updating with new posts. Of course, one is very hard pressed to contact Google about why Google does anything in particular, so who can know without ESP/the Holy Spirit.

I'm not confident that Hyper Cache is caching the feed yet because http://ismyblogworking.com returns: "Your feed seems slow. It took 1.43 seconds to generate the feed." At least now, I can focus in on that more minor issue.

So, what was the "offending" block in the .htaccess file? It was the following:

[more regular text after this block quote, so scroll down beyond it]

# 2010 User-Agent Blacklist
# @ http://perishablepress.com/press/2010/08/09/2010-user-agent-blacklist/

RewriteCond %{HTTP_HOST} !^(127\.0\.0\.0|localhost) [NC]
RewriteCond %{HTTP_USER_AGENT} .*(Firs|exac|Cloak|Detect|uchoo|beaut|ASPSeek|swish|ICS\)|MSIE\ 6\.0\;\ Windows\ NT\;\ DigExt\)|pt\-BR\;\ rv\:1\.9\.0\.3\)\ Firefox\/3\.0|pt\-BR\;\ rv\:1\.9\.0\.18\)\ Firefox\/3\.0|\!susie|\$x0e|\%0a|\%0d|\@\$x|\_irc|\_works|\+select\+|\+union\+|\<\?|1\,\1\,1\,|3gse|4all|4anything|5\.1\;\ xv6875\)|59\.64\.153\.|85\.17\.|88\.0\.106\.|98|a\_browser|a1\ site|abac|abach|abby|aberja|abilon|abont|abot|accept|access|accoo|accoon|aceftp|acme|active|address|adopt|adress|advisor|agent|ahead|aihit|aipbot|alarm|albert|alek|alexa\ toolbar\;\ \(r1\ 1\.5\)|alltop|alma|alot|alpha|america\ online\ browser\ 1\.1|amfi|amfibi|anal|andit|anon|ansearch|answer|answerbus|answerchase|antivirx|apollo|appie|arach|archive|arian|aboutoil|asps|aster|atari|atlocal|atom|atrax|atrop|attrib|autoh|autohot|av\ fetch|avsearch|axod|axon|baboom|baby|back|baid|bali|bandit|barry|basichttp|batch|bdfetch|beat|become|bee|beij|betabot|biglotron|bilgi|bison|bitacle|bitly|blaiz|blitz|blogl|blogscope|blogzice|bloob|blow|bord|boi|bond|boris|bost|bot\.ara|botje|botw|bpimage|brand|brok|broth|browseabit|browsex|bruin|bsalsa|bsdseek|built|bulls|bumble|bunny|busca|busi|buy|bwh3|cafek|cafi|camel|cand|captu|casper|catch|ccbot|ccubee|cd34|ceg|cfnetwork|cgichk|cha0s|chang|chaos|char|char\(|chase\ x|check\_http|checker|checkonly|chek|chill|chttpclient|cipinet|cisco|cita|citeseer|clam|claria|claw|clush|coast|code\.com|cogent|coldfusion|coll|collect|comb|combine|commentreader|common|compan|compatible\-|conc|conduc|contact|control|contype|conv|cool|copi|copy|coral|corn|cosmos|costa|cowbot|cr4nk|craft|cralwer|crank|crap|crawler0|crazy|cres|cs\-cz|cshttp|cuill|CURI|curl|curry|custo|cute|cyber|cz3|czx|daily|dalvik|daobot|dark|darwin|data|daten|dcbot|dcs|dds\ explorer|deep|deps|detect|dex|diam|diibot|dillo|ding|disc|disp|ditto|dlc|doco|dotbot|drag|drec|dsdl|dsok|dts|duck|dumb|eag|earn|earthcom|easydl|ebin|echo|edco|egoto|elnsb5|email|emer|empas|encyclo|enfi|enhan|enterprise\_search|envolk|erck|erocr|eventax|evere|evil|ewh|exploit|expre|extra|eyen|fang|fast|fastbug|faxo|fdse|feed24|feeddisc|feedhub|fetch|filan|fileboo|fimap|find|firebat|firedownload\/1\.2pre\ firefox\/3\.6|firefox\/0|firefox\/1|firefox\/2|firs|flam|flash|flexum|flip|fly|focus|fooky|forum|forv|fost|foto|foun|fount|foxy\/1\;|free|friend|frontpage|fuck|fuer|futile|fyber|gais|galbot|gbpl|gecko\/2001|gecko\/2002|gecko\/2006|gecko\/2009042316|gener|geni|geo|geona|geth|getr|getw|ggl|gira|gluc|gnome|go\!zilla|goforit|goldfire|gonzo|google\ wireless|googlebot\-image|gosearch|got\-it|gozilla|grab|graf|greg|grub|grup|gsa\-cra|gsearch|gt\:\:www|guidebot|guruji|gyps|haha|hailo|harv|hash|hatena|hax|head|helm|herit|heritrix|hgre|hippo|hloader|hmse|hmview|holm|holy|hotbar\ 4\.4\.5\.0|hpprint|httpclient|httpconnect|httplib|human|huron|hverify|hybrid|hyper|iaskspi|ibm\ evv|iccra|ichiro|icopy|ida|ie\/5\.0|ieauto|iempt|iexplore\.exe|ilium|ilse|iltrov|indexer|indy|ineturl|infonav|innerpr|inspect|insuran|intellig|interget|internet\_explorer|internet\x|intraf|ip2|ipsel|irlbot|isc\_sys|isilo|isrccrawler|isspi|jady|jaka|jam|jenn|jet|jiro|jobo|joc|jupit|just|jyx|jyxo|kash|kazo|kbee|kenjin|kernel|keywo|kfsw|kkma|kmc|know|kosmix|krae|krug|ksibot|ktxn|kum|labs|lanshan|lapo|larbin|leech|lets|lexi|lexxe|libby|libcrawl|libcurl|libfetch|libweb|libwww|light|linc|lingue|linkcheck|linklint|linkman|lint|list|litefeeds|livedoor|livejournal|liveup|lmq|locu|london|lone|loop|lork|lth\_|lwp|mac\_f|magi|magp|mail\.ru|main|majest|mam|mama|mana|marketwire|masc|mass|mata|mvi|mcbot|mecha|mechanize|mediapartners|metadata|metalogger|metaspin|metauri|mete|mib\/2\.2|microsoft\.url|microsoft\_internet\_explorer|mido|miggi|miix|mindjet|mindman|mips|mira|mire|miss|mist|mizz|mj12|mlbot|mlm|mnog|moge|moje|mooz|more|mouse|mozdex) [NC]
RewriteRule ^.*$ - [G]
RewriteCond %{HTTP_HOST} !^(127\.0\.0\.0|localhost) [NC]
RewriteCond %{HTTP_USER_AGENT} .*(Windows\ NT\ 6\.1\;\ tr\;\ rv\:1\.9\.2\.6\)|mozilla\/0|mozilla\/1|mozilla\/2|mozilla\/3|mozilla\/4\.61\ \[en\]|mozilla\/firefox|mpf|msie\ 1|msie\ 2|msie\ 3|msie\ 4|msie\ 5|msie\ 6\.0\-|msie\ 6\.0b|msie\ 7\.0a1\;|msie\ 7\.0b\;|msie6xpv1|msiecrawler|msnbot\-media|msnbot\-products|msnptc|msproxy|msrbot|musc|mvac|mwm|my\_age|myapp|mydog|myeng|myie2|mysearch|myurl|nag|name|naver|navr|near|netants|netcach|netcrawl|netfront|netinfo|netmech|netsp|netx|netz|neural|neut|newsbreak|newsgatorinbox|newsrob|newt|next|ng\-s|ng\/2|nice|nikto|nimb|ninja|ninte|nog|noko|nomad|norb|note|npbot|nuse|nutch|nutex|nwsp|obje|ocel|octo|odi3|oegp|offby|offline|omea|omg|omhttp|onfo|onyx|openf|openssl|openu|opera\ 2|opera\ 3|opera\ 4|opera\ 5|opera\ 6|opera\ 7|orac|orbit|oreg|osis|our|outf|owl|p3p\_|page2rss|pagefet|pansci|parser|patw|pavu|pb2pb|pcbrow|pear|peer|pepe|perfect|perl|petit|phoenix\/0\.|php|phras|picalo|piff|pig|pingd|pipe|pirs|plag|planet|plant|platform|playstation|plesk|pluck|plukkie|poe\-com|poirot|pomp|post|postrank|powerset|preload|press|privoxy|probe|program\_shareware|protect|protocol|prowl|proxie|proxy|psbot|pubsub|puf|pulse|punit|purebot|purity|pyq|pyth|query|quest|qweer|radian|rambler|ramp|rapid|rawdog|rawgrunt|reap|reeder|refresh|reget|relevare|repo|requ|request|rese|retrieve|rip|rix|rma|roboz|rocket|rogue|rpt\-http|rsscache|ruby|ruff|rufus|rv\:0\.9\.7\)|salt|sample|sauger|savvy|sbcyds|sbider|sblog|sbp|scagent|scanner|scej\_|sched|schizo|schlong|schmo|scorp|scott|scout|scrawl|screen|screenshot|script|seamonkey\/1\.5a|search17|searchbot|searchme|sega|semto|sensis|seop|seopro|sept|sezn|seznam|share|sharp|shaz|shell|shelo|sherl|shim|shopwiki|silurian|simple|simplepie|siph|sitekiosk|sitescan|sitevigil|sitex|skam|skimp|sledink|sleip|slide|sly|smag|smurf|snag|snapbot|snapshot|snif|snip|snoop|sock|socsci|sogou|sohu|solr|some|soso|spad|span|spbot|speed|sphere|spin|sproose|spurl|sputnik|spyder|squi|sqwid|sqworm|ssm\_ag|stack|stamp|statbot|state|steel|stilo|strateg|stress|strip|style|subot|such|suck|sume|sunos\ 5\.7|sunrise|superbot|superbro|supervi|surf4me|surfbot|survey|susi|suza|suzu|sweep|sygol|synapse|sync2it|systems|szukacz|tagger|tagoo|tagyu|take|talkro|tamu|tandem|tarantula|tbot|tcf|tcs\/1|teamsoft|tecomi|teesoft|teleport|telesoft|tencent|terrawiz|test|texnut|thomas|tiehttp|timebot|timely|tipp|tiscali|titan|tmcrawler|tmhtload|tocrawl|todobr|tongco|toolbar\;\ \(r1|topic|topyx|torrent|track|translate|traveler|treeview|tricus|trivia|trivial|true|tunnel|turing|turnitin|tutorgig|twat|tweak|twice|tygo|ubee|ultraseek|unavail|unf|universal|unknown|upg1|uptime|urlbase|urllib|urly|user\-agent\:|useragent|usyd|vagabo|valet|vamp|vci|veri\~li|verif|versus|via|virtual|visual|void|voyager|vsyn|w0000t|w3search|walhello|walker|wand|waol|watch|wavefire|wbdbot|weather|web\.ima|web2mal|webarchive|webbot|webcat|webcor|webcorp|webcrawl|webdat|webdup|webgo|webind|webis|webitpr|weblea|webmin|webmoney|webp|webql|webrobot|webster|websurf|webtre|webvac|webzip|wells|wep\_s|wget|whiz|widow|win67|windows\-rss|windows\ 2000|windows\ 3|windows\ 95|windows\ 98|windows\ ce|windows\ me|winht|winodws|wish|wizz|wordp|worio|works|world|worth|wwwc|wwwo|wwwster|xaldon|xbot|xenu|xirq|y\!tunnel|yacy|yahoo\-mmaudvid|yahooseeker|yahooysmcm|yamm|yand|yandex|yang|yoono|yori|yotta|yplus\ |ytunnel|zade|zagre|zeal|zebot|zerx|zeus|zhuaxia|zipcode|zixy|zmao) [NC]
RewriteRule ^.*$ - [G]

When I had first added that block, the author of which by the way gives strong warning that it isn't something you set and forget, it had worked fine with the rest of my configuration. I am not disparaging http://perishablepress.com here at all. I like that site very much and will continue to consult there as time permits.

I hope this helps some people out there who are struggling with this 410 issue. Most of the sites on the subject are concerned with advantageous situations using 410.

As for WP Super Cache, I liked it; but even after getting rid of the 410 problem, it still returned info that time stamps don't match. I just thought it was time to try a different plugin or two or three or even four to see if I might get even better results. I did try the W3 Total Cache plugin by the way and found it serviceable but lacking WP Super Cache's ability to handle blogs with static front pages such as mine. That feature may call me back to WP Super Cache yet. There were a few other issues with W3 Total Cache that I don't recall right now off the top of my head. I tried it several months ago and not with this recent go-around.

So, I believe I will reinstall Static Feed again to see what it does now that the 410 issue is cleared away. I'll also give AskApache Crazy Cache another try too for the same reason. That one hasn't been updated in quite awhile though, so I'm not going to hold my breath.

http://www.askapache.com/ is a good source for .htaccess info, but be forewarned: "A little knowledge is a dangerous thing." If you are planning to mess around with your .htaccess file(s), you need to have a good memory and be sure to backup your file before you change it. Nose around too after making changes. Check your site when logged out. Check it with other browsers. Check it with a browser where the browser cache is off or just cleaned. Click around and use the testing sites. They aren't foolproof though. Some of them will attempt to steer you into things that might break plugins you may be using, so keep your wits about you.

If you don't cope well with this stuff, then don't attempt these things. Be satisfied with WordPress.com or some other platform where you don't configured things in the backroom.

Okay, I just re-tried AskApache Crazy Cache. It doesn't appear to be working for me. I'm currently on WordPress 3.0.1. Listing the entire cache via the plugin's settings page renders empty results, and FTPing into /wordpress/wp-content/plugins/hyper-cache/cache/ and refreshing, while AskApache Crazy Cache says it's working building the cache, shows no new cache results. Therefore, I'm about to deactivate and perhaps delete it. Leaving it undeleted though will alert me to any plugin updates. I do that sometimes but the plugin directory can swell and the database can slowdown — use with that in mind.

As for Static Feed, I just tried it again and no luck, so to speak.

So, I'm not sure about Hyper Cache and the caching of feeds. The feed source-code still doesn't indicate anything.

I reset "Expire a cached query after" on the DB Cache Reloaded - Options page to 525600 [that's about a year]. Remember, I want to manually clear the cache when I post. So, the two caching plugins jibe for a year in terms of their expiries. I have Hyper Cache set to 0 [that's zero], so it will never automatically expire. I'll have to wait a bit to see what they do.

I manually ping and manually rebuilt my sitemap too, all for a number of reasons I won't go into here; but suffice it to say, I've discovered that this is the best way for me considering my particular overall configuration and the methods I use (rss for one) to post and such.

Well after a bit, the DB Cache Reloaded cache subdirectories are empty again. I just manually hit the clean buttons in both plugins' settings pages. Now it's more wait to see.

I'm sure there are many aspects of these two plugins that I'm not comprehending, not for lack of trying though. Not all plugins work as advertised, in spite of the best efforts of their creators.

I Googled: "How can you tell Hyper Cache is caching the feed?" (without the quotation marks), but I didn't find anything. If the feed serving time bogs way down though, I'll get into it even if I have to ask at the plugin-creator's site, which I'm sure he wouldn't mind a bit.

If this post helped you, let me know with a comment. Use your real name please (first and last in title case — first letter caps). I won't approve comments that don't conform to this. I'm trying to promote real (less contrived) and non-paranoid behavior, within reason.

Thanks, and Peace,

Tom

Donate


The following should appear at the end of every post:

According to the IRS, "Know the law: Avoid political campaign intervention":

Tax-exempt section 501(c)(3) organizations like churches, universities, and hospitals must follow the law regarding political campaigns. Unfortunately, some don't know the law.

Under the Internal Revenue Code, all section 501(c)(3) organizations are prohibited from participating in any political campaign on behalf of (or in opposition to) any candidate for elective public office. The prohibition applies to campaigns at the federal, state and local level.

Violation of this prohibition may result in denial or revocation of tax-exempt status and the imposition of certain excise taxes. Section 501(c)(3) private foundations are subject to additional restrictions.

Political Campaign Intervention

Political campaign intervention includes any activities that favor or oppose one or more candidates for public office. The prohibition extends beyond candidate endorsements.

Contributions to political campaign funds, public statements of support or opposition (verbal or written) made by or on behalf of an organization, and the distribution of materials prepared by others that support or oppose any candidate for public office all violate the prohibition on political campaign intervention.

Factors in determining whether a communication results in political campaign intervention include the following:

  • Whether the statement identifies one or more candidates for a given public office
  • Whether the statement expresses approval or disapproval of one or more candidates' positions and/or actions
  • Whether the statement is delivered close in time to the election
  • Whether the statement makes reference to voting or an election
  • Whether the issue addressed distinguishes candidates for a given office

Many religious organizations believe, as we do, that the above constitutes a violation of the First Amendment of the US Constitution.

Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances.

That said, we make the following absolutely clear here:

  • The Real Liberal Christian Church and Christian Commons Project not only do not endorse any candidate for any secular office, we say that Christianity forbids voting in such elections.
  • Furthermore, when we discuss any public-office holder's position, policy, action or inaction, we definitely are not encouraging anyone to vote for that office holder's position.
  • We are not trying to influence secular elections but rather want people to come out from that entire fallen system.
  • When we analyze or discuss what is termed "public policy," we do it entirely from a theological standpoint with an eye to educating professing Christians and those to whom we are openly always proselytizing to convert to authentic Christianity.
  • It is impossible for us to fully evangelize and proselytize without directly discussing the pros and cons of public policy and the positions of secular-office holders, hence the unconstitutionality of the IRS code on the matter.
  • We are not rich and wouldn't be looking for a fight regardless. What we cannot do is compromise our faith (which seeks to harm nobody, quite the contrary).
  • We render unto Caesar what is Caesar's. We render unto God what is God's.
  • When Caesar says to us that unless we shut up about the unrighteousness of Caesar's policies and practices, we will lose the ability of people who donate to us to declare their donations as deductions on their federal and state income-tax returns, we say to Caesar that we cannot shut up while exercising our religion in a very reasonable way.
  • We consider the IRS code on this matter as deliberate economic duress (a form of coercion) and a direct attempt by the federal government to censor dissenting, free political and religious speech.
  • It's not freedom of religion if they tax it.

And when they were come to Capernaum, they that received tribute money came to Peter, and said, Doth not your master pay tribute? He saith, Yes. And when he was come into the house, Jesus prevented him, saying, What thinkest thou, Simon? of whom do the kings of the earth take custom or tribute? of their own children, or of strangers? Peter saith unto him, Of strangers. Jesus saith unto him, Then are the children free. (Matthew 17:24-26)

  • Subscribe


  • Tom Usher

    About Tom Usher

    Employment: 2008 - present, website developer and writer. 2015 - present, insurance broker. Education: Arizona State University, Bachelor of Science in Political Science. City University of Seattle, graduate studies in Public Administration. Volunteerism: 2007 - present, president of the Real Liberal Christian Church and Christian Commons Project.
    This entry was posted in Uncategorized. Bookmark the permalink.
    • As of November 6, 2010, at 12:47 PM Pacific Time USA, I cleared the DB Cache Reloaded caches, set the plugin to un-enabled, and deactivated it.

      The plugin was not retaining the cached results even though I had set the expiration way out into the future. It was also returning "1" as the cached query results that according to the plugin author, indicated that the plugin wasn't functioning even though it was activated and enabled.

      It may have had a conflict with Hyper Cache. Plugin conflicts appear to be emphasized by the DB Cache Reloaded author as being the prime reason for not returning 2 or more as the number of queries cached per post.