The Wall Street Journal’s Fancy SEO Tricks

I’m not an SEO expert. So if there were a group of SEO experts standing in the corner, I wouldn’t be among them. I would be among the mere mortals, who apply basically their common sense to how search engines work.

All that said by way of longwinded preamble, I did happen upon a fun realization this morning, in the spirit of “The internet routes around….”

The WSJ does this thing called cloaking. It essentially means they show Google a different website from what they show you. The googlebot sees no paywall and waltzes right in. You hit a “subs only” paywall and get frustrated. Or maybe you pay for the subscription. Still, though, I doubt google pays the subscription, and so even if you see the whole website too, you see a costly website, whereas google sees a free website.

The net result for the WSJ is that it cleverly gets its entire articles indexed, making them easier to find in google, but is able to maintain its paywall strategy. The net result for you and me is that it’s sometimes a pain in the neck to read the WSJ—which is too bad, because it’s a great read. It’s also a pain in the neck to share WSJ articles, as Deputy Managing Editor and Executive Editor Online @alansmurray’s sometimes plaintive “subs only” tweets evince.

But there’s a way around the mess. Actually, there are a couple ways around. One involves the hassle of teaching my mom how to waltz in like google does, and one involves me doing it for her. I prefer the latter.

paywallBut let’s rehearse the former first. Let’s say you hit the paywall. What do you do? You copy the headline, paste it into google, and hit enter. This works way better if you’ve got a search bar in your browser. Once you hit enter, you come to a search results page. You’ll know which link to click because it won’t be blue. Purple means you’ve been there before, so click that link. It will take you back to your article, but you’ll be behind the paywall, gazing at unabridged goodness. It’s not too hard, and the upside it terrific. That said, this procedure is much easier to perform than it is to explain, and the whole thing is pretty unintuitive, so my efforts to spread the word have led to little.

But there’s a better way, for the sharing, a least—a way that involves letting the geekiest among us assume the responsibility of being geeky. It’s natural, and you don’t have to rely on your mother’s ability to route around. Instead, once you decide you want to share a WSJ article, grab the really long URL that sits behind google’s link on its search returns page. They look something like this:

Then push that horribly long URL—itself unfit for sharing in many contexts—into your favorite URL shortener. Send that shortened URL to your mom, or post it to twitter.

No one will ever know the article you’re sharing sits behind WSJ’s grayhat paywall.

LATE UPDATE: I write a follow-up post prompted by @alansmurray’s response, comparing his situation to the one occupied by the folks at Starbucks.

LATER UPDATE: Alex Bennert from the WSJ points out that the WSJ’s fancy trick is in fact sponsored by google and called First Click Free. See his her link below and my reply.

13 Responses to “The Wall Street Journal’s Fancy SEO Tricks”

  1. 1 Wil 2009 August 28 at 1:22 pm

    Actually, I think wrote about this a long time ago (except they were far less subtle: the title of the post was something along the lines of ‘get WSJ for free’.

  2. 2 Alex Bennert 2009 August 28 at 3:03 pm

    Hi Josh, does indeed to First Click Free with Google. This means that we let Google see all of our content and if come to us from a Google search, we’ll let you see your first page free. If you hit a paid wall from a Google search query, please let me know because that is a glitch and must get fixed.
    Alex Bennert

    • 3 Josh Young 2009 August 28 at 4:47 pm

      “First Click Free” strikes me as an odd designation, given that all clicks are free when your users, whose intelligence you presumably respect, simply return to google each time.

  3. 4 Carlo 2009 August 28 at 6:02 pm

    Just pasted this example link into my browser then into tweetie to shorten it and opened that short link in a new tab and bam – nada? Any new info or strategies on this?

  4. 6 Alex Bennert 2009 August 28 at 6:54 pm

    First click free isn’t a “designation.” Nor is it a gray hat tactic. It’s Google’s program for publishers. You can read about it here:

    And yes, there will always be people who are passionate enough about our content that they will game the system to avoid paying a subscription, just like there will always be people who copy music instead of buying it. However, this has nothing to do with my SEO tactics.


    • 7 Josh Young 2009 August 28 at 9:11 pm

      Thanks much for the link, Alex. I appreciate it. I didn’t know google sponsored the tactic, and I’ll stop calling it grayhat.

      However, I still think it’s at worst dishonest and at best a bit creepy. My point about Mr. Murray is that those characteristics of the tactic come into sharper relief on twitter–precisely because it’s a thoroughly human space.

      Also, I completely, utterly reject your claim that my method above is gaming the system. It’s not like giving away coffee at starbucks, and it’s not like ripping copyrighted music. At all.

      See here for my fuller explanation:

  5. 8 Daniel Tunkelang 2009 August 29 at 7:10 pm

    From the comment thread at (by the post’s author, Google Webmaster Trends Analyst John Mueller): “Also, keep in mind that it’s possible to verify Googlebot based on a reverse IP/DNS lookup, so surfing with Googlebot’s user-agent will likely not be that useful.”

    Though personally I’m not aware of any site that takes this step to prevent people from simply spoofing the Googlebot by changing their user agent. Then again, I also don’t know anyone who goes through that much effort to circumvent pay walls for mainstream news sites like the WSJ.

  6. 9 Casey 2009 August 31 at 5:28 pm

    I suspect that if the humble sites that we run got up to such cloaking tricks, Google would de-index them in hours.

  7. 10 Alex Bennert 2009 August 31 at 7:39 pm

    Thanks for adding the update Josh…though I am a her, not a him. :-)

    • 11 Josh Young 2009 August 31 at 8:33 pm

      Okay, so that’s updated too, Alex, and thanks again for stopping by. Apologies for botching your gender. I was thinking about my brother, who shares your name.

      I wonder whether you’re interested in offering a defense of First Click Free. First, does my argument make any sense? If so, do you disagree with it? Where does it go astray?

      I understand you’ve got your employer to look after, so plan to count me unsurprised if you can’t oblige my curiosity. Still, though, I’m interested because I do think First Click Free feels inherently misleading and because it seems like the human context on twitter exacerbates that feeling.

  8. 12 Alex Bennert 2009 September 2 at 3:42 am

    To be honest, I only poked my head in to defend my honor. I’m more of a uniter than a debater so if you still want me to respond, I will (you’ll have to wait until the weekend tho) but I suspect you’ll just be disappointed. So perhaps you’ll let me off the hook this time and I’ll owe you a beer? :-)

  9. 13 Dave. C. 2009 November 12 at 11:49 am

    I consider proper cloaking as a legit form of end-user experience enhancement, albeit true that it also has a long history of us in SEO that sometimes slips a little grey, if not black. On the most part, the practice has been cleaned up over the years and improved to the point of acceptability even to a very reputable publisher such as WSJ.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Josh Young's Facebook profile

What I'm saving.

RSS What I’m reading.

  • An error has occurred; the feed is probably down. Try again later.

%d bloggers like this: