A Guide to Circumventing Newspaper Paywalls

11 min readMay 31, 2021

Introduction
Types of Paywalls
Reading Mode(s)
uBlock Origin — filters
uBlock Origin — blocking cookies
Paywall Removal Websites
Incognito Mode
Internet Archive
Mobile Workarounds

Introduction

In our digital-focused world, news reporting has transitioned from TV telecasters and newspapers to online media. However, the competition for digital ad revenue in these spaces has become hyper-competition, possibly to a fault. Revenue in the advertising space has steadily decreased the past few years while media ad spending has increased during the same timespan. This has forced the hands of online news publishers to move to subscriptions and paywalls.

Types of Paywalls

Let’s go over the two types of paywalls that we will discuss. I like to call them soft paywalls and hard paywalls.

A hard paywall fetches some of an article, renders only that block of text, and covers what would be the remainder of the article with a message. The message usually points you to a subscription plan. Let’s look at an Athletic article:
https://theathletic.com/2622464/2021/05/30/devin-haney-wobbled-against-jorge-linares-sparks-newfound-confidence-in-young-rivals-hes-not-worried/

This article will only deliver your content if you are signed into a paid subscription account. Unfortunately, there is no getting around this.

A soft paywall fetches all of the article’s content. These simply cover the bulk of the article with a message (again, usually asking the user to log in), but disables overflow (scrolling) while hiding the content. These are superficial ways of rendering “premium” content because this paywall is cosmetic and nothing more. Here is an example of such an article: https://www.bloomberg.com/news/articles/2021-05-30/canada-lowers-flags-to-honor-residential-school-children

Using the Google Chrome Inspector, we can see the glaring difference between these paywall types. On a computer, one can hit f12 (Opt+Cmd+I on Mac) or right-click and hit Inspect to find the inspector. Once the inspector is open, navigate to Network and refresh the page. You can filter the network object coming in to just Docs. There will be a list of objects that have entered your browser through your network. In the screenshots below, the first object was the one containing the actual news article.

In the Athletic, you can see that the content fades away:

The rest of the article does not enter the browser, hence the hard paywall.

Here is a Bloomberg article:

You can see that the entire content of the article has successfully fetched, which means that the message on the left side of the screenshot is just covering the article. This is just a soft paywall and is easily mitigated. Getting around these articles is very easy, and there are many ways to accomplish this.

Reading Mode(s)

Reading mode is a feature that many browsers include nowadays to remove unnecessary clutter. This inadvertently removes most soft paywalls in an instant.

In Firefox, there is an icon that you can click in toolbar in mobile and in desktop. See this article for more information: https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free-web-pages

Here is the button in question:

In Chromium based browsers, such as Google Chrome, Brave, Opera, and Edge, an extension can be used for the desktop variants. My favorite extension for reader mode is readermode.io’s Reader Mode. This extension covers your page (instead of re-rendering) with a readable text body of the article. It is just a simple button click away:

Here is the result:

This can be installed through the Chrome Web Store.

If you do not want to use an extension, you can enable Chrome’s secret reader mode by entering chrome:flags in the address bar:

Search for Enable Reader Mode and set this to Enabled:

Restart the browser and you will see Enter Reader Mode in the menu upon visiting a news site:

For desktop Safari users, click on the loose-leaf icon:

Here is Safari’s reader mode:

Another great extension is iamadamdev’s bypass-paywalls-chrome extension. Unfortunately, it is not in the Chrome Web Store and needs to be manually installed. The website for this extension gives instructions on how to install it.

A fork of this extension is Magnolia’s bypass-paywalls-firefox-clean and bypass-paywalls-chrome-clean. These are my current recommendations over the upstream extension.

Here is what the options look like:

Items that are checked will have their cookies removed, along with some extra functionality that this extension employs to stop the paywalls from appearing. However, understand that logins will break for sites that are checked off.

uBlock Origin — filters

My favorite browser extension ever is the content blocker uBlock Origin. It can be used simply as an adblocker, but it also extends well beyond that.

This extension can be downloaded for:

Firefox — https://addons.mozilla.org/en-US/android/addon/ublock-origin/

Chrome — https://chrome.google.com/webstore/detail/ublock-origin/cjpalhdlnbpafiamejdnhcphjbkeiagm

Edge — https://microsoftedge.microsoft.com/addons/detail/ublock-origin/odfafepnkmbhccpbejgmiehpchacaeak

Firefox for Android — https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/

Here is their Github repo.

For this extension, one can use Magnolia’s filter list for adblockers, which includes uBlock Origin. These work really well, but the bypass extensions listed above are more granular in their approach.

To add the filter list, go to the uBlock Origin Dashboard, then navigate to Filter lists. On the bottom of these options is Import, where you can past the raw .txt url of the filter list:

Once you hit Apply changes, this list will now work on every site that it supports. It seems to work by removing the cookies on news sites that are used to track how many times that you’ve visited that site. The bypass extensions utilize JavaScript logic on top of this, but with the Manifest v2 deprecation, this might be limited. It is not possible for uBlock Origin Lite to handle this right now, but who knows what the future holds. I will keep an eye out on how these changes will impact the bypass extensions, along with Chromium browsers’ possible lack of support for the full uBlock Origin extension. As for right now, uBlock Origin and the bypass extensions are my top recommendation.

uBlock Origin — blocking cookies

Some sties try to block many of these methods using cookies. Cookies are files stored by your browser, containing data used by a website. They can be stored, deleted, and expired. Some sites will create cookies in upon your visit, and based on certain criteria- the site will create a paywall. These criteria can include how many times you have visited, from which geolocation you are viewing the site, whether you are logged onto the news sites, etc. To show how this works, let’s look into a couple of extensions:

In Firefox (desktop and Android):

Click on this icon in your addons toolbar:

The Cookie Quick Manager dialog will pop up. Now, if you suspect that a website is using cookies to block articles, you can click “Search Cookies for: [current_website]”:

Next, you will be presented this window:

I recommend doing a trial-and-error approach. Open the article and delete one cookie at a time, refreshing the article after each cookie deletion. To delete a cookie, highlight it in the Cookies column and on the bottom right (at the bottom of the Details column), click on the trash can icon. If a specific cookie is the culprit, then you should see the article work after that cookie is deleted. Note down the cookie, which will be the text BEFORE the colon, for eg:

__tbc:%7Bkpex%7DNl1QZqSN2PPBVoHMBDZL3UBMOEd9hh9pt42_GrF4tgsAkFdcGU6trnNTXjB_MeGQ
Note the italicized portion 👆; the cookie name that you will need is __tbc

In Chromium browsers:

Click on this icon in the extensions toolbar:

Approve permission for Cookie-Editor on the website you are viewing, or for all sites:

Click the Allow button in the modal dialog:

Now, you can access the editor:

Click on the trash can icon to the right of the cookie entry in order to delete the specific cookie. Clicking on the trash can on the bottom of the screenshot will delete all cookies on the site! The cookie name is the text next to the left-side arrow. Again, in this example the cookie name is __tbc

This is all the same as finding the cookie from the browser Developer Tools -> Storage:

Blocking the offending cookie:

Now that we know how to find cookies and delete them using these extensions, let’s take on a real-world offender: Nikkei Asia

When you click on a more than 1 article at the time of this writing
Example article:

https://asia.nikkei.com/Spotlight/Japan-plane-crash/Haneda-air-traffic-control-missed-warning-alert-in-JAL-collision

Take note of the blocked content:

You even get this modal dialog afterwards:

The example cookie shown above is actually the cookie on this site that causes this behavior!

Open up the uBlock Origin dashboard and click on My filters:

Add the following line:

nikkei.com##+js(cookie-remover, __tbc)

The syntax is as follows:

DOMAIN_NAME##+js(cookie-remover, COOKIE_NAME)
In this example, the DOMAIN_NAME is: nikkei.com
The COOKIE_NAME is: __tbc

Hit Apply changes once you have added this line.

Now, refresh the news site.

We have content 🙌:

Note that techniques similar to this are often used by filter lists to bypass soft paywalls. This is how you can create your own custom filter for this purpose!

Paywall Removal Websites

EDIT: Outline seems to be defunct. Try 12ft instead! Remove Paywall is another good option.

Outline is an online tool that generates links to a reading mode equivalent to an article. Let’s take this SI Live article and insert it into Outline:

Here is the result:

https://outline.com/rAtCrA

Incognito Mode

Incognito Mode / Private Browsing mode allows users to browse with a clean slate, so to speak. This mode allows for the navigating of websites without cookies, web storage, nor any cache pre-loaded. Since most news sites use cookies to keep track of how many articles one has read, this can trick the site into thinking that this is your first article read. This does not always work because there are ways for websites to track whether or not you are in private browsing.

The second option in this menu opens up a private tab:

Internet Archive

You can save snapshots of websites using the Internet Archive.

Let’s take this NY Times article as an example.

Log onto the site and paste the url inside of the WaybackMachaine box like so:

The result is a web archive link. For our specific example:

Wayback Machine

The Wayback Machine is an initiative of the Internet Archive, a 501(c)(3) non-profit, building a digital library of…

web.archive.org

You can navigate the dates of all snapshots using the calendars on the page:

You will find the article’s snapshot without interference from paywall block only if the snapshot was saved during a state where the article was fully visible. Here is what you would see:

The web archive makes great for sharing full articles in an easy to access way!

Mobile Workarounds

The battle on the mobile front is one that has much fewer options. Reader mode on Google Chrome for mobile devices is easily my favorite way to read articles by far. To enable it in Chromium browsers:

Navigate to chrome://flags and search for Reader: