Madsack+ Viewer WebApp

updated by 2 min read #reverse engineering

This post describes how the Madsack Media Group’s news sites worked in 2020. The sites have changed since then, and the method most likely no longer works.

Background#

I like to read the local news in the morning to find out what is going on in my hometown. Besides announcements from the local authorities and utilities, there is a lot of news that is hard to find anywhere else. Much of it sits behind a subscription paywall, and most of the articles don’t interest me enough to subscribe.

News sites also want their articles to be found by search engines, and that means delivering the content to crawlers. In 2020, the Madsack sites did this by embedding the whole article as JSON in every page, and only hid it in the browser.

The code is at robert/madsack-plus.

How it worked#

The core of Madsack+ is a single function. It takes the URL of an article and returns the article data that the page embedded for search engines. The version below is tidied up to follow PEP 8; it reflects the sites as they were in 2020.

python The core of Madsack+
import json
import re

import requests
import validators

ARTICLE_PATTERN = re.compile(
    r"Fusion\.globalContent=(.*?);Fusion\.globalContentConfig"
)


def get_madsack_article(url):
    """Return the article JSON embedded in a Madsack page, or an error dict."""
    if not validators.url(url):
        return {"error": True}

    response = requests.get(url, timeout=10)
    if response.status_code != 200:
        return {"error": True}

    try:
        article_json = ARTICLE_PATTERN.findall(response.text)[0]
        return json.loads(article_json)
    except (IndexError, json.JSONDecodeError):
        return {"error": True}

The function works in four steps:

  1. It checks that the URL is valid with validators.url().
  2. It fetches the page with requests and checks for status code 200 (OK).
  3. A regular expression extracts the text between the two literals Fusion.globalContent= and ;Fusion.globalContentConfig in the HTML. That text is a JSON object.
  4. json.loads() turns the JSON into a Python dictionary, which the function returns.

If the URL is invalid, the request fails, the pattern doesn’t match or the JSON doesn’t parse, the function returns a dictionary with the key error set to True.

The JSON contains everything needed to show the whole article.

Conclusion#

The paywall was enforced only in the browser; the full text was already in the page. If you rely on a paywall, check what your pages deliver to crawlers. If you enjoy a local paper’s journalism, consider paying for it.

Sources#

cd ~/projects