mercury-parser/src/extractors/custom/README.md

Custom Parsers

Mercury can extract meaningful content from almost any web site, but custom parsers allow the Mercury parser to find the content more quickly and more accurately than it might otherwise do. Our goal is to include custom parsers as many sites as we can, and we'd love your help!

The basics of parsing a site with a Mercury custom parser

Custom parsers allow you to write CSS selectors that will find the content you're looking for on the page you're testing against. If you've written any CSS or jQuery, CSS selectors should be very familiar to you.

You can query for every field returned by the Mercury Parser:

• title
• author
• content
• date_published
• lead_image_url
• dek
• next_page_url
• excerpt

Using selectors

Basic selectors

To demonstrate, let's start with something simple: Your selector for the page's title might look something like this:

export const ExampleExtractor = {
    ...

    // Order by most likely. Extractor will stop on first occurrence
    title: {
      selectors: [
        'h1.hed',
      ],
    },

    ...

As you might guess, the selectors key provides an array of selectors that Mercury will check to find your title text. In our ExampleExtractor, we're saying that the title can be found in the text of an h1 header with a class name of hed.

The selector you choose should return one element. If more than one element is returned by your selector, it will fail (and Mercury will fall back to its generic extractor).

Selecting an attribute

Sometimes the information you want to return lives in an element's attribute rather than its text — e.g., sometimes a more exact ISO-formatted date/time will be stored in an attribute of an element.

So your element looks like this:

<time class="article-timestamp" datetime="2016-09-02T07:30:01-04:00"></time>

The text you want isn't the text inside a matching element, but rather, inside the datetime attribute. To write a selector that returns an attribute, you provide your custom parser with a two-element array. The first element is your selector; the second element is the attribute you'd like to return.

export const ExampleExtractor = {
    ...

    // This example returns the datetime attribute if it exists
    date_published: {
      selectors: [
        ['time.article-timestamp[datetime]', 'datetime'],
      ],
    },

    ...

This is all you'll need to know to handle most of the fields Mercury parses (titles, authors, date published, etc.). Article content is the exception.

Cleaning content

An article's content can be more complex than the other fields, meaning you sometimes need to do more than just provide the selector(s) in order to return clean content.

For example, sometimes an article's content will contain related content that doesn't translate or render well when you just want to see the article's content. The clean key allows you to provide an array of selectors identifying elements that should be removed from the content.

Here's an example:

export const ExampleExtractor = {
  ...

  content: {
    selectors: [
      'div.g-blocks',
      'article#story',
    ],

    // Selectors to remove from the extracted content
    clean: [
      '.related',
      '.hidden',
    ],
  }

  ...
}

Using transforms

Occasionally, in order to mold the article content to a form that's readable outside the page, you need to transform a few elements inside the content you've chosen. That's where transforms come in.

This example demonstrates a simple tranform that converts h1 headers to h2 headers, along with a more complex transform that transforms lazy-loaded images to images that will render as you would expect outside the context of the site you're extracting from.

export const ExampleExtractor = {
  ...

  content: {
    selectors: [
      'div.article-content',
    ],

    transforms: {
      // In a simple tranform, each key is the selector,
      // and the value, provided it's a string, represents
      // the tag that the matched item should be transformed to.

      // Convert h1s to h2s
      h1: 'h2',

      // If a function is given as the value, it should return a string
      // to convert to or nothing (in which case it will not perform
      // the transformation.

      // Convert lazy-loaded noscript images to figures
      noscript: ($node) => {
        const $children = $node.children();
        if ($children.length === 1 && $children.get(0).tagName === 'img') {
          return 'figure';
        }

        return null;
      },
    },
  },

For much more complex tranforms, you can perform dom manipulation within the tranform function, but this is discouraged unless absolutely necessary. See, for example, the lazy-loaded image transform in the NYTimesExtractor, which transforms the src attribute on the lazy-loaded image.

How to generate a custom parser

Now that you know the basics of how custom extractors work, let's walk through the workflow for how to write and submit one. For our example, we're going to use The New Yorker. (You can find the results of this tutorial in the NewYorkerExtractor source.)

Step 0: Installation

First, you'll need to clone the Mercury Parser repository and install dependencies.

git clone git@github.com:postlight/mercury-parser.git

cd mercury-parser

yarn install

If you don't have already have watchman installed, you'll also need to install that:

brew install watchman

You should also create a new git branch for your custom extractor:

git checkout -b feat-new-yorker-extractor

Now that you're ready to go, take a look at the live custom parsers in src/extractors/custom for examples and to check if the site you want to write a parser for already exists.

Step 1: Generate your custom parser

If we don't already have a parser for the site you want to contribute, you're ready to generate a new custom parser. To do so, run:

yarn generate-parser

This script will prompt you to paste a link to an article you want to parse. The URL you choose will serve as the example your parser will test against. The script will also generate your custom parser and some barebones (and failing) tests for your parser.

For our New Yorker example, we're going to use this story.

When the generator script completes, you'll be prompted to run:

yarn watch:test -- www.newyorker.com

This will run the tests for the parser you just generated, which should fail (which makes sense — you haven't written it yet!). Your goal now is to follow the instructions in the generated www.newyorker.com/index.test.js and www.newyorker.com/index.js files until they pass!

Step 2: Passing your first test: Title extraction

If you look at your parser's test file, you'll see a few instructions to guide you in polishing your parser and making your tests pass.

By default, the first test, which ensures your custom extractor is being selected properly, should be passing. The first failing test checks to see whether your extractor returns the correct title:

it('returns the title', async () => {
  // To pass this test, fill out the title selector
  // in ./src/extractors/custom/www.newyorker.com/index.js.
  const html = fs.readFileSync(
    './fixtures/www.newyorker.com/1475245895852.html'
  );
  const articleUrl =
    'http://www.newyorker.com/tech/elements/hacking-cryptography-and-the-countdown-to-quantum-computing';

  const { title } = await Mercury.parse(articleUrl, html, { fallback: false });

  // Update these values with the expected values from
  // the article.
  assert.equal(title, 'Schrödinger’s Hack');
});

As you can see, to pass this test, we need to fill out our title selector. In order to do this, you need to know what your selector is. To do this, open the html fixture the generator downloaded for you in the fixtures directory. In our example, that file is fixtures/www.newyorker.com/1475248565793.html. Now open that file in your web browser.

The page should look more or less exactly like the site you pointed it to, but this version is downloaded locally for test purposes. (You should always look for selectors using this local fixture rather than the actual web site; some sites re-write elements after the page loads, and we want to make sure we're looking at the page the same way Mercury will be.)

(For the purpose of this guide, we're going to assume you're using Chrome as your default browser; any browser should do, but we're going to refer specifically to Chrome's developer tools in this guide.)

So, back to the title: We want to make sure our test finds the same title we see on the page. In our example, the title of the article is "Hacking, Cryptography, and the Countdown to Quantum Computing." To find the selector (again, in Chrome), press Cmd+Shift+C on OS X, or Ctrl+Shift+C on Windows, then click on the title.

The selector for this title appears to be h1.title. To verify that we're right, click on the Console tab in Chrome's Developer Tools and run the following check:

$$('h1.title');

If that returns only one match (i.e., an array with just one element), and the text of that element looks like the title we want, you're good to go!

Now, in your custom parser file, enter h1.title in your list of selectors for the title extractor:

export const NewYorkerExtractor = {
  domain: 'www.newyorker.com',
  title: {
    selectors: [
      'h1.title',
    ],
  },

  ...

}

Save the file, and... uh oh, our example still fails.

AssertionError: 'Hacking, Cryptography, and the Countdown to Quantum Computing' ==
  'Schrödinger’s Hack';

When Mercury generated our test, it took a guess at the page's title, and in this case, it got it wrong. So update the test with thte title we expect, save it, and your test should pass!

Step 3: Speed it up

We've been moving at a slow pace, but as you can see, once you understand the basics, extracting most items on the page is actually very easy. For example, if you follow the same instructions to find the author selector, you'll find that the .contributors selector will return the correct author (Alex Hutchinson).

For a slightly more complex example, you'll find after a bit of looking that the best place to get the most accurate datetime on the page is in the head of the document, in the value attribute of a meta tag:

<meta value="2016-09-26T14:04:22-04:00" name="article:published_time" />

As explained above, to return an attribute rather than the text inside an element, your selector should be an array where the first element is the element selector and the second element is the attribute you want to return. So, in this example, the date_published selector should look like this:

  ...

  date_published: {
    selectors: [
      ['meta[name="article:published_time"]', 'value'],
    ]
  },

  ...

You can refer to the NewYorkerExtractor to see more the rest of the basic selectors.

Step 4: Content extraction

I've left content extraction for last, since it's often the trickiest, sometimes requiring special passes to clean and transform the content. For the New Yorker, the first part is easy: The selector for this page is clearly div#articleBody. But that's just our first step, because unlike the other tests, where we want to make sure we're matching a simple string, we need to sanity check that the page looks good when it's rendered, and that there aren't any elements returned by our selector that we don't want.

To aid you in previewing the results, you can run the ./preview script to see what the title and content output look like. So, after you've chosen your selector, run the preview script on the URL you're testing:

./preview http://www.newyorker.com/tech/elements/hacking-cryptography-and-the-countdown-to-quantum-computing

This script will open both an html and json file allowing you to preview your results. Luckily for us, the New Yorker content is simple, and doesn't require any unusual cleaning or transformations — at least not in this example. Remember that if you do see content that needs cleaned or transformed in the selected content, you can follow the instructions in the clean and transform sections above.

Submitting a custom extractor

If you've written a custom extractor, please send us a pull request! Passing tests that demonstrate your parser in action will help us evaluate the parser.

Sometimes you may find that the site you're parsing doesn't provide certain information. For example, some sites don't have deks, and in those instances, you don't need to write a selector for that field. If there's a test for a selector you don't need, you can just remove that test and make note of it in your pull request.