* fix: add alternative word count method
* fix: replace pages_rendered key with rendered_pages for consistency
* fix: return first lead_image_url when multiple og:image present
* fix: properly pull image src from lazy loaded img
* fix: allow drop cap character in medium custom extractor
* fix: refined medium parser
* Update more dependencies
Bumps almost everything up, removing almost all warnings from yarn audit. Doesn't touch cheerio or jest, as they require more attention and QA still.
* Adjust more dependencies, tweak build files
Not to be confused with extractor fixtures, which are snapshots of a webpage.
This change removes the pattern of separate JS files that provide "fixtures" for tests, which are used as provided or expected strings in tests. They were inconsistent and disorganized, and generally just served to add indirection to test files. So now all those strings are defined where they are used in their respective tests.
* feat:Add a custom extractor for ma.ttias.be.
When parsing content for cron.weekly issues, such as the one at https://ma.ttias.be/cronweekly/issue-130/, Mercury Parser would remove headings and ordered lists that were part of the content. This resolves that as follows:
* Remove "id" attributes from "h1" and "h2" elements. Those attributes would result in the elements having a low weight.
* Since Mercury Parser demotes "h1" elements to "h2", demote "h2" elements to "h3".
* Add class="entry-content-asset" to "ul" elements to avoid them being removed.
* removed redundant comment.
* feat: Add a custom extractor for engadget.com.
* feat: Add a custom extractor for www.ndtv.com.
* Works, but I need to figure how to make pagination work correctly.
* fixed pagination - would only retrieve first or second page because we would send contentOnly: true on subsequent pages (page 2).
removed failover: true from preview.
* rolled back { fallback: false } option removal
* Clarified comments.
* rolling back yarn.lock changes
Co-authored-by: John Holdun <john@johnholdun.com>
* feat:Add a custom extractor for ma.ttias.be.
When parsing content for cron.weekly issues, such as the one at https://ma.ttias.be/cronweekly/issue-130/, Mercury Parser would remove headings and ordered lists that were part of the content. This resolves that as follows:
* Remove "id" attributes from "h1" and "h2" elements. Those attributes would result in the elements having a low weight.
* Since Mercury Parser demotes "h1" elements to "h2", demote "h2" elements to "h3".
* Add class="entry-content-asset" to "ul" elements to avoid them being removed.
* removed redundant comment.
* feat: Add a custom extractor for engadget.com.
* Works, but I need to figure how to make pagination work correctly.
* fixed pagination - would only retrieve first or second page because we would send contentOnly: true on subsequent pages (page 2).
removed failover: true from preview.
* rolled back { fallback: false } option removal
* Clarified comments.
Co-authored-by: John Holdun <john@johnholdun.com>
* feat:Add a custom extractor for ma.ttias.be.
When parsing content for cron.weekly issues, such as the one at https://ma.ttias.be/cronweekly/issue-130/, Mercury Parser would remove headings and ordered lists that were part of the content. This resolves that as follows:
* Remove "id" attributes from "h1" and "h2" elements. Those attributes would result in the elements having a low weight.
* Since Mercury Parser demotes "h1" elements to "h2", demote "h2" elements to "h3".
* Add class="entry-content-asset" to "ul" elements to avoid them being removed.
* removed redundant comment.
* feat: Add a custom extractor for engadget.com.
Co-authored-by: John Holdun <john@johnholdun.com>
* Implemented custom extractor gruene.de
* Cleaner output of custom extracter www.gruene.de
* Updated fixture for www.gruene.de from real page
* Trying to pick image from og:image -- doesn't work ...
Co-authored-by: John Holdun <john@johnholdun.com>
Removing a couple extraneous CircleCI commands to see if they're still needed. I think one of the removed lines is causing #654 to fail, but let's see.
* feat: Add a custom extractor for pastebin.com
* feat: transforms <li> to <p> in pastebin.com
Co-authored-by: Felipe Canejo <felipecanejo@gmail.com>
Co-authored-by: John Holdun <john@johnholdun.com>
* feat:Add a custom extractor for ma.ttias.be.
When parsing content for cron.weekly issues, such as the one at https://ma.ttias.be/cronweekly/issue-130/, Mercury Parser would remove headings and ordered lists that were part of the content. This resolves that as follows:
* Remove "id" attributes from "h1" and "h2" elements. Those attributes would result in the elements having a low weight.
* Since Mercury Parser demotes "h1" elements to "h2", demote "h2" elements to "h3".
* Add class="entry-content-asset" to "ul" elements to avoid them being removed.
* removed redundant comment.
Co-authored-by: John Holdun <john@johnholdun.com>
* chore(deps-dev): bump karma from 3.1.4 to 6.3.16
Bumps [karma](https://github.com/karma-runner/karma) from 3.1.4 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v3.1.4...v6.3.16)
---
updated-dependencies:
- dependency-name: karma
dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Update CircleCI config
Removing a couple extraneous CircleCI commands to see if they're still needed. I think one of the removed lines is causing #654 to fail, but let's see.
* chore: Update karma-browserify
* Revert "chore: Update CircleCI config"
This reverts commit c474be7433.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: John Holdun <john@johnholdun.com>
These extractors were variously failing tests as I tried updating dependencies. It seems like some of the format detection logic has changed, and making these date detectors more explicit fixes them.
John Holdun
Austin
Michael Ashley
Sarah Doire
Simon Reinhardt
Nate Weaver
John Brayton
Sven Wiegand
dependabot[bot]
Jae Hanley
Samuel Clay
Jim Nielsen
Marco Wiedemeyer
Felipe Canejo
James Shakespeare
Ethan Jucovy
Peter Dave Hello
Joe Moon
Nick Sweeting