Commit Graph

150 Commits (a014e0c9c8bd2e4a65dadf0cc4347620ee8ec214)

Author SHA1 Message Date
PalmerAL a014e0c9c8 exclude graphs from nytimes articles 5 years ago
Radhi Fadlillah 6a5066abe2 Fix tabular data got removed 5 years ago
PalmerAL f70d36852b check itemprop when determining whether a node is a byline 5 years ago
Andres Rey f836a8f291 Add "gdpr" to the list of negative tags 5 years ago
Gijs Kruitbosch 60ef565b67 Don't choke on <meta> tags that do not have a content attribute 5 years ago
Gijs Kruitbosch 30f9670a5f Avoid setAttribute errors from invalid attributes, fixes #392 6 years ago
Gijs 15d411a865
Add comment to indicate duplicate regexes
This comment was added in mozilla-central and seems useful, adding it to keep m-c and github in sync.
6 years ago
Gijs Kruitbosch 977be42d1f Fix link normalization for live HTMLCollections
Newer versions of JSDOM implement getElementsByTagName correctly.
This means it returns a live node list. When calling
`Element.replaceChild` for links inside the loop over that
collection, elements disappear from the list, meaning we miss
every other item. Without this fix, the `clean-links` testcase
breaks.
6 years ago
Gijs Kruitbosch e8bb7f722f Fix whitespace normalization in title metadata
When switching to a newer version of JSDOM, it is more literal
about listing whitespace as part of textContent, including
newlines and not normalizing multiple spaces.

It seems prudent to just always normalize whitespace for titles,
which are guaranteed to be pretty short anyway.
6 years ago
Gijs Kruitbosch 2620542dd1 Split off isProbablyReaderable implementation 6 years ago
Maria Luiza Soares 262fffd703 Retrieve site name on parse, based on meta og:site_name 6 years ago
Gijs 876c81f710 Update sorting function in Readability.js
Simplify sorting function also considering case where arguments are equal

Co-Authored-By: jemrobinson <james.em.robinson@gmail.com>
6 years ago
James Robinson ee18c21fc2 Switched sort function from boolean to explicit -1 and 1 thus avoiding failures to sort when false is evaluated as 0 6 years ago
Dan Burzo 44e90de00b Elements that have no .style (e.g. mathml) are probably visible; fixes #493 6 years ago
Daniel Aleksandersen 3be1aaa01c Recognize Sina Weibo meta tags
http://open.weibo.com/wiki/Weibo_meta_tag
6 years ago
Daniel Aleksandersen 5a69d4a8eb Improve metadata extraction (#478)
* Improve metadata extraction

* Recognize meta[property] as a space-separated list
* Recognize Dulin Core (dc|dcterm): metadata.
* Prefer Dublin Core, Open Graph, Twitter, and HTML in that order.
* _getArticleTitle() is now only used as fallback if document
 doesn't provide good metadata.
6 years ago
Daniel Aleksandersen 0449dbf186 Recognize more iframe video embed video services
* TenCent QQ Video, Alexa Rank 8
* Twitch clips and streams, Alexa Rank 33
* Internet Archive, Alexa Rank 265
* Wikimedia, Alexa Rank 347
6 years ago
Gijs Kruitbosch f782bc5f06 Avoid global flag when looking for metadata using regexes 6 years ago
Gijs Kruitbosch 7cf95bd427 Fix same-line loops and if statements 6 years ago
Gijs Kruitbosch d9f7bb2965 Fix quotes 6 years ago
Gijs Kruitbosch 7d03bec52d Fix issues with finding nytimes content caused by in-article ads 6 years ago
David A Roberts ea4165721f Remove single-cell tables 6 years ago
David A Roberts 72bd1a8532 Don't nest paragraphs 6 years ago
David A Roberts afcc4b8e49 Fix titles not being trimmed sometimes 6 years ago
Gijs Kruitbosch d4b842c82a Match headings on trimmed strings to avoid whitespace causing mismatches 6 years ago
Gijs Kruitbosch 8c02a0d34c Fix #283 and remove hidden nodes 6 years ago
David A Roberts 656a6673d9 Don't put non-phrasing content into paragraphs 6 years ago
David A Roberts 5ae90930cd Don't convert DIVs to Ps when more than 25% links 6 years ago
David A Roberts 9f2c5cb42e Put phrasing content into paragraphs
This removes the need for `p.readability-styled` elements.
6 years ago
Gijs Kruitbosch f4ab856992 Check for a document being passed
This provides a descriptive error message if no document is passed, and
ignores the first argument if the second argument looks like
a reasonable DOM document instance.
6 years ago
David A Roberts 7a24801958 Don't include root html node in candidates
Fixes #435
6 years ago
David A Roberts d60184966c Remove unused URI parameter from constructor 6 years ago
David A Roberts 5ee03bc960 Stop Readability depending on Node.* constants 6 years ago
Andres Rey 6fd816496c Clean <aside> tags on _prepArticle 6 years ago
David A Roberts 8414158fa9 Fix _replaceBrs
Previously, `nextElem` was not actually proceeding to the next element, and therefore aborting the paragraph at the first `<br>` (rather than the first `<br><br>` as the comment indicates).
6 years ago
Joan Espasa Arxer 3ff9a166fb Changed wordThreshold to charThreshold to better reflect the semantics. 6 years ago
Brad Philips 8525c6af36 Fix relative URIs given <base> tags (#422) 6 years ago
Andres Rey 834672ef86 Return longest text after failing to detect text longer than the configured value (#423)
Save extracted text across attempts and return the longest one when all attempts fail, and add a test case from hukumusume
6 years ago
Tom Z?hner 264b8e8968 Remove link elements when preparing article for display 6 years ago
Andres Rey 01ffd0c617 Remove "modal" from strings to remove 7 years ago
Gijs Kruitbosch 807bf05aa3 Fix className usage so it deals correctly with SVG nodes (fixes #412). 7 years ago
Gijs Kruitbosch 092a8aeaff Revert removing ids from elements 7 years ago
Andres Rey 9ce4d87232 Fall back to the original title if after trimming the text we have too many words before the colon. 7 years ago
Andres Rey 5a5c8ba1a2 Add node to elementsToScore when _hasSinglePInsideElement is true 7 years ago
Cameron McCormack 6729538c77 Clean IDs and classes from output. 7 years ago
Björgvin Ragnarsson c3ff1a2d2c remove dead code 7 years ago
Iqbal Ahmed b3fde168cb Allow the word threshold parameter to be configurable 7 years ago
Taylor Hunt b7c32feb25 Remove presentational HTML attributes (#385)
* Remove presentational HTML attributes

Fixes #383

This patch loops through a list of known-presentational attributes in HTML, attempting to remove each from each cleaned element. (Checking for the attribute's existence first seems to just add needless overhead.)

The extra check for the HTML namespace is to avoid removing attributes that inline SVG needs.

* Only remove `width`/`height` for certain elements

Embedded media elements are allowed to have them, but not others.

* Address PR feedback

* Fix loop index formatting
* Only remove `width`/`height` from certain elements
* Combine logic into a single check/remove

* Attempt fixing my recursion

* One weird trick to get your loops to run

* Add inline SVG bailout

Try not to touch any styles for `<svg>`, because it's inherently presentational.

* Update tests to match newly-removed attributes

* Oh those wacky SVGs

The `position:absolute` is a trick to import clipping paths into the document without putting a big 300×150 empty space in it. (`display:none` and such disable the clipPath in some browsers.)

* Whoops, missed some `width`s

* Normalize SVG tagName

JSDOMParser differs from the official DOM here
7 years ago
Gijs Kruitbosch c387d12bc4 Fix eslint issue by unpacking ternary into a proper if statement 7 years ago
Gijs Kruitbosch a63e63c91e Clarify title-splitting code (part of #357)
This adds a number of comments and a wordCount helper function to try to make the code
a bit more readable (which seemed appropriate considering the number of regular expressions).
7 years ago