Archives/readability
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
374 Commits 14 Branches 0 Tags 29 MiB
build-issues-node-js-update
Commit Graph

157 Commits

Author SHA1 Message Date
Andres Rey
5a5c8ba1a2 Add node to elementsToScore when _hasSinglePInsideElement is true 2017-11-22 21:14:14 +00:00
Cameron McCormack
6729538c77 Clean IDs and classes from output. 2017-11-21 10:04:59 +00:00
Björgvin Ragnarsson
c3ff1a2d2c remove dead code 2017-11-02 23:15:13 +00:00
Iqbal Ahmed
b3fde168cb Allow the word threshold parameter to be configurable 2017-09-19 15:38:10 +01:00
Taylor Hunt
b7c32feb25 Remove presentational HTML attributes (#385) 
* Remove presentational HTML attributes

Fixes #383

This patch loops through a list of known-presentational attributes in HTML, attempting to remove each from each cleaned element. (Checking for the attribute's existence first seems to just add needless overhead.)

The extra check for the HTML namespace is to avoid removing attributes that inline SVG needs.

* Only remove `width`/`height` for certain elements

Embedded media elements are allowed to have them, but not others.

* Address PR feedback

* Fix loop index formatting
* Only remove `width`/`height` from certain elements
* Combine logic into a single check/remove

* Attempt fixing my recursion

* One weird trick to get your loops to run

* Add inline SVG bailout

Try not to touch any styles for `<svg>`, because it's inherently presentational.

* Update tests to match newly-removed attributes

* Oh those wacky SVGs

The `position:absolute` is a trick to import clipping paths into the document without putting a big 300×150 empty space in it. (`display:none` and such disable the clipPath in some browsers.)

* Whoops, missed some `width`s

* Normalize SVG tagName

JSDOMParser differs from the official DOM here
 2017-08-09 20:26:34 +01:00
Gijs Kruitbosch
c387d12bc4 Fix eslint issue by unpacking ternary into a proper if statement 2017-07-17 13:25:56 +01:00
Gijs Kruitbosch
a63e63c91e Clarify title-splitting code (part of #357) 
This adds a number of comments and a wordCount helper function to try to make the code
a bit more readable (which seemed appropriate considering the number of regular expressions).
 2017-02-27 14:56:26 +00:00
Farbdose
aed7f10594 Remove general part of title when viewing websites with hierarchical titles 2017-02-27 14:56:26 +00:00
Evan Tseng
c7c26382b9 Bug 1217007 - Remove nodes without any content, r=Gijs 2017-02-22 09:41:49 +00:00
Evan Tseng
1369f923db Merge pull request #348 from gijsk/1322674-table-divs 
Fix bug 1322674 by only conditionally removing content from tables when the tables are layout tables, r=evanxd
 2017-02-09 18:27:01 +08:00
Gijs Kruitbosch
9baea36169 Fix bug 1322674 by not removing content in data tables 2017-01-30 15:19:10 +01:00
Evan Tseng
19f2124c82 Bug 1259763 - Remove h2 when there is only one h2 and its text content substantially equals article title, r=Gijs 2017-01-28 13:41:56 +08:00
Evan Tseng
15e1f03261 Bug 1300697 - Reader View missed first few paragraphs on New York Times website, r=Gijs 2017-01-21 17:46:50 +00:00
andrei-ch
5b1e69bdf2 Bug fix: still not grabbing full content from nytimes.com articles 
Solution: strip one level of empty <DIV> elements so they don’t
obstruct merging adjacent content downstream.
 2017-01-21 17:46:50 +00:00
andrei-ch
486927ebd9 Bug fix: many pages only grab partial content (dirty.ru, nytimes.com) 
1) Avoid conversion of whitespace text nodes into paragraphs. They
create a lot of noise and actually prevent sibling joining logic from
working in many pages.

2) Handle case when adjacent content is actually located in parent's
sibling node instead of top candidate’s sibling.
 2017-01-21 17:46:50 +00:00
Gijs Kruitbosch
2b28d6022f Merge branch 'Treora-commonjs' 2017-01-17 16:18:20 +00:00
Gijs Kruitbosch
f1797fcf7e Merge branch 'commonjs' of https://github.com/Treora/readability into Treora-commonjs 2017-01-17 16:17:04 +00:00
Gerben
43c54e8a71 Fix tiny mistake (typeof dump !== 'undefined') 2017-01-17 14:43:11 +00:00
Evan Tseng
e1c7170d85 Merge pull request #309 from evanxd/bug-1167568 
Bug 1167568 - Find a better topCandidate if there are other nodes scores are high enough, r=Gijs
 2017-01-17 12:08:39 +08:00
Evan Tseng
131d923d38 Bug 1167568 - Find a better topCandidate if there are other nodes scores are high enough, r=Gijs 2017-01-17 11:29:57 +08:00
Gerben
d63576be0f CommonJS support: set module.exports = Readability 2017-01-17 01:38:59 +01:00
Evan Tseng
cec7d90a14 Merge pull request #336 from gijsk/remove-ews-hack 
Remove hack for EWS content management system, r=evanxd
 2017-01-13 13:32:52 +08:00
Gijs
0d3043dacb Remove hack for EWS content management system 
Per issue #322
 2017-01-11 16:48:56 +00:00
Sebastian Hengst
5e9c7a3910 Bug 1322327 - Only remove images which are not descendants of a figure if there is more than one image, update affected tests 2016-12-21 16:31:25 +00:00
andrei-ch
c5ff44d8fe Clean <input>,<textarea>,<select>,<button> elements 2016-12-17 13:37:27 +00:00
Evan Tseng
63230a307a Bug 1142312 - Add two more types of unlikely candidates: cover-wrap and yom-remote, r=Gijs 2016-12-15 11:30:40 +08:00
andrei-ch
4a0d08c56a font-to-span conversion skips half the font elements on 'real' DOMs 2016-12-14 12:18:47 +00:00
Evan Tseng
e84c0c3f07 Bug 1285543 - Only use "og:title" or "twitter:title" if _getArticleTitle does not return a valid title, r=Gijs 2016-12-14 11:34:15 +00:00
Evan Tseng
33dc8fa023 Bug 1255978 - Remove legends candidate, r=Gijs 2016-11-25 11:12:47 +00:00
Evan Tseng
af0aa5c59f Bug 1173548 - Find out text direction from ancestors of final candidate, r=Gijs 2016-11-25 10:24:41 +00:00
Evan Tseng
4fa0d1b207 Bug 1177619 - Score div nodes which have br nodes. r=Gijs 2016-11-01 10:25:57 +00:00
Taylor Hunt
71aa562387 Add microformats2 class names to heuristics (#303) 
Microformats updated their old `hentry` to [a newer
`h-entry`](http://microformats.org/wiki/h-entry).

With the [number of IndieWeb sites breaking into the
ten-thousands](http://tantek.com/2016/190/b1/state-of-indieweb-summit),
this seems like a fair idea.
 2016-08-13 23:06:24 +01:00
Gijs
1a12befa41 Fix code style, tighten up eslint rules (#301) 2016-07-19 21:44:27 +01:00
Ivan Persidsky
fd11f92adb Use a dedicated method and backward iteration for removing nodes (#300) 
This improves compat with "real" DOMs that provide a live NodeList as the return value of getElementsByTagName.
 2016-07-18 12:56:51 +01:00
Gijs Kruitbosch
140d4c4aca Only compute textContent once. 2016-05-04 15:09:21 +01:00
usergit
327bfcb93f exposed textContent to be returned 
this returns the text content only, this is useful as it allows the content to be easily accessible
 2016-05-03 23:59:39 -07:00
Gijs
69b81f5d70 Fix #287: convert getElementsByTagName result to an array (#288) 2016-04-20 08:59:42 +01:00
Gijs Kruitbosch
46b08a5ea5 Address issue #277 by marking 'modal' unlikely+negative 2016-03-17 10:53:57 +00:00
Peter deHaan
b380917b4b Convert nested function declaration to function expression 2016-03-15 10:13:33 -07:00
Gijs Kruitbosch
e830ac9dd8 Fix eslint issues identified in m-c 2016-03-08 14:49:51 +00:00
Gijs Kruitbosch
dffa760c04 Fix issue #267 by ignoring hash URIs when making URIs absolute 2016-03-07 10:32:09 +00:00
Gijs Kruitbosch
a9597efc17 Fix bug 1230050 by checking for the 'hid' class specifically, r?MattN 2016-01-12 15:24:27 +00:00
Gijs
a801846a45 Merge pull request #204 from mozilla/tweak-great-grandparent-scoring 
Updated great grandparent node scoring.
 2015-05-05 23:02:09 +01:00
Nicolas Perriault
ae0833522c Improved embedded video elements detection. 2015-05-05 22:11:11 +02:00
Nicolas Perriault
46304bb5fe Updated great grandparent node scoring. 2015-05-05 18:12:17 +02:00
Nicolas Perriault
88ef3893b5 Fixes #180 - Score intermediary headings. 2015-05-04 08:59:05 +02:00
Nicolas Perriault
dc1b2c9fa0 Refs #195 - Exclude nodes likely to be related content. 2015-05-04 08:51:45 +02:00
Nicolas Perriault
cc18cb5787 Ref #195 - Add support for dailymotion videos. 2015-04-30 15:02:52 +02:00
Nicolas Perriault
9dbc009376 Fixes #113 - Recursive node ancestor scoring. 2015-04-29 22:51:45 +02:00
Nicolas Perriault
44879722b6 Fixes #183 - Preserve list items. 2015-04-28 16:32:04 +02:00
Gijs
79aa2fca87 Merge pull request #189 from mozilla/dont-remove-headings 
Fixes #150 - Keep article intermediary headings.
 2015-04-27 23:36:39 +01:00
Margaret Leibovic
af6da2a87d Merge pull request #190 from mozilla/improved-author-meta-extraction 
Improved author metadata detection.
 2015-04-27 09:11:30 -07:00
Nicolas Perriault
7aee44adb2 Improved author metadata detection. 2015-04-27 17:03:23 +02:00
Gijs Kruitbosch
5f184053cd Make isProbablyReaderable include <pre>, and deal with long <br>-separated paragraphs and/or shorter-than-5-paragraph text and such. 2015-04-27 15:49:03 +01:00
Nicolas Perriault
2451a07a7d Fixes #150 - Keep article intermediary headings. 2015-04-27 15:15:52 +02:00
Margaret Leibovic
319a50b4f0 Fixes #184 - Don't strip class names from article content 2015-04-24 14:49:30 -07:00
Gijs
49e40768aa Merge pull request #185 from mozilla/score-section-tags-by-default 
Fixes #139 #143: Added more weight to section tags.
 2015-04-24 20:12:11 +01:00
Nicolas Perriault
f6ffa6acde Fixes #139 #143: Added more weight to section tags. 2015-04-24 19:55:51 +02:00
Nicolas Perriault
58cd789cd3 Improved title extraction 'algorithm'. 2015-04-24 16:16:10 +01:00
Gijs
b37ff08bc7 Merge pull request #169 from mozilla/clean-footer-tags 
Fixes #163 - Avoid including footer tag contents.
 2015-04-17 16:53:51 +01:00
Nicolas Perriault
12c6a11f67 Fixes #163 - Avoid including footer tag contents. 2015-04-17 17:33:04 +02:00
Nicolas Perriault
6eeabf90c1 Fixes #164 - Add support for title alt semantic metadata. 2015-04-17 15:38:25 +02:00
Gijs Kruitbosch
0ff82de0f4 Implement createTextNode, do more relaxed escaping there, update testcase. 2015-04-13 14:32:49 +01:00
Margaret Leibovic
37a8cd4171 Bug 1147584 - Don't remove unlikely <a> tags, and replace <a> tags with their text content if they won't be useful links 2015-04-09 17:19:59 -07:00
Gijs
a6014f5854 Merge pull request #132 from gijsk/heise-ad-prioritization 
Don't look at banners and skyscrapers, remove <noscript> elements
 2015-04-09 20:12:01 +01:00
Gijs Kruitbosch
a6346a0ad4 Don't look at banners and skyscrapers, remove <noscript> elements 2015-04-09 20:02:46 +01:00
Nicolas Perriault
4424b0bad7 Refs #128 - Add support for options to Readability constructor. r=@gijsk 2015-04-09 11:56:58 +02:00
Nicolas Perriault
4d41f5e4ed Refs #117 - Drop social/share buttons. 2015-04-07 23:00:52 +02:00
Gijs Kruitbosch
7c60dba3b6 Fix Readability.js to work with jsdom's DOM implementation (in particular: no firstElementChild implementation...) 2015-04-03 22:32:13 +01:00
Margaret Leibovic
eb3a8e8dc4 Bug 1150695 - Move isProbablyReaderable function to Readability.js 2015-04-02 17:39:43 -04:00
Nicolas Perriault
f8d37e4276 Don't remove elements containing figures or having them as a parent. 2015-03-31 19:12:45 +02:00
Nicolas Perriault
b6730703a1 Fixes #81 - Keep article images. 2015-03-31 18:46:04 +02:00
Gijs
194a5376c8 Merge pull request #63 from mozilla/preserve-embedded-tweets 
Preserve inline tweets as they're part of article contents.
 2015-03-25 13:57:34 +00:00
Gijs Kruitbosch
b4332328f3 Fix an issue where we don't track scores for the parents appropriately. 2015-03-23 14:57:04 +00:00
Gijs
14b33b69db Merge pull request #65 from mozilla/support-embed-videos 
Fixes #56 - Updated support for embedded Youtube & Vimeo videos.
 2015-03-23 14:48:00 +00:00
Nicolas Perriault
ad52d8ee30 Fixes #53 - Fixed dot-slash relative URI resolution. 2015-03-23 12:54:46 +01:00
Nicolas Perriault
2d5f59f3eb Fixes #56 - Updated support for embedded Youtube & Vimeo videos. 2015-03-23 10:07:04 +01:00
Nicolas Perriault
d83763c8a1 Preserve inline tweets as they're part of article contents. 2015-03-22 18:19:34 +01:00
Nicolas Perriault
cf3dce6cf2 Refs #58 - Stripped embed tags. 2015-03-22 17:52:23 +01:00
Nicolas Perriault
eee224560b Addressed review comments from @Gijsk. 2015-03-21 18:05:58 +01:00
Nicolas Perriault
4f9615cb9a Use forEach when it makes sense. 2015-03-21 15:10:00 +01:00
Gijs Kruitbosch
955951659d Bug 1143725 - fix the Herald Sun website 2015-03-20 17:49:42 -07:00
Gijs Kruitbosch
eb81444946 Improve logic to rely on children instead of childNodes 2015-03-20 17:10:45 -07:00
Margaret Leibovic
3c2d93cd09 Improve byline algorithm 2015-03-20 09:16:28 -07:00
Gijs Kruitbosch
d94f3158d3 Fix readability.js to do a DOM traversal rather than relying on a wonky DOMCollection, fix trims, fix a potential null access, etc. 2015-03-19 18:28:48 -07:00
Margaret Leibovic
fc53e1a315 Set 'name' variable to null in _getExcerpt to avoid old values in future for loop iterations 2015-03-19 14:37:47 -07:00
Margaret Leibovic
2c7c504a36 Merge pull request #32 from gijsk/regex-issues-with-class-and-id-stuff 
Fix regex issues. r=margaret
 2015-03-19 10:15:35 -07:00
Gijs
aec1ce774d Merge pull request #31 from gijsk/testing-generates 
Allow generating tests from the web, make testing more closely match Firefox
 2015-03-19 10:13:36 -07:00
Gijs Kruitbosch
1c42f29aa5 Create a script to generate testcases, actually use our version of JSDOMParser 2015-03-19 10:10:42 -07:00
Gijs
17062c1ccf Fix video regular expression to support https 2015-03-19 06:43:28 -07:00
Gijs
d9f1e884dd Fix regex issues 2015-03-18 22:23:47 -07:00
Margaret Leibovic
98ee8f7463 Merge pull request #27 from gijsk/fix-missing-paragraphs 
Bug 1144441 - avoid leaving out paragraphs. r=margaret
 2015-03-17 18:47:07 -07:00
Gijs Kruitbosch
1d2df4a70e Bug 1144441 - avoid leaving out paragraphs 2015-03-17 18:18:44 -07:00
Margaret Leibovic
a9bd60154d Bug 1144355 - Bail if we don't have a body to parse. r?Gijs 2015-03-17 17:38:35 -07:00
Gijs Kruitbosch
d3f84a1e58 Fix class-related logging exception 2015-03-17 17:13:51 -07:00
Gijs Kruitbosch
ce0ebe24e0 Improve logging of elements 2015-03-17 16:22:26 -07:00
Margaret Leibovic
03d9e36161 Merge pull request #22 from gijsk/fix-empty-classes 
Don't create/leave empty class attributes around all the nodes we're using. r=margaret
 2015-03-17 15:01:44 -07:00
Nicolas Perriault
99f338a03a Added logging to test output. 2015-03-17 22:42:59 +01:00
Gijs Kruitbosch
b62fd27ba6 Don't create/leave empty class attributes around all the nodes we're using. 2015-03-17 13:17:49 -07:00
Gijs Kruitbosch
a563714567 Bug 1127778 - while we're at it, add more logs. 2015-03-17 12:02:03 -07:00