Archives/readability
2
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
386 Commits 14 Branches 0 Tags 29 MiB
89572ad29a
Commit Graph

188 Commits

Author SHA1 Message Date
David A Roberts
d60184966c Remove unused URI parameter from constructor 2018-04-26 10:30:15 +01:00
Andres Rey
3c76104adb Fix engadget test case 2018-03-23 00:27:50 +00:00
Andres Rey
4b99f41ec9 Add engadget test case 2018-03-23 00:27:50 +00:00
Andres Rey
6c5bc62959 Remove aside tags on test cases 2018-03-23 00:27:50 +00:00
David A Roberts
f8d9b1c224 Update test expectations 2018-03-19 13:20:29 +00:00
David A Roberts
8414158fa9 Fix _replaceBrs 
Previously, `nextElem` was not actually proceeding to the next element, and therefore aborting the paragraph at the first `<br>` (rather than the first `<br><br>` as the comment indicates).
 2018-03-19 13:20:29 +00:00
Brad Philips
8525c6af36 Fix relative URIs given <base> tags (#422) 2018-03-02 11:38:14 +00:00
Gijs Kruitbosch
d598baf02b Improve URL handling in JSDOMParser and Readability.js 
This change ups the required node version to 7.0 because it relies on the builtin url module.

We now pass a url when constructing a jsdom document or JSDOMParser document.
Because this is an API change, I'm increasing the package version.

Ultimately, I would like to remove the  argument from the readability constructor. It should
use the documentURI from the document it is passed.
 2018-02-28 11:29:29 +00:00
Andres Rey
834672ef86 Return longest text after failing to detect text longer than the configured value (#423) 
Save extracted text across attempts and return the longest one when all attempts fail, and add a test case from hukumusume
 2018-02-27 14:26:54 +00:00
Tom Z?hner
264b8e8968 Remove link elements when preparing article for display 2018-01-30 15:11:59 +00:00
Andres Rey
fa9d8bda48 Add la-nacion test case 2017-12-11 14:00:48 +00:00
Gijs Kruitbosch
807bf05aa3 Fix className usage so it deals correctly with SVG nodes (fixes #412). 2017-12-05 11:06:39 +00:00
Gijs Kruitbosch
c586aeb404 Fix generate-testcase.js script so it keeps caption classes 2017-11-30 10:41:15 +00:00
Gijs Kruitbosch
ad4dd26448 Update test expectations 2017-11-30 10:41:15 +00:00
Andres Rey
eb895b97a2 Add test case for title and h1 discrepancy 2017-11-27 15:53:38 +00:00
Andres Rey
c2e370c2c7 Add telegraph test case 2017-11-22 21:14:14 +00:00
Cameron McCormack
5ad448f831 Update test expectations. 2017-11-21 10:04:59 +00:00
Cameron McCormack
d88c9afc63 Use a hard coded classesToPreserve in tests. 2017-11-21 10:04:59 +00:00
Björgvin Ragnarsson
c3ff1a2d2c remove dead code 2017-11-02 23:15:13 +00:00
Taylor Hunt
b7c32feb25 Remove presentational HTML attributes (#385) 
* Remove presentational HTML attributes

Fixes #383

This patch loops through a list of known-presentational attributes in HTML, attempting to remove each from each cleaned element. (Checking for the attribute's existence first seems to just add needless overhead.)

The extra check for the HTML namespace is to avoid removing attributes that inline SVG needs.

* Only remove `width`/`height` for certain elements

Embedded media elements are allowed to have them, but not others.

* Address PR feedback

* Fix loop index formatting
* Only remove `width`/`height` from certain elements
* Combine logic into a single check/remove

* Attempt fixing my recursion

* One weird trick to get your loops to run

* Add inline SVG bailout

Try not to touch any styles for `<svg>`, because it's inherently presentational.

* Update tests to match newly-removed attributes

* Oh those wacky SVGs

The `position:absolute` is a trick to import clipping paths into the document without putting a big 300×150 empty space in it. (`display:none` and such disable the clipPath in some browsers.)

* Whoops, missed some `width`s

* Normalize SVG tagName

JSDOMParser differs from the official DOM here
 2017-08-09 20:26:34 +01:00
Gijs Kruitbosch
6f0bd81dfa Provide more useful output when attributes mismatch, r=me,test-only 2017-08-09 20:05:47 +01:00
Evan Tseng
924017c686 Merge pull request #370 from evanxd/bug-1347472 
Bug 1347472 - Tests for WordPress blogs, r=Gijs
 2017-03-17 11:09:14 +08:00
Evan Tseng
f6d0840b82 Bug 1347472 - Tests for WordPress blogs, r=Gijs 2017-03-15 17:46:07 +08:00
Evan Tseng
0e147aa722 Bug 1346717 - Tests for youth.cn, r=Gijs 2017-03-13 17:43:50 +08:00
Evan Tseng
630a0b7367 Bug 1346155 - Tests for gmw.cn, r=Gijs 2017-03-10 10:51:28 +00:00
Evan Tseng
8976fb08bf Bug 1345797 - Tests for yahoo.co.jp, r=Gijs 2017-03-09 18:01:31 +08:00
Evan Tseng
e48451c48c Bug 1342355 - Add tests for wikia.com, r=Gijs 2017-02-24 18:07:24 +08:00
Evan Tseng
c7c26382b9 Bug 1217007 - Remove nodes without any content, r=Gijs 2017-02-22 09:41:49 +00:00
Evan Tseng
0f147374b7 Bug 1323861 - Remove the readScript method, r=Gijs 2017-02-22 09:39:17 +00:00
Evan Tseng
27b97434a3 Merge pull request #352 from evanxd/bug-1338468 
Bug 1338468 - Tests for tumblr.com, r=Gijs
 2017-02-21 18:19:46 +08:00
Evan Tseng
b46e316034 Bug 1338468 - Tests for tumblr.com, r=Gijs 2017-02-10 17:39:25 +08:00
Evan Tseng
bd2d9f8100 Bug 1338475 - Tests for pixnet.net, r=Gijs 2017-02-10 17:35:22 +08:00
Evan Tseng
1369f923db Merge pull request #348 from gijsk/1322674-table-divs 
Fix bug 1322674 by only conditionally removing content from tables when the tables are layout tables, r=evanxd
 2017-02-09 18:27:01 +08:00
Gijs Kruitbosch
66c013c4fb Make it easier to find where errors are coming from based on the error message 2017-01-30 15:19:37 +01:00
Gijs Kruitbosch
9baea36169 Fix bug 1322674 by not removing content in data tables 2017-01-30 15:19:10 +01:00
Evan Tseng
19f2124c82 Bug 1259763 - Remove h2 when there is only one h2 and its text content substantially equals article title, r=Gijs 2017-01-28 13:41:56 +08:00
Evan Tseng
ccc8e9bf4c Bug 1330931 - Add tests for nytimes.com, r=Gijs 2017-01-23 11:56:36 +00:00
Evan Tseng
15e1f03261 Bug 1300697 - Reader View missed first few paragraphs on New York Times website, r=Gijs 2017-01-21 17:46:50 +00:00
Evan Tseng
a58913d975 Bug 1300697 - Add test for New York Times website 2017-01-21 17:46:50 +00:00
Evan Tseng
ad13a63065 Bug 1332236 - Add tests for cnet.com, r=Gijs 2017-01-20 11:17:49 +08:00
Evan Tseng
131d923d38 Bug 1167568 - Find a better topCandidate if there are other nodes scores are high enough, r=Gijs 2017-01-17 11:29:57 +08:00
Sebastian Hengst
5e9c7a3910 Bug 1322327 - Only remove images which are not descendants of a figure if there is more than one image, update affected tests 2016-12-21 16:31:25 +00:00
andrei-ch
c5ff44d8fe Clean <input>,<textarea>,<select>,<button> elements 2016-12-17 13:37:27 +00:00
Evan Tseng
a34d054f45 Merge pull request #332 from gijsk/fix-readability-determination-in-generate-testcase 
Use jsdom for parsing the document to determine readability (fixes #325), r=evanxd
 2016-12-15 17:30:24 +08:00
Evan Tseng
63230a307a Bug 1142312 - Add two more types of unlikely candidates: cover-wrap and yom-remote, r=Gijs 2016-12-15 11:30:40 +08:00
Gijs Kruitbosch
46842048c1 Use jsdom for parsing the document to determine readability (fixes #325) 2016-12-14 13:44:29 +00:00
Gijs Kruitbosch
0ab4ac8556 Fix test failures caused by timeout still being too low 2016-12-14 11:41:38 +00:00
Evan Tseng
e84c0c3f07 Bug 1285543 - Only use "og:title" or "twitter:title" if _getArticleTitle does not return a valid title, r=Gijs 2016-12-14 11:34:15 +00:00
Gijs
c2f7db51f5 Remove textContent from metadata file (fixes #324) (#326) 2016-12-09 13:28:56 -10:00
Evan Tseng
33dc8fa023 Bug 1255978 - Remove legends candidate, r=Gijs 2016-11-25 11:12:47 +00:00
Evan Tseng
af0aa5c59f Bug 1173548 - Find out text direction from ancestors of final candidate, r=Gijs 2016-11-25 10:24:41 +00:00
Evan Tseng
ece0d1ecea Bug 1317930 - Tests for msn.com, r=Gijs 2016-11-16 10:32:55 +00:00
Evan Tseng
1b694cf650 Bug 1310075 - Tests for qq.com. r=Gijs 2016-11-09 12:02:44 +00:00
Evan Tseng
522f39617f Bug 1310074 - Tests for yahoo.com. r=Gijs 2016-11-02 08:54:57 +00:00
Evan Tseng
4fa0d1b207 Bug 1177619 - Score div nodes which have br nodes. r=Gijs 2016-11-01 10:25:57 +00:00
Evan Tseng
8bfd2a978d Bug 1310073 - Tests for wikipedia.org. r=Gijs 2016-10-28 10:59:56 +01:00
Gijs
1a12befa41 Fix code style, tighten up eslint rules (#301) 2016-07-19 21:44:27 +01:00
Gijs Kruitbosch
46b08a5ea5 Address issue #277 by marking 'modal' unlikely+negative 2016-03-17 10:53:57 +00:00
Gijs Kruitbosch
a4d1e9ca12 Fix oversight in comment removal code exposed by better/newer jsdom implementation 2016-03-10 15:09:36 +00:00
Gijs Kruitbosch
e830ac9dd8 Fix eslint issues identified in m-c 2016-03-08 14:49:51 +00:00
Gijs Kruitbosch
dffa760c04 Fix issue #267 by ignoring hash URIs when making URIs absolute 2016-03-07 10:32:09 +00:00
Wes Johnston
f87a12400b Reuse test from pull request #239 which passes without modifications (modified by @gijsk to pass in the current XHTML test environment) 2016-01-25 11:18:47 +00:00
Gijs Kruitbosch
2e1cb3f467 Fix issue #251 by making JSDOMParser expect XML and stop making excuses for 'self-closed' things, when all that does is cause trouble 2016-01-22 19:57:45 +00:00
Gijs
a801846a45 Merge pull request #204 from mozilla/tweak-great-grandparent-scoring 
Updated great grandparent node scoring.
 2015-05-05 23:02:09 +01:00
Nicolas Perriault
ae0833522c Improved embedded video elements detection. 2015-05-05 22:11:11 +02:00
Nicolas Perriault
46304bb5fe Updated great grandparent node scoring. 2015-05-05 18:12:17 +02:00
Nicolas Perriault
88ef3893b5 Fixes #180 - Score intermediary headings. 2015-05-04 08:59:05 +02:00
Nicolas Perriault
dc1b2c9fa0 Refs #195 - Exclude nodes likely to be related content. 2015-05-04 08:51:45 +02:00
Nicolas Perriault
cc18cb5787 Ref #195 - Add support for dailymotion videos. 2015-04-30 15:02:52 +02:00
Nicolas Perriault
9dbc009376 Fixes #113 - Recursive node ancestor scoring. 2015-04-29 22:51:45 +02:00
Nicolas Perriault
44879722b6 Fixes #183 - Preserve list items. 2015-04-28 16:32:04 +02:00
Alexis Métaireau
5912e0c872 Add Firefox User-Agent when generating the test case. 2015-04-28 08:45:42 +02:00
Gijs
79aa2fca87 Merge pull request #189 from mozilla/dont-remove-headings 
Fixes #150 - Keep article intermediary headings.
 2015-04-27 23:36:39 +01:00
Margaret Leibovic
af6da2a87d Merge pull request #190 from mozilla/improved-author-meta-extraction 
Improved author metadata detection.
 2015-04-27 09:11:30 -07:00
Nicolas Perriault
7aee44adb2 Improved author metadata detection. 2015-04-27 17:03:23 +02:00
Gijs Kruitbosch
5f184053cd Make isProbablyReaderable include <pre>, and deal with long <br>-separated paragraphs and/or shorter-than-5-paragraph text and such. 2015-04-27 15:49:03 +01:00
Nicolas Perriault
2451a07a7d Fixes #150 - Keep article intermediary headings. 2015-04-27 15:15:52 +02:00
Margaret Leibovic
319a50b4f0 Fixes #184 - Don't strip class names from article content 2015-04-24 14:49:30 -07:00
Gijs
49e40768aa Merge pull request #185 from mozilla/score-section-tags-by-default 
Fixes #139 #143: Added more weight to section tags.
 2015-04-24 20:12:11 +01:00
Nicolas Perriault
f6ffa6acde Fixes #139 #143: Added more weight to section tags. 2015-04-24 19:55:51 +02:00
Nicolas Perriault
58cd789cd3 Improved title extraction 'algorithm'. 2015-04-24 16:16:10 +01:00
Nicolas Perriault
de89036cd5 Fixes #130 - Using js-beautify for HTML formatting. 2015-04-21 10:30:48 +02:00
Gijs
b37ff08bc7 Merge pull request #169 from mozilla/clean-footer-tags 
Fixes #163 - Avoid including footer tag contents.
 2015-04-17 16:53:51 +01:00
Nicolas Perriault
12c6a11f67 Fixes #163 - Avoid including footer tag contents. 2015-04-17 17:33:04 +02:00
Nicolas Perriault
6eeabf90c1 Fixes #164 - Add support for title alt semantic metadata. 2015-04-17 15:38:25 +02:00
Gijs Kruitbosch
0ff82de0f4 Implement createTextNode, do more relaxed escaping there, update testcase. 2015-04-13 14:32:49 +01:00
Margaret Leibovic
37a8cd4171 Bug 1147584 - Don't remove unlikely <a> tags, and replace <a> tags with their text content if they won't be useful links 2015-04-09 17:19:59 -07:00
Gijs
a6014f5854 Merge pull request #132 from gijsk/heise-ad-prioritization 
Don't look at banners and skyscrapers, remove <noscript> elements
 2015-04-09 20:12:01 +01:00
Gijs Kruitbosch
a6346a0ad4 Don't look at banners and skyscrapers, remove <noscript> elements 2015-04-09 20:02:46 +01:00
Nicolas Perriault
4424b0bad7 Refs #128 - Add support for options to Readability constructor. r=@gijsk 2015-04-09 11:56:58 +02:00
Gijs Kruitbosch
c53ca31907 Fixed test result output being sent at once 2015-04-08 15:24:34 +01:00
Nicolas Perriault
4d41f5e4ed Refs #117 - Drop social/share buttons. 2015-04-07 23:00:52 +02:00
Nicolas Perriault
d725ebc953 Fixes #99: JSDOMParser tag name case handling. r=@gijsk 2015-04-07 14:19:54 +02:00
Gijs Kruitbosch
a04f9d9a68 Fix embedded video testcase to work with both jsdom and JSDOMParser 2015-04-03 22:42:29 +01:00
Gijs Kruitbosch
225adef137 Turn off JS when using JSDOM in tests. 2015-04-03 22:42:29 +01:00
Gijs Kruitbosch
b1f6c62b25 Fix tests for readability to also work with jsdom 
For instance, jsdom's more spec-compliant parsing causes issues with auto-closing elements (lifehacker article) and with not having self-closing <img> and <br> tags. The former was fixed by removing offending markup, the latter by adjusting JSDOMParser to be more sane, and the expected outputs to cope with this.

Finally, JSDOMParser automatically drops comments. The test code needed to manually do this in the jsdom case.
 2015-04-03 22:41:55 +01:00
Gijs Kruitbosch
62801faa0e Fix JSDOMParser to do some HTML escaping and identify as JSDOMParser + add tests 2015-04-03 22:32:06 +01:00
Nicolas Perriault
3b636b59f0 Added readerable value to test pages metadata. 2015-04-03 15:14:49 +02:00
Margaret Leibovic
6b7834f798 add test for isProbablyReaderable 2015-04-02 17:40:06 -04:00
Nicolas Perriault
2bfa678cab Added separate command for reference benchmark. 2015-04-01 22:35:06 +01:00