* chore: add missing fields to package.json
* feat: add postlight org scope to package name
* feat: automate npm publish
* test: npm publish without filters
* fix: add docker image
* test: change directory
* test: add working directory
* fix: defaults syntax
* test: add workspace
* fix: attach workspace
* fix: use standard mercury email
* fix: use ISO time format and preserve original timezone offset
* fix: do not match time zone offset
* chore: move babel runtime-corejs2 to prod deps
* chore: uncomment config to deploy on git tag
* feat: publish to npm public
* adding browser-request
It doesn't seem to impact the build, but technically it should be there
so for good measure, why not...
* chore: roll version back to original state
* dx: remove commented code and obvious comments that can be looked up
* dx: remove commented out eslint options
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove test block as all its code was commented out
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove regex example comments
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out import
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* dx: remove commented out code
* chore: remove empty files
* chore: re-prettier code that may have missed it
* added back nec comments
* chore: update .nvmrc
* added prettier and pre-commit hooks
* update docker image to new node
* add karma-cli to get web tests working
* explictly install karma... seems to fix problem
* remove pre-built phantomjs
* swap install order
* fix: scrub meta viewport tags
They leak to the parent page when using the web version of Mercury
Parser.
* chore: build
* fix: keep DOM in memory to avoid conflicts
* feat: prospect magazine parser
Couldn’t find a way to parse the date but I think it’s good otherwise.
* fix: pulls date
* fix: add timezone
* fix: generalize
* feat: forward.com parser
LGTM although image didn’t show up in preview
* feat: also pull imge into content
* fix: generalize selectors
* fix: generalize selector
* feat: qdaily parser
Firstly — I accidentally tried to generate the parser on the master
branch, and I’m not sure where it is, maybe floating in the nether
world.
On to the parser — this one was a bit tricky because things were in
Chinese! The content appears to be parsing (as seen in preview) but
it’s not passing the test. I noticed the second “ ‘ “ mark isn’t
appearing on the parser side.
Additionally, some of the lazy loading images aren’t appearing in the
preview (I cleaned the wrong lazy load images that appeared), so
someone will probably have to work on that (I don’t know how to do
transforms yet).
* fix tests
* fix: selector generalization
* feat: gothamist extractor
* feat: add other gothamist network sites
* fix: try getting date another way
* fix: add gothamist timezone
* fix: generalize selectors
* fix: h1 is inside entry-header, needs to be specific because of another h1 on the page
* fix: general and specific selector
* feat: natgeo parser
For some reason, the local copy of the article didn’t grab the author
name in it, so I couldn’t figure out how to parse it. The generic
parser took a name of an author of a paper mentioned in the article,
and thought that was the author name, which was funny.
I cleaned a large block quote that didn’t make sense as it was shown in
the preview, although I noticed that the Mercury chrome extension
didn’t even display it.
* fix: add date_published transform
* fix: date_published assertion
* disable: author assertion, generlize author selector
* rm: author assertion
* fix: image lead
* fix: guard agaist missing img url
* fix: generalize dek and title selectors
* feat: natgeo parser
Same as the news.nationalgeographic.com parser - for some reason the
author name doesn’t appear to be getting pulled into the local copy of
the file.
* fix: content assertion
* fix: generalize author byline
* disable: author assertion
* rm: author assertion
* fix: image lead, handles image-group
* fix: guard agaist missing img url
* fix: generalize dek and title selectors
* feat: howtogeek extractor
This one is a bit tricky - the author and date info appear in a comment
section at the bottom. Was able to parse the author but not the date
info. Halp
* howtogeek update
Thanks to @fdsimms I was able to parse the date, but not sure what to
test it against, so I left it blank.
* fix: date_published assertion, it was comparing against empty string
* fix: timezone
* amend: generalize author selector
* feat: today parser
This looks fine — there are a couple of lines of “Related” but they are
within the body (and don’t have their own classes) so I couldn’t clean
them out.
* fix: fix content assertion
* feat: westernjournalism parser
Adjacent sibling selector FTW!
Image not displaying in preview.
* feat: fix assertion, body does not include _Advertisement_ subtext
* feat: npr parser
Lead image appears in preview, but the test fails for some reason.
AssertionError: null ==
'https://media.npr.org/assets/img/2016/12/15/gettyimages-540681598_wide-
8b160732b96c083dc115134c3c019f3ac73586ba.jpg?s=1400'
Looks okay otherwise.
* feat: transformed figures/figcaptions, improved date_published and
addressed NPR's bad image metadata
* feat: fortune parser
For some reason, the dek doesn’t appear in the local version of the
article I selected. I tried parsing the meta tag containing
og:description but it’s not working, and the description is slightly
longer than the dek in the original article.
I’m not sure why, but for the lead image, the meta tag for og:image is
not parsing the image url.
:(
* feat: fortune redesigned, so re-did extractor
* fix: added timezone
* feat: qz parser
I couldn’t figure out how to parse the date, but otherwise should be
fine. I added a clean for the div.article-aside element based on what I
saw in how the chrome extension worked.
* feat: updated content to grab top image
test: date is null :/
* feat: dmagazine parser
I’m sorry to have failed you. :-( These are the issues I encountered:
1) author - does not have a unique selector to distinguish it from the
date, couldn’t parse it
2) date - no meta data in the head
3) no meta og:image in the head (my go to), so I couldn’t get the image
test to pass, but it appears to be parsing. The caption below it is the
same size as the body copy in the preview. I couldn’t figure out how to
“transform” it to caption size.
* feat: update date, image, and author selectors and corresponding tests
* feat: generalized content selector
* feat: reuters parser
Date parses correctly but fails test because of format discrepancy.
Author tags are nested within the content, which is why the author
names are appearing twice. I wasn’t sure how to address this.
Additionally, the location appears twice, so I cleaned the location
tags from the content.
* test: fix date format
* transform .article-subtitle to h4; cleaning author but leaving location
* feat: mashable parser
As usual the date is giving me issues because of formatting
discrepancies:
AssertionError: '2016-12-13T22:33:06.000Z' == '2016-12-14T03:33:06.000Z'
Not sure how we wanna deal with Twitter card embeds that don’t show up?
Also, image credits did not show up in preview.
* test: fixed date format
* transforming .image-credit to figcaption
* feat: chicago tribune parser
Date is parsing but failing the test because:
AssertionError: '2016-12-13T21:45:00.000Z' == '2016-12-13T13:45:00-0800'
I tried to insert a line of code for Time Zone but I’m a n00b so I
don’t think I did it right.
No image showing up in the preview.
* fix: remove timezone from date_published extractor
* test: update unit tests to assert the correct value for date_published
Bloomberg has several templates. I'm supporting three different templates here, but I'm not sure that this is complete by any means.
It's also worth noting that SVGs don't make it through the parser terribly well for many reasons. One, for example, is that a lot of SVGs require custom CSS in order for them to make sense. I'm not sure this is something we can expect to address in the parser.
the links at the bottom of the stories feel a little spammy because of how we treat links vs. the way they are displayed on the Times, would like to clean them
* feat: extractor for the verge's standard article template
* feat: basic support for the verge feature template
* feat: allow multiple links to be previewed
* feat: content selector arrays
Content selector arrays allow custom parsers to select multiple elements
to match and include in the result.
* feat: updated verge parser to use multimatch selectors
* lint fix
* cleanup test builds
Big undertaking to support Mercury in the browser. Builds are working and all tests are passing both for web and node builds. Most code is closely shared.
This commit also swaps in yarn for npm and tweaks circle ci a bit.
* appveyor.yml first go
* changing node
* ps
* narrow it down
* trying this
* fix airbnb module
* trying with yarn
* logging
* hybrid?
* trying yarn w/circle
* bump workers?
* build off?
* updating script
* tweaking script for appveyor
* bumping maxworkers
* cleaning up
* build step?
* yarn it
* added appveyor badge
Jordan Hotmann
kik0220
Olli Sulopuisto
Ben Ubois
Adam Pash
George Haddad
Wajeeh Zantout
Toufic Mouallem
Jad Termsani
Ralph Jbeily
Jeremy Mack
Janet
Kevin Ngao
Silas Burton
dviramontes
Matt