You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
searxng/searx/engines
Markus Heiser 9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
..
1337x.py [mod] the bittorent search engines are available only in the files category 3 years ago
__init__.py [fix] sys.exit(1) when there is duplicate engine name 3 years ago
ahmia.py
apkmirror.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
archlinux.py
artic.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
arxiv.py
bandcamp.py
base.py
bing.py
bing_images.py
bing_news.py
bing_videos.py
btdigg.py [mod] the bittorent search engines are available only in the files category 3 years ago
ccengine.py
command.py
core.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
currency_convert.py
dailymotion.py
deezer.py
demo_offline.py [docs] add engine "Demo Offline Engine" 3 years ago
demo_online.py [docs] add engine "Demo Online Engine" 3 years ago
deviantart.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
dictzone.py [fix] typo: online_dictionnary --> online_dictionary 3 years ago
digbt.py
digg.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
docker_hub.py [enh] add engine - Docker Hub 3 years ago
doku.py
duckduckgo.py
duckduckgo_definitions.py [pylint] searx/engines/duckduckgo_definitions.py 3 years ago
duckduckgo_images.py
duden.py
dummy-offline.py
dummy.py
ebay.py
elasticsearch.py
etools.py
fdroid.py
flickr.py
flickr_noapi.py
framalibre.py
freesound.py
frinkiac.py
genius.py [fix] engine genius should not use the video template 3 years ago
gentoo.py
gigablast.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
github.py
google.py [mod] google - get_lang_info add documentataion & comments 3 years ago
google_images.py [fix] log messages from: google- images, news, scholar, videos 3 years ago
google_news.py [fix] google news - send CONSENT Cookie to not be redirected 3 years ago
google_scholar.py [fix] log messages from: google- images, news, scholar, videos 3 years ago
google_videos.py [fix] log messages from: google- images, news, scholar, videos 3 years ago
ina.py
invidious.py
json_engine.py
kickass.py [mod] the bittorent search engines are available only in the files category 3 years ago
loc.py
mediathekviewweb.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
mediawiki.py
meilisearch.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
microsoft_academic.py
mixcloud.py
mysql_server.py [enh] add MySQL engine 3 years ago
not_evil.py
nyaa.py [mod] the bittorent search engines are available only in the files category 3 years ago
opensemantic.py
openstreetmap.py [fix] openstreetmap - fix some minor whitespace & indentation issues 3 years ago
pdbe.py
peertube.py [fix] peertube fetch supported languages 3 years ago
photon.py
piratebay.py [mod] the bittorent search engines are available only in the files category 3 years ago
postgresql.py [enh] add PostgreSQL engine 3 years ago
pubmed.py
qwant.py fix Qwant's fetch_languages function 3 years ago
recoll.py
reddit.py
redis_server.py [fix] enigine redis - avoid error when the engine is loaded 3 years ago
rumble.py
scanr_structures.py
searchcode_code.py
searx_engine.py
semantic_scholar.py
sepiasearch.py
seznam.py
sjp.py
solidtorrents.py [fix] URL to solidtorrent result page 3 years ago
solr.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
soundcloud.py
spotify.py
springer.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
sqlite.py [enh] add offline engine for sqlite database 3 years ago
stackoverflow.py
startpage.py
tokyotoshokan.py [mod] the bittorent search engines are available only in the files category 3 years ago
torrentz.py [mod] the bittorent search engines are available only in the files category 3 years ago
translated.py [fix] typo: online_dictionnary --> online_dictionary 3 years ago
unsplash.py [fix] unsplash engine - 'searx:result: invalid title:' 3 years ago
vimeo.py
wikidata.py
wikipedia.py
wolframalpha_api.py
wolframalpha_noapi.py
wordnik.py
www1x.py
xpath.py [enh] XPath engine - add time safe-search support 3 years ago
yacy.py
yahoo.py
yahoo_news.py [pylint] tag PYLINT_FILES by comment `# lint: pylint` 3 years ago
yggtorrent.py [mod] the bittorent search engines are available only in the files category 3 years ago
youtube_api.py
youtube_noapi.py