You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
searxng/searx/engines
Markus Heiser 9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
..
1337x.py
__init__.py
ahmia.py
apkmirror.py
archlinux.py
artic.py
arxiv.py
bandcamp.py
base.py
bing.py
bing_images.py
bing_news.py
bing_videos.py
btdigg.py
ccengine.py
command.py
core.py
currency_convert.py [enh] engines: add about variable 4 years ago
dailymotion.py
deezer.py
demo_offline.py
demo_online.py
deviantart.py
dictzone.py
digbt.py
digg.py
docker_hub.py
doku.py
duckduckgo.py
duckduckgo_definitions.py
duckduckgo_images.py
duden.py
dummy-offline.py
dummy.py
ebay.py
elasticsearch.py
etools.py
fdroid.py
flickr.py
flickr_noapi.py
framalibre.py
freesound.py
frinkiac.py
genius.py
gentoo.py
gigablast.py
github.py
google.py
google_images.py
google_news.py
google_scholar.py
google_videos.py [fix] log messages from: google- images, news, scholar, videos 3 years ago
ina.py
invidious.py
json_engine.py
kickass.py
loc.py
mediathekviewweb.py
mediawiki.py
meilisearch.py
microsoft_academic.py
mixcloud.py
mysql_server.py
not_evil.py
nyaa.py
opensemantic.py
openstreetmap.py
pdbe.py
peertube.py
photon.py
piratebay.py
postgresql.py
pubmed.py
qwant.py
recoll.py
reddit.py
redis_server.py
rumble.py
scanr_structures.py
searchcode_code.py
searx_engine.py
semantic_scholar.py
sepiasearch.py
seznam.py
sjp.py
solidtorrents.py
solr.py
soundcloud.py
spotify.py
springer.py
sqlite.py
stackoverflow.py
startpage.py
tokyotoshokan.py
torrentz.py
translated.py
unsplash.py
vimeo.py
wikidata.py
wikipedia.py
wolframalpha_api.py
wolframalpha_noapi.py
wordnik.py
www1x.py
xpath.py
yacy.py
yahoo.py
yahoo_news.py
yggtorrent.py
youtube_api.py
youtube_noapi.py