You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
searxng/searx/engines
Markus Heiser 14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
11 months ago
..
9gag.py
1337x.py
__init__.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
ahmia.py
annas_archive.py [fix] engine annas archive - fetch traits (modified xpath selectors) 12 months ago
apkmirror.py
apple_app_store.py
apple_maps.py
archlinux.py [fix] spelling 1 year ago
artic.py
arxiv.py
bandcamp.py
base.py
bilibili.py [fix] bilibili new api used 12 months ago
bing.py [fix] Revision of the Bing engines 12 months ago
bing_images.py [fix] Revision of the Bing engines 12 months ago
bing_news.py [fix] Revision of the Bing engines 12 months ago
bing_videos.py [fix] Revision of the Bing engines 12 months ago
brave.py [fix] engine brave - fetch traits (modified settings menu) 12 months ago
bt4g.py [fix] spelling 1 year ago
btdigg.py
chefkoch.py [feat] engine: implementation of chefkoch.de 1 year ago
command.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
core.py
crossref.py [fix] engine - Crossref 1 year ago
currency_convert.py [fix] engine - currency fix and simplify 12 months ago
dailymotion.py [fix] engine & network issues / documentation and type annotations 1 year ago
deepl.py
deezer.py
demo_offline.py
demo_online.py
deviantart.py [fix] engine deviantart: review of the result-scrapper 1 year ago
dictzone.py
digbt.py
docker_hub.py
doku.py
duckduckgo.py [fix] ddg-lite & ddg-extra: don't send empty vqd value 11 months ago
duckduckgo_definitions.py [mod] engine duckduckgo definitions: support for answer source 1 year ago
duckduckgo_extra.py [fix] ddg-lite & ddg-extra: don't send empty vqd value 11 months ago
duckduckgo_weather.py [fix] engine duckduckgo weather api changes 1 year ago
duden.py
dummy-offline.py
dummy.py
ebay.py
elasticsearch.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
emojipedia.py [fix] emojipedia: fix engine 12 months ago
fdroid.py
flickr.py
flickr_noapi.py
framalibre.py
freesound.py
frinkiac.py
genius.py
gentoo.py
github.py
google.py [fix] spelling 1 year ago
google_images.py [fix] engine - google images error when no results 1 year ago
google_news.py [fix] spelling 1 year ago
google_play.py [mod] engine google_play: raise error on unsupported category 1 year ago
google_scholar.py
google_videos.py [fix] engine google_video: google has changed the layout of the rsponse 1 year ago
hackernews.py [feat] implement hackernews engine - news.ycombinator.com 12 months ago
imdb.py [mod] add movies category for tmdb, imdb and moviepilot 12 months ago
imgur.py [feat] engine: implementation of imgur 1 year ago
ina.py
internet_archive_scholar.py [fix] internet archive scholar: crash when there's no title 1 year ago
invidious.py [fix] engine - invidious thumbnails 1 year ago
jisho.py
json_engine.py
kickass.py [fix] kickass: crash when no results 12 months ago
lemmy.py [mod] utils.py: add markdown_to_text helper function 1 year ago
lingva.py
loc.py [mod] library of congress: fix engine 1 year ago
mastodon.py [feat] engine: implementation of mastodon 12 months ago
material_icons.py [feat] engine: implementation of google icons/material design icons 1 year ago
mediathekviewweb.py
mediawiki.py [mod] engine: wikimedia - improve results, add addition settings & doc 1 year ago
meilisearch.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
metacpan.py
mixcloud.py
mongodb.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
moviepilot.py [fix] engine - moviepilot instead of thumbnail use img_src 12 months ago
mrs.py [fix] matrixrooms add proper MRS integration 12 months ago
mwmbl.py [doc] add documentation of Mwmbl engine & autocompleter 1 year ago
mysql_server.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
nyaa.py
odysee.py [fix] spelling 1 year ago
opensemantic.py
openstreetmap.py [fix] spelling 1 year ago
openverse.py
pdbe.py
peertube.py [fix] engine & network issues / documentation and type annotations 1 year ago
photon.py
pinterest.py [feat] engine: implementation of pinterest 12 months ago
piped.py [mod] piped: always show video length if available 12 months ago
piratebay.py
postgresql.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
pubmed.py [fix] pubmed content being None 1 year ago
qwant.py [fix] engine qwant (web-lite) - ignore advertising adds 1 year ago
radio_browser.py [mod] engine - simplify region & lang handling, make filters configurable 12 months ago
recoll.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
reddit.py
redis_server.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
rumble.py [fix] rumble redirect 1 year ago
scanr_structures.py
searchcode_code.py
searx_engine.py
seekr.py [mod] improve seekr engines and add documentation 1 year ago
semantic_scholar.py
sepiasearch.py
seznam.py [fix] engine seznam: fix issues reported by black & pylint 1 year ago
sjp.py
solidtorrents.py [fix] solidtorrents redirects 1 year ago
solr.py [doc] rearranges Settings & Engines docs for better readability 1 year ago
soundcloud.py
spotify.py
springer.py
sqlite.py [fix] spelling 1 year ago
stackexchange.py
startpage.py [fix] engine & network issues / documentation and type annotations 1 year ago
svgrepo.py [feat] new engine: svgrepo 1 year ago
tagesschau.py [mod] tagesschau: add option to only use tagesschau urls 12 months ago
tineye.py
tokyotoshokan.py
torznab.py [fix] spelling 1 year ago
translated.py
unsplash.py
vimeo.py
wallhaven.py [feat] engine: implementation of wallhaven 1 year ago
wikicommons.py [fix] engine: wikicommons - don't quoute ':|' in URL parameters 1 year ago
wikidata.py wikipedia wikidata infobox + disable wikisource (#2806) 1 year ago
wikipedia.py wikipedia wikidata infobox + disable wikisource (#2806) 1 year ago
wolframalpha_api.py
wolframalpha_noapi.py
wordnik.py
wttr.py [mod] clarify the difference of the default category and subgrouping 1 year ago
www1x.py
xpath.py [fix] spelling 1 year ago
yacy.py [mod] yacy: use official instance by default and fix crashes 12 months ago
yahoo.py [typo] solved a typo in yahoo error message. 12 months ago
yahoo_news.py
youtube_api.py
youtube_noapi.py
yummly.py [feat] engine: implementation of Yummly 1 year ago
zlibrary.py [feat] engine: re-enables z-library (zlibrary-global.se) 1 year ago