Commit Graph

57 Commits (8c5cb5039e84f37437c308f7f82c9c6f3502b010)

Author SHA1 Message Date
Alexandre Flament d14994dc73 [httpx] replace searx.poolrequests by searx.network
settings.yml:

* outgoing.networks:
   * can contains network definition
   * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
     keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
   * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
   * local_addresses can be "192.168.0.1/24" (it supports IPv6)
   * support_ipv4 & support_ipv6: both True by default
     see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
   * either a full network description
   * either reference an existing network

* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
3 years ago
Adam Tauber 5f450fda74 [enh] add year filter to duckduckgo 3 years ago
Alexandre Flament c22d4c764c [fix] duckduckgo engine: "!ddg !g" do not redirect to google
* searx understand "!ddg !g time" as : send "!g time" to DDG
* !g a DDG bang for Google: DDG return a HTTP redirect to Google

This commit adds a the allows_redirect param not to follow HTTP redirect.

The DDG engine returns a empty result as before without HTTP redirect.
3 years ago
Alexandre Flament d2dac11392 [mod] duckduckgo engine: better support of the language preference
After the main request, send a second to https://duckduckgo.com/t/sl_h

See https://github.com/searx/searx/issues/2259
3 years ago
Alexandre Flament ca93a01844 [mod] dynamically set language_support variable
The language_support variable is set to True by default,
and set to False in only 5 engines.

Except the documentation and the /config URL, this variable is not used.

This commit remove the variable definition in the engines, and
set value according to supported_languages length: False when the length is 0,
True otherwise.

Close #2485
3 years ago
Alexandre Flament a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
4 years ago
Alexandre Flament b00d108673 [mod] pylint: numerous minor code fixes 4 years ago
Alexandre Flament 3038052c79 [mod] remove unused import
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url  # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
4 years ago
Alexandre Flament cfd21bc475 [fix] fix duckduckgo engine
- remove paging support: a "vqd" parameter is required between each request. This parameter is uniq for each request
- update the URL (no redirect), use the POST method
- language support: works if there is no more than request per minute, otherwise it is ignored !
4 years ago
Alexandre Flament 2006eb4680 [mod] move extract_text, extract_url to searx.utils 4 years ago
Dalf 1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 4 years ago
Gordon Quad 385e9b5c9e add correction support for duckduckgo 4 years ago
Dalf 85b3723345 [mod] speed optimization
compile XPath only once
avoid redundant call to urlparse
get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
5 years ago
Adam Tauber ed1c1bdb04 [fix] pep8 5 years ago
Adam Tauber 94ea9d6622 [fix] duckduckgo paging - closes #1677 5 years ago
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages"
This reverts commit 4d1770398a.
6 years ago
Marc Abonce Seguin 772c048d01 refactor engine's search language handling
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
6 years ago
Marc Abonce Seguin d1eae9359f fix fetch_langauges to be more accurate
Add languages supported by either all default general engines or 10 engines.
6 years ago
Marc Abonce Seguin 216716db70 [fix] change duckduckgo url to avoid error response 6 years ago
marc 4d1770398a remove 'all' option from search languages 7 years ago
marc 44085e31d0 update engines_languages.json and languages.py
Also, fix fetch_languages.py so it can run on python3.
7 years ago
Adam Tauber 077d8efed8 [fix] use poolrequests in ddg engines 7 years ago
marc 405e5c8f24 [fix] duckduckgo images doesn't fail with countryless language 7 years ago
marc 3b950929b1 [fix] fix duckduckgo's offset
First page now starts with 0 offset,
rather than starting on the 30th result.

DuckDuckGo returns 30 results on each page.
7 years ago
marc c65a409f0d add duckduckgo images engine 7 years ago
Adam Tauber 52e615dede [enh] py3 compatibility 7 years ago
Noémi Ványi 742e4dfdcc [fix] fix paging of duckduckgo 8 years ago
marc af35eee10b tests for _fetch_supported_languages in engines
and refactor method to make it testable without making requests
8 years ago
marc e0c270bd72 tests for language support in engines 8 years ago
marc f62ce21f50 [mod] fetch supported languages for several engines
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
8 years ago
marc a11948c71b Add language support for more engines. 8 years ago
marc 149802c569 [enh] add supported_languages on engines and auto-generate languages.py 8 years ago
Noémi Ványi b034356825 add year filter to engines with time range support && tests
Following engines does not support "Last year":
 * Bing News
 * DeviantArt
 * DuckDuckGo
 * Yahoo
 * YouTube (noapi)
8 years ago
Adam Tauber 350a84520d [fix] time range detection 8 years ago
Noemi Vanyi f13b9fa36a add time range search for duckduckgo 8 years ago
marc aac4ed6971 [fix] guess country code if not specified in duckduckgo engine 8 years ago
a01200356 8fc4b8156a [fix] multilingual duckduckgo
only works if both country and language are set
8 years ago
a01200356 751c9a346e [fix] duckduckgo's xpaths changed
test_duckduckgo modified to reflect changes in duckduckgo's html
8 years ago
Adam Tauber 4184cece4a [fix] duckduckgo unicode url - #419 9 years ago
Emmanuel Benazera 78a69e4c98 ddg encoding of URLs appears to be broken, revealed when trying to pickled the results to disk 9 years ago
Alexandre Flament 4689fe341c update versions.cfg to use the current up-to-date packages 9 years ago
Cqoicebordel efe6dead55 Duckduckgo unit test 10 years ago
dalf 7c13d630e4 [fix] pep8 : engines (errors E121, E127, E128 and E501 still exist) 10 years ago
dalf 64b75e6d82 [enh] duckduckgo engine: add language support 10 years ago
Thomas Pointhuber 8eb064dea1 add little note to duckduckgo engine 10 years ago
Thomas Pointhuber 80f98d6041 add little comment 10 years ago
Thomas Pointhuber e6e4de8ba0 rewrite duckduckgo engine and add comments 10 years ago
Adam Tauber 52a817f5c1 [fix] pep8 10 years ago
Adam Tauber a8a0de644d [fix] indexing 10 years ago
Adam Tauber 3854703d95 [mod] ddg engine mods 10 years ago