Commit Graph

328 Commits (master)

Author SHA1 Message Date
Alexandre Flament b4b81a5e1a [enh] settings.yml: add use_default_settings option (2nd version) 4 years ago
Alexandre Flament 3786920df9 [enh] Add multiple outgoing proxies
credits go to @bauruine see https://github.com/searx/searx/pull/1958
4 years ago
Alexandre Flament 9c25cd99c4 [mod] use github actions instead of travis
fix https://github.com/searx/searx/issues/2279
4 years ago
Alexandre Flament 230a5ecd04 [fix] fix a test_standalone_searx test case
If test_engines_init.py runs before test_standalone_searx.py, the engine list is not empty.
It makes test_get_search_query flaky.

This commit initializes the engline list in test_standalone_searx.py
4 years ago
Alexandre Flament 3038052c79 [mod] remove unused import
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url  # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
4 years ago
Alexandre Flament e1bd617669
Merge pull request #2296 from rachmadaniHaryono/feature/suppress-output
suppress test output
4 years ago
Alexandre Flament 46b454277f
Merge pull request #2309 from dalf/mod_search_repr
[mod] searx.search: EngineRef, SearchQuery: add __repr__ and __eq__ methods
4 years ago
Alexandre Flament 8fc74d0d7b [mod] searx.search: EngineRef, SearchQuery: add __repr__ and __eq__ methods 4 years ago
Alexandre Flament b3a3ccf2db [fix] fix of / and /search
* URL / : the index page displayed the selected or the default category.
* URL / : when the q parameter is set using the URL, the redirect includes the URL query.
* URL /search : an empty query doesn't raise an exception.
4 years ago
rachmadaniHaryono 4d4b6750bc chg: test: suppress output 4 years ago
rachmadani haryono c03e4c86bc
Feature/standalone searx update (#1591)
* chg: dev: update standalone_searx

parent d8a5df721b33dd8a7cc9e21dba4060f21d629f69
author rachmadaniHaryono <foreturiga@gmail.com> 1603896594 +0800
committer rachmadaniHaryono <foreturiga@gmail.com> 1603896619 +0800

chg: dev: debug engine_shortcuts
chg: dev: only initilize if engine is given
chg: dev: split main
chg: dev: standalone_searx
chg: dev: update standalone_searx
chg: doc: remove unnecessary log
chg: test: differentiate travis
chg: test: disable shortcut
chg: test: use default engine settings
fix: dev: category choices
fix: dev: duplicate engine shortcut
fix: dev: travis python3
fix: test:  use empty string as shortcut
fix: test: apkm
fix: test: engine shortcut
fix: test: mypy
fix: test: parameter
fix: test: pep8
fix: test: py2 compatibilities
fix: test: searx settings
fix: test: travis engines
new: dev: deduplicate engine
new: dev: main receive engines parameter
new: dev: parse_argument accept engines parameter
new: dev: split search query from get_result func
new: test: basic result case
Suggestions: use RawTextQuery to make the suggestions URLs. Update all themes accordingly.

* new: doc: searx import and init

* chg: dev: parse_argument

- doc
- run on __main__
- simple parse_args

* chg: doc: module

* chg: dev: import section

- remove unused python path modification
- new required package

* chg: dev: script run

- parse_argument func return directly parsed results
- main func return dict instead json text
- dump directly on sys.stdout.write

* chg: dev: get_search_query and get_search_query func

* chg: dev: main func

- move inner function outside
- return dict instead of json text

* new: dev: add utils to doc sys path

* new: doc: standalone_searx

* fix: doc: run script

* chg: dev: mypy type hint

* chg: dev: SearchQuery don't have attr engines

* chg: dev: reset engines __init__

* chg: test: unit test update

* chg: dev: pylint and flake8

* new: test: standalone_searx

* chg: dev: main func and doc

* chg: dev: import and type hint

* new: dev: main func

- remove get_result func
- single func which just translate dict

* chg: test: put mypy on dev requirement

* chg: doc: update

* new: doc: add standalone_searx module member

* chg: doc: shell command line

* chg: dev: remove mypy

* chg: doc: module docstring
4 years ago
Marc Abonce Seguin 8d71420b45 [mod] separate index and search routes
This makes it easier to separately handle search and index requests
from a web server or from a reverse proxy.

If a request to index contains a query, a permanent redirect HTTP response
is returned. This should give some level of backwards compatibility
for users that have set a searx instance in their browser's search bar.
4 years ago
Adam Tauber db703a0283
Merge pull request #565 from MarcAbonce/onions
New category: Onions
4 years ago
Adam Tauber 2aef38c3b9 [fix] resolve query_parts regression 4 years ago
a01200356 c3daa08537 [enh] Add onions category with Ahmia, Not Evil and Torch
Xpath engine and results template changed to account for the fact that
archive.org doesn't cache .onions, though some onion engines migth have
their own cache.

Disabled by default. Can be enabled by setting the SOCKS proxies to
wherever Tor is listening and setting using_tor_proxy as True.

Requires Tor and updating packages.

To avoid manually adding the timeout on each engine, you can set
extra_proxy_timeout to account for Tor's (or whatever proxy used) extra
time.
4 years ago
Noémi Ványi 33e139cae6 Let admins lock user preferences 4 years ago
Adam Tauber 6beb84efb8 [fix] adjust query unit tests 4 years ago
Venca24 35577051e2 [fix] hash plugin test 4 years ago
Venca24 69e5a58058 [fix] code style 4 years ago
Venca24 c9593c8ffd [enh] add plugin converting strings into hash digests 4 years ago
Noémi Ványi 116f7a6daa Force admins to set secret_key if debug mode is disabled
This commit also enables debug mode for unit tests.
4 years ago
Alexandre Flament 2006eb4680 [mod] move extract_text, extract_url to searx.utils 4 years ago
Alexandre Flament 485a502b88 [mod] add typing and __slots__ 4 years ago
Alexandre Flament 691d12726b [mod] check the engine tokens in searx/webadapter.py instead of searx/search.py 4 years ago
Alexandre Flament 2929495112 [mod] add searx.search.EngineRef
was previously a Dict with two or three keys: name, category, from_bang
make clear that this is a engine reference (see tests/unit/test_search.py for example)
all variables using this class are renamed accordingly.
4 years ago
Alexandre Flament 2dbc0de0cd [mod] add searx/webadapter.py
* move searx.search.get_search_query_from_webapp to searx.webadapter
* move searx.query.SearchQuery to searx.search
4 years ago
Alexandre Flament ad0758e52a [mod] add searx/webutils.py
contains utility functions and classes used only by webapp.py
4 years ago
Alexandre Flament 6deb85072a [fix] searx.utils.HTMLTextExtractor: invalid HTML don't raise an Exception
Close #2188
4 years ago
Alexandre Flament df12ed6e55 [mod] searx.RawTextQuery: the constructor call parse_query 4 years ago
Dalf c225db45c8 Drop Python 2 (4/n): SearchQuery.query is a str instead of bytes 4 years ago
Dalf 7888377743 Drop Python 2 (3/n): objects 4 years ago
Dalf 1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 4 years ago
Noémi Ványi f0ca1c3483
[enh] Add command line engines: git grep, find, etc. (#2128)
A new "base" engine called command is introduced. It is the foundation for all command line engines for now.
You can use this engine to create your own command line engine.

Add some engines (commented out to make sure no one enables anything accidentally):
* git grep: This engine lets you grep in the searx repo.
* locate: If locate is installed and initialized, you can search on the FS.
* find: You can find files with a specific name from where you started searx.
* pattern search in files: This engine utilizes the command fgrep.
* regex search in files: This engine runs `grep` to find a file based on its contents.
4 years ago
Alexandre Flament b329058c1a Revert "[enh] test: load each engine to check for syntax errors"
This reverts commit 4fb3ed2c63.
4 years ago
Dalf 4fb3ed2c63 [enh] test: load each engine to check for syntax errors 4 years ago
Lukas van den Berk 4829a76aae
Created new plugin type custom_results. Added new plugin bang_redirect (#2027)
* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Refactored getting search query. Also changed bang operator to ! and is now working.

* Removed prints

* Removed temporary bangs_redirect.js file. Updated plugin documentation

* Added unit test for the bangs plugin

* Fixed a unit test and added 2 more for bangs plugin

* Changed back to default settings.yml

* Added myself to AUTHORS.rst

* Refacored working of custom plugin.

* Refactored _get_bangs_data from list to dict to improve search speed.

* Decoupled bangs plugin from webserver with redirect_url

* Refactored bangs unit tests

* Fixed unit test bangs. Removed dubbel parsing in bangs.py

* Removed a dumb print statement

* Refactored bangs plugin to core engine.

* Removed bangs plugin.

* Refactored external bangs unit tests from plugin to core.

* Removed custom_results/bangs documentation from plugins.rst

* Added newline in settings.yml so the PR stays clean.

* Changed searx/plugins/__init__.py back to the old file

* Removed newline search.py

* Refactored get_external_bang_operator from utils to external_bang.py

* Removed unnecessary import form test_plugins.py

* Removed _parseExternalBang and _isExternalBang from query.py

* Removed get_external_bang_operator since it was not necessary

* Simplified external_bang.py

* Simplified external_bang.py

* Moved external_bangs unit tests to test_webapp.py. Fixed return in search with external_bang

* Refactored query parsing to unicode to support python2

* Refactored query parsing to unicode to support python2

* Refactored bangs plugin to core engine.

* Refactored search parameter to search_query in external_bang.py
4 years ago
Markus Heiser eae3481688 [fix] commit 2c6531b2 breaks the unit test, this is a hotfix
commit 2c6531b2 does not only break the unit test, it is a significant change of
the data model and the searx search-syntax model (UI) without any discussion nor
documentation.

At the end, adding routes to instant answers is a nice feature but commit
2c6531b2 leaf some questions open.

In that sense, this patch is only a hotfix not a assessment.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
4 years ago
Adam Tauber 8e727ac77f [fix] update csv unit test 4 years ago
Noémi Ványi 99435381a8 [enh] introduce private engines
This PR adds a new setting to engines named `tokens`.
It expects a list of tokens which lets searx validate
if the request should be accepted or not.
4 years ago
Adam Tauber 52ccaa7acc [mod] remove useless engine unit tests
These tests are not able to detect engine errors if the upstream
site changes.
4 years ago
Marc Abonce Seguin ccaf6ca02c [fix] update xpaths for new google results page 5 years ago
Marc Abonce Seguin 9299355570 add seedpeer again 5 years ago
Adam Tauber 3c425f09c1 [fix] remove useless engine tests 5 years ago
Adam Tauber e74bdf8429 [fix] engine test 5 years ago
Léo Bourrel 88261e111c Fix bing engine results count (#1387)
This PR fixes the result count from bing which was throwing an (hidden) error and add a validation to avoid reading more results than avalaible.

For example :
If there is 100 results from some search and we try to get results from 120 to 130, Bing will send back the results from 0 to 10 and no error. If we compare results count with the first parameter of the request we can avoid this "invalid" results.
5 years ago
Dalf fcc9587ee9 [fix] fdroid engine 5 years ago
Dalf 9ff5001816 [fix] arxiv engine 5 years ago
Alexandre Flament 333e54943d
[fix] fix monkey patch in test_webapp.py (#1667)
at the end of test_webapp.py, the monkey patch of searx.search.Search was not revert which lead to side effects on other tests
close #1663
5 years ago
Alexandre Flament 72029d27de
[enh] Add timeout limit per request (#1640)
The new url parameter "timeout_limit" set timeout limit defined in second.
Example "timeout_limit=1.5" means the timeout limit is 1.5 seconds.

In addition, the query can start with <[number] to set the timeout limit.

For number between 0 and 99, the unit is the second :
Example: "<30 searx" means the timeout limit is 3 seconds

For number above 100, the unit is the millisecond:
Example: "<850 searx" means the timeout is 850 milliseconds.

In addition, there is a new optional setting: outgoing.max_request_timeout.
If not set, the user timeout can't go above searx configuration (as before: the max timeout of selected engine for a query).

If the value is set, the user can set a timeout between 0 and max_request_timeout using
<[number] or timeout_limit query parameter.

Related to #1077
Updated version of PR #1413 from @isj-privacore
5 years ago
Alexandre Flament 2179079a91
[fix] fix flickr_noapi decoding (#1655)
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
5 years ago
Dalf 6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection.
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
5 years ago
Frank de Lange cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571)
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
5 years ago
Frank de Lange 204a2cbbf0 [fix] bing_videos (#1579)
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
5 years ago
Frank de Lange 11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568)
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
5 years ago
volth eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 5 years ago
rachmadani haryono 3b1122c5fa [fix] fix duden engine (#1594) 5 years ago
Alexandre Flament 554a21e1d0
[enh] Add Server-Timing header (#1637)
Server Timing specification: https://www.w3.org/TR/server-timing/

In the browser Dev Tools, focus on the main request, there are the responses per engine in the Timing tab.
5 years ago
rachmadani haryono ec88fb8a0f [fix] secret_key can be bytes instead of a string (#1602)
Fix #1600
In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
5 years ago
rachmadani haryono 8f44014627 [fix] preference query parameter decoding (#1599)
Fix issue #1598
5 years ago
rachmadani haryono ac357b12e3
Merge branch 'master' into feature/fix-config 5 years ago
Dalf ffe0972f91 Remove some engines : subtitleseeker, seedpeer, swisscows
http://www.subtitleseeker.com and http://www.seedpeer.eu don't exist anymore.
https://swisscows.ch/ has change : the engine needs to be updated
5 years ago
rachmadaniHaryono 9afc1b1e83 new: dev: test for config api 5 years ago
Marc Abonce Seguin 3e1c2153f7 [fix] duckduckgo images requests 5 years ago
Marc Abonce Seguin f2d49a6971 [fix] get youtube results from js object
Results are not appearing in the html document anymore,
instead they are found inside an object embedded in a script.
5 years ago
d-tux f1814079f0
Merge branch 'master' into engines/unsplash 5 years ago
Marc Abonce Seguin 626a8e9ac9 [fix] unicode error with WolframAlpha API engine 5 years ago
d-tux 329172f66e
Merge branch 'master' into engines/unsplash 5 years ago
Noémi Ványi 97351a2c72 fix after rebase 5 years ago
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages"
This reverts commit 4d1770398a.
5 years ago
Marc Abonce Seguin 0169b63e84 [fix] fetch google's supported languages 5 years ago
Marc Abonce Seguin 5568f24d6c [fix] check language aliases when setting search language 5 years ago
Noémi Ványi 1ea56576dc
Merge branch 'master' into devel_google_videos 5 years ago
Noémi Ványi 5b81f7c2cc
Merge branch 'master' into bugfix_startpage 5 years ago
Venca24 0e493db2fb [fix] google videos test 5 years ago
Léo Bourrel 73ec2ba74e Update bing image engine with new image json format 6 years ago
Michael Pfitzner 83a06d276d change rest of test_startpage.py 6 years ago
Michael Pfitzner 5a20d9ecd1 change the html in test_startpage.py to look more like the actual startpage site 6 years ago
Venca24 cee15f0375 [fix] google videos test 6 years ago
Denis Wernert b9ada93b3a Removes what looks like tracking parameters 6 years ago
Denis Wernert ee07a5e750 Adds a unit test for the unsplash engine 6 years ago
Adam Tauber baacfac32b [fix] remove publicly unavailable 500px engine - #1338 6 years ago
dadosch b575f898c0 duden.de engine 6 years ago
Léo Bourrel 0a37f90990 Fix wikidata tests with updated path to media 6 years ago
Adam Tauber a3b83b73cb [fix] remove obsolete test 6 years ago
Noémi Ványi 7fd7bd75ff follow up tests 6 years ago
rinpatch 059c2ccb95
Fix acgsou encoding error on python2 and add unicode characters to unit testt 6 years ago
rinpatch b7eb05dac6
Fix unit test url 6 years ago
rinpatch a79c676f27
Remove Japanese characters 6 years ago
rinpatch 1ac0c90037
Fix unit test 6 years ago
rinpatch dcc9fdb47f
Added unit test 6 years ago
rinpatch 06e070aee2 Fix string length 6 years ago
rinpatch c40d8e1d15 Unit test 6 years ago
Marc Abonce Seguin f7f9c50393 [fix] force English results in Google when using en-US 6 years ago
Marc Abonce Seguin 9687786226 update unit tests for google news and wikidata 6 years ago
Marc Abonce Seguin 772c048d01 refactor engine's search language handling
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
6 years ago
Noémi Ványi 3ef8533f4d fix unit tests 6 years ago
marc 4d1770398a remove 'all' option from search languages 7 years ago
Adam Tauber 9e79ad4bde [fix] pdbe test 7 years ago
Adam Tauber 060c797739 [fix] change domain name in google engine test 7 years ago
pyrrh0n1c a1d1a50149 Fixed unittests. 7 years ago
pyrrh0n1c 4340c0b16c Fixed typos. 7 years ago
Adam Tauber 00a7041daa
Merge pull request #1068 from Apply55gx/genius
Continuation of PR #1004 (seems abandoned)
7 years ago
Noémi Ványi 5954a8e16a minor fix of BASE engine 7 years ago
jibe-b df0d915806 [add] pubmed engine 7 years ago
Noémi Ványi 9c2b7a82f0 minor fixes of arxiv
Closes #1050
7 years ago
jibe-b 3e3672e079 [add] arxiv engine 7 years ago
Apply55gx d800e3fcfa Merge pull request #1 from asciimoo/master
-
7 years ago
marc 54f0ab1f3b test fetch langauges function on bing images 7 years ago
marc 44085e31d0 update engines_languages.json and languages.py
Also, fix fetch_languages.py so it can run on python3.
7 years ago
marc a524dbb823 [fix] language support for bing images and videos 7 years ago
jibe-b d081eee3f8 [add] unit_test for base engine 7 years ago
misnyo c3232b0e1a Merge branch 'master' into nyaa 7 years ago
misnyo 01330f71cd [fix] nyaa.si fixed 7 years ago
misnyo 3182ba7069 [fix] google news dom xpath fix 7 years ago
Adam Tauber e74aaa781e Merge pull request #1016 from misnyo/blekko
[mod]blekko images removed
7 years ago
misnyo 0607b167f8 [fix] faroo json api and image layout fixed 7 years ago
misnyo f139f9cfdb [mod]blekko images removed 7 years ago
misnyo bf3f9a91fa [fix]torrentz unit_tests timezone fix 7 years ago
misnyo 2098e1b7ff [fix]torrentz unit_tests timezone fix 7 years ago
misnyo d14e2781b2 [fix]torrentz search engine fixed for new version 7 years ago
woorst 636b760196 remove unicode characters 7 years ago
woorst 62b2f79ce7 fix line lengths for pep8 standards 7 years ago
woorst 2434c29dc5 New engine: Genius (lyrics) 7 years ago
marc 3ca9cad927 add bing videos engine 7 years ago
marc 856dfc3018 add google videos 7 years ago
Noémi Ványi 2242000bd4 change unresponsive_engines to a set to eliminate duplication of errors 7 years ago
Noémi Ványi 243d3e4298 show engine errors in infobox && add new error alert 7 years ago
marc 405e5c8f24 [fix] duckduckgo images doesn't fail with countryless language 7 years ago
marc 3b950929b1 [fix] fix duckduckgo's offset
First page now starts with 0 offset,
rather than starting on the 30th result.

DuckDuckGo returns 30 results on each page.
7 years ago
marc c65a409f0d add duckduckgo images engine 7 years ago
Adam Tauber 8db527c1d2 [fix] use raw response with etree.parsefromstring - Unicode strings with encoding declaration are not supported 7 years ago
marc 9ee8e552da [fix] bing images 7 years ago
Alexandre Flament f5128c7cb9 [mod] add/modify image fetching for bing_news, qwant and twitter engines 7 years ago
Adam Tauber 52e615dede [enh] py3 compatibility 7 years ago
François Revol 45d15bd6f0 Add framalibre engine
framalibre.org is a catalogue of Free Software, edited by Framasoft.

For now we pass the thumbnail as img_src as it doesn't seem to be used
for IT...
7 years ago
marc 805fb02ed1 add language support for qwant
closes issue #863
7 years ago
Alexandre Flament e67dfaaac7 Merge branch 'master' into flask_perimeter 7 years ago
Alexandre Flament 7fdfeca3a4 [mod] add a __common__ template that can't be selected but that provides a common place for shared templates.
What has been moved into this template :
* opensearch*.xml is always the same whatever the themes.
* the text inside */about.html
7 years ago
David A Roberts 1d30141c20 [enh] show spelling corrections 7 years ago
Alexandre Flament 84a2c97a65 [mod] searx uses flask framework only in webapp.py. Make migration to another framework easier. 8 years ago
Adam Tauber 6bf9c398a7 [fix] use english as default language in bing
If no language is specified, bing returns results with multiple languages
for one query which isn't really useful. Setting english as default
insted if nothing.
8 years ago
marc 1175b3906f change language list to only include languages with a minimum of engines
that support them.
users can still query lesser supported through the :lang_code bang.
8 years ago
marc af35eee10b tests for _fetch_supported_languages in engines
and refactor method to make it testable without making requests
8 years ago
marc e0c270bd72 tests for language support in engines 8 years ago
marc f62ce21f50 [mod] fetch supported languages for several engines
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
8 years ago
marc 92c6e88ad3 small fixes 8 years ago
marc a11948c71b Add language support for more engines. 8 years ago
Noémi Ványi b034356825 add year filter to engines with time range support && tests
Following engines does not support "Last year":
 * Bing News
 * DeviantArt
 * DuckDuckGo
 * Yahoo
 * YouTube (noapi)
8 years ago
Noémi Ványi c59c76e6ee add year to time range to engines which support "Last year"
Engines:
 * Bing images
 * Flickr (noapi)
 * Google
 * Google Images
 * Google News
8 years ago
Adam Tauber 9b2bd6847f [fix] 500px unit tests 8 years ago