Commit Graph

116 Commits

Author SHA1 Message Date
Alexandre Flament
c4067a9d02 [mod] searx.search.Search: split the search method into different methods 2020-09-22 12:14:46 +02:00
Alexandre Flament
df12ed6e55 [mod] searx.RawTextQuery: the constructor call parse_query 2020-09-12 15:25:58 +02:00
Dalf
7888377743 Drop Python 2 (3/n): objects 2020-09-10 10:39:04 +02:00
Dalf
1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Adam Tauber
4a913247b2 [enh] add option to configure proxies per engine - closes #1827 2020-09-01 15:59:22 +02:00
asht
3b8bd0770b
Change isAlive() to is_alive() as isAlive() is unsupported in newer Python versions 2020-07-31 00:28:54 +03:00
Lukas van den Berk
4829a76aae
Created new plugin type custom_results. Added new plugin bang_redirect (#2027)
* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Refactored getting search query. Also changed bang operator to ! and is now working.

* Removed prints

* Removed temporary bangs_redirect.js file. Updated plugin documentation

* Added unit test for the bangs plugin

* Fixed a unit test and added 2 more for bangs plugin

* Changed back to default settings.yml

* Added myself to AUTHORS.rst

* Refacored working of custom plugin.

* Refactored _get_bangs_data from list to dict to improve search speed.

* Decoupled bangs plugin from webserver with redirect_url

* Refactored bangs unit tests

* Fixed unit test bangs. Removed dubbel parsing in bangs.py

* Removed a dumb print statement

* Refactored bangs plugin to core engine.

* Removed bangs plugin.

* Refactored external bangs unit tests from plugin to core.

* Removed custom_results/bangs documentation from plugins.rst

* Added newline in settings.yml so the PR stays clean.

* Changed searx/plugins/__init__.py back to the old file

* Removed newline search.py

* Refactored get_external_bang_operator from utils to external_bang.py

* Removed unnecessary import form test_plugins.py

* Removed _parseExternalBang and _isExternalBang from query.py

* Removed get_external_bang_operator since it was not necessary

* Simplified external_bang.py

* Simplified external_bang.py

* Moved external_bangs unit tests to test_webapp.py. Fixed return in search with external_bang

* Refactored query parsing to unicode to support python2

* Refactored query parsing to unicode to support python2

* Refactored bangs plugin to core engine.

* Refactored search parameter to search_query in external_bang.py
2020-07-03 13:25:04 +00:00
Noémi Ványi
bce3830b8d [fix] translate engine errors to Occitan when configured 2020-04-17 23:39:48 +02:00
Noémi Ványi
99435381a8 [enh] introduce private engines
This PR adds a new setting to engines named `tokens`.
It expects a list of tokens which lets searx validate
if the request should be accepted or not.
2020-02-08 11:47:39 +01:00
Noémi Ványi
5796dc60c9 fix pep 8 check 2019-10-16 15:52:48 +02:00
Noémi Ványi
a6f20caf32 add initial support for offline engines && command engine 2019-10-16 15:52:48 +02:00
Dalf
ba80a0ba38 [fix] fix paging for the oscar theme after PR #1640 2019-08-02 20:04:37 +02:00
Alexandre Flament
72029d27de
[enh] Add timeout limit per request (#1640)
The new url parameter "timeout_limit" set timeout limit defined in second.
Example "timeout_limit=1.5" means the timeout limit is 1.5 seconds.

In addition, the query can start with <[number] to set the timeout limit.

For number between 0 and 99, the unit is the second :
Example: "<30 searx" means the timeout limit is 3 seconds

For number above 100, the unit is the millisecond:
Example: "<850 searx" means the timeout is 850 milliseconds.

In addition, there is a new optional setting: outgoing.max_request_timeout.
If not set, the user timeout can't go above searx configuration (as before: the max timeout of selected engine for a query).

If the value is set, the user can set a timeout between 0 and max_request_timeout using
<[number] or timeout_limit query parameter.

Related to #1077
Updated version of PR #1413 from @isj-privacore
2019-08-02 13:50:51 +02:00
Dalf
7d8000d965 [fix] "!wp !wp test" send only one request to wikipedia.
Note that "!general !wp test" is not fixed because the category for "!wp" is "none".
2019-07-27 08:52:30 +02:00
Alexandre Flament
554a21e1d0
[enh] Add Server-Timing header (#1637)
Server Timing specification: https://www.w3.org/TR/server-timing/

In the browser Dev Tools, focus on the main request, there are the responses per engine in the Timing tab.
2019-07-17 10:38:45 +02:00
Dalf
c7220b40ea Suggestions: use RawTextQuery to make the suggestions URLs. Update all themes accordingly. 2019-07-17 09:05:29 +02:00
Noémi Ványi
36790168a1 add again accidentally removed import to search.py 2019-01-25 20:54:23 +01:00
Noémi Ványi
b63d645a52 Revert "remove 'all' option from search languages"
This reverts commit 4d1770398a.
2019-01-07 21:19:00 +01:00
Adam Tauber
c5c812899f [fix] pep8 2018-08-19 15:32:32 +02:00
Adam Tauber
f7bb45e891 [enh] admin configurable suspend time of errored engines - closes #1266 2018-08-19 15:30:12 +02:00
Noémi Ványi
8cc529e9a3 forward category to engine without highlighting on the ui 2018-03-22 11:02:24 +01:00
marc
4d1770398a remove 'all' option from search languages 2017-12-06 01:20:15 -06:00
Adam Tauber
1435ae7e96 [enh] display engine error messages 2017-12-05 20:38:32 +01:00
Alexandre Flament
c1cfe97851 [enh] timeout and total HTTP time are managed by searx.poolrequests 2017-11-22 18:33:35 +01:00
Noémi Ványi
80331cc8e6 add timeout errors in search_multiple_requests 2017-07-20 13:30:56 +02:00
Noémi Ványi
243d3e4298 show engine errors in infobox && add new error alert 2017-07-12 16:29:11 +02:00
Adam Tauber
52e615dede [enh] py3 compatibility 2017-05-15 12:02:30 +02:00
marc
fd65c12921 make search language handling less strict
languages.py can change, so users may query on a language that is not
on the list anymore, even if it is still recognized by a few engines.

also made no and nb the same because they seem to return the same,
though most engines will only support one or the other.
2017-03-18 23:44:21 +01:00
Alexandre Flament
e67dfaaac7 Merge branch 'master' into flask_perimeter 2017-01-27 13:47:59 +00:00
Adam Tauber
e7aa98902f [fix] correct search parameter check 2017-01-21 20:21:32 +01:00
Adam Tauber
2522254fb2 [fix] search time range value check 2017-01-21 20:13:55 +01:00
Alexandre Flament
15eef0ebdb [enh] validate input and raise an exception inside search.py. The exception message is output in json and rss format. 2017-01-20 18:52:47 +01:00
Alexandre Flament
84a2c97a65 [mod] searx uses flask framework only in webapp.py. Make migration to another framework easier. 2017-01-02 12:06:04 +01:00
Alexandre Flament
5a9551925d [fix] fix bugs introduced by commit 0a2fde19d0 from dalf/searchpy4 2016-12-30 18:10:14 +01:00
Alexandre Flament
28d51fd063 [mod] timeout in log are readable (the timeouts are compare to the start_time of the request). 2016-12-30 17:37:46 +01:00
Alexandre Flament
3a8ab5880a [mod] result_container.extend sets result['engine'] = engine_name for each result 2016-12-30 17:18:02 +01:00
Alexandre Flament
0a2fde19d0 [mod] add a search_one_request_safe function wrapper to call search_one_request. All exceptions are catched in this wrapper. 2016-12-30 11:51:49 +01:00
marc
7fba3d1179 Merge branch 'master' into languages 2016-12-16 22:14:36 -06:00
Adam Tauber
cf5c7d050a [fix] engine setting if no categories specified 2016-12-15 11:59:21 +01:00
marc
1b46ecd21a [fix] broken language selector 2016-12-13 21:36:40 -06:00
Adam Tauber
75aa46a6b3 [fix] remove unused variable 2016-12-10 00:15:53 +01:00
Adam Tauber
0205838f25 [enh] handle engine response crashes 2016-12-10 00:15:33 +01:00
Alexandre Flament
e48f07a367 Merge branch 'master' into searchpy2 2016-12-09 23:11:45 +01:00
Adam Tauber
971ed0abd1 [enh] add quick answer functionality with an example answerer 2016-11-19 20:53:51 +01:00
Alexandre Flament
219f047bf3 Merge branch 'master' into searchpy2 2016-11-06 09:44:07 +01:00
Adam Tauber
7d26b3f8f8 [fix] suspend end time check 2016-11-06 02:53:09 +01:00
Alexandre Flament
01e2648e93 Simplify search.py, basically updated PR #518
The timeouts in settings.yml is about the total time (not only the HTTP request but also the prepare the request and parsing the response)
It was more or less the case before since the threaded_requests function ignores the thread after the timeout even the HTTP request is ended.

New / changed stats :
* page_load_time : record the HTTP request time
* page_load_count: the number of HTTP request
* engine_time : the execution total time of an engine
* engine_time_count : the number of "engine_time" measure

The avg response times in the preferences are the engine response time (engine_load_time / engine_load_count)

To sum up :
* Search.search() filters the engines that can't process the request
* Search.search() call search_multiple_requests function
* search_multiple_requests creates one thread per engine, each thread runs the search_one_request function
* search_one_request calls the request function, make the HTTP request, calls the response function, extends the result_container
* search_multiple_requests waits for the the thread to finish (or timeout)
2016-11-05 13:45:20 +01:00
Alexandre Flament
58a6c045c8 [mod] replace references request.request_data (GET or POST parameters) by request.form (based on merge of POST and GET parameters) 2016-11-02 14:52:22 +01:00
dalf
fbb080f358 Change plugin API :
- pre_search(request, search)
- post_search(request, search)
- on_result(request, search, result)

with
- request is the Flask request
- search a searx.Search instance
- result a searx result as usual
2016-11-02 14:22:16 +01:00
dalf
67e11c42b9 Clean up the architecture
Purposes :
- isolate the plugins calls
- distinction between parsing the web request and running the search (Search class). To be able to test code easily, to run searx code outside a web server, to filter the search query parameters with plugins more easily, etc...

Details :
- request.request_data contains request.form or request.args (initialize inside pre_request() function)
- Query class is renamed RawTextQuery
- SearchQuery class defines all search parameters
- get_search_query_from_webapp create a SearchQuery instance (basically the previous Search.__init__ code)
- Search class and SearchWithPlugins class takes a SearchQuery instance as class constructor parameter
- SearchWithPlugins class inherites from Search class, and run plugins
- A dedicated function search_with_plugins executes plugins to have a well define locals() (which is used by the plugins code).
- All plugins code is executed inside the try...except block (webapp.py, index function)
- advanced_search HTTP parameter value stays in webapp.py (it is only part of UI)
- multiple calls to result_container.get_ordered_results() doesn't compute the order multiple time (note : this method was call only once before)
- paging value is stored in the result_container class (compute in the extend method)
- test about engine.suspend_end_time is done during search method call (instead of __init__)
- check that the format parameter value is one of these : html, rss, json, rss (before the html value was assumed but some text formatting wasn't not done)
2016-11-02 14:22:16 +01:00