Commit Graph

141 Commits (dbed8da284f0e91bef0fb42606ae10241505b364)

Author SHA1 Message Date
Alexandre Flament f79b0fce06 [enh] limiter plugin
can replace filtron:
* rate limite the number of request per IP and per (IP, User-Agent)
* block some bots

use Redis
data stored in Redis never contains the IP addresses, only HMAC using the secret_key

Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2 years ago
Martin Fischer 6d43cf7952 [typing] add optional attrs to Plugin 3 years ago
Martin Fischer bb06758a7b [refactor] add type hints & remove Setting._post_init
Previously the Setting classes used a horrible _post_init
hack that prevented proper type checking.
3 years ago
Markus Heiser 3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser fcdc2c2cd2 [format.python] disable py code formatting for some hunks of code
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 5731b6b700 [mod] searx.plugins.prepare_package_resources() - use generators
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser aa5a5147b2 [fix] searx.plugins.initialize() - don't miss module & module-name
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament 2b4fef7118 plugins: refactor initialization
add a new function "init" call when the app starts.
The function can:
* return False to disable the plugin.
* modify the Flask app.
3 years ago
Alexandre Flament 0f43b39eac [enh] add hostname_replace plugin
* backport of https://github.com/searx/searx/pull/2724
* allow to remove result if the replacement is the boolean value false
3 years ago
Alexandre Flament b941763e20 [mod] ahmia_filter: use on_result instead of post_search
see commit 6c9ae7911e9639bc46cd53af215734b4bdb61ba9
3 years ago
Alexandre Flament fc20c561bf [mod] oa_doi_rewrite plugin: get_doi_resolver: remove args parameter
doi_resolvers.get_value('preferences') already contains the value from
request.args.get('doi_resolver')
3 years ago
Alexandre Flament 3f3b5d6181 [mod] plugins: minor change
required attributes: display a different message
when the attribute has the wrong type
3 years ago
Alexandre Flament 881659ca9d [mod] oscar theme: /preferences : HTML detail order match visual tabs
First details about the general tab, then detail about UI tab, etc...
No functionnal change
3 years ago
Alexandre Flament f83b64270c [mod] oscar theme /preferences: reorganize the preferences
close #115
3 years ago
Alexandre Flament 4b07df62e5 [mod] move all default settings into searx.settings_defaults 3 years ago
Markus Heiser 247c46c6b0 [drop] plugin HTTPS rewrite - rule base is to old (update is to big)
The usefulness of the _HTTPS rewrite_ plugin is questionable:

- the 36 rule files have not been updated since 2015 [1]
- actual there are 23760 rule files in the https-everywhere repo [2]

For the first, we can remove this plugin.  For a complete new implementation, it
might be good to know that there is a project "https-everywhere : Privacy for
Pythons" [3]

related:  https://github.com/return42/searx-next/issues/8

[1] https://github.com/return42/searx-next/tree/d187a1d/searx/plugins/https_rules
[2] https://github.com/EFForg/https-everywhere/tree/master/src/chrome/content/rules
[3] https://github.com/jayvdb/https-everywhere-py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
3nprob 2ca0aa4f29 Remove hubsbpot tracking URL params
More information here:
https://knowledge.hubspot.com/settings/how-do-i-create-a-tracking-url
https://knowledge.hubspot.com/ads/ad-tracking-in-hubspot
https://knowledge.hubspot.com/ctas/calls-to-action-frequently-asked-questions
https://meta.stackexchange.com/questions/263392/what-are-these-very-long-parameters-in-so-careers-feedback-always-welcome-url
3 years ago
Markus Heiser ebfd0eb2b7 [fix] default_doi_resolver in preferences
Instead of a hard-coded `oadoi.org` default, use the default value from
`settings.yml`.

Fix an issue in the themes: The replacement 'current_doi_resolver' contains the
doi_resolver_url, not the name of the DOI resolver.  Compare return value of::

    searx.plugins.oa_doi_rewrite.get_doi_resolver(...)

Fix a typo in `get_doi_resolver(..)`:  suggested by @kvch:

  *L32 should set doi_resolver not doi_resolvers*

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament b00d108673 [mod] pylint: numerous minor code fixes 4 years ago
Alexandre Flament 3038052c79 [mod] remove unused import
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url  # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
4 years ago
Alexandre Flament 5e7060053c [mod] ahmia_filter.py: minor changes
- use result['parsed_url']
- load ahmia_blacklist.txt in searx.datae
4 years ago
Marc Abonce Seguin 32957cdf49 add Ahmia filter plugin for onion results 4 years ago
Venca24 2b93e70a26 [fix] code style 4 years ago
Venca24 1cbcf2ccb6 [mod] adapt hash plugin to current version of searx 4 years ago
Venca24 40c552c11e [fix] hash plugin 4 years ago
Venca24 69e5a58058 [fix] code style 4 years ago
Venca24 1ea9438f5d [fix] hash plugin 4 years ago
Venca24 c9593c8ffd [enh] add plugin converting strings into hash digests 4 years ago
Adam Tauber a05c660e30 [enh] add ability to set enabled plugins from settings - closes #1613 #778 4 years ago
Dalf c225db45c8 Drop Python 2 (4/n): SearchQuery.query is a str instead of bytes 4 years ago
Dalf 1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 4 years ago
Noémi Ványi 74013c212a Revert "[enh] new plugin to open results on new browser tabs"
This reverts commit ab6a7601cb.

Closes #2037
4 years ago
Adam Tauber 93ac4db312 [enh] copy atime_ns and mtime_ns of external plugin resources 4 years ago
Adam Tauber 164e4725f3 [mod] add dedicated directory to external plugins with gitignore file 4 years ago
Adam Tauber 3f60bb470a [fix] correct imported modules 4 years ago
Adam Tauber dff879d07d [fix] resolve minor review issues 4 years ago
Adam Tauber e45a269ecb [mod] create static directory for each external plugin 4 years ago
Adam Tauber 81e9c75534 [fix] pep8 4 years ago
Adam Tauber 7b93d11d84 [fix] add plugin sha sum check to be able to copy updated resources 4 years ago
Adam Tauber 1f2dc6c647 [enh] add external plugin support 4 years ago
Lukáš Kucharczyk 74e4e5e507
Allow localization of "Self Informations"
Allows the string "Self Informations" in the plugin of the same name to be localized by wrapping it with gettext().
4 years ago
Adam Tauber 2c6531b233 [enh] add routing directions to osm search - closes #254 4 years ago
Marc Abonce Seguin dede46a2d5 fix out of range error in tracker remover plugin 5 years ago
Noémi Ványi a6f20caf32 add initial support for offline engines && command engine 5 years ago
Adam Tauber ed1c1bdb04 [fix] pep8 5 years ago
Adam Tauber 28c75c187f [fix] update query params sanitization - closes #722 5 years ago
Étienne Deparis 0d4d58f794 [enh] Add preference section to two upstream plugins for simple theme integration 6 years ago
Adam Tauber 19f18a2638
Merge pull request #1360 from Popolon/master
get_doi_resolver return doi resolver url instead of doi resolver name
6 years ago
Alex 3ba0d0516e simple theme update
- npm package update
- apply #1226
- implement vim help dialog
- display cookies and search URL with preferences
- allow to enable / disable Open Access DOI rewrite
- add a clear text button on the left of the search button
- implement #1011 : the HTML title page is not set when using POST
- remove searx/static/themes/simple/img/loader.gif
- use full width when only there are only images as result
6 years ago
Popolon 00b8aa113f remove double spaces 6 years ago
Popolon 3a160d3aed return doi resolver url instead of doi resolver name 6 years ago
Noémi Ványi 1b10abfc92 minor fixes of doi resolver
Closes #1047
7 years ago
jibe-b 575159b194 [enh] oa_doi_rewrite plugin broadens doai_rewrite 7 years ago
Alexandre Flament 10a24bdc2c [enh] add simple theme (WIP) 7 years ago
Adam Tauber 52e615dede [enh] py3 compatibility 7 years ago
Alexandre Flament 84a2c97a65 [mod] searx uses flask framework only in webapp.py. Make migration to another framework easier. 8 years ago
Adam Tauber 88dfee858e [fix] rewrite missing variable 8 years ago
dalf fbb080f358 Change plugin API :
- pre_search(request, search)
- post_search(request, search)
- on_result(request, search, result)

with
- request is the Flask request
- search a searx.Search instance
- result a searx result as usual
8 years ago
dalf 67e11c42b9 Clean up the architecture
Purposes :
- isolate the plugins calls
- distinction between parsing the web request and running the search (Search class). To be able to test code easily, to run searx code outside a web server, to filter the search query parameters with plugins more easily, etc...

Details :
- request.request_data contains request.form or request.args (initialize inside pre_request() function)
- Query class is renamed RawTextQuery
- SearchQuery class defines all search parameters
- get_search_query_from_webapp create a SearchQuery instance (basically the previous Search.__init__ code)
- Search class and SearchWithPlugins class takes a SearchQuery instance as class constructor parameter
- SearchWithPlugins class inherites from Search class, and run plugins
- A dedicated function search_with_plugins executes plugins to have a well define locals() (which is used by the plugins code).
- All plugins code is executed inside the try...except block (webapp.py, index function)
- advanced_search HTTP parameter value stays in webapp.py (it is only part of UI)
- multiple calls to result_container.get_ordered_results() doesn't compute the order multiple time (note : this method was call only once before)
- paging value is stored in the result_container class (compute in the extend method)
- test about engine.suspend_end_time is done during search method call (instead of __init__)
- check that the format parameter value is one of these : html, rss, json, rss (before the html value was assumed but some text formatting wasn't not done)
8 years ago
Noemi Vanyi aa227180e2 remove self info answers from pages after first one 8 years ago
David A Roberts 678b9e994a [enh] infinite scroll
resolves #649
8 years ago
David A Roberts 117d36b66e Add Crossref search engine and DOAI rewrite plugin 8 years ago
stepshal b3ab221b98 Fix anomalous backslash in string 8 years ago
Adam Tauber 257e52954a [enh][fix] update to latest dependencies ++ fix tests & travis test runner
WARNING: dependency changes
8 years ago
Kirill Isakov 3246541bdc [WIP] Add vim-hotkeys plugin 8 years ago
Adam Tauber 4e5af8d87b [mod] move plugin static files to static/plugins 9 years ago
Adam Tauber ab6a7601cb [enh] new plugin to open results on new browser tabs 9 years ago
Adam Tauber bd22e9a336 [fix] pep8 compatibilty 9 years ago
Adam Tauber d2fd56612c [fix] rename after result refactor 9 years ago
Adam Tauber 6424a7702d Merge branch 'remove_trackers_plugin' of https://github.com/Cqoicebordel/searx into Cqoicebordel-remove_trackers_plugin
Conflicts:
	searx/plugins/__init__.py
9 years ago
Cqoicebordel 8911233e3e Use parsed_url 9 years ago
Cqoicebordel e93f5314d7 A bit of cleanup of the code
- regexes in a array
- regexes applied only on the last part of the url
9 years ago
Luc Didry 0fe362bacf Change self plugin to self_info + change self_info description 9 years ago
Luc Didry 41fd748cdf Merge self_ip and self_useragent into one plugin 9 years ago
Luc Didry 538029dc14 Add a self user agent plugin
Just like with the "ip" query, duckduckgo gives the server's
information with the "user agent" query.
This corrects this behavior by adding a plugin based on self_ip.py plugin.
9 years ago
Cqoicebordel 617495cca8 Add a plugin to remove trackers from results URLs 9 years ago
Adam Tauber 3683b35877 [fix] plugin description 9 years ago
Adam Tauber ff68546c69 [mod] enable category select plugin by default 9 years ago
Adam Tauber 33516256fd [fix] load https rules 9 years ago
Adam Tauber d2a636f75d [mod] https rewrite pluginification 9 years ago
Adam Tauber 146928a749 [doc] plugin license infos 9 years ago
Adam Tauber 973c97c85b [enh] new plugin: search on category select (currently only in oscar theme)
TODO
purge mootools from default/courgette and integrate jquery ++ this theme
9 years ago
Adam Tauber dd84814b68 [enh] plugins: client side dependency support 9 years ago
Adam Tauber 54f1bc0425 [fix] call ip plugin as a post search callback 9 years ago
Adam Tauber a57a011e94 [fix] remove ip plugin pre_search override - closes #275 9 years ago
Adam Tauber bf5d6f56c6 [enh] plugin attribute type check 9 years ago
Adam Tauber 13ea0a20ae [enh] gettext in self ip plugin 9 years ago
Adam Tauber 9eeb36c787 [enh] selfip plugin description 9 years ago
Adam Tauber 9cb66be29c [enh] plugin id 9 years ago
Adam Tauber cae22bfc76 [enh] per user plugin switch 9 years ago
Adam Tauber 00cc4dcbf4 [enh] plugin support basics ++ self ip plugin 9 years ago