Alexandre Flament
4224607c62
searx.utils.html_to_text: replace <br/> by a space
2 years ago
Alexandre Flament
2d5929cc59
[mod] searx.utils: more typing
3 years ago
Alexandre Flament
0eacc46ee3
[mod] add documentation about searx.utils
...
This module is a toolbox for the engines.
Is should be documented.
In addition, searx/utils.py is checked by pylint.
3 years ago
Markus Heiser
3d96a9839a
[format.python] initial formatting of the python code
...
This patch was generated by black [1]::
make format.python
[1] https://github.com/psf/black
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Marc Abonce Seguin
66b7be0965
[fix] fix match_language issue to make zh-TW match to zh-Hant-TW
...
pybabel separates locales with underscores but we use hyphens
everywhere babel doesn't directly touch
3 years ago
Markus Heiser
de0249ddae
[fix] don't mix loaded modules with imported modules (sys.modules)
...
The utils.load_module() function is used to load a python file (aka module) and
return the module's namespace. SearXNG uses this function to load *engines and
answerers* from arbitrary locations with arbitrary modifications. These are not
real python modules and it is not intended to mix this *engines and answerers*
with the python modules registered in sys.modules.
Closes: https://github.com/searxng/searxng/issues/312
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/312
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament
697ebeddcc
[mod] searx.utils.dict_subset: rewrite with comprehension
3 years ago
Alexandre Flament
4b43775c91
version based on the git repository
...
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.
It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
3 years ago
Alexandre Flament
92c8a8829f
[fix] strip spaces from searx user agent
...
h11 (used by httpx) rejects HTTP request with a trailing space in HTTP headers
3 years ago
Alexandre Flament
4b07df62e5
[mod] move all default settings into searx.settings_defaults
3 years ago
Markus Heiser
96b223023a
[mod] utils.get_value() - avoidance of a recursion
...
In a comment [1] dalf suggested to avoid a recursion of get_value()
[1] https://github.com/searxng/searxng/pull/99#discussion_r640833716
Suggested-by: Alexandre Flament <alex@al-f.net>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser
6ed4616da9
[enh] add settings option to enable/disable search formats
...
Access to formats can be denied by settings configuration::
search:
formats: [html, csv, json, rss]
Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament
eaa694fb7d
[enh] replace requests by httpx
3 years ago
Alexandre Flament
3f8ebf70b1
[fix] pylint: use "raise ... from ..."
4 years ago
Alexandre Flament
de887c6347
[mod] bing_news: use eval_xpath_getindex
...
remove unused function searx.utils.list_get
4 years ago
Alexandre Flament
1d0c368746
[enh] record details exception per engine
...
add an new API /stats/errors
4 years ago
Alexandre Flament
b00d108673
[mod] pylint: numerous minor code fixes
4 years ago
Alexandre Flament
3038052c79
[mod] remove unused import
...
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
4 years ago
Alexandre Flament
ca593728af
[mod] duckduckgo_definitions: display only user friendly attributes / URL
...
various bug fixes
4 years ago
Alexandre Flament
a9dc54bebc
[mod] Add searx.data module
...
Instead of loading the data/*.json in different location,
load these files in the new searx.data module.
4 years ago
Alexandre Flament
15013e64d8
[fix] drop Python 2: use importlib instead of imp.load_source
...
imp.load_source is not documented in Python 3
see documentation : https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
partial fix of https://github.com/searx/searx/issues/1674
4 years ago
Alexandre Flament
8f914a28fa
[mod] searx.utils.normalize_url: remove Yahoo hack
...
* The hack for Yahoo URLs is not necessary anymore. (see searx.engines.yahoo.parse_url)
* move the URL normalization in extract_url to normalize_url
4 years ago
Alexandre Flament
c1d10bde02
[mod] searx/utils.py: add docstring
4 years ago
Alexandre Flament
2006eb4680
[mod] move extract_text, extract_url to searx.utils
4 years ago
Alexandre Flament
ad0758e52a
[mod] add searx/webutils.py
...
contains utility functions and classes used only by webapp.py
4 years ago
Alexandre Flament
6deb85072a
[fix] searx.utils.HTMLTextExtractor: invalid HTML don't raise an Exception
...
Close #2188
4 years ago
Alexandre Flament
bdac99d4f0
Drop Python 2 (5/n): searx.utils.is_valid_lang, input parameter is a str instead of bytes
...
Fix bug in translated.py and dictzone.py
4 years ago
Dalf
c225db45c8
Drop Python 2 (4/n): SearchQuery.query is a str instead of bytes
4 years ago
Dalf
1022228d95
Drop Python 2 (1/n): remove unicode string and url_utils
4 years ago
Dalf
85b3723345
[mod] speed optimization
...
compile XPath only once
avoid redundant call to urlparse
get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
5 years ago
Noémi Ványi
5796dc60c9
fix pep 8 check
5 years ago
Noémi Ványi
a6f20caf32
add initial support for offline engines && command engine
5 years ago
Adam Tauber
72459b246b
[fix] convert bytes type to string in language detection (fixes dictzone)
5 years ago
Alexandre Flament
2179079a91
[fix] fix flickr_noapi decoding ( #1655 )
...
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
5 years ago
Dalf
7e201cbf65
[mod] use cache in _match_language function to speed up searx start time significantly
5 years ago
rachmadani haryono
ec88fb8a0f
[fix] secret_key can be bytes instead of a string ( #1602 )
...
Fix #1600
In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
5 years ago
Alex
50c836864a
fetch_firefox_version.py : compatible with Python 3 and minor fixes.
6 years ago
Alexandre Flament
066bd916bf
[mod] fetch firefox versions in a standalone script
6 years ago
Adam Tauber
d51732c0e5
Merge pull request #1303 from MarcAbonce/bing
...
Fix bing "garbage" results
6 years ago
Marc Abonce Seguin
c7000cd1df
[fix] update user agent versions
...
this fixes duckduckgo error response
6 years ago
Adam Tauber
aef2b07969
[fix] add basestring for py3
6 years ago
Marc Abonce Seguin
75b276f408
fix bing "garbage" results (issue #1275 )
6 years ago
Marc Abonce Seguin
772c048d01
refactor engine's search language handling
...
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.
Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
7 years ago
Adam Tauber
0969e50c5b
[fix] convert json engine result attributes to string - closes #1006
7 years ago
Adam Tauber
b5071fea6a
[fix] remove trailing 0x00 from csv output
7 years ago
Adam Tauber
3d6c67951a
[fix] resurrect csv output in py2
7 years ago
Noémi Ványi
e73cb14889
fix hmac python3 compatibility
7 years ago
misnyo
33fd938016
[mod] int_or_zero refactored to searx_utils
7 years ago
potato
9b82cb1908
[fix] is_valid_lang fixed for new languages.py + dictzone engine encoding
7 years ago
Alexandre Flament
9c91ab33f8
[mod] settings.yml can be /etc/searx/settings.yml
...
The exact order is
* first from SEARX_SETTINGS_PATH,
* if not found then from searx code base,
* if not found then from /etc/searx/settings.yml
* if not found an exception stops searx loading
7 years ago