Commit Graph

122 Commits

Author SHA1 Message Date
Markus Heiser
40956e8e6b [data.all] add searx_extra/update/update_osm_keys_tags.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09 18:08:23 +02:00
Markus Heiser
393c06f537 [enh] add test.yamllint - lint yaml files
Usage::

    make test.yamllint

    ./manage test.yamllint

test.yamllint is also added to the `test` makefile target.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-05 17:41:24 +02:00
Markus Heiser
d6c321b830 [mod] docker-entrypoint.sh - add script to make test.sh (shellcheck)
Add script docker-entrypoint.sh to shellcheck and try to _simplify_ and
_normalize_ some parts:

- fix issues reported by shellcheck
- don't mix tab and space indent
- command 'help' replaced by '-h': ./dockerfiles/docker-entrypoint.sh -h
- replace printf in help() by 'cat <<EOF'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-17 12:02:24 +02:00
Alexandre Flament
20580bcbd4 [docker] multiarch support: linux/amd64,linux/arm64,linux/arm/v7
make docker.buildx : build and push multiarch build.
(it can't be only build)

use buildx with the --cache-from and --cache-to options to cache the layers
(only the last built is cached)
2021-04-28 14:40:00 +02:00
Markus Heiser
abd423cbf8 [fix] Makefile fix target 'test' & 'test.sh'
Renamed 'test.sh' to 'test.shell' seems to solve this issue on my ubu204::

    $ make test.sh
    ...
    TEST      robot
    ...
    INSTALL   gecko.driver
    ...
    Running 7 tests
    cat test.sh >test
    cat: test.sh: Datei oder Verzeichnis nicht gefunden
    make: *** [<eingebaut>: test] Fehler 1

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Alexandre Flament
d01741c9a2
Merge pull request #15 from return42/add-springer
Add a search engine for Springer Nature
2021-04-22 13:23:31 +02:00
Pierre Chevalier
a80bf1ba97 [enh] Add Springer Nature engine
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].

To test this PR, first get your API key following this page:

   https://dev.springernature.com/signup

In searx/engines/springer.py at line 24, add this API key.  I left my own key,
commented out in the line aboce.  Feel free to use it, if needed.

[1] https://www.springernature.com/
[2] https://dev.springernature.com/
2021-04-22 12:35:25 +02:00
habsinn
41a2e3785e [enh] add engine using API from "The Art Institute of Chicago" 2021-04-22 12:25:43 +02:00
Noémi Ványi
07f5edce3d Add Meilisearch engine
Website: https://www.meilisearch.com/
2021-04-06 21:57:05 +02:00
Noémi Ványi
ff527e2681 Add Solr engine 2021-03-13 21:18:09 +01:00
Alexandre Flament
9292571304
Merge pull request #2346 from dalf/upgrade-oscar
[mod] oscar: upgrade dependencies
2021-03-13 09:29:13 +01:00
Alexandre Flament
92dd5e245e
Merge pull request #2626 from mikeri/solidtorrents
Add Solid Torrents engine
2021-03-12 19:45:22 +01:00
Alexandre Flament
af3e969c5a
Merge pull request #2642 from return42/fix-apkmirror
[fix] APKMirror engine - update xpath selectors and fix img_src
2021-03-11 09:48:31 +01:00
Alexandre Flament
8b650e6a2d
Merge pull request #2643 from return42/fix-makefile
[fix] make targets engines.languages and useragents.update
2021-03-11 09:44:30 +01:00
Alexandre Flament
86912e2272 [mod] oscar: get bootstrap and typeahead from NPM 2021-03-11 09:33:01 +01:00
Markus Heiser
96422e5c9f [fix] APKMirror engine - update xpath selectors and fix img_src
BTW: make the code slightly more readable

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-09 08:34:57 +01:00
Markus Heiser
c0d3183593 [fix] make targets engines.languages and useragents.update
Since [PR 2600] is merged the update scripts for languages and useragent has
been moved to folder:

    searx_extra/update/

[PR 2600] https://github.com/searx/searx/pull/2600

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-08 16:14:33 +01:00
Markus Heiser
d2faea423a [fix] rewrite Yahoo-News engine
Many things have been changed since last review of this engine.  This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-03-08 11:43:34 +01:00
Michael Ilsaas
5549d58de3 Add Solid Torrents engine 2021-03-07 18:14:30 +01:00
Alexandre Flament
b8cd326464 Add searx_extra package
Split the utils directory into:
* searx_extra contains update scripts, standalone_searx.py
* utils contains the files to build and setup searx.
2021-03-04 11:59:14 +01:00
Markus Heiser
d48e2e7b0b [enh] google scholar - python implementation of the engine
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-01 15:16:37 +01:00
Alexandre Flament
7c1847d5f2 [mod] add utils/fetch_external_bangs.py
Based on duckduckgo bangs
Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-24 18:48:36 +01:00
Markus Heiser
bc1be3f0e9 [enh] add engine MediathekViewWeb (API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-09 13:08:01 +01:00
Markus Heiser
8c45f1149d [hardening] github workflows - corrupted cache
aka: ensure that 'make test' works as expected

The cache contains a copy './local' which is - under some circumstance -
corrupted.  It is not possible to clear the cache [1] (see the top of the page).

Ensure that 'make test' works as expected [2] even if

- the python interpreter is missing
- the virtualenv exists but pyyaml is missing

To hardening when the workflow cache fails, this patch adds the new target
'travis.test' into the workflow.  This target probes to import a python module
'yaml'.  If this fails the virtualenv will be completely new build.

[1] https://github.com/actions/cache/issues/2#issuecomment-673493515
[2] https://github.com/searx/searx/pull/2517#discussion_r567240235

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-01 16:58:04 +01:00
Markus Heiser
923b490022 [mod] add Makfile targets for search.checker.<engine_name>
To check all engines:

    make search.checker

To check a engine 'google news' replace space by underline:

    make search.checker.google_news

To see HTTP requests and more use SEARX_DEBUG:

    make SEARX_DEBUG=1 search.checker.google_news

To filter out HTTP redirects:

    make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
    ...
    Engine google news                   Checking
    https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
    https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
    --
    https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
    https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
    --

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-26 11:46:36 +01:00
Markus Heiser
89b3050b5c [fix] revise of the google-Video engine
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-24 09:39:30 +01:00
Markus Heiser
baec54c492 [fix] revise of the google-news engine
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-22 18:49:45 +01:00
Alexandre Flament
f7e11fd722
Merge pull request #2459 from dalf/update-python
Update python
2021-01-12 11:02:58 +01:00
Alexandre Flament
d54034a5e6 [mod] add Python 3.9 support 2021-01-12 09:53:26 +01:00
Markus Heiser
9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Alexandre Flament
14c7cc0e11 [mod] Makefile: make CONTACT_URL optional 2020-12-18 09:54:03 +01:00
BBaoVanC
19fce74443
Add link to contact instance maintainer to footer of each page (#2391) 2020-12-18 09:53:28 +01:00
Alexandre Flament
9bc1856e2b [mod] themes: remove legacy, courgette and pix-art themes 2020-12-17 11:33:28 +01:00
Markus Heiser
6b0a896f01 [mod] digg - pylint searx/engines/digg.py
Eliminate redundant file names which are tested by test.pylint and ignored by
test.pep8

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-02 20:59:30 +01:00
Alexandre Flament
8aa2a7556a [enh] activate pylint
There are 3 invocations:
* one with the default .pylintrc file
* one for searx/engines with some disabled checks and some additional-builtins
* one for the all the code except the engines with some disabled checks
2020-12-01 15:21:19 +01:00
Markus Heiser
c71d214b0c [refactor] deviantart - improve results and clean up source code
Devian's request and response forms has been changed.

- fixed title
- fixed time_range_dict to 'popular-*-***'
- use image from <noscript> if exists
- drop obsolete "http to https, remove domain sharding"
- use query URL https://www.deviantart.com/search/deviations?page=5&q=foo
- add searx/engines/deviantart.py to pylint check (test.pylint)

Error pattern::

    There DEBUG:searx:result: invalid title: {'url': 'https://www.deviantart.com/  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-11-14 17:09:56 +01:00
rachmadaniHaryono
5d015b58f7 new: test: ignored warning and errors 2020-11-06 18:54:53 +08:00
rachmadaniHaryono
d422f60529 chg: test: Makefile pep8 test 2020-11-06 08:40:21 +08:00
Markus Heiser
4a15d675d5 [fix] make.run - use SEARX_DEBUG=1 instead of modifing settings.yml
Reloading webapp.py when a file is changed exit with error::

    $ make run
    ....
    INFO:werkzeug: * Detected change in 'searx/searx/engines/deviantart.py', reloading
    INFO:werkzeug: * Restarting with stat
    ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
    make: *** [Makefile:69: run] Error 1

Patch suggested by @dalf [1]

[1] https://github.com/searx/searx/issues/2278#issuecomment-719732579

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-10-31 15:40:55 +01:00
Alexandre Flament
f204e4903d [fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs 2020-09-28 16:44:14 +02:00
Alexandre Flament
c2a6f14516 [fix] Update Sphinx package versions
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
2020-09-28 13:27:50 +02:00
Marc Abonce Seguin
c86504b47a fix utils/fetch_languages to work with new languages 2020-09-22 11:37:44 +02:00
Dalf
1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser
f14a7add31 Merge branch 'master' of https://github.com/asciimoo/searx into csp-oscar-theme 2020-06-30 15:37:39 +02:00
Markus Heiser
4eab5cf854
Merge branch 'master' into gigablast 2020-06-19 04:51:48 +00:00
Markus Heiser
ca1c3bd15d
Merge branch 'master' into csp-oscar-theme 2020-06-19 04:51:02 +00:00
Markus Heiser
15924bcb67 [fix] buildprocess of the docs by adding sphinx-doc-prebuilds target
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs

BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:44:22 +02:00
Markus Heiser
d44b4eda64 [fix] gigablaste engine: switch from pep8 to pylint test
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-29 08:52:11 +02:00
Markus Heiser
4954e56b69 [fix] Makefile: node.env is a prerequisite of lessc and grunt
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-15 09:57:09 +02:00