Commit Graph

456 Commits (57b8f340a6689fafd0c8d92aebd2b263b090af18)

Author SHA1 Message Date
Markus Heiser 2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament f523fd3ea7
Merge pull request #211 from MarcAbonce/onions_v3_fix_searxng
Update onion engines to v3
3 years ago
Marc Abonce Seguin 1b05ea6a6b update onion engines to v3
remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
3 years ago
Markus Heiser 0a9cd08bf1 [enh] add mongodb offline engine
Cherry-Pick: https://github.com/searx/searx/commit/198aad43
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 420e3caba3 [mod] enable Qwant engines (web, news, images, video) by default
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA.  Let
us enable Qwant engines by default.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser ef6e1bd6b9 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 2767046065 [fix] drop engine dogpile
The engine was added in commit a4b07460 but now it shows new issues [1].

In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]

  Powered by technology, Dogpile returns all the best results from leading
  search engines including Google and Yahoo!

Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough.   It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.

With this patch the dogpile engine is removed

[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 513c73a309 [drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/

Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Allen 3b2e0a93b7 [enh] add gpodder.net (JSON)
Engine just for Podcasts.  An API which returns Podcasts and their Info like:
website, author etc.

Upstream query example: https://gpodder.net/search.json?q=linux
3 years ago
Markus Heiser d5eedf1b61 [yamllint] fix some warning issues reported by yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Allen 9648d71ad8 [enh] add woxikon.de synonyme (Xpath)
Added synonyme.woxikon.de using the xpath engine.  Adds a site which returns
word synonyms although just in German.

Depending on the query not all synonyms are shown because of not the best xpath
selection.  But should do the job just fine.

Upstream example query: https://synonyme.woxikon.de/synonyme/test.php
3 years ago
Markus Heiser bb98a07b50
Merge pull request #196 from return42/fix-brave
[fix] brave engine: shows descriptions with their correct URLs
3 years ago
Markus Heiser 9d97b2e854 [fix] brave engine: shows descriptions with their correct URLs
BTW add about section to the YAML configuration

It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.

Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 7e8d1f9935
Merge pull request #178 from return42/whaleslide.com
[enh] Add WhaleSlide engine (JSON)
3 years ago
Alexandre Flament 7308709b8b
Merge pull request #179 from return42/brave
[enh] add Brave-Search engine (XPath)
3 years ago
Markus Heiser 6787e5a36b [fix] decoding of saved preferences in the URL
To compress saved preferences in the URL was introduced in 5f758b2d3 and
slightly fixed in 8f4401462.  But the main fail was not fixed; The decompress
function returns a binary string and this binary should first be decoded to a
string before it is passed to urllib.parse_qs.

BTW: revert the hot-fix from 5973491

Related-to: https://github.com/searxng/searxng/issues/166
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser e0365d987b [enh] add Brave-Search engine (XPath)
Add https://search.brave.com which was requested by @kaonashi696 [1].
This patch was suggested by @eagle-dogtooth [2].

[1] https://github.com/searx/searx/issues/2857
[2] https://github.com/searx/searx/issues/2857#issuecomment-867199241

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Allen e2094cb226 [enh] Add WhaleSlide engine (JSON)
Add WhaleSlide engine [1] / merged from [2]

[1] https://whaleslide.com
[2] https://github.com/searx/searx/pull/2861
3 years ago
Émilien Devos 63ebe00554
Set use_mobile_ui to false for Google engine 3 years ago
Markus Heiser 5973491701 [fix] UnicodeEncodeError for preferences ULR
Closes: https://github.com/searxng/searxng/issues/166

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 0ef6aa5126 [docs] add documentation from the sources of the google engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament 7a5c36408a [mod] google: add "use_mobile_ui" parameter to use mobile endpoint.
disable by default, it has to be enabled in settings.yml

related to  #159
3 years ago
Markus Heiser 96d4dbf74d [fix] typo in a searx/settings.yml
remove the erroneously leading underline

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 5a7b12ee6f [yamllint] searx/settings.yml
Fixed messages reported by::

    make test.yamllint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Adam Tauber e4b6558339 [enh] add redis offline engine / https://redis.io/
Slightly modified merge of commit [97269be6], [01a8a5814a] and [c8d2b5eb] from
searx.

[97269be6] https://github.com/searx/searx/commit/97269be6
[01a8a581] https://github.com/searx/searx/commit/01a8a581
[c8d2b5eb] https://github.com/searx/searx/commit/c8d2b5eb

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament 6ca74386d6
[fix] mankier: fix paging
See https://github.com/searxng/searxng/pull/111#issuecomment-851641967
3 years ago
Kyle Anthony Williams d6a2d4f969 [enh] add engine - Docker Hub
Slightly modified merge of commit [1cb1d3ac] from searx [PR 2543]:

      This adds Docker Hub .. as a search engine .. the engine's favicon was
      downloaded from the Docker Hub website with wget and converted to a PNG
      with ImageMagick .. It supports the parsing of URLs, titles, content,
      published dates, and thumbnails of Docker images.

[1cb1d3ac] https://github.com/searx/searx/pull/2543/commits/1cb1d3ac
[PR 2543] https://github.com/searx/searx/pull/2543

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament e239369f1f
Merge pull request #109 from return42/sql-engines
Merge MySQL and PostgreSQL from searx
3 years ago
Alexandre Flament c22c7cf35b
Merge pull request #111 from return42/mankier
[enh] add engine ManKier (JSON) / ManKier: Linux man pages
3 years ago
Jordan Webb a8f474b23f [enh] add engine PyPi (XPath)
Add a XPath configuration for PyPI (Python Package Index)

Slightly modified merge of [66d06b05] from searx's [PR 2830].

[PR 2830] https://github.com/searx/searx/pull/2830
[66d06b05] https://github.com/searx/searx/pull/2830/commits/66d06b05

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Jordan Webb 52672d887b [enh] add engine ManKier (JSON) / ManKier: Linux man pages
Slightly modified merge of [a881e742aa] from searx's [PR 2829].

[PR 2829] https://github.com/searx/searx/pull/2829
[a881e742aa] https://github.com/searx/searx/pull/2829/commits/a881e742aa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Noémi Ványi 87a01a1736 [enh] add MySQL engine
Slightly modified merge of [c00a33fe] from searx.

[c00a33fe] c00a33feee

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Noémi Ványi 324aa96062 [enh] add PostgreSQL engine
Slightly modified merge of [22079ff] from searx.

[22079ff] 22079ffdef

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Allen a4b0746044 [enh] add metasearch engin dogpile.com (XPath)
Add Dogpile engine [1] / merged from [2]

[1] https://www.dogpile.com
[2] https://github.com/searx/searx/pull/2822
3 years ago
LL Productions FR 15f1d6a80f [enh] add engine Wikimini (fr.wikimini.org)
Online encyclopedia for children (only language fr), merged from [1]

[1] https://github.com/searx/searx/pull/2819
3 years ago
Alexandre Flament 83ccf7b04d
Merge pull request #99 from return42/webapp-misc
[enh] add settings option to enable/disable search formats
3 years ago
Markus Heiser 6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 25b5797a0c
Merge pull request #103 from searxng/add-sqlite-engine2
[enh] add offline engine for sqlite database
3 years ago
Alexandre Flament 2ea34a3c36 [enh] add offline engine for sqlite database
To test & demonstrate this implementation download:

  https://liste.mediathekview.de/filmliste-v2.db.bz2

and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite
engine named "demo"::

    - name : demo
      engine : sqlite
      shortcut: demo
      categories: general
      result_template: default.html
      database : searx/data/filmliste-v2.db
      query_str :  >-
        SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
               COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
               description AS content
          FROM film
         WHERE title LIKE :wildcard OR description LIKE :wildcard
         ORDER BY duration DESC
      disabled : False

Query to test: "!demo concert"

This is a rewrite of the implementation from commit [1]

[1] searx/searx@8e90a21

Suggested-by: @virtadpt searx/searx#2808
3 years ago
Markus Heiser 97d173eda0 [mod] unsplash engine - activated by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 57cdf180fe [fix] engine archive is - search_url has been changed
BTW: set soft_max_redirects from *archive is* by 1, to prevent logging::

    DEBUG:httpx._client:HTTP Request: GET https://archive.is/search/?q=www.python.org "HTTP/2 302 Found"
    DEBUG:httpx._client:HTTP Request: GET https://archive.is/www.python.org "HTTP/2 200 OK"
    DEBUG:searx:archive is: ErrorContext('searx/search/processors/online.py', 110, 'count_error(self.engine_name,', None, '1 redirects, maximum: 0', ('200', 'OK', 'archive.is')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 5253df946b
[mod] option to enable or disable "proxy" button next to each result (#54)
* [mod] option to enable or disable "proxy" button next to each result

Closes: https://github.com/searxng/searxng/issues/51
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Alexandre Flament <alex@al-f.net>
3 years ago
Alexandre Flament 51494849fc
Merge pull request #23 from searxng/metrics-stats-engines
Metrics stats engines
3 years ago
Alexandre Flament df41b77121 [mod] /stats : detail per engine
allow to submit a github issue including the technical details
(exceptions, errors, warning, checker result)
3 years ago
Markus Heiser 28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Markus Heiser 6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
3 years ago
spongebob33 7528e38c8a add core.ac.uk engine 3 years ago
Alexandre Flament 3755cf2eba More branding 3 years ago
Alexandre Flament 9d84025e4f searxng brand 3 years ago