Commit Graph

1612 Commits (be0d756ad5dab0a477cea0ebcb5391a350955cf0)

Author SHA1 Message Date
Jinyuan Huang ae28d429c9 [fix] bilibili new api used 1 year ago
jazzzooo 1a66d74673 [fix] engine - kickass update url, fix parsing, use multiple mirrors 1 year ago
Markus Heiser b428ccc5a0 [fix] engine brave - fetch traits (modified settings menu)
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 3a456b1282 [fix] engine annas archive - fetch traits (modified xpath selectors)
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Émilien (perso) 1851f27154
[mod] remove twitter (#2843) 1 year ago
Bnyro 75c9de02d1 [feat] engine: implementation of imgur 1 year ago
Bnyro fb72f71f0a [fix] internet archive scholar: crash when there's no title 1 year ago
Markus Heiser 71358e9c67 Revert "[fix] engine - duckduckgo vqd edge-case"
This reverts commit 102502a4f0.
1 year ago
Bnyro 51236ae47a [feat] engine: implementation of chefkoch.de 1 year ago
jazzzooo 8bcca0e620 [fix] engine - brave don't show ads 1 year ago
jazzzooo b729542a66 [fix] engine - google images error when no results 1 year ago
Bnyro cc2e0537a3 [feat] engine: implementation of google icons/material design icons 1 year ago
Bnyro c999cfb422 [feat] engine: implementation of wallhaven 1 year ago
jazzzooo 102502a4f0 [fix] engine - duckduckgo vqd edge-case 1 year ago
Markus Heiser 043dcbf7c5 [fix] engine qwant (web-lite) - ignore advertising adds
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Émilien (perso) ad725ce7d7
wikipedia wikidata infobox + disable wikisource (#2806)
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro efd3a2d6d1 [feat] engine: implementation of internet archive scholar 1 year ago
jazzzooo 223b3487c3 [fix] spelling 1 year ago
Markus Heiser a9b6963971 [fix] engine - qwant delivers only 5 pages maximum
all qwant engines (incl qwant-lite) delivers only 5 pages maximum

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
jazzzooo da1446c5ed [fix] engine - qwant wrong error type 1 year ago
Markus Heiser 7398d525c8 [fix] qwant: subsequent fix of commit d9dbcedeb
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser d9dbcedeb6 [feat] implementation of qwant lite for web search
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro b4e0d2eedc [feat] engine: implemenation of moviepilot (de) 1 year ago
jazzzooo 7dfcc3386e [fix] tagesschau videos 1 year ago
jazzzooo ec540a967a [fix] brave.videos 1 year ago
jazzzooo 27477f51fd [fix] brave.news 1 year ago
Justas Zabulionis 41ef73ca3a [fix] rumble redirect 1 year ago
Justas Zabulionis be888810ba [fix] pubmed content being None 1 year ago
Justas Zabulionis 92d39de410 [fix] solidtorrents redirects 1 year ago
Justas Zabulionis cf8a6cf6db [fix] solidtorrents pagination 1 year ago
Justas Zabulionis 8172f89075 [fix] solidtorrents 1 year ago
jazzzooo 74600c028d [fix] engine - Crossref
Crossref was broken on result types journal-issue and component .. The old code
had lots of assumptions, and broke during parsing.  Now the assumptions are more
explicit and checked them with the API.
1 year ago
Bnyro 3568a3cafb [feat] odysee: implement fetch_traits for language support 1 year ago
Bnyro 09c61dabc9 [mod] odysee: time range support 1 year ago
jazzzooo b98907e91f [fix] engine - piped.music incorrect timestamps 1 year ago
jazzzooo 6039dbf211 [fix] engine - invidious thumbnails 1 year ago
jazzzooo b2fd6304bf [fix] engine - openstreetmap currency rendering 1 year ago
jazzzooo 54a3e03b45 [fix] engine - openstreetmap currency matching 1 year ago
Bnyro 64d9587ac8 [feat] new engine: svgrepo 1 year ago
jazzzooo b189578b6b [fix] engine - brave 1 year ago
Bnyro f182abd6f8 [mod] library of congress: fix engine 1 year ago
Bnyro e73a6f5d14 [fix] engine deviantart: review of the result-scrapper
The deviantart site changed and hence deviantart is currently unusable.
1 year ago
Alexandre Flament d07c006aed Replace chompjs with pure Python code
The new implementation is good enough for the current usage (brave)
1 year ago
Bnyro 9e83c0dedc [feat] engine: implementation of Yummly
Co-authored-by: Markus Heiser <markus.heiser@damarit.de>
1 year ago
Bnyro a3d7e9c285 [mod] utils.py: add markdown_to_text helper function 1 year ago
Hackurei 1f21ac7d62 [feat] engine: implementation of bilibili https://www.bilibili.com 1 year ago
Markus Heiser 696c35d2c3 [fix] engine - duckduckgo_images / determination of vqd value incorrect
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
bonswouar 4fb26cd96d [fix] engine duckduckgo weather api changes 1 year ago
Markus Heiser 01be9e0e20 [fix] engine: wikicommons - don't quoute ':|' in URL parameters
From [1]: It seems to be because of [2] For some reason it gets url encoded
twice, resulting in

- ``filetype%253Abitmap%257Cdrawing+birds`` instead of
- ``filetype:bitmap%7Cdrawing+birds``

[1] https://github.com/searxng/searxng/issues/2707
[2] https://github.com/searxng/searxng/blob/master/searx/engines/wikicommons.py#L43

Closes: #2707
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 4f8895c6de [fix] follow-up of 4da7003ae / add missing review from @Bnyro
[1] https://github.com/searxng/searxng/pull/2656#pullrequestreview-1607956209

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Hackurei 4da7003ae0 [feat] engine: implementation of odysee 1 year ago
Bnyro 9c4e9d3814 [feat] implementation of Wikimedia commons for images 1 year ago
Alexandre Flament faa4280e1a [mod] bing: resolve redirect without additional requests
Remove the usage of searx.network.multi_requests
The results from Bing contains the target URL encoded in base64
See the u parameter, remove the first two character "a1", and done.

Also add a comment the check of the result_len / pageno
( from https://github.com/searx/searx/pull/1387 )
1 year ago
Markus Heiser b0d2cd5ca9 [doc] add documentation of Mwmbl engine & autocompleter
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 19300a5659 [mod] engine mwmbl: add link to official api docs 1 year ago
Alexandre Flament e16c007c22 [fix] openstreetmap engine
It seems there is an API change:
extratags can be either a dictionnary or None.

This commit avoid crash when extratags is None

Test query "!osm gare du nord"
1 year ago
Markus Heiser 0647f83a3e [fix] google engine: don't overspecify the search query to Google
The method EngineTraits.get_region(..) returns engine's region string
that **best fits** to SearXNG's locale.  This means it returns a
region (country) if only a language is set in the locale.  By example the method
returns for a locale tag `es` a region `ES`.

Google's search parameter `cr` restricts search results to documents originating
in a particular country / in case of a locale tag (language) as described above,
this argument should be unset in the query send to Google.

Closes: https://github.com/searxng/searxng/issues/2672
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 4b42644579 [fix] engine google_video: google has changed the layout of the rsponse
Closes: https://github.com/searxng/searxng/issues/2664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro c59ae91b76 [feat] engine: implementation of mwmbl 1 year ago
Markus Heiser c741fc6f00 [mod] currency_convert: support for showing the answer source url
Show URL of the ddg-search page, not the URL of a (generic) Javascript.  The
latter one is not usefull for the user.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser e2744520f8 [mod] google: support for showing the answer source url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 5ec7df3480 [mod] engine duckduckgo definitions: support for answer source 1 year ago
Bnyro 64bc98b5fb [mod] brave: support for showing the answer source url 1 year ago
Markus Heiser 9100a48541 [mod] improve seekr engines and add documentation
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 2bab658d39 [feat] engine: implementation of seekr for news, images and videos 1 year ago
Bnyro e25d1c7288 [feat] engine: implementation of German news, Tagesschau
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 834e1c3f12 [mod] engine lemmy: increase thumbnail quality to align with theme 1 year ago
Markus Heiser c381fc001f [mod] settings: remove lemmy from categ 'general' & enable by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser fda111c0c9 [mod] engine lemmy: add more info fields to the result items
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 224f2250ae [feat] engine: support for lemmy communities, posts, comments and users 1 year ago
Bnyro 9f82c39610 [mod] engine google_play: raise error on unsupported category 1 year ago
Bnyro 0a99dc85b9 [mod] engine brave: raise error on unsupported category 1 year ago
allendema_searxng_pi c00c0c5434 [mod] remove discontinued petalsearch engines 1 year ago
Markus Heiser b8352eca0c [mod] brave engines: add fetch_traits() / improve language support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 460bbe5b81 [mod] implement brave (WEB) engine to replace XPath configuration
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro d151497db3 [feat] engine: brave - support for news 1 year ago
Bnyro cae06f2781 [feat] engine: brave - support for videos 1 year ago
Bnyro 73364e158e [feat] engine: brave - support for images 1 year ago
Markus Heiser 1d0abb7157 [doc] engine bt4g: add documentation to docs/dev/engines/online/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Emilien Devos 0fc8f99ecc [feat] new engine: bt4g added & enabled and disable by default btdigg
Disable btdigg because on most SearXNG instances, SearXNG is blocked by btdigg
due to cloudflare too many requests.

This impementation did not parse the HTML page because there is an API in
XML (RSS).  The RSS feed provides fewer data like amount of seeders/leechers and
the files in the torrent file. It's a tradeoff for a "stable" engine as the XML
from RSS content will change way less than the HTML page.

Closes: https://github.com/searxng/searxng/issues/2553
1 year ago
Markus Heiser db522cf76d [mod] engine: wikimedia - improve results, add addition settings & doc
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 7d8c20c80d [feat] new engine: wikispecies 1 year ago
Markus Heiser 1b030d4b41 [doc] engine: Yacy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
zutto ca518c6803 add option to change yacy search mode 1 year ago
Markus Heiser 203f1f0928 [fix] engine piped: 'invalid content'
SearXNG does not allow a None value in the content field of a result item.

If the key (shortDescription, uploaderName) in the JSON response from piped
exists but is set to None, SearXNG ignores this result item::

  DEBUG   searx    : result: invalid content: { ..,  'content': None,  ..}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 207fcc0c8c [mod] engine piped: add paging support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser ef5831cd84 [mod] engine piped: split into two dedicated engiens for video & music
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 7aa95d2d52 [doc] engine piped: add documentation to docs/dev/engines/online/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Bnyro 636bfdac68 [feat] engine: implementation of Piped 1 year ago
Paolo Basso cada89ee36 [feat] engine: re-enables z-library (zlibrary-global.se)
- re-enables z-library as the new domain zlibrary-global.se is now available
  from the open web.   The announcement of the domain:

    https://www.reddit.com/r/zlibrary/comments/13whe08/mod_note_zlibraryglobalse_domain_is_officially/

  It is an official domain, it requires to log in to the "personal" subdomain
  only to download files, but the search works.

- changes the result template of zlibrary to paper.html, filling the appropriate fields
- implements language filtering for zlibrary
- implement zlibrary custom filters (engine traits)
- refactor and document the zlibrary engine
1 year ago
Markus Heiser 5720844fcd [doc] rearranges Settings & Engines docs for better readability
We have built up detailed documentation of the *settings* and the *engines* over
the past few years.  However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.

This patch rearranges the Settings & Engines documentation for better
readability.

To review new ordered docs::

   make docs.clean docs.live

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser 87e7926ae9 [fix] engine: Anna's Archive - grep results from '.js-scroll-hidden' elements
The renderuing of the WEB page is very strange; except the firts position all
other positions of Anna's result page are enclosed in SGML comments.  These
cooments are *uncommented* by some JS code, see query of the class
'.js-scroll-hidden' in Anna's HTML template [1].

[1] https://annas-software.org/AnnaArchivist/annas-archive/-/blob/main/allthethings/templates/macros/md5_list.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser e2df6b77a3 [mod] engine: Anna's Archive - additionl settings (content, sort, ext)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser eafc2906f1 [mod] engine: Anna's Archive - fetch search arguments from search form
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Paolo Basso 7adb9090e5 [mod] engine: Anna's Archive - add language support 1 year ago
Paolo Basso e5637fe7b9 [feat] engine: implementation of Anna's Archive
Anna's Archive [1] is a free non-profit online shadow library metasearch engine
providing access to a variety of book resources (also via IPFS), created by a
team of anonymous archivists [2].

[1] https://annas-archive.org/
[2] https://annas-software.org/AnnaArchivist/annas-archive
1 year ago
Paolo Basso 401561cb58 [mod] engine torznab - refactor & option to hide links
- torznab engine using types and clearer code
- torznab option to hide torrent and magnet links.
- document the torznab engine
- add myself to authors

Closes: https://github.com/searxng/searxng/issues/1124
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser da7c30291d [fix] Google API changed
It seems that Google is rolling out a modified WEB API [1][2].

In the past there was only the UI language in the `hl` argument but nowadays it
seems a combination of the UI language and the "search region" is mixed in this
argument and the `gl` argument has been removed.  I'm very surprised that google
is starting to mix the parameters of the UI with the parameters of the search
index.

This patch modifies the get_google_info(..) function.  Beside Google-WEB this
function is also used by other Google services, here are some examples to test
region & language of ..

- Google-WEB:    `!go dragon boat :en-CA`
- Google-News:   `!gon dragon boat :en-CA`
- Google-Videos: `!gov bmw :en-CA`
- Goolge-Images  `!goi bmw :en-CA`

- [1] https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
- [2] https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
Markus Heiser e8706fb738 [fix] engine & network issues / documentation and type annotations
This patch fixes some quirks and issues related to the engines and the network.
Each engine has its own network and this network was broken for the following
engines[1]:

- archlinux
- bing
- dailymotion
- duckduckgo
- google
- peertube
- startpage
- wikipedia

Since the files have been touched anyway, the type annotaions of the engine
modules has also been completed so that error messages from the type checker are
no longer reported.

Related and (partial) fixed issue:

- [1] https://github.com/searxng/searxng/issues/762#issuecomment-1605323861
- [2] https://github.com/searxng/searxng/issues/2513
- [3] https://github.com/searxng/searxng/issues/2515

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1 year ago
pankaj 4900c091a6 use logger.warning
logger.warn() is depricated.
logger.warning is already being used in some files.
1 year ago