Commit Graph

391 Commits (28286cf3f2308113bf440fb6e7cf326c6ed07889)

Author SHA1 Message Date
Adam Tauber 06b754ad67 [mod] increase lobste.rs engine timeout to avoid timeouts most of the time 3 years ago
Adam Tauber 6255b33c9d [fix] rewrite hoogle to use html/xpath instead of json
the json response has been changed and it contains html chunks which is
not compatible with our json engine, so we have to switch to html/xpath
parsing
3 years ago
Adam Tauber 45f0e1a859 [fix] update geektimes.ru url - it redirects to habr.com 3 years ago
Adam Tauber 50ba2b9e87 [fix] update google play movies xpath 3 years ago
Adam Tauber 88657fe9c2 [fix] update google play apps xpath 3 years ago
James Higginbotham ce6eb81a71
Update settings.yml to enable HTTP for yacy
Added a line to the yacy entry to enable HTTP if the local yacy instance isn't using HTTPS. Otherwise, an error will be thrown in the logs: "No connection adapters were found for 'http://localhost:8090/yacysearch.json...'". This is likely related to ticket #2641 that forces HTTPS by default.
3 years ago
Adam Tauber dd34ac396c
Merge pull request #2652 from kvch/solr-engine
Add Apache Solr engine
3 years ago
Alexandre Flament 5b176b3496
Merge pull request #2659 from MarcAbonce/onions-http-fix
Fix HTTP error in onion engines
3 years ago
Marc Abonce Seguin f4a0a4d756 fix HTTP error in onion engines
regression from https://github.com/searx/searx/pull/2641
most onion websites only serve HTTP, so it must be enabled
3 years ago
Markus Heiser 3703ebb22a [drop] Acgsou engine - www.acgsou.com no longer exists
- https://www.acgsou.com/ acgsou.com is redirected to 36dm.club
- @rinpatch do not plan on maintaining the engine [1]

[1] https://github.com/searx/searx/pull/1283#issuecomment-798783585

Signed-off-by: Markus Heiser <markus@darmarit.de>
3 years ago
Noémi Ványi ff527e2681 Add Solr engine 3 years ago
Alexandre Flament 92dd5e245e
Merge pull request #2626 from mikeri/solidtorrents
Add Solid Torrents engine
3 years ago
Alexandre Flament 99e0651cea [mod] by default allow only HTTPS, not HTTP
Related to https://github.com/searx/searx/pull/2373
3 years ago
Michael Ilsaas 5549d58de3 Add Solid Torrents engine 3 years ago
Markus Heiser d48e2e7b0b [enh] google scholar - python implementation of the engine
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
GazoilKerozen 5f6ac3afa2
Add Freesound engine (#2596)
Add freesound engine with player.

Co-authored-by: Gazoil <maildeguzel@gmail.com>
3 years ago
datagram1 1d0a32a2c5 Added rumble.com video search engine. TODO video embedding.
Update rumble.py

some lines too long.

Disable Rumble engine

disabled : True

PEP8 fix

change line spacing
3 years ago
Emilien Devos 4b37e10dd9 fix yggtorrent url + add it back 3 years ago
Alexandre Flament 45027765e3
Merge pull request #2566 from dalf/remove-yandex
[remove] yandex engine
3 years ago
Alexandre Flament d76660463b
Merge pull request #2562 from dalf/mod-json-engine
[mod] json_engine: add content_html_to_text and title_html_to_text
3 years ago
Alexandre Flament 2b60d0d243
Merge pull request #2564 from dalf/fix-seznam
[fix] fix seznam engine
3 years ago
Alexandre Flament 74c8b5606f
Merge pull request #2541 from return42/mediathekviewweb
[enh] add engine MediathekViewWeb (API)
3 years ago
Alexandre Flament 5d9db6c2f7 [remove] yandex engine 3 years ago
Alexandre Flament 35dd069402 [fix] fix seznam engine
no paging support
3 years ago
Alexandre Flament ff84a1af35 [mod] json_engine: add content_html_to_text and title_html_to_text
Some JSON API returns HTML in either in the HTML or the content.
This commit adds two new parameters to the json_engine:
content_html_to_text and title_html_to_text, False by default.

If True, then the searx.utils.html_to_text removes the HTML tags.

Update crossref, openairedatasets and openairepublications engines
3 years ago
Alexandre Flament 436d366448
Merge pull request #2544 from mrwormo/congresslibrary
[Engine] Add Library of Congress engine
3 years ago
Markus Heiser bc1be3f0e9 [enh] add engine MediathekViewWeb (API)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
mrwormo 051da88328 Add Library of Congress engine 3 years ago
Alexandre Flament 9211cdfe9b [upd] remove google_play_music engine
Google Play Music has been replaced by Youtube music.
3 years ago
Noémi Ványi ac309f5b8d Fix naver engine
Closes #2540
3 years ago
Markus Heiser 41c03cf011 [drop] metager - xpath engine won't work anymore
The new version of MetaGer needs to reload the reults (into a iframe) with a
unique tag (see HTML response below).

Implementing a dedicated metager-engine for searx makes no sense to me. The
great days of MetaGer seems to be ended.  I remember the good old days this
project started in the 90's of the last century.  But in the last few years it
becomes more and more crap.  As the name suggested, MetaGer was made for
germans in the first place.  They have added a english and spain translation but
the i18n is very poor compared to what searx offers.

It's a pity, lets drop MetaGer.

This is the first response, the id (b82679980656899ba5a17ffd02a56846) is unique
for each query:

    $ curl "https://metager.org/meta/meta.ger3?eingabe=foo&submit-query=&focus=web"
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <link rel="stylesheet" href="/index.css?id=b82679980656899ba5a17ffd02a56846">
        <script src="/index.js?id=b82679980656899ba5a17ffd02a56846"></script>
    <title>foo - MetaGer</title>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
    </head>
    <body>
        <iframe id="mg-framed" src="https://metager.org/meta/meta.ger3?eingabe=foo&amp;submit-query=&amp;focus=web&amp;mgv=b82679980656899ba5a17ffd02a56846" autofocus="true" onload="this.contentWindow.focus();"></iframe>
     </body>

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
mrwormo c4c1636b18 Add Creative Commons search engine 3 years ago
Markus Heiser e436287385 [mod] checker: add some additional tests
BTW: fix indentation by 2 spaces

The additional tests has been commented out in the google engines to not release
any CAPTCHA issues.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Alexandre Flament 73c86f9bf2 [mod] checker: disable by default 3 years ago
Alexandre Flament 80d7411f2c
Merge pull request #2452 from kvch/add-wilby-engine
Add wiby.me engine
3 years ago
Alexandre Flament b405646749
Merge pull request #2451 from mrwormo/invidious-engine
[Fix] Invidious Engine
3 years ago
Alexandre Flament a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
3 years ago
mrwormo 2dff3887f0 [fix] Invidious engine by enabling requests by randomly picking amongst working instances 3 years ago
Alexandre Flament 3a9f513521 [enh] checker: background check
See settings.yml for the options
SIGUSR1 signal starts the checker.
The result is available at /stats/checker
3 years ago
Alexandre Flament ca0889d488 [enh] checker: wikidata & ddd: add specific tests 3 years ago
Alexandre Flament 16a889dd8f [enh] checker: add rosebud test 3 years ago
Alexandre Flament 424e6abc7e [mod] settings.yml: move brand settings to a dedicated section 3 years ago
Markus Heiser 9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Noémi Ványi a6dd1de4a8 Add wiby.me engine
Closes #2339
3 years ago
Markus Heiser b0bb0a3a0f [fix] Library Genesis links shifted by 1 #1998
Fixes: #1998
Suggested-by: @linuxmue
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
3 years ago
Émilien Devos fc6cfc3b58
Remove voat due to its shutdown
Voat shutted down on December 25th, 2020 at 12 noon PST: https://voat.co/host/voat/static/inactive.min.html?ReturnUrl=/
3 years ago
Alexandre Flament 36600118fb
Merge pull request #2372 from dalf/remove-broken-engines
[remove] remove searchcode_doc and twitter
4 years ago
Alexandre Flament d703119d3a [enh] add raise_for_httperror
check HTTP response:
* detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time.
* otherwise raise HTTPError as before

the check is done in poolrequests.py (was before in search.py).

update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
4 years ago
Alexandre Flament 1c9e7cef50 [remove] remove searchcode_doc and twitter
* twitter: the API has changed. the engine needs to rewritten.
* searchcode_doc: the API about documentation doesn't exist anymore.
4 years ago
Alexandre Flament 42a194898b
Merge pull request #2360 from dalf/update-libgen
[mod] libgen: update the URL to http://libgen.rs/
4 years ago