forked from Archives/searxng
Compare commits
No commits in common. 'master' and 'v0.16.0' have entirely different histories.
@ -0,0 +1,31 @@
|
||||
comment: false
|
||||
coverage:
|
||||
status:
|
||||
project:
|
||||
default:
|
||||
# basic
|
||||
target: auto
|
||||
threshold: null
|
||||
base: auto
|
||||
# advanced
|
||||
branches: null
|
||||
if_no_uploads: error
|
||||
if_not_found: success
|
||||
if_ci_failed: error
|
||||
only_pulls: false
|
||||
flags: null
|
||||
paths: null
|
||||
patch:
|
||||
default:
|
||||
# basic
|
||||
target: auto
|
||||
threshold: null
|
||||
base: auto
|
||||
# advanced
|
||||
branches: null
|
||||
if_no_uploads: error
|
||||
if_not_found: success
|
||||
if_ci_failed: error
|
||||
only_pulls: false
|
||||
flags: null
|
||||
paths: null
|
||||
@ -1,12 +1,10 @@
|
||||
*.gif -diff
|
||||
*.png -diff
|
||||
*.min.css -diff
|
||||
*.min.js -diff
|
||||
*.css.map -diff
|
||||
*.js.map -diff
|
||||
*.eot -diff
|
||||
*.svg -diff
|
||||
*.ttf -diff
|
||||
*.woff -diff
|
||||
*.woff2 -diff
|
||||
messages.mo -diff
|
||||
searx/static/css/bootstrap.min.css -diff
|
||||
searx/static/js/bootstrap.min.js -diff
|
||||
searx/static/themes/oscar/css/logicodev.min.css -diff
|
||||
searx/static/themes/oscar/css/leaflet.min.css -diff
|
||||
searx/static/themes/oscar/css/pointhi.min.css -diff
|
||||
searx/static/themes/oscar/js/searx.min.js -diff
|
||||
searx/static/themes/simple/css/searx.min.css -diff
|
||||
searx/static/themes/simple/css/searx-rtl.min.css -diff
|
||||
searx/static/themes/simple/js/searx.min.js -diff
|
||||
searx/static/themes/simple/js/searx.min.js.map -diff
|
||||
|
||||
@ -1,39 +0,0 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Report a bug in SearXNG
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
|
||||
|
||||
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
|
||||
<!-- If you are running on master branch using git execute this command
|
||||
in order to fetch the latest commit ID:
|
||||
```
|
||||
git log -1
|
||||
```
|
||||
If you are using searxng-docker then look at the bottom of the SearXNG page
|
||||
and check for the version after "Powered by SearXNG"
|
||||
|
||||
Please also stipulate if you are using a forked version of SearXNG and
|
||||
include a link to the fork source code.
|
||||
-->
|
||||
**How did you install SearXNG?**
|
||||
<!-- Did you install SearXNG using the official wiki or using searxng-docker
|
||||
or manually by executing the searx/webapp.py file? -->
|
||||
**What happened?**
|
||||
<!-- A clear and concise description of what the bug is. -->
|
||||
|
||||
**How To Reproduce**
|
||||
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
|
||||
|
||||
**Expected behavior**
|
||||
<!-- A clear and concise description of what you expected to happen. -->
|
||||
|
||||
**Screenshots & Logs**
|
||||
<!-- If applicable, add screenshots, logs to help explain your problem. -->
|
||||
|
||||
**Additional context**
|
||||
<!-- Add any other context about the problem here. -->
|
||||
@ -1,5 +0,0 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: Questions & Answers (Q&A)
|
||||
url: https://github.com/searxng/searxng/discussions/categories/q-a
|
||||
about: Ask questions and find answers
|
||||
@ -1,31 +0,0 @@
|
||||
---
|
||||
name: Engine request
|
||||
about: Request a new engine in SearXNG
|
||||
title: ''
|
||||
labels: enhancement, engine request
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
|
||||
|
||||
**Working URL to the engine**
|
||||
<!-- Please check if the engine is responding correctly before submitting it. -->
|
||||
|
||||
**Why do you want to add this engine?**
|
||||
<!-- What's special about this engine? Is it open source or libre? -->
|
||||
|
||||
**Features of this engine**
|
||||
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->
|
||||
|
||||
**How can SearXNG fetch the information from this engine?**
|
||||
<!-- List API URL, example code (using the correct markdown) and more
|
||||
that could be useful for the developers in order to implement this engine.
|
||||
If you don't know what to write, let this part blank.>
|
||||
|
||||
**Applicable category of this engine**
|
||||
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:
|
||||
general, files, images, it, map, music, news, science, social media and videos.
|
||||
You can add multiple categories at the same time. -->
|
||||
|
||||
**Additional context**
|
||||
<!-- Add any other context about this engine here. -->
|
||||
@ -1,21 +0,0 @@
|
||||
---
|
||||
name: Feature request
|
||||
about: Request a new feature in SearXNG
|
||||
title: ''
|
||||
labels: enhancement
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
|
||||
|
||||
**Is your feature request related to a problem? Please describe.**
|
||||
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
|
||||
|
||||
**Describe the solution you'd like**
|
||||
<!-- A clear and concise description of what you want to happen. -->
|
||||
|
||||
**Describe alternatives you've considered**
|
||||
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
|
||||
|
||||
**Additional context**
|
||||
<!-- Add any other context or screenshots about the feature request here. -->
|
||||
@ -1,17 +0,0 @@
|
||||
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "friday"
|
||||
open-pull-requests-limit: 5
|
||||
target-branch: "master"
|
||||
- package-ecosystem: "npm"
|
||||
directory: "/searx/static/themes/simple"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "friday"
|
||||
open-pull-requests-limit: 5
|
||||
target-branch: "master"
|
||||
@ -1,67 +0,0 @@
|
||||
name: "Update searx.data"
|
||||
on:
|
||||
schedule:
|
||||
- cron: "59 23 28 * *"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
updateData:
|
||||
name: Update data - ${{ matrix.fetch }}
|
||||
runs-on: ubuntu-20.04
|
||||
if: ${{ github.repository_owner == 'searxng'}}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
fetch:
|
||||
- update_ahmia_blacklist.py
|
||||
- update_currencies.py
|
||||
- update_external_bangs.py
|
||||
- update_firefox_version.py
|
||||
- update_languages.py
|
||||
- update_wikidata_units.py
|
||||
- update_engine_descriptions.py
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Install Ubuntu packages
|
||||
run: |
|
||||
sudo ./utils/searxng.sh install packages
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
make V=1 install
|
||||
|
||||
- name: Fetch data
|
||||
env:
|
||||
FETCH_SCRIPT: ./searxng_extra/update/${{ matrix.fetch }}
|
||||
run: |
|
||||
V=1 ./manage pyenv.cmd python "$FETCH_SCRIPT"
|
||||
|
||||
- name: Create Pull Request
|
||||
id: cpr
|
||||
uses: peter-evans/create-pull-request@v3
|
||||
with:
|
||||
commit-message: Update searx.data - ${{ matrix.fetch }}
|
||||
committer: searxng-bot <noreply@github.com>
|
||||
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
|
||||
signoff: false
|
||||
branch: update_data_${{ matrix.fetch }}
|
||||
delete-branch: true
|
||||
draft: false
|
||||
title: 'Update searx.data - ${{ matrix.fetch }}'
|
||||
body: |
|
||||
Update searx.data - ${{ matrix.fetch }}
|
||||
labels: |
|
||||
data
|
||||
|
||||
- name: Check outputs
|
||||
run: |
|
||||
echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
|
||||
echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"
|
||||
@ -1,214 +0,0 @@
|
||||
name: Integration
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["master"]
|
||||
pull_request:
|
||||
branches: ["master"]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
python:
|
||||
name: Python ${{ matrix.python-version }}
|
||||
runs-on: ubuntu-20.04
|
||||
strategy:
|
||||
matrix:
|
||||
os: [ubuntu-20.04]
|
||||
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
- name: Install Ubuntu packages
|
||||
run: |
|
||||
sudo ./utils/searxng.sh install packages
|
||||
sudo apt install firefox
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('requirements*.txt', 'setup.py') }}
|
||||
- name: Install Python dependencies
|
||||
if: steps.cache-python.outputs.cache-hit != 'true'
|
||||
run: |
|
||||
make V=1 install
|
||||
make V=1 gecko.driver
|
||||
- name: Run tests
|
||||
run: make V=1 ci.test
|
||||
- name: Test coverage
|
||||
run: make V=1 test.coverage
|
||||
- name: Store coverage result
|
||||
uses: actions/upload-artifact@v2
|
||||
with:
|
||||
name: coverage-${{ matrix.python-version }}
|
||||
path: coverage/
|
||||
retention-days: 60
|
||||
|
||||
themes:
|
||||
name: Themes
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
- name: Install Ubuntu packages
|
||||
run: sudo ./utils/searxng.sh install buildhost
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
|
||||
- name: Install node dependencies
|
||||
run: make V=1 node.env
|
||||
- name: Build themes
|
||||
run: make V=1 themes.all
|
||||
|
||||
documentation:
|
||||
name: Documentation
|
||||
runs-on: ubuntu-20.04
|
||||
permissions:
|
||||
contents: write # for JamesIves/github-pages-deploy-action to push changes in repo
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
fetch-depth: '0'
|
||||
persist-credentials: false
|
||||
- name: Install Ubuntu packages
|
||||
run: sudo ./utils/searxng.sh install buildhost
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
|
||||
- name: Build documentation
|
||||
run: |
|
||||
make V=1 docs.clean docs.html
|
||||
- name: Deploy
|
||||
if: github.ref == 'refs/heads/master'
|
||||
uses: JamesIves/github-pages-deploy-action@3.7.1
|
||||
with:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
BRANCH: gh-pages
|
||||
FOLDER: dist/docs
|
||||
CLEAN: true # Automatically remove deleted files from the deploy branch
|
||||
SINGLE_COMMIT: True
|
||||
COMMIT_MESSAGE: build from commit ${{ github.sha }}
|
||||
|
||||
babel:
|
||||
name: Update translations branch
|
||||
runs-on: ubuntu-20.04
|
||||
if: ${{ github.repository_owner == 'searxng' && github.ref == 'refs/heads/master' }}
|
||||
needs:
|
||||
- python
|
||||
- themes
|
||||
- documentation
|
||||
permissions:
|
||||
contents: write # for make V=1 weblate.push.translations
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
fetch-depth: '0'
|
||||
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
|
||||
- name: weblate & git setup
|
||||
env:
|
||||
WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
|
||||
run: |
|
||||
mkdir -p ~/.config
|
||||
echo "${WEBLATE_CONFIG}" > ~/.config/weblate
|
||||
git config --global user.email "searxng-bot@users.noreply.github.com"
|
||||
git config --global user.name "searxng-bot"
|
||||
- name: Update transations
|
||||
id: update
|
||||
run: |
|
||||
git restore utils/brand.env
|
||||
make V=1 weblate.push.translations
|
||||
|
||||
dockers:
|
||||
name: Docker
|
||||
if: github.ref == 'refs/heads/master'
|
||||
needs:
|
||||
- python
|
||||
- themes
|
||||
- documentation
|
||||
env:
|
||||
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout
|
||||
if: env.DOCKERHUB_USERNAME != null
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
# make sure "make docker.push" can get the git history
|
||||
fetch-depth: '0'
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
|
||||
- name: Set up QEMU
|
||||
if: env.DOCKERHUB_USERNAME != null
|
||||
uses: docker/setup-qemu-action@v1
|
||||
- name: Set up Docker Buildx
|
||||
if: env.DOCKERHUB_USERNAME != null
|
||||
uses: docker/setup-buildx-action@v1
|
||||
- name: Login to DockerHub
|
||||
if: env.DOCKERHUB_USERNAME != null
|
||||
uses: docker/login-action@v1
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
- name: Build and push
|
||||
if: env.DOCKERHUB_USERNAME != null
|
||||
run: make -e GIT_URL=$(git remote get-url origin) docker.buildx
|
||||
@ -1,28 +0,0 @@
|
||||
name: "Security checks"
|
||||
on:
|
||||
schedule:
|
||||
- cron: "42 05 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
dockers:
|
||||
name: Trivy ${{ matrix.image }}
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Run Trivy vulnerability scanner
|
||||
uses: aquasecurity/trivy-action@master
|
||||
with:
|
||||
image-ref: 'searxng/searxng:latest'
|
||||
ignore-unfixed: false
|
||||
vuln-type: 'os,library'
|
||||
severity: 'UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL'
|
||||
format: 'sarif'
|
||||
output: 'trivy-results.sarif'
|
||||
|
||||
- name: Upload Trivy scan results to GitHub Security tab
|
||||
uses: github/codeql-action/upload-sarif@v2
|
||||
with:
|
||||
sarif_file: 'trivy-results.sarif'
|
||||
@ -1,59 +0,0 @@
|
||||
name: "Update translations"
|
||||
on:
|
||||
schedule:
|
||||
- cron: "05 07 * * 5"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
babel:
|
||||
name: "create PR for additons from weblate"
|
||||
runs-on: ubuntu-20.04
|
||||
if: ${{ github.repository_owner == 'searxng' && github.ref == 'refs/heads/master' }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
fetch-depth: '0'
|
||||
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
architecture: 'x64'
|
||||
- name: Cache Python dependencies
|
||||
id: cache-python
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
./local
|
||||
./.nvm
|
||||
./node_modules
|
||||
key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
|
||||
- name: weblate & git setup
|
||||
env:
|
||||
WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
|
||||
run: |
|
||||
mkdir -p ~/.config
|
||||
echo "${WEBLATE_CONFIG}" > ~/.config/weblate
|
||||
git config --global user.email "searxng-bot@users.noreply.github.com"
|
||||
git config --global user.name "searxng-bot"
|
||||
- name: Merge and push transation updates
|
||||
run: |
|
||||
make V=1 weblate.translations.commit
|
||||
- name: Create Pull Request
|
||||
id: cpr
|
||||
uses: peter-evans/create-pull-request@v3
|
||||
with:
|
||||
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
|
||||
commit-message: Update translations
|
||||
committer: searxng-bot <searxng-bot@users.noreply.github.com>
|
||||
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
|
||||
signoff: false
|
||||
branch: translations_update
|
||||
delete-branch: true
|
||||
draft: false
|
||||
title: 'Update translations'
|
||||
body: |
|
||||
Update translations
|
||||
labels: |
|
||||
translation
|
||||
@ -1,25 +1,26 @@
|
||||
# to sync with .dockerignore & pyrightconfig.json
|
||||
# to sync with .dockerignore
|
||||
.coverage
|
||||
coverage/
|
||||
.installed.cfg
|
||||
engines.cfg
|
||||
env
|
||||
searx-ve
|
||||
robot_log.html
|
||||
robot_output.xml
|
||||
robot_report.html
|
||||
test_basic/
|
||||
setup.cfg
|
||||
|
||||
*.pyc
|
||||
*/*.pyc
|
||||
*~
|
||||
*.swp
|
||||
geckodriver.log
|
||||
|
||||
.coverage
|
||||
coverage/
|
||||
node_modules/
|
||||
|
||||
.tx/
|
||||
|
||||
.nvm/
|
||||
cache/
|
||||
build/
|
||||
dist/
|
||||
local/
|
||||
gh-pages/
|
||||
*.egg-info/
|
||||
|
||||
/package-lock.json
|
||||
/node_modules/
|
||||
|
||||
.idea/
|
||||
|
||||
searx/version_frozen.py
|
||||
searx.egg-info/
|
||||
|
||||
@ -0,0 +1,3 @@
|
||||
strictness: high
|
||||
ignore-paths:
|
||||
- bootstrap.py
|
||||
@ -0,0 +1,56 @@
|
||||
language: python
|
||||
sudo: false
|
||||
cache:
|
||||
- pip
|
||||
- npm
|
||||
- directories:
|
||||
- $HOME/.cache/pip
|
||||
|
||||
addons:
|
||||
firefox: "latest"
|
||||
|
||||
install:
|
||||
- ./manage.sh install_geckodriver ~/drivers
|
||||
- export PATH=~/drivers:$PATH
|
||||
- ./manage.sh npm_packages
|
||||
- ./manage.sh update_dev_packages
|
||||
- pip install codecov
|
||||
script:
|
||||
- ./manage.sh styles
|
||||
- ./manage.sh grunt_build
|
||||
- ./manage.sh tests
|
||||
after_success:
|
||||
- ./manage.sh py_test_coverage
|
||||
- codecov
|
||||
|
||||
stages:
|
||||
- test
|
||||
- name: docker
|
||||
if: branch = master AND type != pull_request AND env(DOCKER_USERNAME) IS present
|
||||
|
||||
jobs:
|
||||
include:
|
||||
- python: "2.7"
|
||||
- python: "3.5"
|
||||
- python: "3.6"
|
||||
- stage: docker
|
||||
python: "3.6"
|
||||
git:
|
||||
depth: false
|
||||
services:
|
||||
- docker
|
||||
addons: []
|
||||
before_install: true
|
||||
install: true
|
||||
script:
|
||||
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
|
||||
- ./manage.sh docker_build push
|
||||
after_success: true
|
||||
|
||||
notifications:
|
||||
irc:
|
||||
channels:
|
||||
- "irc.freenode.org#searx"
|
||||
template:
|
||||
- "%{repository}/#%{build_number}/%{branch} (%{author}): %{message} %{build_url}"
|
||||
on_success: change
|
||||
@ -1,3 +0,0 @@
|
||||
[weblate]
|
||||
url = https://translate.codeberg.org/api/
|
||||
translation = searxng/searxng
|
||||
@ -1,16 +0,0 @@
|
||||
extends: default
|
||||
|
||||
rules:
|
||||
|
||||
indentation:
|
||||
spaces: 2
|
||||
|
||||
# 120 chars should be enough, but don't fail if a line is longer
|
||||
line-length:
|
||||
max: 120
|
||||
level: warning
|
||||
allow-non-breakable-words: true
|
||||
|
||||
# we don't have multiple document per file
|
||||
document-start: disable
|
||||
document-end: disable
|
||||
@ -1,49 +0,0 @@
|
||||
# How to contribute
|
||||
|
||||
## Resources in the documentation
|
||||
|
||||
* [Development quickstart](https://docs.searxng.org/dev/contribution_guide.html)
|
||||
* [Contribution guide](https://docs.searxng.org/dev/contribution_guide.html)
|
||||
|
||||
## Submitting PRs
|
||||
|
||||
Please follow the provided PR template when writing a description for your changes.
|
||||
|
||||
Do not take criticism personally. When you get feedback, it is about your work,
|
||||
not your character, personality, etc. Keep in mind we all want to make the project better.
|
||||
|
||||
When something is not clear, please ask questions to clear things up.
|
||||
|
||||
If you would like to introduce a big architectural changes or do a refactoring
|
||||
either in the codebase or the development tools, please open an issue with a proposal
|
||||
first. This way we can think together about the problem and probably come up
|
||||
with a better solution.
|
||||
|
||||
## Coding conventions and guidelines
|
||||
|
||||
### Commit messages
|
||||
|
||||
* Always write descriptive commit messages ("fix bug" is not acceptable).
|
||||
* Use the present tense ("Add feature" not "Added feature").
|
||||
* Use the imperative mood ("Move cursor to..." not "Moves cursor to...").
|
||||
* Limit the first line to 72 characters or less.
|
||||
* Include the number of the issue you are fixing.
|
||||
|
||||
### Coding guidelines
|
||||
|
||||
As a Python project, we must follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) and [PEP 20](https://www.python.org/dev/peps/pep-0020/) guidelines.
|
||||
|
||||
Furthermore, follow the Clean code conventions. The most important
|
||||
in this project are the following rules:
|
||||
|
||||
* Simpler is better. [KISS principle](https://en.wikipedia.org/wiki/KISS_principle)
|
||||
* Be consistent.
|
||||
* Every function must do one thing.
|
||||
* Use descriptive names for functions and variables.
|
||||
* Always look for the root cause.
|
||||
* Keep configurable data high level.
|
||||
* Avoid negative conditionals.
|
||||
* Prefer fewer arguments.
|
||||
* Do not add obvious comment to code.
|
||||
* Do not comment out code, just delete lines.
|
||||
|
||||
@ -1,104 +1,89 @@
|
||||
# -*- coding: utf-8; mode: makefile-gmake -*-
|
||||
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
.DEFAULT_GOAL=help
|
||||
export MTOOLS=./manage
|
||||
export GIT_URL=https://github.com/asciimoo/searx
|
||||
export SEARX_URL=https://searx.me
|
||||
export DOCS_URL=https://asciimoo.github.io/searx
|
||||
|
||||
PYOBJECTS = searx
|
||||
DOC = docs
|
||||
PY_SETUP_EXTRAS ?= \[test\]
|
||||
|
||||
PYDIST=./dist/py
|
||||
PYBUILD=./build/py
|
||||
|
||||
include utils/makefile.include
|
||||
include utils/makefile.python
|
||||
include utils/makefile.sphinx
|
||||
|
||||
all: clean install
|
||||
|
||||
PHONY += help
|
||||
|
||||
help:
|
||||
@./manage --help
|
||||
@echo '----'
|
||||
@echo 'run - run developer instance'
|
||||
@echo 'install - developer install of SearxNG into virtualenv'
|
||||
@echo 'uninstall - uninstall developer installation'
|
||||
@echo 'clean - clean up working tree'
|
||||
@echo 'search.checker - check search engines'
|
||||
@echo 'test - run shell & CI tests'
|
||||
@echo 'test.shell - test shell scripts'
|
||||
@echo 'ci.test - run CI tests'
|
||||
@echo ' test - run developer tests'
|
||||
@echo ' docs - build documentation'
|
||||
@echo ' docs-live - autobuild HTML documentation while editing'
|
||||
@echo ' run - run developer instance'
|
||||
@echo ' install - developer install (./local)'
|
||||
@echo ' uninstall - uninstall (./local)'
|
||||
@echo ' gh-pages - build docs & deploy on gh-pages branch'
|
||||
@echo ' clean - drop builds and environments'
|
||||
@echo ''
|
||||
@$(MAKE) -s -f utils/makefile.include make-help
|
||||
@echo ''
|
||||
@$(MAKE) -s -f utils/makefile.python python-help
|
||||
|
||||
PHONY += install
|
||||
install: pyenvinstall
|
||||
|
||||
PHONY += uninstall
|
||||
uninstall: pyenvuninstall
|
||||
|
||||
PHONY += clean
|
||||
clean: pyclean
|
||||
$(call cmd,common_clean)
|
||||
|
||||
PHONY += run
|
||||
run: install
|
||||
$(Q)./manage webapp.run
|
||||
run: pyenvinstall
|
||||
$(Q) ( \
|
||||
sed -i -e "s/debug : False/debug : True/g" ./searx/settings.yml ; \
|
||||
sleep 2 ; \
|
||||
xdg-open http://127.0.0.1:8888/ ; \
|
||||
sleep 3 ; \
|
||||
sed -i -e "s/debug : True/debug : False/g" ./searx/settings.yml ; \
|
||||
) &
|
||||
$(PY_ENV)/bin/python ./searx/webapp.py
|
||||
|
||||
PHONY += install uninstall
|
||||
install uninstall:
|
||||
$(Q)./manage pyenv.$@
|
||||
# docs
|
||||
# ----
|
||||
|
||||
PHONY += clean
|
||||
clean: py.clean docs.clean node.clean nvm.clean test.clean
|
||||
$(Q)./manage build_msg CLEAN "common files"
|
||||
$(Q)find . -name '*.orig' -exec rm -f {} +
|
||||
$(Q)find . -name '*.rej' -exec rm -f {} +
|
||||
$(Q)find . -name '*~' -exec rm -f {} +
|
||||
$(Q)find . -name '*.bak' -exec rm -f {} +
|
||||
|
||||
lxc.clean:
|
||||
$(Q)rm -rf lxc-env
|
||||
|
||||
PHONY += search.checker search.checker.%
|
||||
search.checker: install
|
||||
$(Q)./manage pyenv.cmd searx-checker -v
|
||||
|
||||
search.checker.%: install
|
||||
$(Q)./manage pyenv.cmd searx-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))"
|
||||
|
||||
PHONY += test ci.test test.shell
|
||||
ci.test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.pybabel
|
||||
test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.shell
|
||||
test.shell:
|
||||
$(Q)shellcheck -x -s dash \
|
||||
dockerfiles/docker-entrypoint.sh
|
||||
$(Q)shellcheck -x -s bash \
|
||||
utils/brand.env \
|
||||
$(MTOOLS) \
|
||||
utils/lib.sh \
|
||||
utils/lib_nvm.sh \
|
||||
utils/lib_static.sh \
|
||||
utils/lib_go.sh \
|
||||
utils/lib_redis.sh \
|
||||
utils/filtron.sh \
|
||||
utils/searx.sh \
|
||||
utils/searxng.sh \
|
||||
utils/morty.sh \
|
||||
utils/lxc.sh \
|
||||
utils/lxc-searxng.env
|
||||
$(Q)$(MTOOLS) build_msg TEST "$@ OK"
|
||||
|
||||
|
||||
# wrap ./manage script
|
||||
|
||||
MANAGE += buildenv
|
||||
MANAGE += weblate.translations.commit weblate.push.translations
|
||||
MANAGE += data.all data.languages data.useragents data.osm_keys_tags
|
||||
MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean
|
||||
MANAGE += docker.build docker.push docker.buildx
|
||||
MANAGE += gecko.driver
|
||||
MANAGE += node.env node.clean
|
||||
MANAGE += py.build py.clean
|
||||
MANAGE += pyenv pyenv.install pyenv.uninstall
|
||||
MANAGE += pypi.upload pypi.upload.test
|
||||
MANAGE += format.python
|
||||
MANAGE += test.yamllint test.pylint test.pyright test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean
|
||||
MANAGE += themes.all themes.simple themes.simple.test pygments.less
|
||||
MANAGE += static.build.commit static.build.drop static.build.restore
|
||||
MANAGE += nvm.install nvm.clean nvm.status nvm.nodejs
|
||||
|
||||
PHONY += $(MANAGE)
|
||||
|
||||
$(MANAGE):
|
||||
$(Q)$(MTOOLS) $@
|
||||
|
||||
# short hands of selected targets
|
||||
|
||||
PHONY += docs docker themes
|
||||
|
||||
docs: docs.html
|
||||
docker: docker.build
|
||||
themes: themes.all
|
||||
PHONY += docs
|
||||
docs: pyenvinstall sphinx-doc
|
||||
$(call cmd,sphinx,html,docs,docs)
|
||||
|
||||
PHONY += docs-live
|
||||
docs-live: pyenvinstall sphinx-live
|
||||
$(call cmd,sphinx_autobuild,html,docs,docs)
|
||||
|
||||
$(GH_PAGES)::
|
||||
@echo "doc available at --> $(DOCS_URL)"
|
||||
|
||||
# test
|
||||
# ----
|
||||
|
||||
PHONY += test test.pylint test.pep8 test.unit test.robot
|
||||
|
||||
# TODO: balance linting with pylint
|
||||
test: test.pep8 test.unit test.robot
|
||||
- make pylint
|
||||
|
||||
test.pep8: pyenvinstall
|
||||
$(PY_ENV_ACT); ./manage.sh pep8_check
|
||||
|
||||
test.unit: pyenvinstall
|
||||
$(PY_ENV_ACT); ./manage.sh unit_tests
|
||||
|
||||
test.robot: pyenvinstall
|
||||
$(PY_ENV_ACT); ./manage.sh install_geckodriver
|
||||
$(PY_ENV_ACT); ./manage.sh robot_tests
|
||||
|
||||
.PHONY: $(PHONY)
|
||||
|
||||
@ -1,25 +0,0 @@
|
||||
## What does this PR do?
|
||||
|
||||
<!-- MANDATORY -->
|
||||
|
||||
<!-- explain the changes in your PR, algorithms, design, architecture -->
|
||||
|
||||
## Why is this change important?
|
||||
|
||||
<!-- MANDATORY -->
|
||||
|
||||
<!-- explain the motivation behind your PR -->
|
||||
|
||||
## How to test this PR locally?
|
||||
|
||||
<!-- commands to run the tests or instructions to test the changes-->
|
||||
|
||||
## Author's checklist
|
||||
|
||||
<!-- additional notes for reviewiers -->
|
||||
|
||||
## Related issues
|
||||
|
||||
<!--
|
||||
Closes #234
|
||||
-->
|
||||
@ -1,10 +0,0 @@
|
||||
# Security Policy
|
||||
|
||||
We love responsible reports of (potential) security issues in SearXNG.
|
||||
|
||||
You can contact us at security@searxng.org.
|
||||
|
||||
Be sure to provide as much information as possible and if found
|
||||
also reproduction steps of the identified vulnerability. Also
|
||||
add the specific URL of the project as well as code you found
|
||||
the issue in to your report.
|
||||
@ -1,6 +1,3 @@
|
||||
[extractors]
|
||||
searxng_msg = searx.babel_extract.extract
|
||||
[ignore: **/node_modules/**]
|
||||
[python: **.py]
|
||||
[jinja2: **/templates/**.html]
|
||||
[searxng_msg: **/searxng.msg]
|
||||
extensions=jinja2.ext.autoescape,jinja2.ext.with_
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[theme]
|
||||
inherit = pocoo
|
||||
stylesheet = searxng.css
|
||||
stylesheet = searx.css
|
||||
|
||||
[options]
|
||||
touch_icon =
|
||||
@ -1,30 +1,33 @@
|
||||
digraph G {
|
||||
|
||||
node [style=filled, shape=box, fillcolor="#ffffcc", fontname=Sans];
|
||||
node [style=filled, shape=box, fillcolor="#ffffcc", fontname="Sans"];
|
||||
edge [fontname="Sans"];
|
||||
|
||||
browser [label="browser", shape=tab, fillcolor=aliceblue];
|
||||
rp [label="reverse proxy"];
|
||||
static [label="static files", shape=folder, href="url to configure static files", fillcolor=lightgray];
|
||||
uwsgi [label="uwsgi", shape=parallelogram href="https://docs.searxng.org/utils/searx.sh.html"]
|
||||
redis [label="redis DB", shape=cylinder];
|
||||
searxng1 [label="SearXNG #1", fontcolor=blue3];
|
||||
searxng2 [label="SearXNG #2", fontcolor=blue3];
|
||||
searxng3 [label="SearXNG #3", fontcolor=blue3];
|
||||
searxng4 [label="SearXNG #4", fontcolor=blue3];
|
||||
browser [label="Browser", shape=Mdiamond];
|
||||
rp [label="Reverse Proxy", href="url to configure reverse proxy"];
|
||||
filtron [label="Filtron", href="https://github.com/asciimoo/filtron"];
|
||||
morty [label="Morty", href="https://github.com/asciimoo/morty"];
|
||||
static [label="Static files", href="url to configure static files"];
|
||||
uwsgi [label="uwsgi", href="url to configure uwsgi"]
|
||||
searx1 [label="Searx #1"];
|
||||
searx2 [label="Searx #2"];
|
||||
searx3 [label="Searx #3"];
|
||||
searx4 [label="Searx #4"];
|
||||
|
||||
browser -> rp [label="HTTPS"]
|
||||
|
||||
subgraph cluster_searxng {
|
||||
label = "SearXNG instance" fontname=Sans;
|
||||
subgraph cluster_searx {
|
||||
label = "Searx instance" fontname="Sans";
|
||||
bgcolor="#fafafa";
|
||||
{ rank=same; static rp };
|
||||
rp -> static [label="optional: reverse proxy serves static files", fillcolor=slategray, fontcolor=slategray];
|
||||
rp -> uwsgi [label="http:// (tcp) or unix:// (socket)"];
|
||||
uwsgi -> searxng1 -> redis;
|
||||
uwsgi -> searxng2 -> redis;
|
||||
uwsgi -> searxng3 -> redis;
|
||||
uwsgi -> searxng4 -> redis;
|
||||
rp -> morty [label="optional: images and HTML pages proxy"];
|
||||
rp -> static [label="optional: reverse proxy serves directly static files"];
|
||||
rp -> filtron [label="HTTP"];
|
||||
filtron -> uwsgi [label="HTTP"];
|
||||
uwsgi -> searx1;
|
||||
uwsgi -> searx2;
|
||||
uwsgi -> searx3;
|
||||
uwsgi -> searx4;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -0,0 +1,71 @@
|
||||
.. _engines generic:
|
||||
|
||||
=======
|
||||
Engines
|
||||
=======
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`settings engine`
|
||||
- :ref:`engine settings`
|
||||
- :ref:`engine file`
|
||||
|
||||
============= =========== ==================== ============
|
||||
:ref:`engine settings` :ref:`engine file`
|
||||
------------------------- ---------------------------------
|
||||
Name (cfg) Categories
|
||||
------------------------- ---------------------------------
|
||||
Engine .. Paging support **P**
|
||||
------------------------- -------------------- ------------
|
||||
Shortcut **S** Language support **L**
|
||||
Timeout **TO** Time range support **TR**
|
||||
Disabled **D** Offline **O**
|
||||
------------- ----------- -------------------- ------------
|
||||
Safe search **SS**
|
||||
------------- ----------- ---------------------------------
|
||||
Weigth **W**
|
||||
------------- ----------- ---------------------------------
|
||||
Disabled **D**
|
||||
============= =========== =================================
|
||||
|
||||
Configuration defaults (at built time):
|
||||
|
||||
.. _configured engines:
|
||||
|
||||
.. jinja:: webapp
|
||||
|
||||
.. flat-table:: Engines configured at built time (defaults)
|
||||
:header-rows: 1
|
||||
:stub-columns: 2
|
||||
|
||||
* - Name (cfg)
|
||||
- S
|
||||
- Engine
|
||||
- TO
|
||||
- Categories
|
||||
- P
|
||||
- L
|
||||
- SS
|
||||
- D
|
||||
- TR
|
||||
- O
|
||||
- W
|
||||
- D
|
||||
|
||||
{% for name, mod in engines.items() %}
|
||||
|
||||
* - {{name}}
|
||||
- !{{mod.shortcut}}
|
||||
- {{mod.__name__}}
|
||||
- {{mod.timeout}}
|
||||
- {{", ".join(mod.categories)}}
|
||||
- {{(mod.paging and "y") or ""}}
|
||||
- {{(mod.language_support and "y") or ""}}
|
||||
- {{(mod.safesearch and "y") or ""}}
|
||||
- {{(mod.disabled and "y") or ""}}
|
||||
- {{(mod.time_range_support and "y") or ""}}
|
||||
- {{(mod.offline and "y") or ""}}
|
||||
- {{mod.weight or 1 }}
|
||||
- {{(mod.disabled and "y") or ""}}
|
||||
|
||||
{% endfor %}
|
||||
@ -1,79 +0,0 @@
|
||||
.. _engine command:
|
||||
|
||||
====================
|
||||
Command Line Engines
|
||||
====================
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`command.py <searx/engines/command.py>`
|
||||
- :ref:`offline engines`
|
||||
|
||||
With *command engines* administrators can run engines to integrate arbitrary
|
||||
shell commands.
|
||||
|
||||
When creating and enabling a ``command`` engine on a public instance, you must
|
||||
be careful to avoid leaking private data. The easiest solution is to limit the
|
||||
access by setting ``tokens`` as described in section :ref:`private engines`.
|
||||
|
||||
The engine base is flexible. Only your imagination can limit the power of this
|
||||
engine (and maybe security concerns). The following options are available:
|
||||
|
||||
``command``:
|
||||
A comma separated list of the elements of the command. A special token
|
||||
``{{QUERY}}`` tells where to put the search terms of the user. Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
['ls', '-l', '-h', '{{QUERY}}']
|
||||
|
||||
``delimiter``:
|
||||
A mapping containing a delimiter ``char`` and the *titles* of each element in
|
||||
``keys``.
|
||||
|
||||
``parse_regex``:
|
||||
A dict containing the regular expressions for each result key.
|
||||
|
||||
``query_type``:
|
||||
|
||||
The expected type of user search terms. Possible values: ``path`` and
|
||||
``enum``.
|
||||
|
||||
``path``:
|
||||
Checks if the user provided path is inside the working directory. If not,
|
||||
the query is not executed.
|
||||
|
||||
``enum``:
|
||||
Is a list of allowed search terms. If the user submits something which is
|
||||
not included in the list, the query returns an error.
|
||||
|
||||
``query_enum``:
|
||||
A list containing allowed search terms if ``query_type`` is set to ``enum``.
|
||||
|
||||
``working_dir``:
|
||||
|
||||
The directory where the command has to be executed. Default: ``./``
|
||||
|
||||
``result_separator``:
|
||||
The character that separates results. Default: ``\n``
|
||||
|
||||
The example engine below can be used to find files with a specific name in the
|
||||
configured working directory:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: find
|
||||
engine: command
|
||||
command: ['find', '.', '-name', '{{QUERY}}']
|
||||
query_type: path
|
||||
shortcut: fnd
|
||||
delimiter:
|
||||
chars: ' '
|
||||
keys: ['line']
|
||||
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
|
||||
@ -1,75 +0,0 @@
|
||||
.. _configured engines:
|
||||
|
||||
==================
|
||||
Configured Engines
|
||||
==================
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`engines-dev`
|
||||
- :ref:`settings engine`
|
||||
|
||||
Explanation of the :ref:`general engine configuration` shown in the table
|
||||
:ref:`configured engines`.
|
||||
|
||||
.. jinja:: searx
|
||||
|
||||
SearXNG supports {{engines | length}} search engines (of which {{enabled_engine_count}} are enabled by default).
|
||||
|
||||
{% for category, engines in categories_as_tabs.items() %}
|
||||
|
||||
{{category}} search engines
|
||||
---------------------------------------
|
||||
|
||||
{% for group, engines in engines | group_engines_in_tab %}
|
||||
|
||||
{% if loop.length > 1 %}
|
||||
{{group}}
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
{% endif %}
|
||||
|
||||
.. flat-table::
|
||||
:header-rows: 2
|
||||
:stub-columns: 1
|
||||
|
||||
* - :cspan:`5` Engines configured by default (in :ref:`settings.yml <engine settings>`)
|
||||
- :cspan:`3` :ref:`Supported features <engine file>`
|
||||
|
||||
* - Name
|
||||
- Shortcut
|
||||
- Module
|
||||
- Disabled
|
||||
- Timeout
|
||||
- Weight
|
||||
- Paging
|
||||
- Language
|
||||
- Safe search
|
||||
- Time range
|
||||
|
||||
{% for mod in engines %}
|
||||
|
||||
* - `{{mod.name}} <{{mod.about and mod.about.website}}>`_
|
||||
- ``!{{mod.shortcut}}``
|
||||
- {%- if 'searx.engines.' + mod.__name__ in documented_modules %}
|
||||
:py:mod:`~searx.engines.{{mod.__name__}}`
|
||||
{%- else %}
|
||||
:origin:`{{mod.__name__}} <searx/engines/{{mod.__name__}}.py>`
|
||||
{%- endif %}
|
||||
- {{(mod.disabled and "y") or ""}}
|
||||
{%- if mod.about and mod.about.language %}
|
||||
({{mod.about.language | upper}})
|
||||
{%- endif %}
|
||||
- {{mod.timeout}}
|
||||
- {{mod.weight or 1 }}
|
||||
{% if mod.engine_type == 'online' %}
|
||||
- {{(mod.paging and "y") or ""}}
|
||||
- {{(mod.language_support and "y") or ""}}
|
||||
- {{(mod.safesearch and "y") or ""}}
|
||||
- {{(mod.time_range_support and "y") or ""}}
|
||||
{% else %}
|
||||
- :cspan:`3` not applicable ({{mod.engine_type}})
|
||||
{% endif %}
|
||||
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
@ -1,23 +0,0 @@
|
||||
.. _engines and settings:
|
||||
|
||||
==================
|
||||
Engines & Settings
|
||||
==================
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`settings engine`
|
||||
- :ref:`engine settings` & :ref:`engine file`
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
settings
|
||||
configured_engines
|
||||
private-engines
|
||||
recoll
|
||||
sql-engines
|
||||
nosql-engines
|
||||
search-indexer-engines
|
||||
command-line-engines
|
||||
searx.engines.xpath
|
||||
@ -1,135 +0,0 @@
|
||||
===============
|
||||
NoSQL databases
|
||||
===============
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- `NoSQL databases <https://en.wikipedia.org/wiki/NoSQL>`_
|
||||
- `redis.io <https://redis.io/>`_
|
||||
- `MongoDB <https://www.mongodb.com>`_
|
||||
|
||||
The following `NoSQL databases`_ are supported:
|
||||
|
||||
- :ref:`engine redis_server`
|
||||
- :ref:`engine mongodb`
|
||||
|
||||
All of the engines above are just commented out in the :origin:`settings.yml
|
||||
<searx/settings.yml>`, as you have to set various options and install
|
||||
dependencies before using them.
|
||||
|
||||
By default, the engines use the ``key-value`` template for displaying results /
|
||||
see :origin:`simple <searx/templates/simple/result_templates/key-value.html>`
|
||||
theme. If you are not satisfied with the original result layout, you can use
|
||||
your own template, set ``result_template`` attribute to ``{template_name}`` and
|
||||
place the templates at::
|
||||
|
||||
searx/templates/{theme_name}/result_templates/{template_name}
|
||||
|
||||
Furthermore, if you do not wish to expose these engines on a public instance, you
|
||||
can still add them and limit the access by setting ``tokens`` as described in
|
||||
section :ref:`private engines`.
|
||||
|
||||
|
||||
Configure the engines
|
||||
=====================
|
||||
|
||||
`NoSQL databases`_ are used for storing arbitrary data without first defining
|
||||
their structure.
|
||||
|
||||
|
||||
Extra Dependencies
|
||||
------------------
|
||||
|
||||
For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
|
||||
install additional packages in Python's Virtual Environment of your SearXNG
|
||||
instance. To switch into the environment (:ref:`searxng-src`) you can use
|
||||
:ref:`searxng.sh`::
|
||||
|
||||
$ sudo utils/searxng.sh instance cmd bash
|
||||
(searxng-pyenv)$ pip install ...
|
||||
|
||||
|
||||
.. _engine redis_server:
|
||||
|
||||
Redis Server
|
||||
------------
|
||||
|
||||
.. _redis: https://github.com/andymccurdy/redis-py#installation
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- ``pip install`` redis_
|
||||
- redis.io_
|
||||
- :origin:`redis_server.py <searx/engines/redis_server.py>`
|
||||
|
||||
|
||||
Redis is an open source (BSD licensed), in-memory data structure (key value
|
||||
based) store. Before configuring the ``redis_server`` engine, you must install
|
||||
the dependency redis_.
|
||||
|
||||
Select a database to search in and set its index in the option ``db``. You can
|
||||
either look for exact matches or use partial keywords to find what you are
|
||||
looking for by configuring ``exact_match_only``. You find an example
|
||||
configuration below:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
# Required dependency: redis
|
||||
|
||||
- name: myredis
|
||||
shortcut : rds
|
||||
engine: redis_server
|
||||
exact_match_only: false
|
||||
host: '127.0.0.1'
|
||||
port: 6379
|
||||
enable_http: true
|
||||
password: ''
|
||||
db: 0
|
||||
|
||||
.. _engine mongodb:
|
||||
|
||||
MongoDB
|
||||
-------
|
||||
|
||||
.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- ``pip install`` pymongo_
|
||||
- MongoDB_
|
||||
- :origin:`mongodb.py <searx/engines/mongodb.py>`
|
||||
|
||||
MongoDB_ is a document based database program that handles JSON like data.
|
||||
Before configuring the ``mongodb`` engine, you must install the dependency
|
||||
redis_.
|
||||
|
||||
In order to query MongoDB_, you have to select a ``database`` and a
|
||||
``collection``. Furthermore, you have to select a ``key`` that is going to be
|
||||
searched. MongoDB_ also supports the option ``exact_match_only``, so configure
|
||||
it as you wish. Below is an example configuration for using a MongoDB
|
||||
collection:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
# MongoDB engine
|
||||
# Required dependency: pymongo
|
||||
|
||||
- name: mymongo
|
||||
engine: mongodb
|
||||
shortcut: md
|
||||
exact_match_only: false
|
||||
host: '127.0.0.1'
|
||||
port: 27017
|
||||
enable_http: true
|
||||
results_per_page: 20
|
||||
database: 'business'
|
||||
collection: 'reviews' # name of the db collection
|
||||
key: 'name' # key in the collection to search for
|
||||
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
|
||||
|
||||
@ -1,49 +0,0 @@
|
||||
.. _private engines:
|
||||
|
||||
============================
|
||||
Private Engines (``tokens``)
|
||||
============================
|
||||
|
||||
Administrators might find themselves wanting to limit access to some of the
|
||||
enabled engines on their instances. It might be because they do not want to
|
||||
expose some private information through :ref:`offline engines`. Or they would
|
||||
rather share engines only with their trusted friends or colleagues.
|
||||
|
||||
To solve this issue the concept of *private engines* exists.
|
||||
|
||||
|
||||
A new option was added to engines named `tokens`. It expects a list of
|
||||
strings. If the user making a request presents one of the tokens of an engine,
|
||||
they can access information about the engine and make search requests.
|
||||
|
||||
Example configuration to restrict access to the Arch Linux Wiki engine:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: arch linux wiki
|
||||
engine: archlinux
|
||||
shortcut: al
|
||||
tokens: [ 'my-secret-token' ]
|
||||
|
||||
|
||||
Unless a user has configured the right token, the engine is going
|
||||
to be hidden from him/her. It is not going to be included in the
|
||||
list of engines on the Preferences page and in the output of
|
||||
`/config` REST API call.
|
||||
|
||||
Tokens can be added to one's configuration on the Preferences page
|
||||
under "Engine tokens". The input expects a comma separated list of
|
||||
strings.
|
||||
|
||||
The distribution of the tokens from the administrator to the users
|
||||
is not carved in stone. As providing access to such engines
|
||||
implies that the admin knows and trusts the user, we do not see
|
||||
necessary to come up with a strict process. Instead,
|
||||
we would like to add guidelines to the documentation of the feature.
|
||||
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
|
||||
@ -1,50 +0,0 @@
|
||||
.. _engine recoll:
|
||||
|
||||
=============
|
||||
Recoll Engine
|
||||
=============
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- `Recoll <https://www.lesbonscomptes.com/recoll/>`_
|
||||
- `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
|
||||
- :origin:`searx/engines/recoll.py`
|
||||
|
||||
Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_
|
||||
does not offer WEB or API access, this can be achieved using recoll-webui_
|
||||
|
||||
|
||||
Configuration
|
||||
=============
|
||||
|
||||
You must configure the following settings:
|
||||
|
||||
``base_url``:
|
||||
Location where recoll-webui can be reached.
|
||||
|
||||
``mount_prefix``:
|
||||
Location where the file hierarchy is mounted on your *local* filesystem.
|
||||
|
||||
``dl_prefix``:
|
||||
Location where the file hierarchy as indexed by recoll can be reached.
|
||||
|
||||
``search_dir``:
|
||||
Part of the indexed file hierarchy to be search, if empty the full domain is
|
||||
searched.
|
||||
|
||||
|
||||
Example
|
||||
=======
|
||||
|
||||
Scenario:
|
||||
|
||||
#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
|
||||
#. the Recoll search interface can be reached at https://recoll.example.org/ and
|
||||
#. the contents of this filesystem can be reached though https://download.example.org/reference
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
base_url: https://recoll.example.org/
|
||||
mount_prefix: /export/documents
|
||||
dl_prefix: https://download.example.org
|
||||
search_dir: ''
|
||||
@ -1,136 +0,0 @@
|
||||
====================
|
||||
Local Search Engines
|
||||
====================
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- `Comparison to alternatives
|
||||
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
|
||||
|
||||
Administrators might find themselves wanting to integrate locally running search
|
||||
engines. The following ones are supported for now:
|
||||
|
||||
* `Elasticsearch`_
|
||||
* `Meilisearch`_
|
||||
* `Solr`_
|
||||
|
||||
Each search engine is powerful, capable of full-text search. All of the engines
|
||||
above are added to ``settings.yml`` just commented out, as you have to
|
||||
``base_url`` for all them.
|
||||
|
||||
Please note that if you are not using HTTPS to access these engines, you have to enable
|
||||
HTTP requests by setting ``enable_http`` to ``True``.
|
||||
|
||||
Furthermore, if you do not want to expose these engines on a public instance, you
|
||||
can still add them and limit the access by setting ``tokens`` as described in
|
||||
section :ref:`private engines`.
|
||||
|
||||
.. _engine meilisearch:
|
||||
|
||||
MeiliSearch
|
||||
===========
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`meilisearch.py <searx/engines/meilisearch.py>`
|
||||
- `MeiliSearch <https://www.meilisearch.com>`_
|
||||
- `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
|
||||
- `Install MeiliSearch
|
||||
<https://docs.meilisearch.com/learn/getting_started/installation.html>`_
|
||||
|
||||
MeiliSearch_ is aimed at individuals and small companies. It is designed for
|
||||
small-scale (less than 10 million documents) data collections. E.g. it is great
|
||||
for storing web pages you have visited and searching in the contents later.
|
||||
|
||||
The engine supports faceted search, so you can search in a subset of documents
|
||||
of the collection. Furthermore, you can search in MeiliSearch_ instances that
|
||||
require authentication by setting ``auth_token``.
|
||||
|
||||
Here is a simple example to query a Meilisearch instance:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: meilisearch
|
||||
engine: meilisearch
|
||||
shortcut: mes
|
||||
base_url: http://localhost:7700
|
||||
index: my-index
|
||||
enable_http: true
|
||||
|
||||
|
||||
.. _engine elasticsearch:
|
||||
|
||||
Elasticsearch
|
||||
=============
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
|
||||
- `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
|
||||
- `Elasticsearch Guide
|
||||
<https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
|
||||
- `Install Elasticsearch
|
||||
<https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
|
||||
|
||||
Elasticsearch_ supports numerous ways to query the data it is storing. At the
|
||||
moment the engine supports the most popular search methods (``query_type``):
|
||||
|
||||
- ``match``,
|
||||
- ``simple_query_string``,
|
||||
- ``term`` and
|
||||
- ``terms``.
|
||||
|
||||
If none of the methods fit your use case, you can select ``custom`` query type
|
||||
and provide the JSON payload to submit to Elasticsearch in
|
||||
``custom_query_json``.
|
||||
|
||||
The following is an example configuration for an Elasticsearch_ instance with
|
||||
authentication configured to read from ``my-index`` index.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: elasticsearch
|
||||
shortcut: es
|
||||
engine: elasticsearch
|
||||
base_url: http://localhost:9200
|
||||
username: elastic
|
||||
password: changeme
|
||||
index: my-index
|
||||
query_type: match
|
||||
# custom_query_json: '{ ... }'
|
||||
enable_http: true
|
||||
|
||||
.. _engine solr:
|
||||
|
||||
Solr
|
||||
====
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`solr.py <searx/engines/solr.py>`
|
||||
- `Solr <https://solr.apache.org>`_
|
||||
- `Solr Resources <https://solr.apache.org/resources.html>`_
|
||||
- `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
|
||||
|
||||
Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But
|
||||
instead of searching in indices, you can search in collections.
|
||||
|
||||
This is an example configuration for searching in the collection
|
||||
``my-collection`` and get the results in ascending order.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: solr
|
||||
engine: solr
|
||||
shortcut: slr
|
||||
base_url: http://localhost:8983
|
||||
collection: my-collection
|
||||
sort: asc
|
||||
enable_http: true
|
||||
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
|
||||
@ -1,9 +0,0 @@
|
||||
.. _xpath engine:
|
||||
|
||||
============
|
||||
XPath Engine
|
||||
============
|
||||
|
||||
.. automodule:: searx.engines.xpath
|
||||
:members:
|
||||
|
||||
@ -1,679 +0,0 @@
|
||||
.. _settings.yml:
|
||||
|
||||
================
|
||||
``settings.yml``
|
||||
================
|
||||
|
||||
This page describe the options possibilities of the :origin:`searx/settings.yml`
|
||||
file.
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`use_default_settings.yml`
|
||||
- :ref:`search API`
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
.. _settings location:
|
||||
|
||||
settings.yml location
|
||||
=====================
|
||||
|
||||
The initial ``settings.yml`` we be load from these locations:
|
||||
|
||||
1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
|
||||
2. ``/etc/searxng/settings.yml``
|
||||
|
||||
If these files don't exist (or are empty or can't be read), SearXNG uses the
|
||||
:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to
|
||||
see how you can simplify your *user defined* ``settings.yml``.
|
||||
|
||||
|
||||
.. _settings global:
|
||||
|
||||
Global Settings
|
||||
===============
|
||||
|
||||
.. _settings brand:
|
||||
|
||||
``brand:``
|
||||
----------
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
brand:
|
||||
issue_url: https://github.com/searxng/searxng/issues
|
||||
docs_url: https://docs.searxng.org
|
||||
public_instances: https://searx.space
|
||||
wiki_url: https://github.com/searxng/searxng/wiki
|
||||
|
||||
``issue_url`` :
|
||||
If you host your own issue tracker change this URL.
|
||||
|
||||
``docs_url`` :
|
||||
If you host your own documentation change this URL.
|
||||
|
||||
``public_instances`` :
|
||||
If you host your own https://searx.space change this URL.
|
||||
|
||||
``wiki_url`` :
|
||||
Link to your wiki (or ``false``)
|
||||
|
||||
.. _settings general:
|
||||
|
||||
``general:``
|
||||
------------
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
general:
|
||||
debug: false
|
||||
instance_name: "SearXNG"
|
||||
privacypolicy_url: false
|
||||
donation_url: https://docs.searxng.org/donate.html
|
||||
contact_url: false
|
||||
enable_metrics: true
|
||||
|
||||
``debug`` : ``$SEARXNG_DEBUG``
|
||||
Allow a more detailed log if you run SearXNG directly. Display *detailed* error
|
||||
messages in the browser too, so this must be deactivated in production.
|
||||
|
||||
``donation_url`` :
|
||||
At default the donation link points to the `SearXNG project
|
||||
<https://docs.searxng.org/donate.html>`_. Set value to ``true`` to use your
|
||||
own donation page written in the :ref:`searx/info/en/donate.md
|
||||
<searx.infopage>` and use ``false`` to disable the donation link altogether.
|
||||
|
||||
``privacypolicy_url``:
|
||||
Link to privacy policy.
|
||||
|
||||
``contact_url``:
|
||||
Contact ``mailto:`` address or WEB form.
|
||||
|
||||
``enable_metrics``:
|
||||
Enabled by default. Record various anonymous metrics availabled at ``/stats``,
|
||||
``/stats/errors`` and ``/preferences``.
|
||||
|
||||
.. _settings search:
|
||||
|
||||
``search:``
|
||||
-----------
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
search:
|
||||
safe_search: 0
|
||||
autocomplete: ""
|
||||
default_lang: ""
|
||||
ban_time_on_fail: 5
|
||||
max_ban_time_on_fail: 120
|
||||
formats:
|
||||
- html
|
||||
|
||||
``safe_search``:
|
||||
Filter results.
|
||||
|
||||
- ``0``: None
|
||||
- ``1``: Moderate
|
||||
- ``2``: Strict
|
||||
|
||||
``autocomplete``:
|
||||
Existing autocomplete backends, leave blank to turn it off.
|
||||
|
||||
- ``dbpedia``
|
||||
- ``duckduckgo``
|
||||
- ``google``
|
||||
- ``startpage``
|
||||
- ``swisscows``
|
||||
- ``qwant``
|
||||
- ``wikipedia``
|
||||
|
||||
``default_lang``:
|
||||
Default search language - leave blank to detect from browser information or
|
||||
use codes from :origin:`searx/languages.py`.
|
||||
|
||||
``languages``:
|
||||
List of available languages - leave unset to use all codes from
|
||||
:origin:`searx/languages.py`. Otherwise list codes of available languages.
|
||||
The ``all`` value is shown as the ``Default language`` in the user interface
|
||||
(in most cases, it is meant to send the query without a language parameter ;
|
||||
in some cases, it means the English language) Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
languages:
|
||||
- all
|
||||
- en
|
||||
- en-US
|
||||
- de
|
||||
- it-IT
|
||||
- fr
|
||||
- fr-BE
|
||||
|
||||
``ban_time_on_fail``:
|
||||
Ban time in seconds after engine errors.
|
||||
|
||||
``max_ban_time_on_fail``:
|
||||
Max ban time in seconds after engine errors.
|
||||
|
||||
``formats``:
|
||||
Result formats available from web, remove format to deny access (use lower
|
||||
case).
|
||||
|
||||
- ``html``
|
||||
- ``csv``
|
||||
- ``json``
|
||||
- ``rss``
|
||||
|
||||
.. _settings server:
|
||||
|
||||
``server:``
|
||||
-----------
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
server:
|
||||
base_url: false # set custom base_url (or false)
|
||||
port: 8888
|
||||
bind_address: "127.0.0.1" # address to listen on
|
||||
secret_key: "ultrasecretkey" # change this!
|
||||
limiter: false
|
||||
image_proxy: false # proxying image results through SearXNG
|
||||
default_http_headers:
|
||||
X-Content-Type-Options : nosniff
|
||||
X-XSS-Protection : 1; mode=block
|
||||
X-Download-Options : noopen
|
||||
X-Robots-Tag : noindex, nofollow
|
||||
Referrer-Policy : no-referrer
|
||||
|
||||
.. sidebar:: buildenv
|
||||
|
||||
Changing a value tagged by :ref:`buildenv <make buildenv>`, needs to
|
||||
rebuild instance's environment :ref:`utils/brand.env <make buildenv>`.
|
||||
|
||||
``base_url`` : :ref:`buildenv SEARXNG_URL <make buildenv>`
|
||||
The base URL where SearXNG is deployed. Used to create correct inbound links.
|
||||
If you change the value, don't forget to rebuild instance's environment
|
||||
(:ref:`utils/brand.env <make buildenv>`)
|
||||
|
||||
``port`` & ``bind_address``: :ref:`buildenv SEARXNG_PORT & SEARXNG_BIND_ADDRESS <make buildenv>`
|
||||
Port number and *bind address* of the SearXNG web application if you run it
|
||||
directly using ``python searx/webapp.py``. Doesn't apply to SearXNG running on
|
||||
Apache or Nginx.
|
||||
|
||||
``secret_key`` : ``$SEARXNG_SECRET``
|
||||
Used for cryptography purpose.
|
||||
|
||||
.. _limiter:
|
||||
|
||||
``limiter`` :
|
||||
Rate limit the number of request on the instance, block some bots. The
|
||||
:ref:`limiter plugin` requires a :ref:`settings redis` database.
|
||||
|
||||
.. _image_proxy:
|
||||
|
||||
``image_proxy`` :
|
||||
Allow your instance of SearXNG of being able to proxy images. Uses memory space.
|
||||
|
||||
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
|
||||
|
||||
``default_http_headers`` :
|
||||
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
|
||||
|
||||
|
||||
.. _settings ui:
|
||||
|
||||
``ui:``
|
||||
-------
|
||||
|
||||
.. _cache busting:
|
||||
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
ui:
|
||||
static_use_hash: false
|
||||
default_locale: ""
|
||||
query_in_title: false
|
||||
infinite_scroll: false
|
||||
center_alignment: false
|
||||
cache_url: https://web.archive.org/web/
|
||||
default_theme: simple
|
||||
theme_args:
|
||||
simple_style: auto
|
||||
|
||||
.. _static_use_hash:
|
||||
|
||||
``static_use_hash`` :
|
||||
Enables `cache busting`_ of static files.
|
||||
|
||||
``default_locale`` :
|
||||
SearXNG interface language. If blank, the locale is detected by using the
|
||||
browser language. If it doesn't work, or you are deploying a language
|
||||
specific instance of searx, a locale can be defined using an ISO language
|
||||
code, like ``fr``, ``en``, ``de``.
|
||||
|
||||
``query_in_title`` :
|
||||
When true, the result page's titles contains the query it decreases the
|
||||
privacy, since the browser can records the page titles.
|
||||
|
||||
``infinite_scroll``:
|
||||
When true, automatically loads the next page when scrolling to bottom of the current page.
|
||||
|
||||
``center_alignment`` : default ``false``
|
||||
When enabled, the results are centered instead of being in the left (or RTL)
|
||||
side of the screen. This setting only affects the *desktop layout*
|
||||
(:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
|
||||
|
||||
.. cache_url:
|
||||
|
||||
``cache_url`` : ``https://web.archive.org/web/``
|
||||
URL prefix of the internet archive or cache, don't forgett trailing slash (if
|
||||
needed). The default is https://web.archive.org/web/ alternatives are:
|
||||
|
||||
- https://webcache.googleusercontent.com/search?q=cache:
|
||||
- https://archive.today/
|
||||
|
||||
``default_theme`` :
|
||||
Name of the theme you want to use by default on your SearXNG instance.
|
||||
|
||||
``theme_args.simple_style``:
|
||||
Style of simple theme: ``auto``, ``light``, ``dark``
|
||||
|
||||
``results_on_new_tab``:
|
||||
Open result links in a new tab by default.
|
||||
|
||||
|
||||
.. _settings redis:
|
||||
|
||||
``redis:``
|
||||
----------
|
||||
|
||||
.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
|
||||
|
||||
A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
|
||||
will find a description to test your redis connection in SerXNG. When using
|
||||
sockets, don't forget to check the access rights on the socket::
|
||||
|
||||
ls -la /usr/local/searxng-redis/run/redis.sock
|
||||
srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
|
||||
|
||||
In this example read/write access is given to the *searxng-redis* group. To get
|
||||
access rights to redis instance (the socket), your SearXNG (or even your
|
||||
developer) account needs to be added to the *searxng-redis* group.
|
||||
|
||||
``url``
|
||||
URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
|
||||
|
||||
redis://[[username]:[password]]@localhost:6379/0
|
||||
rediss://[[username]:[password]]@localhost:6379/0
|
||||
unix://[[username]:[password]]@/path/to/socket.sock?db=0
|
||||
|
||||
.. admonition:: Tip for developers
|
||||
|
||||
To set up a local redis instance, first set the socket path of the Redis DB
|
||||
in your YAML setting:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
redis:
|
||||
url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
|
||||
|
||||
Then use the following commands to install the redis instance ::
|
||||
|
||||
$ ./manage redis.build
|
||||
$ sudo -H ./manage redis.install
|
||||
$ sudo -H ./manage redis.addgrp "${USER}"
|
||||
# don't forget to logout & login to get member of group
|
||||
|
||||
|
||||
.. _settings outgoing:
|
||||
|
||||
``outgoing:``
|
||||
-------------
|
||||
|
||||
Communication with search engines.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
outgoing:
|
||||
request_timeout: 2.0 # default timeout in seconds, can be override by engine
|
||||
max_request_timeout: 10.0 # the maximum timeout in seconds
|
||||
useragent_suffix: "" # information like an email address to the administrator
|
||||
pool_connections: 100 # Maximum number of allowable connections, or null
|
||||
# for no limits. The default is 100.
|
||||
pool_maxsize: 10 # Number of allowable keep-alive connections, or null
|
||||
# to always allow. The default is 10.
|
||||
enable_http2: true # See https://www.python-httpx.org/http2/
|
||||
# uncomment below section if you want to use a custom server certificate
|
||||
# see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
|
||||
# and https://www.python-httpx.org/compatibility/#ssl-configuration
|
||||
# verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
|
||||
#
|
||||
# uncomment below section if you want to use a proxyq see: SOCKS proxies
|
||||
# https://2.python-requests.org/en/latest/user/advanced/#proxies
|
||||
# are also supported: see
|
||||
# https://2.python-requests.org/en/latest/user/advanced/#socks
|
||||
#
|
||||
# proxies:
|
||||
# all://:
|
||||
# - http://proxy1:8080
|
||||
# - http://proxy2:8080
|
||||
#
|
||||
# using_tor_proxy: true
|
||||
#
|
||||
# Extra seconds to add in order to account for the time taken by the proxy
|
||||
#
|
||||
# extra_proxy_timeout: 10.0
|
||||
#
|
||||
|
||||
``request_timeout`` :
|
||||
Global timeout of the requests made to others engines in seconds. A bigger
|
||||
timeout will allow to wait for answers from slow engines, but in consequence
|
||||
will slow SearXNG reactivity (the result page may take the time specified in the
|
||||
timeout to load). Can be override by :ref:`settings engine`
|
||||
|
||||
``useragent_suffix`` :
|
||||
Suffix to the user-agent SearXNG uses to send requests to others engines. If an
|
||||
engine wish to block you, a contact info here may be useful to avoid that.
|
||||
|
||||
``keepalive_expiry`` :
|
||||
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
|
||||
|
||||
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
|
||||
|
||||
``proxies`` :
|
||||
Define one or more proxies you wish to use, see `httpx proxies`_.
|
||||
If there are more than one proxy for one protocol (http, https),
|
||||
requests to the engines are distributed in a round-robin fashion.
|
||||
|
||||
``source_ips`` :
|
||||
If you use multiple network interfaces, define from which IP the requests must
|
||||
be made. Example:
|
||||
|
||||
* ``0.0.0.0`` any local IPv4 address.
|
||||
* ``::`` any local IPv6 address.
|
||||
* ``192.168.0.1``
|
||||
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
|
||||
* ``fe80::60a2:1691:e5a2:ee1f``
|
||||
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
|
||||
* ``[ 192.168.0.1, fe80::/126 ]``
|
||||
|
||||
``retries`` :
|
||||
Number of retry in case of an HTTP error. On each retry, SearXNG uses an
|
||||
different proxy and source ip.
|
||||
|
||||
``retry_on_http_error`` :
|
||||
Retry request on some HTTP status code.
|
||||
|
||||
Example:
|
||||
|
||||
* ``true`` : on HTTP status code between 400 and 599.
|
||||
* ``403`` : on HTTP status code 403.
|
||||
* ``[403, 429]``: on HTTP status code 403 and 429.
|
||||
|
||||
``enable_http2`` :
|
||||
Enable by default. Set to ``false`` to disable HTTP/2.
|
||||
|
||||
.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
|
||||
.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
|
||||
|
||||
``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
|
||||
Allow to specify a path to certificate.
|
||||
see `httpx verification defaults`_.
|
||||
|
||||
In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
|
||||
``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
|
||||
see `httpx ssl configuration`_.
|
||||
|
||||
``max_redirects`` :
|
||||
30 by default. Maximum redirect before it is an error.
|
||||
|
||||
``categories_as_tabs:``
|
||||
-----------------------
|
||||
|
||||
A list of the categories that are displayed as tabs in the user interface.
|
||||
Categories not listed here can still be searched with the :ref:`search-syntax`.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
categories_as_tabs:
|
||||
general:
|
||||
images:
|
||||
videos:
|
||||
news:
|
||||
map:
|
||||
music:
|
||||
it:
|
||||
science:
|
||||
files:
|
||||
social media:
|
||||
|
||||
.. _settings engine:
|
||||
|
||||
Engine settings
|
||||
===============
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`configured engines`
|
||||
- :ref:`engines-dev`
|
||||
|
||||
In the code example below a *full fledged* example of a YAML setup from a dummy
|
||||
engine is shown. Most of the options have a default value or even are optional.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: example engine
|
||||
engine: example
|
||||
shortcut: demo
|
||||
base_url: 'https://{language}.example.com/'
|
||||
send_accept_language_header: false
|
||||
categories: general
|
||||
timeout: 3.0
|
||||
api_key: 'apikey'
|
||||
disabled: false
|
||||
language: en_US
|
||||
tokens: [ 'my-secret-token' ]
|
||||
weigth: 1
|
||||
display_error_messages: true
|
||||
about:
|
||||
website: https://example.com
|
||||
wikidata_id: Q306656
|
||||
official_api_documentation: https://example.com/api-doc
|
||||
use_official_api: true
|
||||
require_api_key: true
|
||||
results: HTML
|
||||
enable_http: false
|
||||
enable_http2: false
|
||||
retries: 1
|
||||
retry_on_http_error: true # or 403 or [404, 429]
|
||||
max_connections: 100
|
||||
max_keepalive_connections: 10
|
||||
keepalive_expiry: 5.0
|
||||
proxies:
|
||||
http:
|
||||
- http://proxy1:8080
|
||||
- http://proxy2:8080
|
||||
https:
|
||||
- http://proxy1:8080
|
||||
- http://proxy2:8080
|
||||
- socks5://user:password@proxy3:1080
|
||||
- socks5h://user:password@proxy4:1080
|
||||
|
||||
``name`` :
|
||||
Name that will be used across SearXNG to define this engine. In settings, on
|
||||
the result page...
|
||||
|
||||
``engine`` :
|
||||
Name of the python file used to handle requests and responses to and from this
|
||||
search engine.
|
||||
|
||||
``shortcut`` :
|
||||
Code used to execute bang requests (in this case using ``!bi``)
|
||||
|
||||
``base_url`` : optional
|
||||
Part of the URL that should be stable across every request. Can be useful to
|
||||
use multiple sites using only one engine, or updating the site URL without
|
||||
touching at the code.
|
||||
|
||||
``send_accept_language_header`` :
|
||||
Several engines that support languages (or regions) deal with the HTTP header
|
||||
``Accept-Language`` to build a response that fits to the locale. When this
|
||||
option is activated, the language (locale) that is selected by the user is used
|
||||
to build and send a ``Accept-Language`` header in the request to the origin
|
||||
search engine.
|
||||
|
||||
``categories`` : optional
|
||||
Define in which categories this engine will be active. Most of the time, it is
|
||||
defined in the code of the engine, but in a few cases it is useful, like when
|
||||
describing multiple search engine using the same code.
|
||||
|
||||
``timeout`` : optional
|
||||
Timeout of the search with the current search engine. **Be careful, it will
|
||||
modify the global timeout of SearXNG.**
|
||||
|
||||
``api_key`` : optional
|
||||
In a few cases, using an API needs the use of a secret key. How to obtain them
|
||||
is described in the file.
|
||||
|
||||
``disabled`` : optional
|
||||
To disable by default the engine, but not deleting it. It will allow the user
|
||||
to manually activate it in the settings.
|
||||
|
||||
``language`` : optional
|
||||
If you want to use another language for a specific engine, you can define it
|
||||
by using the full ISO code of language and country, like ``fr_FR``, ``en_US``,
|
||||
``de_DE``.
|
||||
|
||||
``tokens`` : optional
|
||||
A list of secret tokens to make this engine *private*, more details see
|
||||
:ref:`private engines`.
|
||||
|
||||
``weigth`` : default ``1``
|
||||
Weighting of the results of this engine.
|
||||
|
||||
``display_error_messages`` : default ``true``
|
||||
When an engine returns an error, the message is displayed on the user interface.
|
||||
|
||||
``network`` : optional
|
||||
Use the network configuration from another engine.
|
||||
In addition, there are two default networks:
|
||||
|
||||
- ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
|
||||
- ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
|
||||
|
||||
.. note::
|
||||
|
||||
A few more options are possible, but they are pretty specific to some
|
||||
engines, and so won't be described here.
|
||||
|
||||
|
||||
Example: Multilingual Search
|
||||
----------------------------
|
||||
|
||||
SearXNG does not support true multilingual search. You have to use the language
|
||||
prefix in your search query when searching in a different language.
|
||||
|
||||
But there is a workaround: By adding a new search engine with a different
|
||||
language, SearXNG will search in your default and other language.
|
||||
|
||||
Example configuration in settings.yml for a German and English speaker:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
search:
|
||||
default_lang : "de"
|
||||
...
|
||||
|
||||
engines:
|
||||
- name : google english
|
||||
engine : google
|
||||
language : en
|
||||
...
|
||||
|
||||
When searching, the default google engine will return German results and
|
||||
"google english" will return English results.
|
||||
|
||||
|
||||
.. _settings use_default_settings:
|
||||
|
||||
use_default_settings
|
||||
====================
|
||||
|
||||
.. sidebar:: ``use_default_settings: true``
|
||||
|
||||
- :ref:`settings location`
|
||||
- :ref:`use_default_settings.yml`
|
||||
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
|
||||
|
||||
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
|
||||
and can relied on the default configuration :origin:`searx/settings.yml` using:
|
||||
|
||||
``use_default_settings: true``
|
||||
|
||||
``server:``
|
||||
In the following example, the actual settings are the default settings defined
|
||||
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
|
||||
the ``bind_address``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "ultrasecretkey" # change this!
|
||||
bind_address: "0.0.0.0"
|
||||
|
||||
``engines:``
|
||||
With ``use_default_settings: true``, each settings can be override in a
|
||||
similar way, the ``engines`` section is merged according to the engine
|
||||
``name``. In this example, SearXNG will load all the engine and the arch linux
|
||||
wiki engine has a :ref:`token <private engines>`:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "ultrasecretkey" # change this!
|
||||
engines:
|
||||
- name: arch linux wiki
|
||||
tokens: ['$ecretValue']
|
||||
|
||||
``engines:`` / ``remove:``
|
||||
It is possible to remove some engines from the default settings. The following
|
||||
example is similar to the above one, but SearXNG doesn't load the the google
|
||||
engine:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings:
|
||||
engines:
|
||||
remove:
|
||||
- google
|
||||
server:
|
||||
secret_key: "ultrasecretkey" # change this!
|
||||
engines:
|
||||
- name: arch linux wiki
|
||||
tokens: ['$ecretValue']
|
||||
|
||||
``engines:`` / ``keep_only:``
|
||||
As an alternative, it is possible to specify the engines to keep. In the
|
||||
following example, SearXNG has only two engines:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings:
|
||||
engines:
|
||||
keep_only:
|
||||
- google
|
||||
- duckduckgo
|
||||
server:
|
||||
secret_key: "ultrasecretkey" # change this!
|
||||
engines:
|
||||
- name: google
|
||||
tokens: ['$ecretValue']
|
||||
- name: duckduckgo
|
||||
tokens: ['$ecretValue']
|
||||
@ -1,166 +0,0 @@
|
||||
.. _sql engines:
|
||||
|
||||
===========
|
||||
SQL Engines
|
||||
===========
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- `SQLite <https://www.sqlite.org/index.html>`_
|
||||
- `PostgreSQL <https://www.postgresql.org>`_
|
||||
- `MySQL <https://www.mysql.com>`_
|
||||
|
||||
With the *SQL engines* you can bind SQL databases into SearXNG. The following
|
||||
Relational Database Management System (RDBMS) are supported:
|
||||
|
||||
- :ref:`engine sqlite`
|
||||
- :ref:`engine postgresql`
|
||||
- :ref:`engine mysql_server`
|
||||
|
||||
All of the engines above are just commented out in the :origin:`settings.yml
|
||||
<searx/settings.yml>`, as you have to set the required attributes for the
|
||||
engines, e.g. ``database:`` ...
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: ...
|
||||
engine: {sqlite|postgresql|mysql_server}
|
||||
database: ...
|
||||
result_template: {template_name}
|
||||
query_str: ...
|
||||
|
||||
By default, the engines use the ``key-value`` template for displaying results /
|
||||
see :origin:`simple <searx/templates/simple/result_templates/key-value.html>`
|
||||
theme. If you are not satisfied with the original result layout, you can use
|
||||
your own template, set ``result_template`` attribute to ``{template_name}`` and
|
||||
place the templates at::
|
||||
|
||||
searx/templates/{theme_name}/result_templates/{template_name}
|
||||
|
||||
If you do not wish to expose these engines on a public instance, you can still
|
||||
add them and limit the access by setting ``tokens`` as described in section
|
||||
:ref:`private engines`.
|
||||
|
||||
|
||||
Configure the engines
|
||||
=====================
|
||||
|
||||
The configuration of the new database engines are similar. You must put a valid
|
||||
SQL-SELECT query in ``query_str``. At the moment you can only bind at most one
|
||||
parameter in your query. By setting the attribute ``limit`` you can define how
|
||||
many results you want from the SQL server. Basically, it is the same as the
|
||||
``LIMIT`` keyword in SQL.
|
||||
|
||||
Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines
|
||||
rely on these keywords during paging. If you want to configure the number of
|
||||
returned results use the option ``limit``.
|
||||
|
||||
.. _engine sqlite:
|
||||
|
||||
SQLite
|
||||
------
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`sqlite.py <searx/engines/sqlite.py>`
|
||||
|
||||
.. _MediathekView: https://mediathekview.de/
|
||||
|
||||
SQLite is a small, fast and reliable SQL database engine. It does not require
|
||||
any extra dependency. To demonstrate the power of database engines, here is a
|
||||
more complex example which reads from a MediathekView_ (DE) movie database. For
|
||||
this example of the SQlite engine download the database:
|
||||
|
||||
- https://liste.mediathekview.de/filmliste-v2.db.bz2
|
||||
|
||||
and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g
|
||||
Query to test: ``!mediathekview concert``
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: mediathekview
|
||||
engine: sqlite
|
||||
disabled: False
|
||||
categories: general
|
||||
result_template: default.html
|
||||
database: searx/data/filmliste-v2.db
|
||||
query_str: >-
|
||||
SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
|
||||
COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
|
||||
description AS content
|
||||
FROM film
|
||||
WHERE title LIKE :wildcard OR description LIKE :wildcard
|
||||
ORDER BY duration DESC
|
||||
|
||||
|
||||
Extra Dependencies
|
||||
------------------
|
||||
|
||||
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
|
||||
install additional packages in Python's Virtual Environment of your SearXNG
|
||||
instance. To switch into the environment (:ref:`searxng-src`) you can use
|
||||
:ref:`searxng.sh`::
|
||||
|
||||
$ sudo utils/searxng.sh instance cmd bash
|
||||
(searxng-pyenv)$ pip install ...
|
||||
|
||||
|
||||
.. _engine postgresql:
|
||||
|
||||
PostgreSQL
|
||||
----------
|
||||
|
||||
.. _psycopg2: https://www.psycopg.org/install
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`postgresql.py <searx/engines/postgresql.py>`
|
||||
- ``pip install`` psycopg2_
|
||||
|
||||
PostgreSQL is a powerful and robust open source database. Before configuring
|
||||
the PostgreSQL engine, you must install the dependency ``psychopg2``. You can
|
||||
find an example configuration below:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: my_database
|
||||
engine: postgresql
|
||||
database: my_database
|
||||
username: searxng
|
||||
password: password
|
||||
query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
|
||||
|
||||
.. _engine mysql_server:
|
||||
|
||||
MySQL
|
||||
-----
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :origin:`mysql_server.py <searx/engines/mysql_server.py>`
|
||||
- ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
|
||||
|
||||
MySQL is said to be the most popular open source database. Before enabling MySQL
|
||||
engine, you must install the package ``mysql-connector-python``.
|
||||
|
||||
The authentication plugin is configurable by setting ``auth_plugin`` in the
|
||||
attributes. By default it is set to ``caching_sha2_password``. This is an
|
||||
example configuration for querying a MySQL server:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name: my_database
|
||||
engine: mysql_server
|
||||
database: my_database
|
||||
username: searxng
|
||||
password: password
|
||||
limit: 5
|
||||
query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
|
||||
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
|
||||
|
||||
@ -0,0 +1,148 @@
|
||||
==========================
|
||||
How to protect an instance
|
||||
==========================
|
||||
|
||||
Searx depens on external search services. To avoid the abuse of these services
|
||||
it is advised to limit the number of requests processed by searx.
|
||||
|
||||
An application firewall, ``filtron`` solves exactly this problem. Information
|
||||
on how to install it can be found at the `project page of filtron
|
||||
<https://github.com/asciimoo/filtron>`__.
|
||||
|
||||
|
||||
Sample configuration of filtron
|
||||
===============================
|
||||
|
||||
An example configuration can be find below. This configuration limits the access
|
||||
of:
|
||||
|
||||
- scripts or applications (roboagent limit)
|
||||
- webcrawlers (botlimit)
|
||||
- IPs which send too many requests (IP limit)
|
||||
- too many json, csv, etc. requests (rss/json limit)
|
||||
- the same UserAgent of if too many requests (useragent limit)
|
||||
|
||||
.. code:: json
|
||||
|
||||
[{
|
||||
"name":"search request",
|
||||
"filters":[
|
||||
"Param:q",
|
||||
"Path=^(/|/search)$"
|
||||
],
|
||||
"interval":"<time-interval-in-sec (int)>",
|
||||
"limit":"<max-request-number-in-interval (int)>",
|
||||
"subrules":[
|
||||
{
|
||||
"name":"roboagent limit",
|
||||
"interval":"<time-interval-in-sec (int)>",
|
||||
"limit":"<max-request-number-in-interval (int)>",
|
||||
"filters":[
|
||||
"Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"
|
||||
],
|
||||
"actions":[
|
||||
{
|
||||
"name":"block",
|
||||
"params":{
|
||||
"message":"Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"botlimit",
|
||||
"limit":0,
|
||||
"stop":true,
|
||||
"filters":[
|
||||
"Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"
|
||||
],
|
||||
"actions":[
|
||||
{
|
||||
"name":"block",
|
||||
"params":{
|
||||
"message":"Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"IP limit",
|
||||
"interval":"<time-interval-in-sec (int)>",
|
||||
"limit":"<max-request-number-in-interval (int)>",
|
||||
"stop":true,
|
||||
"aggregations":[
|
||||
"Header:X-Forwarded-For"
|
||||
],
|
||||
"actions":[
|
||||
{
|
||||
"name":"block",
|
||||
"params":{
|
||||
"message":"Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"rss/json limit",
|
||||
"interval":"<time-interval-in-sec (int)>",
|
||||
"limit":"<max-request-number-in-interval (int)>",
|
||||
"stop":true,
|
||||
"filters":[
|
||||
"Param:format=(csv|json|rss)"
|
||||
],
|
||||
"actions":[
|
||||
{
|
||||
"name":"block",
|
||||
"params":{
|
||||
"message":"Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name":"useragent limit",
|
||||
"interval":"<time-interval-in-sec (int)>",
|
||||
"limit":"<max-request-number-in-interval (int)>",
|
||||
"aggregations":[
|
||||
"Header:User-Agent"
|
||||
],
|
||||
"actions":[
|
||||
{
|
||||
"name":"block",
|
||||
"params":{
|
||||
"message":"Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}]
|
||||
|
||||
|
||||
|
||||
Route request through filtron
|
||||
=============================
|
||||
|
||||
Filtron can be started using the following command:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ filtron -rules rules.json
|
||||
|
||||
It listens on ``127.0.0.1:4004`` and forwards filtered requests to
|
||||
``127.0.0.1:8888`` by default.
|
||||
|
||||
Use it along with ``nginx`` with the following example configuration.
|
||||
|
||||
.. code:: nginx
|
||||
|
||||
location / {
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
proxy_pass http://127.0.0.1:4004/;
|
||||
}
|
||||
|
||||
Requests are coming from port 4004 going through filtron and then forwarded to
|
||||
port 8888 where a searx is being run.
|
||||
@ -1,388 +0,0 @@
|
||||
.. _installation apache:
|
||||
|
||||
======
|
||||
Apache
|
||||
======
|
||||
|
||||
.. _Apache: https://httpd.apache.org/
|
||||
.. _Apache Debian:
|
||||
https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x):
|
||||
.. _apache2.README.Debian:
|
||||
https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian
|
||||
.. _Apache Arch Linux:
|
||||
https://wiki.archlinux.org/index.php/Apache_HTTP_Server
|
||||
.. _Apache Fedora:
|
||||
https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html
|
||||
.. _Apache directives:
|
||||
https://httpd.apache.org/docs/trunk/mod/directives.html
|
||||
.. _Getting Started:
|
||||
https://httpd.apache.org/docs/current/en/getting-started.html
|
||||
.. _Terms Used to Describe Directives:
|
||||
https://httpd.apache.org/docs/current/en/mod/directive-dict.html
|
||||
.. _Configuration Files:
|
||||
https://httpd.apache.org/docs/current/en/configuring.html
|
||||
.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost
|
||||
.. _LoadModule:
|
||||
https://httpd.apache.org/docs/mod/mod_so.html#loadmodule
|
||||
.. _IncludeOptional:
|
||||
https://httpd.apache.org/docs/mod/core.html#includeoptional
|
||||
.. _DocumentRoot:
|
||||
https://httpd.apache.org/docs/trunk/mod/core.html#documentroot
|
||||
.. _Location:
|
||||
https://httpd.apache.org/docs/trunk/mod/core.html#location
|
||||
.. _uWSGI Apache support:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Apache.html
|
||||
.. _mod_proxy_uwsgi:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi
|
||||
.. _mod_proxy_http:
|
||||
https://httpd.apache.org/docs/current/mod/mod_proxy_http.html
|
||||
.. _mod_proxy:
|
||||
https://httpd.apache.org/docs/current/mod/mod_proxy.html
|
||||
|
||||
|
||||
This section explains how to set up a SearXNG instance using the HTTP server Apache_.
|
||||
If you did use the :ref:`installation scripts` and do not have any special preferences
|
||||
you can install the :ref:`SearXNG site <apache searxng site>` using
|
||||
:ref:`searxng.sh <searxng.sh overview>`:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/searxng.sh install apache
|
||||
|
||||
If you have special interests or problems with setting up Apache, the following
|
||||
section might give you some guidance.
|
||||
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- `Apache Arch Linux`_
|
||||
- `Apache Debian`_
|
||||
- `apache2.README.Debian`_
|
||||
- `Apache Fedora`_
|
||||
- `Apache directives`_
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
The Apache HTTP server
|
||||
======================
|
||||
|
||||
If Apache_ is not installed, install it now. If apache_ is new to you, the
|
||||
`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe
|
||||
Directives`_ documentation gives first orientation. There is also a list of
|
||||
`Apache directives`_ *to keep in the pocket*.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H apt-get install apache2
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H pacman -S apache
|
||||
sudo -H systemctl enable httpd
|
||||
sudo -H systemctl start http
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H dnf install httpd
|
||||
sudo -H systemctl enable httpd
|
||||
sudo -H systemctl start httpd
|
||||
|
||||
Now at http://localhost you should see some kind of *Welcome* or *Test* page.
|
||||
How this default site is configured, depends on the linux distribution
|
||||
(compare `Apache directives`_).
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
less /etc/apache2/sites-enabled/000-default.conf
|
||||
|
||||
In this file, there is a line setting the `DocumentRoot`_ directive:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
DocumentRoot /var/www/html
|
||||
|
||||
And the *welcome* page is the HTML file at ``/var/www/html/index.html``.
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
less /etc/httpd/conf/httpd.conf
|
||||
|
||||
In this file, there is a line setting the `DocumentRoot`_ directive:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
DocumentRoot "/srv/http"
|
||||
<Directory "/srv/http">
|
||||
Options Indexes FollowSymLinks
|
||||
AllowOverride None
|
||||
Require all granted
|
||||
</Directory>
|
||||
|
||||
The *welcome* page of Arch Linux is a page showing the directory located
|
||||
at ``DocumentRoot``. This *directory* page is generated by the Module
|
||||
`mod_autoindex <https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html>`_:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
LoadModule autoindex_module modules/mod_autoindex.so
|
||||
...
|
||||
Include conf/extra/httpd-autoindex.conf
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code:: bash
|
||||
|
||||
less /etc/httpd/conf/httpd.conf
|
||||
|
||||
In this file, there is a line setting the ``DocumentRoot`` directive:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
DocumentRoot "/var/www/html"
|
||||
...
|
||||
<Directory "/var/www">
|
||||
AllowOverride None
|
||||
# Allow open access:
|
||||
Require all granted
|
||||
</Directory>
|
||||
|
||||
On fresh installations, the ``/var/www`` is empty and the *default
|
||||
welcome page* is shown, the configuration is located at::
|
||||
|
||||
less /etc/httpd/conf.d/welcome.conf
|
||||
|
||||
|
||||
.. _Debian's Apache layout:
|
||||
|
||||
Debian's Apache layout
|
||||
----------------------
|
||||
|
||||
Be aware, Debian's Apache layout is quite different from the standard Apache
|
||||
configuration. For details look at the apache2.README.Debian_
|
||||
(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on
|
||||
Debian:
|
||||
|
||||
* :man:`apache2ctl`: Apache HTTP server control interface
|
||||
* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules
|
||||
* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations
|
||||
* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites
|
||||
|
||||
.. _apache modules:
|
||||
|
||||
Apache modules
|
||||
--------------
|
||||
|
||||
To load additional modules, in most distributions you have to un-comment the
|
||||
lines with the corresponding LoadModule_ directive, except in :ref:`Debian's
|
||||
Apache layout`.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
:ref:`Debian's Apache layout` uses :man:`a2enmod` and :man:`a2dismod` to
|
||||
activate or disable modules:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H a2enmod ssl
|
||||
sudo -H a2enmod headers
|
||||
sudo -H a2enmod proxy
|
||||
sudo -H a2enmod proxy_http
|
||||
sudo -H a2enmod proxy_uwsgi
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_
|
||||
directives:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
LoadModule ssl_module modules/mod_ssl.so
|
||||
LoadModule headers_module modules/mod_headers.so
|
||||
LoadModule proxy_module modules/mod_proxy.so
|
||||
LoadModule proxy_http_module modules/mod_proxy_http.so
|
||||
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_
|
||||
directives:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
LoadModule ssl_module modules/mod_ssl.so
|
||||
LoadModule headers_module modules/mod_headers.so
|
||||
LoadModule proxy_module modules/mod_proxy.so
|
||||
LoadModule proxy_http_module modules/mod_proxy_http.so
|
||||
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
|
||||
|
||||
|
||||
.. _apache sites:
|
||||
|
||||
Apache sites
|
||||
------------
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
In :ref:`Debian's Apache layout` you create a ``searxng.conf`` with the
|
||||
``<Location /searxng >`` directive and save this file in the *sites
|
||||
available* folder at ``/etc/apache2/sites-available``. To enable the
|
||||
``searxng.conf`` use :man:`a2ensite`:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H a2ensite searxng.conf
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_
|
||||
directive:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
IncludeOptional sites-enabled/*.conf
|
||||
|
||||
Create two folders, one for the *available sites* and one for the *enabled sites*:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
mkdir -p /etc/httpd/sites-available
|
||||
mkdir -p /etc/httpd/sites-enabled
|
||||
|
||||
Create configuration at ``/etc/httpd/sites-available`` and place a
|
||||
symlink to ``sites-enabled``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H ln -s /etc/httpd/sites-available/searxng.conf \
|
||||
/etc/httpd/sites-enabled/searxng.conf
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_
|
||||
directive:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
IncludeOptional sites-enabled/*.conf
|
||||
|
||||
Create two folders, one for the *available sites* and one for the *enabled sites*:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
mkdir -p /etc/httpd/sites-available
|
||||
mkdir -p /etc/httpd/sites-enabled
|
||||
|
||||
Create configuration at ``/etc/httpd/sites-available`` and place a
|
||||
symlink to ``sites-enabled``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H ln -s /etc/httpd/sites-available/searxng.conf \
|
||||
/etc/httpd/sites-enabled/searxng.conf
|
||||
|
||||
|
||||
.. _apache searxng site:
|
||||
|
||||
Apache's SearXNG site
|
||||
=====================
|
||||
|
||||
.. _mod_uwsgi: https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi
|
||||
|
||||
.. sidebar:: uWSGI
|
||||
|
||||
Use mod_proxy_uwsgi_ / don't use the old mod_uwsgi_ anymore.
|
||||
|
||||
To proxy the incoming requests to the SearXNG instance Apache needs the
|
||||
mod_proxy_ module (:ref:`apache modules`).
|
||||
|
||||
.. sidebar:: HTTP headers
|
||||
|
||||
With ProxyPreserveHost_ the incoming ``Host`` header is passed to the proxied
|
||||
host.
|
||||
|
||||
Depending on what your SearXNG installation is listening on, you need a http
|
||||
mod_proxy_http_) or socket (mod_proxy_uwsgi_) communication to upstream.
|
||||
|
||||
The :ref:`installation scripts` installs the :ref:`reference setup
|
||||
<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default.
|
||||
You can install and activate your own ``searxng.conf`` like shown in
|
||||
:ref:`apache sites`.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: socket
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START apache socket
|
||||
:end-before: END apache socket
|
||||
|
||||
.. group-tab:: http
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START apache http
|
||||
:end-before: END apache http
|
||||
|
||||
.. _restart apache:
|
||||
|
||||
Restart service:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart apache2
|
||||
sudo -H service uwsgi restart searxng
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart httpd
|
||||
sudo -H systemctl restart uwsgi@searxng
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart httpd
|
||||
sudo -H touch /etc/uwsgi.d/searxng.ini
|
||||
|
||||
|
||||
disable logs
|
||||
============
|
||||
|
||||
For better privacy you can disable Apache logs. In the examples above activate
|
||||
one of the lines and `restart apache`_:
|
||||
|
||||
.. code:: apache
|
||||
|
||||
SetEnvIf Request_URI "/searxng" dontlog
|
||||
# CustomLog /dev/null combined env=dontlog
|
||||
|
||||
The ``CustomLog`` directive disables logs for the entire (virtual) server, use it
|
||||
when the URL of the service does not have a path component (``/searxng``), so when
|
||||
SearXNG is located at root (``/``).
|
||||
@ -1,194 +0,0 @@
|
||||
.. _installation docker:
|
||||
|
||||
================
|
||||
Docker Container
|
||||
================
|
||||
|
||||
.. _ENTRYPOINT: https://docs.docker.com/engine/reference/builder/#entrypoint
|
||||
.. _searxng/searxng @dockerhub: https://hub.docker.com/r/searxng/searxng
|
||||
.. _searxng-docker: https://github.com/searxng/searxng-docker
|
||||
.. _[caddy]: https://hub.docker.com/_/caddy
|
||||
.. _Redis: https://redis.io/
|
||||
|
||||
----
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- `searxng/searxng @dockerhub`_
|
||||
- :origin:`Dockerfile`
|
||||
- `Docker overview <https://docs.docker.com/get-started/overview>`_
|
||||
- `Docker Cheat Sheet <https://docs.docker.com/get-started/docker_cheatsheet.pdf>`_
|
||||
- `Alpine Linux <https://alpinelinux.org>`_
|
||||
`(wiki) <https://en.wikipedia.org/wiki/Alpine_Linux>`__
|
||||
`apt packages <https://pkgs.alpinelinux.org/packages>`_
|
||||
- Alpine's ``/bin/sh`` is :man:`dash`
|
||||
|
||||
**If you intend to create a public instance using Docker, use our well maintained
|
||||
docker container**
|
||||
|
||||
- `searxng/searxng @dockerhub`_.
|
||||
|
||||
.. sidebar:: hint
|
||||
|
||||
The rest of this article is of interest only to those who want to create and
|
||||
maintain their own Docker images.
|
||||
|
||||
The sources are hosted at searxng-docker_ and the container includes:
|
||||
|
||||
- a HTTPS reverse proxy `[caddy]`_ and
|
||||
- a Redis_ DB
|
||||
|
||||
The `default SearXNG setup <https://github.com/searxng/searxng-docker/blob/master/searxng/settings.yml>`_
|
||||
of this container:
|
||||
|
||||
- enables :ref:`limiter <limiter>` to protect against bots
|
||||
- enables :ref:`image proxy <image_proxy>` for better privacy
|
||||
- enables :ref:`cache busting <static_use_hash>` to save bandwith
|
||||
|
||||
----
|
||||
|
||||
|
||||
Get Docker
|
||||
==========
|
||||
|
||||
If you plan to build and maintain a docker image by yourself, make sure you have
|
||||
`Docker installed <https://docs.docker.com/get-docker/>`_. On Linux don't
|
||||
forget to add your user to the docker group (log out and log back in so that
|
||||
your group membership is re-evaluated):
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo usermod -a -G docker $USER
|
||||
|
||||
|
||||
searxng/searxng
|
||||
===============
|
||||
|
||||
.. sidebar:: ``docker run``
|
||||
|
||||
- `-\-rm <https://docs.docker.com/engine/reference/run/#clean-up---rm>`__
|
||||
automatically clean up when container exits
|
||||
- `-d <https://docs.docker.com/engine/reference/run/#detached--d>`__ start
|
||||
detached container
|
||||
- `-v <https://docs.docker.com/engine/reference/run/#volume-shared-filesystems>`__
|
||||
mount volume ``HOST:CONTAINER``
|
||||
|
||||
The docker image is based on :origin:`Dockerfile` and available from
|
||||
`searxng/searxng @dockerhub`_. Using the docker image is quite easy, for
|
||||
instance you can pull the `searxng/searxng @dockerhub`_ image and deploy a local
|
||||
instance using `docker run <https://docs.docker.com/engine/reference/run/>`_:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ mkdir my-instance
|
||||
$ cd my-instance
|
||||
$ export PORT=8080
|
||||
$ docker pull searxng/searxng
|
||||
$ docker run --rm \
|
||||
-d -p ${PORT}:8080 \
|
||||
-v "${PWD}/searxng:/etc/searxng" \
|
||||
-e "BASE_URL=http://localhost:$PORT/" \
|
||||
-e "INSTANCE_NAME=my-instance" \
|
||||
searxng/searxng
|
||||
2f998.... # container's ID
|
||||
|
||||
Open your WEB browser and visit the URL:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ xdg-open "http://localhost:$PORT"
|
||||
|
||||
Inside ``${PWD}/searxng``, you will find ``settings.yml`` and ``uwsgi.ini``. You
|
||||
can modify these files according to your needs and restart the Docker image.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ docker container restart 2f998
|
||||
|
||||
Use command ``container ls`` to list running containers, add flag `-a
|
||||
<https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list
|
||||
exited containers also. With ``container stop`` a running container can be
|
||||
stoped. To get rid of a container use ``container rm``:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ docker container ls
|
||||
CONTAINER ID IMAGE COMMAND CREATED ...
|
||||
2f998d725993 searxng/searxng "/sbin/tini -- /usr/…" 7 minutes ago ...
|
||||
|
||||
$ docker container stop 2f998
|
||||
$ docker container rm 2f998
|
||||
|
||||
.. sidebar:: Warning
|
||||
|
||||
This might remove all docker items, not only those from SearXNG.
|
||||
|
||||
If you won't use docker anymore and want to get rid of all conatiners & images
|
||||
use the following *prune* command:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ docker stop $(docker ps -aq) # stop all containers
|
||||
$ docker system prune # make some housekeeping
|
||||
$ docker rmi -f $(docker images -q) # drop all images
|
||||
|
||||
|
||||
shell inside container
|
||||
----------------------
|
||||
|
||||
.. sidebar:: Bashism
|
||||
|
||||
- `A tale of two shells: bash or dash <https://lwn.net/Articles/343924/>`_
|
||||
- `How to make bash scripts work in dash <http://mywiki.wooledge.org/Bashism>`_
|
||||
- `Checking for Bashisms <https://dev.to/bowmanjd/writing-bash-scripts-that-are-not-only-bash-checking-for-bashisms-and-testing-with-dash-1bli>`_
|
||||
|
||||
Like in many other distributions, Alpine's `/bin/sh
|
||||
<https://wiki.ubuntu.com/DashAsBinSh>`__ is :man:`dash`. Dash is meant to be
|
||||
`POSIX-compliant <https://pubs.opengroup.org/onlinepubs/9699919799>`__.
|
||||
Compared to debian, in the Alpine image :man:`bash` is not installed. The
|
||||
:origin:`dockerfiles/docker-entrypoint.sh` script is checked *against dash*
|
||||
(``make tests.shell``).
|
||||
|
||||
To open a shell inside the container:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ docker exec -it 2f998 sh
|
||||
|
||||
|
||||
Build the image
|
||||
===============
|
||||
|
||||
It's also possible to build SearXNG from the embedded :origin:`Dockerfile`::
|
||||
|
||||
$ git clone https://github.com/searxng/searxng.git
|
||||
$ cd searxng
|
||||
$ make docker.build
|
||||
...
|
||||
Successfully built 49586c016434
|
||||
Successfully tagged searxng/searxng:latest
|
||||
Successfully tagged searxng/searxng:1.0.0-209-9c823800-dirty
|
||||
|
||||
$ docker images
|
||||
REPOSITORY TAG IMAGE ID CREATED SIZE
|
||||
searxng/searxng 1.0.0-209-9c823800-dirty 49586c016434 13 minutes ago 308MB
|
||||
searxng/searxng latest 49586c016434 13 minutes ago 308MB
|
||||
alpine 3.13 6dbb9cc54074 3 weeks ago 5.61MB
|
||||
|
||||
|
||||
Command line
|
||||
============
|
||||
|
||||
.. sidebar:: docker run
|
||||
|
||||
Use flags ``-it`` for `interactive processes
|
||||
<https://docs.docker.com/engine/reference/run/#foreground>`__.
|
||||
|
||||
In the :origin:`Dockerfile` the ENTRYPOINT_ is defined as
|
||||
:origin:`dockerfiles/docker-entrypoint.sh`
|
||||
|
||||
.. code:: sh
|
||||
|
||||
docker run --rm -it searxng/searxng -h
|
||||
|
||||
.. program-output:: ../dockerfiles/docker-entrypoint.sh -h
|
||||
@ -1,252 +0,0 @@
|
||||
.. _installation nginx:
|
||||
|
||||
=====
|
||||
NGINX
|
||||
=====
|
||||
|
||||
.. _nginx:
|
||||
https://docs.nginx.com/nginx/admin-guide/
|
||||
.. _nginx server configuration:
|
||||
https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers
|
||||
.. _nginx beginners guide:
|
||||
https://nginx.org/en/docs/beginners_guide.html
|
||||
.. _Getting Started wiki:
|
||||
https://www.nginx.com/resources/wiki/start/
|
||||
.. _uWSGI support from nginx:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html
|
||||
.. _uwsgi_params:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx
|
||||
.. _SCRIPT_NAME:
|
||||
https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name
|
||||
|
||||
This section explains how to set up a SearXNG instance using the HTTP server nginx_.
|
||||
If you have used the :ref:`installation scripts` and do not have any special preferences
|
||||
you can install the :ref:`SearXNG site <nginx searxng site>` using
|
||||
:ref:`searxng.sh <searxng.sh overview>`:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/searxng.sh install nginx
|
||||
|
||||
If you have special interests or problems with setting up nginx, the following
|
||||
section might give you some guidance.
|
||||
|
||||
|
||||
.. sidebar:: further reading
|
||||
|
||||
- nginx_
|
||||
- `nginx beginners guide`_
|
||||
- `nginx server configuration`_
|
||||
- `Getting Started wiki`_
|
||||
- `uWSGI support from nginx`_
|
||||
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
The nginx HTTP server
|
||||
=====================
|
||||
|
||||
If nginx_ is not installed, install it now.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H apt-get install nginx
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
sudo -H pacman -S nginx-mainline
|
||||
sudo -H systemctl enable nginx
|
||||
sudo -H systemctl start nginx
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
sudo -H dnf install nginx
|
||||
sudo -H systemctl enable nginx
|
||||
sudo -H systemctl start nginx
|
||||
|
||||
Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you
|
||||
see a *Fedora Webserver - Test Page*. The test page comes from the default
|
||||
`nginx server configuration`_. How this default site is configured,
|
||||
depends on the linux distribution:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
less /etc/nginx/nginx.conf
|
||||
|
||||
There is one line that includes site configurations from:
|
||||
|
||||
.. code:: nginx
|
||||
|
||||
include /etc/nginx/sites-enabled/*;
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
less /etc/nginx/nginx.conf
|
||||
|
||||
There is a configuration section named ``server``:
|
||||
|
||||
.. code-block:: nginx
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name localhost;
|
||||
# ...
|
||||
}
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
less /etc/nginx/nginx.conf
|
||||
|
||||
There is one line that includes site configurations from:
|
||||
|
||||
.. code:: nginx
|
||||
|
||||
include /etc/nginx/conf.d/*.conf;
|
||||
|
||||
|
||||
.. _nginx searxng site:
|
||||
|
||||
NGINX's SearXNG site
|
||||
====================
|
||||
|
||||
Now you have to create a configuration file (``searxng.conf``) for the SearXNG
|
||||
site. If nginx_ is new to you, the `nginx beginners guide`_ is a good starting
|
||||
point and the `Getting Started wiki`_ is always a good resource *to keep in the
|
||||
pocket*.
|
||||
|
||||
Depending on what your SearXNG installation is listening on, you need a http or socket
|
||||
communication to upstream.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: socket
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START nginx socket
|
||||
:end-before: END nginx socket
|
||||
|
||||
.. group-tab:: http
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START nginx http
|
||||
:end-before: END nginx http
|
||||
|
||||
The :ref:`installation scripts` installs the :ref:`reference setup
|
||||
<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
Create configuration at ``/etc/nginx/sites-available/`` and place a
|
||||
symlink to ``sites-enabled``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H ln -s /etc/nginx/sites-available/searxng.conf \
|
||||
/etc/nginx/sites-enabled/searxng.conf
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
In the ``/etc/nginx/nginx.conf`` file, in the ``server`` section add a
|
||||
`include <https://nginx.org/en/docs/ngx_core_module.html#include>`_
|
||||
directive:
|
||||
|
||||
.. code:: nginx
|
||||
|
||||
server {
|
||||
# ...
|
||||
include /etc/nginx/default.d/*.conf;
|
||||
# ...
|
||||
}
|
||||
|
||||
Create two folders, one for the *available sites* and one for the *enabled sites*:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
mkdir -p /etc/nginx/default.d
|
||||
mkdir -p /etc/nginx/default.apps-available
|
||||
|
||||
Create configuration at ``/etc/nginx/default.apps-available`` and place a
|
||||
symlink to ``default.d``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \
|
||||
/etc/nginx/default.d/searxng.conf
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
Create a folder for the *available sites*:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
mkdir -p /etc/nginx/default.apps-available
|
||||
|
||||
Create configuration at ``/etc/nginx/default.apps-available`` and place a
|
||||
symlink to ``conf.d``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \
|
||||
/etc/nginx/conf.d/searxng.conf
|
||||
|
||||
Restart services:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart nginx
|
||||
sudo -H service uwsgi restart searxng
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart nginx
|
||||
sudo -H systemctl restart uwsgi@searxng
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo -H systemctl restart nginx
|
||||
sudo -H touch /etc/uwsgi.d/searxng.ini
|
||||
|
||||
|
||||
Disable logs
|
||||
============
|
||||
|
||||
For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``.
|
||||
|
||||
.. code:: nginx
|
||||
|
||||
http {
|
||||
# ...
|
||||
access_log /dev/null;
|
||||
error_log /dev/null;
|
||||
# ...
|
||||
}
|
||||
@ -1,62 +0,0 @@
|
||||
.. _installation scripts:
|
||||
|
||||
===================
|
||||
Installation Script
|
||||
===================
|
||||
|
||||
.. sidebar:: Update the OS first!
|
||||
|
||||
To avoid unwanted side effects, update your OS before installing SearXNG.
|
||||
|
||||
The following will install a setup as shown in :ref:`the reference architecture
|
||||
<arch public>`. First you need to get a clone of the repository. The clone is only needed for
|
||||
the installation procedure and some maintenance tasks.
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- :ref:`toolboxing`
|
||||
|
||||
Jump to a folder that is readable by *others* and start to clone SearXNG,
|
||||
alternatively you can create your own fork and clone from there.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ cd ~/Downloads
|
||||
$ git clone https://github.com/searxng/searxng.git searxng
|
||||
$ cd searxng
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- :ref:`inspect searxng`
|
||||
|
||||
To install a SearXNG :ref:`reference setup <use_default_settings.yml>`
|
||||
including a :ref:`uWSGI setup <architecture uWSGI>` as described in the
|
||||
:ref:`installation basic` and in the :ref:`searxng uwsgi` section type:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/searxng.sh install all
|
||||
|
||||
.. attention::
|
||||
|
||||
For the installation procedure, use a *sudoer* login to run the scripts. If
|
||||
you install from ``root``, take into account that the scripts are creating a
|
||||
``searxng`` user. In the installation procedure this new created user does
|
||||
need read access to the cloned SearXNG repository, which is not the case if you clone
|
||||
it into a folder below ``/root``!
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- :ref:`update searxng`
|
||||
|
||||
.. _caddy: https://hub.docker.com/_/caddy
|
||||
|
||||
When all services are installed and running fine, you can add SearXNG to your
|
||||
HTTP server. We do not have any preferences for the HTTP server, you can use
|
||||
whatever you prefer.
|
||||
|
||||
We use caddy in our :ref:`docker image <installation docker>` and we have
|
||||
implemented installation procedures for:
|
||||
|
||||
- :ref:`installation nginx`
|
||||
- :ref:`installation apache`
|
||||
@ -1,132 +0,0 @@
|
||||
.. _installation basic:
|
||||
|
||||
=========================
|
||||
Step by step installation
|
||||
=========================
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
In this section we show the setup of a SearXNG instance that will be installed
|
||||
by the :ref:`installation scripts`.
|
||||
|
||||
.. _install packages:
|
||||
|
||||
Install packages
|
||||
================
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START distro-packages
|
||||
:end-before: END distro-packages
|
||||
|
||||
.. hint::
|
||||
|
||||
This installs also the packages needed by :ref:`searxng uwsgi`
|
||||
|
||||
.. _create searxng user:
|
||||
|
||||
Create user
|
||||
===========
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START create user
|
||||
:end-before: END create user
|
||||
|
||||
.. _searxng-src:
|
||||
|
||||
Install SearXNG & dependencies
|
||||
==============================
|
||||
|
||||
Start a interactive shell from new created user and clone SearXNG:
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START clone searxng
|
||||
:end-before: END clone searxng
|
||||
|
||||
In the same shell create *virtualenv*:
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START create virtualenv
|
||||
:end-before: END create virtualenv
|
||||
|
||||
To install SearXNG's dependencies, exit the SearXNG *bash* session you opened above
|
||||
and start a new one. Before installing, check if your *virtualenv* was sourced
|
||||
from the login (*~/.profile*):
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START manage.sh update_packages
|
||||
:end-before: END manage.sh update_packages
|
||||
|
||||
.. tip::
|
||||
|
||||
Open a second terminal for the configuration tasks and leave the ``(searx)$``
|
||||
terminal open for the tasks below.
|
||||
|
||||
|
||||
.. _use_default_settings.yml:
|
||||
|
||||
Configuration
|
||||
=============
|
||||
|
||||
.. sidebar:: ``use_default_settings: True``
|
||||
|
||||
- :ref:`settings global`
|
||||
- :ref:`settings location`
|
||||
- :ref:`settings use_default_settings`
|
||||
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
|
||||
|
||||
To create a initial ``/etc/searxng/settings.yml`` we recommend to start with a
|
||||
copy of the file :origin:`utils/templates/etc/searxng/settings.yml`. This setup
|
||||
:ref:`use default settings <settings use_default_settings>` from
|
||||
:origin:`searx/settings.yml` and is shown in the tab *"Use default settings"*
|
||||
below. This setup:
|
||||
|
||||
- enables :ref:`limiter <limiter>` to protect against bots
|
||||
- enables :ref:`image proxy <image_proxy>` for better privacy
|
||||
- enables :ref:`cache busting <static_use_hash>` to save bandwith
|
||||
|
||||
Modify the ``/etc/searxng/settings.yml`` to your needs:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Use default settings
|
||||
|
||||
.. literalinclude:: ../../utils/templates/etc/searxng/settings.yml
|
||||
:language: yaml
|
||||
:end-before: # hostname_replace:
|
||||
|
||||
To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml`
|
||||
|
||||
.. group-tab:: searx/settings.yml
|
||||
|
||||
.. literalinclude:: ../../searx/settings.yml
|
||||
:language: yaml
|
||||
:end-before: # hostname_replace:
|
||||
|
||||
To see the entire file jump to :origin:`searx/settings.yml`
|
||||
|
||||
For a *minimal setup* you need to set ``server:secret_key``.
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng config
|
||||
:end-before: END searxng config
|
||||
|
||||
|
||||
Check
|
||||
=====
|
||||
|
||||
To check your SearXNG setup, optional enable debugging and start the *webapp*.
|
||||
SearXNG looks at the exported environment ``$SEARXNG_SETTINGS_PATH`` for a
|
||||
configuration file.
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START check searxng installation
|
||||
:end-before: END check searxng installation
|
||||
|
||||
If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the
|
||||
debug option in ``settings.yml``. You can now exit SearXNG user bash session (enter exit
|
||||
command twice). At this point SearXNG is not demonized; uwsgi allows this.
|
||||
|
||||
@ -1,268 +0,0 @@
|
||||
.. _searxng uwsgi:
|
||||
|
||||
=====
|
||||
uWSGI
|
||||
=====
|
||||
|
||||
.. sidebar:: further reading
|
||||
|
||||
- `systemd.unit`_
|
||||
- `uWSGI Emperor`_
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
|
||||
.. _One service per app in systemd:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd
|
||||
.. _uWSGI Emperor:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html
|
||||
.. _uwsgi ini file:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files
|
||||
.. _systemd unit template:
|
||||
http://0pointer.de/blog/projects/instances.html
|
||||
|
||||
|
||||
Origin uWSGI
|
||||
============
|
||||
|
||||
.. _Tyrant mode:
|
||||
https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html#tyrant-mode-secure-multi-user-hosting
|
||||
|
||||
How uWSGI is implemented by distributors varies. The uWSGI project itself
|
||||
recommends two methods:
|
||||
|
||||
1. `systemd.unit`_ template file as described here `One service per app in systemd`_:
|
||||
|
||||
There is one `systemd unit template`_ on the system installed and one `uwsgi
|
||||
ini file`_ per uWSGI-app placed at dedicated locations. Take archlinux and a
|
||||
``searxng.ini`` as example::
|
||||
|
||||
systemd template unit: /usr/lib/systemd/system/uwsgi@.service
|
||||
contains: [Service]
|
||||
ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/%I.ini
|
||||
|
||||
SearXNG application: /etc/uwsgi/searxng.ini
|
||||
links to: /etc/uwsgi/apps-available/searxng.ini
|
||||
|
||||
The SearXNG app (template ``/etc/uwsgi/%I.ini``) can be maintained as known
|
||||
from common systemd units:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ systemctl enable uwsgi@searxng
|
||||
$ systemctl start uwsgi@searxng
|
||||
$ systemctl restart uwsgi@searxng
|
||||
$ systemctl stop uwsgi@searxng
|
||||
|
||||
2. The `uWSGI Emperor`_ which fits for maintaining a large range of uwsgi
|
||||
apps and there is a `Tyrant mode`_ to secure multi-user hosting.
|
||||
|
||||
The Emperor mode is a special uWSGI instance that will monitor specific
|
||||
events. The Emperor mode (the service) is started by a (common, not template)
|
||||
systemd unit.
|
||||
|
||||
The Emperor service will scan specific directories for `uwsgi ini file`_\s
|
||||
(also know as *vassals*). If a *vassal* is added, removed or the timestamp is
|
||||
modified, a corresponding action takes place: a new uWSGI instance is started,
|
||||
reload or stopped. Take Fedora and a ``searxng.ini`` as example::
|
||||
|
||||
to install & start SearXNG instance create --> /etc/uwsgi.d/searxng.ini
|
||||
to reload the instance edit timestamp --> touch /etc/uwsgi.d/searxng.ini
|
||||
to stop instance remove ini --> rm /etc/uwsgi.d/searxng.ini
|
||||
|
||||
|
||||
Distributors
|
||||
============
|
||||
|
||||
The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors
|
||||
mostly offer their users, even if they differ in the way they implement both
|
||||
modes and their defaults. Another point they might differ in is the packaging of
|
||||
plugins (if so, compare :ref:`install packages`) and what the default python
|
||||
interpreter is (python2 vs. python3).
|
||||
|
||||
While archlinux does not start a uWSGI service by default, Fedora (RHEL) starts
|
||||
a Emperor in `Tyrant mode`_ by default (you should have read :ref:`uWSGI Tyrant
|
||||
mode pitfalls`). Worth to know; debian (ubuntu) follow a complete different
|
||||
approach, read see :ref:`Debian's uWSGI layout`.
|
||||
|
||||
.. _Debian's uWSGI layout:
|
||||
|
||||
Debian's uWSGI layout
|
||||
---------------------
|
||||
|
||||
.. _uwsgi.README.Debian:
|
||||
https://salsa.debian.org/uwsgi-team/uwsgi/-/raw/debian/latest/debian/uwsgi.README.Debian
|
||||
|
||||
Be aware, Debian's uWSGI layout is quite different from the standard uWSGI
|
||||
configuration. Your are familiar with :ref:`Debian's Apache layout`? .. they do a
|
||||
similar thing for the uWSGI infrastructure. The folders are::
|
||||
|
||||
/etc/uwsgi/apps-available/
|
||||
/etc/uwsgi/apps-enabled/
|
||||
|
||||
The `uwsgi ini file`_ is enabled by a symbolic link::
|
||||
|
||||
ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/
|
||||
|
||||
More details can be found in the uwsgi.README.Debian_
|
||||
(``/usr/share/doc/uwsgi/README.Debian.gz``). Some commands you should know on
|
||||
Debian:
|
||||
|
||||
.. code:: none
|
||||
|
||||
Commands recognized by init.d script
|
||||
====================================
|
||||
|
||||
You can issue to init.d script following commands:
|
||||
* start | starts daemon
|
||||
* stop | stops daemon
|
||||
* reload | sends to daemon SIGHUP signal
|
||||
* force-reload | sends to daemon SIGTERM signal
|
||||
* restart | issues 'stop', then 'start' commands
|
||||
* status | shows status of daemon instance (running/not running)
|
||||
|
||||
'status' command must be issued with exactly one argument: '<confname>'.
|
||||
|
||||
Controlling specific instances of uWSGI
|
||||
=======================================
|
||||
|
||||
You could control specific instance(s) by issuing:
|
||||
|
||||
SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi <command> <confname> <confname>...
|
||||
|
||||
where:
|
||||
* <command> is one of 'start', 'stop' etc.
|
||||
* <confname> is the name of configuration file (without extension)
|
||||
|
||||
For example, this is how instance for /etc/uwsgi/apps-enabled/hello.xml is
|
||||
started:
|
||||
|
||||
SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi start hello
|
||||
|
||||
|
||||
.. _uWSGI maintenance:
|
||||
|
||||
uWSGI maintenance
|
||||
=================
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-description ubuntu-20.04
|
||||
:end-before: END searxng uwsgi-description ubuntu-20.04
|
||||
|
||||
.. hotfix: a bug group-tab need this comment
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-description arch
|
||||
:end-before: END searxng uwsgi-description arch
|
||||
|
||||
.. hotfix: a bug group-tab need this comment
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-description fedora
|
||||
:end-before: END searxng uwsgi-description fedora
|
||||
|
||||
|
||||
.. _uwsgi setup:
|
||||
|
||||
uWSGI setup
|
||||
===========
|
||||
|
||||
Create the configuration ini-file according to your distribution and restart the
|
||||
uwsgi application. As shown below, the :ref:`installation scripts` installs by
|
||||
default:
|
||||
|
||||
- a uWSGI setup that listens on a socket and
|
||||
- enables :ref:`cache busting <static_use_hash>`.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-appini ubuntu-20.04
|
||||
:end-before: END searxng uwsgi-appini ubuntu-20.04
|
||||
|
||||
.. hotfix: a bug group-tab need this comment
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-appini arch
|
||||
:end-before: END searxng uwsgi-appini arch
|
||||
|
||||
.. hotfix: a bug group-tab need this comment
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst
|
||||
:start-after: START searxng uwsgi-appini fedora
|
||||
:end-before: END searxng uwsgi-appini fedora
|
||||
|
||||
|
||||
.. _uWSGI Tyrant mode pitfalls:
|
||||
|
||||
Pitfalls of the Tyrant mode
|
||||
===========================
|
||||
|
||||
The implementation of the process owners and groups in the `Tyrant mode`_ is
|
||||
somewhat unusual and requires special consideration. In `Tyrant mode`_ mode the
|
||||
Emperor will run the vassal using the UID/GID of the vassal configuration file
|
||||
(user and group of the app ``.ini`` file).
|
||||
|
||||
.. _#2099@uWSGI: https://github.com/unbit/uwsgi/issues/2099
|
||||
.. _#752@uWSGI: https://github.com/unbit/uwsgi/pull/752
|
||||
.. _#2425uWSGI: https://github.com/unbit/uwsgi/issues/2425
|
||||
|
||||
Without option ``emperor-tyrant-initgroups=true`` in ``/etc/uwsgi.ini`` the
|
||||
process won't get the additional groups, but this option is not available in
|
||||
2.0.x branch (see `#2099@uWSGI`_) the feature `#752@uWSGI`_ has been merged (on
|
||||
Oct. 2014) to the master branch of uWSGI but had never been released; the last
|
||||
major release is from Dec. 2013, since the there had been only bugfix releases
|
||||
(see `#2425uWSGI`_). To shorten up:
|
||||
|
||||
**In Tyrant mode, there is no way to get additional groups, and the uWSGI
|
||||
process misses additional permissions that may be needed.**
|
||||
|
||||
For example on Fedora (RHEL): If you try to install a redis DB with socket
|
||||
communication and you want to connect to it from the SearXNG uWSGI, you will see a
|
||||
*Permission denied* in the log of your instance::
|
||||
|
||||
ERROR:searx.redisdb: [searxng (993)] can't connect redis DB ...
|
||||
ERROR:searx.redisdb: Error 13 connecting to unix socket: /usr/local/searxng-redis/run/redis.sock. Permission denied.
|
||||
ERROR:searx.plugins.limiter: init limiter DB failed!!!
|
||||
|
||||
Even if your *searxng* user of the uWSGI process is added to additional groups
|
||||
to give access to the socket from the redis DB::
|
||||
|
||||
$ groups searxng
|
||||
searxng : searxng searxng-redis
|
||||
|
||||
To see the effective groups of the uwsgi process, you have to look at the status
|
||||
of the process, by example::
|
||||
|
||||
$ ps -aef | grep '/usr/sbin/uwsgi --ini searxng.ini'
|
||||
searxng 93 92 0 12:43 ? 00:00:00 /usr/sbin/uwsgi --ini searxng.ini
|
||||
searxng 186 93 0 12:44 ? 00:00:01 /usr/sbin/uwsgi --ini searxng.ini
|
||||
|
||||
Here you can see that the additional "Groups" of PID 186 are unset (missing gid
|
||||
of ``searxng-redis``)::
|
||||
|
||||
$ cat /proc/186/task/186/status
|
||||
...
|
||||
Uid: 993 993 993 993
|
||||
Gid: 993 993 993 993
|
||||
FDSize: 128
|
||||
Groups:
|
||||
...
|
||||
@ -0,0 +1,26 @@
|
||||
=========================
|
||||
How to setup result proxy
|
||||
=========================
|
||||
|
||||
.. _morty: https://github.com/asciimoo/morty
|
||||
.. _morty's README: https://github.com/asciimoo/morty
|
||||
|
||||
By default searx can only act as an image proxy for result images, but it is
|
||||
possible to proxify all the result URLs with an external service, morty_.
|
||||
|
||||
To use this feature, morty has to be installed and activated in searx's
|
||||
``settings.yml``.
|
||||
|
||||
Add the following snippet to your ``settings.yml`` and restart searx:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
result_proxy:
|
||||
url : http://127.0.0.1:3000/
|
||||
key : your_morty_proxy_key
|
||||
|
||||
``url``
|
||||
Is the address of the running morty service.
|
||||
|
||||
``key``
|
||||
Is an optional argument, see `morty's README`_ for more information.
|
||||
@ -0,0 +1,181 @@
|
||||
.. _settings.yml:
|
||||
|
||||
================
|
||||
``settings.yml``
|
||||
================
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`search API`
|
||||
|
||||
This page describe the options possibilities of the settings.yml file.
|
||||
|
||||
.. _settings global:
|
||||
|
||||
Global Settings
|
||||
===============
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
server:
|
||||
port : 8888
|
||||
secret_key : "ultrasecretkey" # change this!
|
||||
debug : False # debug mode, only for development
|
||||
request_timeout : 2.0 # seconds
|
||||
base_url : False # set custom base_url (or False)
|
||||
themes_path : "" # custom ui themes path
|
||||
default_theme : oscar # ui theme
|
||||
useragent_suffix : "" # suffix of searx_useragent, could contain
|
||||
# informations like admins email address
|
||||
image_proxy : False # proxying image results through searx
|
||||
default_locale : "" # default interface locale
|
||||
|
||||
# uncomment below section if you want to use a proxy
|
||||
|
||||
#outgoing_proxies :
|
||||
# http : http://127.0.0.1:8080
|
||||
# https: http://127.0.0.1:8080
|
||||
|
||||
# uncomment below section only if you have more than one network interface
|
||||
# which can be the source of outgoing search requests
|
||||
|
||||
#source_ips:
|
||||
# - 1.1.1.1
|
||||
# - 1.1.1.2
|
||||
|
||||
locales:
|
||||
en : English
|
||||
de : Deutsch
|
||||
he : Hebrew
|
||||
hu : Magyar
|
||||
fr : Français
|
||||
es : Español
|
||||
it : Italiano
|
||||
nl : Nederlands
|
||||
ja : 日本語 (Japanese)
|
||||
tr : Türkçe
|
||||
ru : Russian
|
||||
ro : Romanian
|
||||
|
||||
|
||||
``port`` :
|
||||
Port number of the searx web application if you run it directly using ``python
|
||||
searx/webapp.py``. Doesn't apply to searx running on Apache or Nginx.
|
||||
|
||||
``secret_key`` :
|
||||
Used for cryptography purpose.
|
||||
|
||||
``debug`` :
|
||||
Allow a more detailed log if you run searx directly. Display *detailed* error
|
||||
messages in the browser too, so this must be deactivated in production.
|
||||
|
||||
``request_timeout`` :
|
||||
Global timeout of the requests made to others engines in seconds. A bigger
|
||||
timeout will allow to wait for answers from slow engines, but in consequence
|
||||
will slow searx reactivity (the result page may take the time specified in the
|
||||
timeout to load)
|
||||
|
||||
``base_url`` :
|
||||
The base URL where searx is deployed. Used to create correct inbound links.
|
||||
|
||||
``themes_path`` :
|
||||
Path to where the themes are located. If you didn't develop anything, leave it
|
||||
blank.
|
||||
|
||||
``default_theme`` :
|
||||
Name of the theme you want to use by default on you searx instance.
|
||||
|
||||
``useragent_suffix`` :
|
||||
Suffix to the user-agent searx uses to send requests to others engines. If an
|
||||
engine wish to block you, a contact info here may be useful to avoid that.
|
||||
|
||||
``image_proxy`` :
|
||||
Allow your instance of searx of being able to proxy images. Uses memory space.
|
||||
|
||||
``default_locale`` :
|
||||
Aearx interface language. If blank, the locale is detected by using the
|
||||
browser language. If it doesn't work, or you are deploying a language
|
||||
specific instance of searx, a locale can be defined using an ISO language
|
||||
code, like ``fr``, ``en``, ``de``.
|
||||
|
||||
.. _requests proxies: http://docs.python-requests.org/en/latest/user/advanced/#proxies
|
||||
.. _PR SOCKS support: https://github.com/kennethreitz/requests/pull/478
|
||||
|
||||
``outgoing_proxies`` :
|
||||
Define a proxy you wish to use, see `requests proxies`_. SOCKS proxies are
|
||||
not supported / see `PR SOCKS support`.
|
||||
|
||||
``source_ips`` :
|
||||
If you use multiple nework interfaces, define from which IP the requests must
|
||||
be made.
|
||||
|
||||
``locales`` :
|
||||
Locales codes and their names. Available translations of searx interface.
|
||||
|
||||
|
||||
.. _settings engine:
|
||||
|
||||
Engine settings
|
||||
===============
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`engines-dev`
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- name : bing
|
||||
engine : bing
|
||||
shortcut : bi
|
||||
base_url : 'https://{language}.wikipedia.org/'
|
||||
categories : general
|
||||
timeout : 3.0
|
||||
api_key : 'apikey'
|
||||
disabled : True
|
||||
language : en_US
|
||||
|
||||
``name`` :
|
||||
Name that will be used accross searx to define this engine. In settings, on
|
||||
the result page...
|
||||
|
||||
``engine`` :
|
||||
Name of the python file used to handle requests and responses to and from this
|
||||
search engine.
|
||||
|
||||
``shortcut`` :
|
||||
Code used to execute bang requests (in this case using ``!bi`` or ``?bi``)
|
||||
|
||||
``base_url`` : optional
|
||||
Part of the URL that should be stable accross every request. Can be useful to
|
||||
use multiple sites using only one engine, or updating the site URL without
|
||||
touching at the code.
|
||||
|
||||
``categories`` : optional
|
||||
Define in which categories this engine will be active. Most of the time, it is
|
||||
defined in the code of the engine, but in a few cases it is useful, like when
|
||||
describing multiple search engine using the same code.
|
||||
|
||||
``timeout`` : optional
|
||||
Timeout of the search with the current search engine. **Be careful, it will
|
||||
modify the global timeout of searx.**
|
||||
|
||||
``api_key`` : optional
|
||||
In a few cases, using an API needs the use of a secret key. How to obtain them
|
||||
is described in the file.
|
||||
|
||||
``disabled`` : optional
|
||||
To disable by default the engine, but not deleting it. It will allow the user
|
||||
to manually activate it in the settings.
|
||||
|
||||
``language`` : optional
|
||||
If you want to use another language for a specific engine, you can define it
|
||||
by using the full ISO code of language and country, like ``fr_FR``, ``en_US``,
|
||||
``de_DE``.
|
||||
|
||||
``weigth`` : default ``1``
|
||||
Weighting of the results of this engine.
|
||||
|
||||
.. note::
|
||||
|
||||
A few more options are possible, but they are pretty specific to some
|
||||
engines, and so won't be described here.
|
||||
@ -1,136 +0,0 @@
|
||||
===================
|
||||
SearXNG maintenance
|
||||
===================
|
||||
|
||||
.. sidebar:: further read
|
||||
|
||||
- :ref:`toolboxing`
|
||||
- :ref:`uWSGI maintenance`
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
.. _update searxng:
|
||||
|
||||
How to update
|
||||
=============
|
||||
|
||||
How to update depends on the :ref:`installation` method. If you have used the
|
||||
:ref:`installation scripts`, use the ``update`` command from the :ref:`searxng.sh`
|
||||
script.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
sudo -H ./utils/searxng.sh instance update
|
||||
|
||||
.. _inspect searxng:
|
||||
|
||||
How to inspect & debug
|
||||
======================
|
||||
|
||||
How to debug depends on the :ref:`installation` method. If you have used the
|
||||
:ref:`installation scripts`, use the ``inspect`` command from the :ref:`searxng.sh`
|
||||
script.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
sudo -H ./utils/searxng.sh instance inspect
|
||||
|
||||
.. _migrate and stay tuned:
|
||||
|
||||
Migrate and stay tuned!
|
||||
=======================
|
||||
|
||||
.. sidebar:: info
|
||||
|
||||
- :pull:`1332`
|
||||
- :pull:`456`
|
||||
- :pull:`A comment about rolling release <446#issuecomment-954730358>`
|
||||
|
||||
SearXNG is a *rolling release*; each commit to the master branch is a release.
|
||||
SearXNG is growing rapidly, the services and opportunities are change every now
|
||||
and then, to name just a few:
|
||||
|
||||
- Bot protection has been switched from filtron to SearXNG's :ref:`limiter
|
||||
<limiter>`, this requires a :ref:`Redis <settings redis>` database.
|
||||
|
||||
- The image proxy morty is no longer needed, it has been replaced by the
|
||||
:ref:`image proxy <image_proxy>` from SearXNG.
|
||||
|
||||
- To save bandwith :ref:`cache busting <static_use_hash>` has been implemented.
|
||||
To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi
|
||||
setup`.
|
||||
|
||||
To stay tuned and get in use of the new features, instance maintainers have to
|
||||
update the SearXNG code regularly (see :ref:`update searxng`). As the above
|
||||
examples show, this is not always enough, sometimes services have to be set up
|
||||
or reconfigured and sometimes services that are no longer needed should be
|
||||
uninstalled.
|
||||
|
||||
.. hint::
|
||||
|
||||
First of all: SearXNG is installed by the script :ref:`searxng.sh`. If you
|
||||
have old filtron, morty or searx setup you should consider complete
|
||||
uninstall/reinstall.
|
||||
|
||||
Here you will find a list of changes that affect the infrastructure. Please
|
||||
check to what extent it is necessary to update your installations:
|
||||
|
||||
:pull:`1595`: ``[fix] uWSGI: increase buffer-size``
|
||||
Re-install uWSGI (:ref:`searxng.sh`) or fix your uWSGI ``searxng.ini``
|
||||
file manually.
|
||||
|
||||
|
||||
remove obsolete services
|
||||
------------------------
|
||||
|
||||
If your searx instance was installed *"Step by step"* or by the *"Installation
|
||||
scripts"*, you need to undo the installation procedure completely. If you have
|
||||
morty & filtron installed, it is recommended to uninstall these services also.
|
||||
In case of scripts, to uninstall use the scripts from the origin you installed
|
||||
searx from or try::
|
||||
|
||||
$ sudo -H ./utils/filtron.sh remove all
|
||||
$ sudo -H ./utils/morty.sh remove all
|
||||
$ sudo -H ./utils/searx.sh remove all
|
||||
|
||||
.. hint::
|
||||
|
||||
If you are migrate from searx take into account that the ``.config.sh`` is no
|
||||
longer used.
|
||||
|
||||
If you upgrade from searx or from before :pull:`1332` has been merged and you
|
||||
have filtron and/or morty installed, don't forget to remove HTTP sites.
|
||||
|
||||
Apache::
|
||||
|
||||
$ sudo -H ./utils/filtron.sh apache remove
|
||||
$ sudo -H ./utils/morty.sh apache remove
|
||||
|
||||
nginx::
|
||||
|
||||
$ sudo -H ./utils/filtron.sh nginx remove
|
||||
$ sudo -H ./utils/morty.sh nginx remove
|
||||
|
||||
|
||||
|
||||
Check after Installation
|
||||
------------------------
|
||||
|
||||
Once you have done your installation, you can run a SearXNG *check* procedure,
|
||||
to see if there are some left overs. In this example there exists a *old*
|
||||
``/etc/searx/settings.yml``::
|
||||
|
||||
$ sudo -H ./utils/searxng.sh instance check
|
||||
|
||||
SearXNG checks
|
||||
--------------
|
||||
ERROR: settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/
|
||||
INFO: [OK] (old) account 'searx' does not exists
|
||||
INFO: [OK] (old) account 'filtron' does not exists
|
||||
INFO: [OK] (old) account 'morty' does not exists
|
||||
...
|
||||
INFO searx.redisdb : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock'
|
||||
INFO searx.redisdb : connected to Redis
|
||||
@ -0,0 +1,43 @@
|
||||
=============================================================
|
||||
Searx admin interface
|
||||
=============================================================
|
||||
|
||||
.. _searx-admin: https://github.com/kvch/searx-admin#searx-admin
|
||||
.. _NLnet Foundation: https://nlnet.nl/
|
||||
|
||||
manage your instance from your browser
|
||||
|
||||
.. sidebar:: Installation
|
||||
|
||||
Installation guide can be found in the repository of searx-admin_.
|
||||
|
||||
One of the biggest advantages of searx is being extremely customizable. But at
|
||||
first it can be daunting to newcomers. A barrier of taking advantage of this
|
||||
feature is our ugly settings file which is sometimes hard to understand and
|
||||
edit.
|
||||
|
||||
To make self-hosting searx more accessible a new tool is introduced, called
|
||||
``searx-admin``. It is a web application which is capable of managing your
|
||||
instance and manipulating its settings via a web UI. It aims to replace editing
|
||||
of ``settings.yml`` for less experienced administrators or people who prefer
|
||||
graphical admin interfaces.
|
||||
|
||||
.. figure:: searx-admin-engines.png
|
||||
:alt: Screenshot of engine list
|
||||
|
||||
Configuration page of engines
|
||||
|
||||
Since ``searx-admin`` acts as a supervisor for searx, we have decided to
|
||||
implement it as a standalone tool instead of part of searx. Another reason for
|
||||
making it a standalone tool is that the codebase and dependencies of searx
|
||||
should not grow because of a fully optional feature, which does not affect
|
||||
existing instances.
|
||||
|
||||
|
||||
Acknowledgements
|
||||
================
|
||||
|
||||
This development was sponsored by `NLnet Foundation`_.
|
||||
|
||||
| Happy hacking.
|
||||
| kvch // 2017.08.22 21:25
|
||||
@ -0,0 +1,10 @@
|
||||
====
|
||||
Blog
|
||||
====
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
python3
|
||||
admin
|
||||
intro-offline
|
||||
@ -0,0 +1,77 @@
|
||||
===============================
|
||||
Preparation for offline engines
|
||||
===============================
|
||||
|
||||
Offline engines
|
||||
===============
|
||||
|
||||
To extend the functionality of searx, offline engines are going to be
|
||||
introduced. An offline engine is an engine which does not need Internet
|
||||
connection to perform a search and does not use HTTP to communicate.
|
||||
|
||||
Offline engines can be configured as online engines, by adding those to the
|
||||
`engines` list of :origin:`settings.yml <searx/settings.yml>`. Thus, searx
|
||||
finds the engine file and imports it.
|
||||
|
||||
Example skeleton for the new engines:
|
||||
|
||||
.. code:: python
|
||||
|
||||
from subprocess import PIPE, Popen
|
||||
|
||||
categories = ['general']
|
||||
offline = True
|
||||
|
||||
def init(settings):
|
||||
pass
|
||||
|
||||
def search(query, params):
|
||||
process = Popen(['ls', query], stdout=PIPE)
|
||||
return_code = process.wait()
|
||||
if return_code != 0:
|
||||
raise RuntimeError('non-zero return code', return_code)
|
||||
|
||||
results = []
|
||||
line = process.stdout.readline()
|
||||
while line:
|
||||
result = parse_line(line)
|
||||
results.append(results)
|
||||
|
||||
line = process.stdout.readline()
|
||||
|
||||
return results
|
||||
|
||||
|
||||
Development progress
|
||||
====================
|
||||
|
||||
First, a proposal has been created as a Github issue. Then it was moved to the
|
||||
wiki as a design document. You can read it here: :wiki:`Offline-engines`.
|
||||
|
||||
In this development step, searx core was prepared to accept and perform offline
|
||||
searches. Offline search requests are scheduled together with regular offline
|
||||
requests.
|
||||
|
||||
As offline searches can return arbitrary results depending on the engine, the
|
||||
current result templates were insufficient to present such results. Thus, a new
|
||||
template is introduced which is caplable of presenting arbitrary key value pairs
|
||||
as a table. You can check out the pull request for more details see
|
||||
:pull:`1700`.
|
||||
|
||||
Next steps
|
||||
==========
|
||||
|
||||
Today, it is possible to create/run an offline engine. However, it is going to be publicly available for everyone who knows the searx instance. So the next step is to introduce token based access for engines. This way administrators are able to limit the access to private engines.
|
||||
|
||||
Acknowledgement
|
||||
===============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
|
||||
|
||||
.. _Search and Discovery Fund: https://nlnet.nl/discovery
|
||||
.. _NLnet Foundation: https://nlnet.nl/
|
||||
|
||||
|
||||
| Happy hacking.
|
||||
| kvch // 2019.10.21 17:03
|
||||
|
||||
@ -0,0 +1,68 @@
|
||||
============================
|
||||
Introducing Python 3 support
|
||||
============================
|
||||
|
||||
.. _Python 2.7 clock: https://pythonclock.org/
|
||||
|
||||
.. sidebar:: Python 2.7 to 3 upgrade
|
||||
|
||||
This chapter exists of historical reasons. Python 2.7 release schedule ends
|
||||
(`Python 2.7 clock`_) after 11 years Python 3 exists
|
||||
|
||||
As most operation systems are coming with Python3 installed by default. So it is
|
||||
time for searx to support Python3. But don't worry support of Python2.7 won't be
|
||||
dropped.
|
||||
|
||||
.. image:: searxpy3.png
|
||||
:scale: 50 %
|
||||
:alt: hurray
|
||||
:align: center
|
||||
|
||||
|
||||
How to run searx using Python 3
|
||||
===============================
|
||||
|
||||
Please make sure that you run at least Python 3.5.
|
||||
|
||||
To run searx, first a Python3 virtualenv should be created. After entering the
|
||||
virtualenv, dependencies must be installed. Then run searx with python3 instead
|
||||
of the usual python command.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
virtualenv -p python3 venv3
|
||||
source venv3/bin/activate
|
||||
pip3 install -r requirements.txt
|
||||
python3 searx/webapp.py
|
||||
|
||||
|
||||
If you want to run searx using Python2.7, you don't have to do anything
|
||||
differently as before.
|
||||
|
||||
Fun facts
|
||||
=========
|
||||
|
||||
- 115 files were changed when implementing the support for both Python versions.
|
||||
|
||||
- All of the dependencies was compatible except for the robotframework used for
|
||||
browser tests. Thus, these tests were migrated to splinter. So from now on
|
||||
both versions are being tested on Travis and can be tested locally.
|
||||
|
||||
If you found bugs
|
||||
=================
|
||||
|
||||
Please open an issue on `GitHub`_. Make sure that you mention your Python
|
||||
version in your issue, so we can investigate it properly.
|
||||
|
||||
.. _GitHub: https://github.com/asciimoo/searx/issues
|
||||
|
||||
Acknowledgment
|
||||
==============
|
||||
|
||||
This development was sponsored by `NLnet Foundation`_.
|
||||
|
||||
.. _NLnet Foundation: https://nlnet.nl/
|
||||
|
||||
|
||||
| Happy hacking.
|
||||
| kvch // 2017.05.13 22:57
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
@ -1,53 +0,0 @@
|
||||
.. START create user
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H useradd --shell /bin/bash --system \\
|
||||
--home-dir \"$SERVICE_HOME\" \\
|
||||
--comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
|
||||
|
||||
$ sudo -H mkdir \"$SERVICE_HOME\"
|
||||
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
|
||||
|
||||
.. END create user
|
||||
|
||||
.. START install go
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: os: linux / arch: amd64
|
||||
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ cat > \"$GO_ENV\" <<EOF
|
||||
export GOPATH=${SERVICE_HOME}/go-apps
|
||||
export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
|
||||
EOF
|
||||
$ sudo -i -u \"${SERVICE_USER}\"
|
||||
(${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
|
||||
(${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
|
||||
(${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
|
||||
\"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
|
||||
(${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
|
||||
(${SERVICE_USER}) $ which go
|
||||
${SERVICE_HOME}/local/go/bin/go
|
||||
|
||||
.. END install go
|
||||
|
||||
.. START install filtron
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo -i -u \"${SERVICE_USER}\"
|
||||
(${SERVICE_USER}) $ go get -v -u github.com/searxng/filtron
|
||||
|
||||
.. END install filtron
|
||||
@ -1,53 +0,0 @@
|
||||
.. START create user
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H useradd --shell /bin/bash --system \\
|
||||
--home-dir \"$SERVICE_HOME\" \\
|
||||
--comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
|
||||
|
||||
$ sudo -H mkdir \"$SERVICE_HOME\"
|
||||
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
|
||||
|
||||
.. END create user
|
||||
|
||||
.. START install go
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: os: linux / arch: amd64
|
||||
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ cat > \"$GO_ENV\" <<EOF
|
||||
export GOPATH=${SERVICE_HOME}/go-apps
|
||||
export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
|
||||
EOF
|
||||
$ sudo -i -u \"${SERVICE_USER}\"
|
||||
(${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
|
||||
(${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
|
||||
(${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
|
||||
\"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
|
||||
(${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
|
||||
(${SERVICE_USER}) $ which go
|
||||
${SERVICE_HOME}/local/go/bin/go
|
||||
|
||||
.. END install go
|
||||
|
||||
.. START install morty
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ sudo -i -u \"${SERVICE_USER}\"
|
||||
(${SERVICE_USER}) $ go get -v -u github.com/asciimoo/morty
|
||||
|
||||
.. END install morty
|
||||
@ -1,203 +0,0 @@
|
||||
.. template evaluated by: ./utils/searxng.sh searxng.doc.rst
|
||||
.. hint: all dollar-names are variables, dollar sign itself is quoted by: \\$
|
||||
|
||||
.. START distro-packages
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H apt-get install -y \\
|
||||
${debian}
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H pacman -S --noconfirm \\
|
||||
${arch}
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H dnf install -y \\
|
||||
${fedora}
|
||||
|
||||
.. END distro-packages
|
||||
|
||||
.. START build-packages
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Ubuntu / debian
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H apt-get install -y \\
|
||||
${debian_build}
|
||||
|
||||
.. group-tab:: Arch Linux
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H pacman -S --noconfirm \\
|
||||
${arch_build}
|
||||
|
||||
.. group-tab:: Fedora / RHEL
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H dnf install -y \\
|
||||
${fedora_build}
|
||||
|
||||
.. END build-packages
|
||||
|
||||
.. START create user
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H useradd --shell /bin/bash --system \\
|
||||
--home-dir \"$SERVICE_HOME\" \\
|
||||
--comment 'Privacy-respecting metasearch engine' \\
|
||||
$SERVICE_USER
|
||||
|
||||
$ sudo -H mkdir \"$SERVICE_HOME\"
|
||||
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
|
||||
|
||||
.. END create user
|
||||
|
||||
.. START clone searxng
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H -u ${SERVICE_USER} -i
|
||||
(${SERVICE_USER})$ git clone \"$GIT_URL\" \\
|
||||
\"$SEARXNG_SRC\"
|
||||
|
||||
.. END clone searxng
|
||||
|
||||
.. START create virtualenv
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
(${SERVICE_USER})$ python3 -m venv \"${SEARXNG_PYENV}\"
|
||||
(${SERVICE_USER})$ echo \". ${SEARXNG_PYENV}/bin/activate\" \\
|
||||
>> \"$SERVICE_HOME/.profile\"
|
||||
|
||||
.. END create virtualenv
|
||||
|
||||
.. START manage.sh update_packages
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H -u ${SERVICE_USER} -i
|
||||
|
||||
(${SERVICE_USER})$ command -v python && python --version
|
||||
$SEARXNG_PYENV/bin/python
|
||||
Python 3.8.1
|
||||
|
||||
# update pip's boilerplate ..
|
||||
pip install -U pip
|
||||
pip install -U setuptools
|
||||
pip install -U wheel
|
||||
pip install -U pyyaml
|
||||
|
||||
# jump to SearXNG's working tree and install SearXNG into virtualenv
|
||||
(${SERVICE_USER})$ cd \"$SEARXNG_SRC\"
|
||||
(${SERVICE_USER})$ pip install -e .
|
||||
|
||||
|
||||
.. END manage.sh update_packages
|
||||
|
||||
.. START searxng config
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: Use default settings
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H mkdir -p \"$(dirname ${SEARXNG_SETTINGS_PATH})\"
|
||||
$ sudo -H cp \"$SEARXNG_SRC/utils/templates/etc/searxng/settings.yml\" \\
|
||||
\"${SEARXNG_SETTINGS_PATH}\"
|
||||
|
||||
.. group-tab:: minimal setup
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ sudo -H sed -i -e \"s/ultrasecretkey/\$(openssl rand -hex 16)/g\" \\
|
||||
\"$SEARXNG_SETTINGS_PATH\"
|
||||
|
||||
.. END searxng config
|
||||
|
||||
.. START check searxng installation
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: bash
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
# enable debug ..
|
||||
$ sudo -H sed -i -e \"s/debug : False/debug : True/g\" \"$SEARXNG_SETTINGS_PATH\"
|
||||
|
||||
# start webapp
|
||||
$ sudo -H -u ${SERVICE_USER} -i
|
||||
(${SERVICE_USER})$ cd ${SEARXNG_SRC}
|
||||
(${SERVICE_USER})$ export SEARXNG_SETTINGS_PATH=\"${SEARXNG_SETTINGS_PATH}\"
|
||||
(${SERVICE_USER})$ python searx/webapp.py
|
||||
|
||||
# disable debug
|
||||
$ sudo -H sed -i -e \"s/debug : True/debug : False/g\" \"$SEARXNG_SETTINGS_PATH\"
|
||||
|
||||
Open WEB browser and visit http://$SEARXNG_INTERNAL_HTTP . If you are inside a
|
||||
container or in a script, test with curl:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: WEB browser
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
$ xdg-open http://$SEARXNG_INTERNAL_HTTP
|
||||
|
||||
.. group-tab:: curl
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ curl --location --verbose --head --insecure $SEARXNG_INTERNAL_HTTP
|
||||
|
||||
* Trying 127.0.0.1:8888...
|
||||
* TCP_NODELAY set
|
||||
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
|
||||
> HEAD / HTTP/1.1
|
||||
> Host: 127.0.0.1:8888
|
||||
> User-Agent: curl/7.68.0
|
||||
> Accept: */*
|
||||
>
|
||||
* Mark bundle as not supporting multiuse
|
||||
* HTTP 1.0, assume close after body
|
||||
< HTTP/1.0 200 OK
|
||||
HTTP/1.0 200 OK
|
||||
...
|
||||
|
||||
.. END check searxng installation
|
||||
@ -1,402 +1,267 @@
|
||||
|
||||
.. _engines-dev:
|
||||
|
||||
===============
|
||||
Engine Overview
|
||||
Engine overview
|
||||
===============
|
||||
|
||||
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`configured engines`
|
||||
- :ref:`settings engine`
|
||||
|
||||
.. contents::
|
||||
:depth: 3
|
||||
:backlinks: entry
|
||||
|
||||
SearXNG is a metasearch-engine_, so it uses different search engines to provide
|
||||
searx is a metasearch-engine_, so it uses different search engines to provide
|
||||
better results.
|
||||
|
||||
Because there is no general search API which could be used for every search
|
||||
engine, an adapter has to be built between SearXNG and the external search
|
||||
engine, an adapter has to be built between searx and the external search
|
||||
engines. Adapters are stored under the folder :origin:`searx/engines`.
|
||||
|
||||
.. _general engine configuration:
|
||||
.. contents::
|
||||
:depth: 3
|
||||
:backlinks: entry
|
||||
|
||||
General Engine Configuration
|
||||
general engine configuration
|
||||
============================
|
||||
|
||||
It is required to tell SearXNG the type of results the engine provides. The
|
||||
arguments can be set in the engine file or in the settings file (normally
|
||||
``settings.yml``). The arguments in the settings file override the ones in the
|
||||
engine file.
|
||||
It is required to tell searx the type of results the engine provides. The
|
||||
arguments can be set in the engine file or in the settings file
|
||||
(normally ``settings.yml``). The arguments in the settings file override
|
||||
the ones in the engine file.
|
||||
|
||||
It does not matter if an option is stored in the engine file or in the settings.
|
||||
However, the standard way is the following:
|
||||
It does not matter if an option is stored in the engine file or in the
|
||||
settings. However, the standard way is the following:
|
||||
|
||||
.. _engine file:
|
||||
|
||||
Engine File
|
||||
engine file
|
||||
-----------
|
||||
|
||||
.. table:: Common options in the engine module
|
||||
:width: 100%
|
||||
|
||||
======================= =========== ========================================================
|
||||
argument type information
|
||||
======================= =========== ========================================================
|
||||
categories list pages, in which the engine is working
|
||||
paging boolean support multiple pages
|
||||
time_range_support boolean support search time range
|
||||
engine_type str - ``online`` :ref:`[ref] <demo online engine>` by
|
||||
default, other possibles values are:
|
||||
- ``offline`` :ref:`[ref] <offline engines>`
|
||||
- ``online_dictionary``
|
||||
- ``online_currency``
|
||||
======================= =========== ========================================================
|
||||
======================= =========== ===========================================
|
||||
argument type information
|
||||
======================= =========== ===========================================
|
||||
categories list pages, in which the engine is working
|
||||
paging boolean support multible pages
|
||||
language_support boolean support language choosing
|
||||
time_range_support boolean support search time range
|
||||
offline boolean engine runs offline
|
||||
======================= =========== ===========================================
|
||||
|
||||
.. _engine settings:
|
||||
|
||||
Engine ``settings.yml``
|
||||
-----------------------
|
||||
settings.yml
|
||||
------------
|
||||
|
||||
For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`.
|
||||
======================= =========== ===========================================
|
||||
argument type information
|
||||
======================= =========== ===========================================
|
||||
name string name of search-engine
|
||||
engine string name of searx-engine
|
||||
(filename without ``.py``)
|
||||
shortcut string shortcut of search-engine
|
||||
timeout string specific timeout for search-engine
|
||||
======================= =========== ===========================================
|
||||
|
||||
.. table:: Common options in the engine setup (``settings.yml``)
|
||||
:width: 100%
|
||||
|
||||
======================= =========== ==================================================
|
||||
argument type information
|
||||
======================= =========== ==================================================
|
||||
name string name of search-engine
|
||||
engine string name of searxng-engine (file name without ``.py``)
|
||||
enable_http bool enable HTTP (by default only HTTPS is enabled).
|
||||
shortcut string shortcut of search-engine
|
||||
timeout string specific timeout for search-engine
|
||||
display_error_messages boolean display error messages on the web UI
|
||||
proxies dict set proxies for a specific engine
|
||||
(e.g. ``proxies : {http: socks5://proxy:port,
|
||||
https: socks5://proxy:port}``)
|
||||
======================= =========== ==================================================
|
||||
overrides
|
||||
---------
|
||||
|
||||
.. _engine overrides:
|
||||
A few of the options have default values in the engine, but are often
|
||||
overwritten by the settings. If ``None`` is assigned to an option in the engine
|
||||
file, it has to be redefined in the settings, otherwise searx will not start
|
||||
with that engine.
|
||||
|
||||
Overrides
|
||||
---------
|
||||
The naming of overrides is arbitrary. But the recommended overrides are the
|
||||
following:
|
||||
|
||||
A few of the options have default values in the namespace of engine's python
|
||||
modul, but are often overwritten by the settings. If ``None`` is assigned to an
|
||||
option in the engine file, it has to be redefined in the settings, otherwise
|
||||
SearXNG will not start with that engine (global names with a leading underline can
|
||||
be ``None``).
|
||||
======================= =========== ===========================================
|
||||
argument type information
|
||||
======================= =========== ===========================================
|
||||
base_url string base-url, can be overwritten to use same
|
||||
engine on other URL
|
||||
number_of_results int maximum number of results per request
|
||||
language string ISO code of language and country like en_US
|
||||
api_key string api-key if required by engine
|
||||
======================= =========== ===========================================
|
||||
|
||||
Here is an very simple example of the global names in the namespace of engine's
|
||||
module:
|
||||
example code
|
||||
------------
|
||||
|
||||
.. code:: python
|
||||
|
||||
# engine dependent config
|
||||
categories = ['general']
|
||||
paging = True
|
||||
_non_overwritten_global = 'foo'
|
||||
|
||||
language_support = True
|
||||
|
||||
.. table:: The naming of overrides is arbitrary / recommended overrides are:
|
||||
:width: 100%
|
||||
|
||||
======================= =========== ===========================================
|
||||
argument type information
|
||||
======================= =========== ===========================================
|
||||
base_url string base-url, can be overwritten to use same
|
||||
engine on other URL
|
||||
number_of_results int maximum number of results per request
|
||||
language string ISO code of language and country like en_US
|
||||
api_key string api-key if required by engine
|
||||
======================= =========== ===========================================
|
||||
|
||||
.. _engine request:
|
||||
|
||||
Making a Request
|
||||
making a request
|
||||
================
|
||||
|
||||
To perform a search an URL have to be specified. In addition to specifying an
|
||||
URL, arguments can be passed to the query.
|
||||
|
||||
.. _engine request arguments:
|
||||
|
||||
Passed Arguments (request)
|
||||
--------------------------
|
||||
passed arguments
|
||||
----------------
|
||||
|
||||
These arguments can be used to construct the search query. Furthermore,
|
||||
parameters with default value can be redefined for special purposes.
|
||||
|
||||
====================== ============ ========================================================================
|
||||
argument type default-value, information
|
||||
====================== ============ ========================================================================
|
||||
url string ``''``
|
||||
method string ``'GET'``
|
||||
headers set ``{}``
|
||||
data set ``{}``
|
||||
cookies set ``{}``
|
||||
verify boolean ``True``
|
||||
headers.User-Agent string a random User-Agent
|
||||
category string current category, like ``'general'``
|
||||
started datetime current date-time
|
||||
pageno int current pagenumber
|
||||
language string specific language code like ``'en_US'``, or ``'all'`` if unspecified
|
||||
====================== ============ ========================================================================
|
||||
|
||||
parsed arguments
|
||||
----------------
|
||||
|
||||
The function ``def request(query, params):`` always returns the ``params``
|
||||
variable. Inside searx, the following paramters can be used to specify a search
|
||||
request:
|
||||
|
||||
============ =========== =========================================================
|
||||
argument type information
|
||||
============ =========== =========================================================
|
||||
url string requested url
|
||||
method string HTTP request method
|
||||
headers set HTTP header information
|
||||
data set HTTP data information (parsed if ``method != 'GET'``)
|
||||
cookies set HTTP cookies
|
||||
verify boolean Performing SSL-Validity check
|
||||
============ =========== =========================================================
|
||||
|
||||
|
||||
example code
|
||||
------------
|
||||
|
||||
.. code:: python
|
||||
|
||||
# search-url
|
||||
base_url = 'https://example.com/'
|
||||
search_string = 'search?{query}&page={page}'
|
||||
|
||||
# do search-request
|
||||
def request(query, params):
|
||||
search_path = search_string.format(
|
||||
query=urlencode({'q': query}),
|
||||
page=params['pageno'])
|
||||
|
||||
params['url'] = base_url + search_path
|
||||
|
||||
return params
|
||||
|
||||
|
||||
returned results
|
||||
================
|
||||
|
||||
.. table:: If the ``engine_type`` is ``online``
|
||||
:width: 100%
|
||||
|
||||
====================== ============== ========================================================================
|
||||
argument type default-value, information
|
||||
====================== ============== ========================================================================
|
||||
url str ``''``
|
||||
method str ``'GET'``
|
||||
headers set ``{}``
|
||||
data set ``{}``
|
||||
cookies set ``{}``
|
||||
verify bool ``True``
|
||||
headers.User-Agent str a random User-Agent
|
||||
category str current category, like ``'general'``
|
||||
safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict)
|
||||
time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year``
|
||||
pageno int current pagenumber
|
||||
language str specific language code like ``'en_US'``, or ``'all'`` if unspecified
|
||||
====================== ============== ========================================================================
|
||||
|
||||
|
||||
.. table:: If the ``engine_type`` is ``online_dictionary``, in addition to the
|
||||
``online`` arguments:
|
||||
:width: 100%
|
||||
|
||||
====================== ============== ========================================================================
|
||||
argument type default-value, information
|
||||
====================== ============== ========================================================================
|
||||
from_lang str specific language code like ``'en_US'``
|
||||
to_lang str specific language code like ``'en_US'``
|
||||
query str the text query without the languages
|
||||
====================== ============== ========================================================================
|
||||
|
||||
.. table:: If the ``engine_type`` is ``online_currency```, in addition to the
|
||||
``online`` arguments:
|
||||
:width: 100%
|
||||
|
||||
====================== ============== ========================================================================
|
||||
argument type default-value, information
|
||||
====================== ============== ========================================================================
|
||||
amount float the amount to convert
|
||||
from str ISO 4217 code
|
||||
to str ISO 4217 code
|
||||
from_name str currency name
|
||||
to_name str currency name
|
||||
====================== ============== ========================================================================
|
||||
|
||||
|
||||
Specify Request
|
||||
---------------
|
||||
|
||||
The function :py:func:`def request(query, params):
|
||||
<searx.engines.demo_online.request>` always returns the ``params`` variable, the
|
||||
following parameters can be used to specify a search request:
|
||||
|
||||
.. table::
|
||||
:width: 100%
|
||||
|
||||
=================== =========== ==========================================================================
|
||||
argument type information
|
||||
=================== =========== ==========================================================================
|
||||
url str requested url
|
||||
method str HTTP request method
|
||||
headers set HTTP header information
|
||||
data set HTTP data information
|
||||
cookies set HTTP cookies
|
||||
verify bool Performing SSL-Validity check
|
||||
allow_redirects bool Follow redirects
|
||||
max_redirects int maximum redirects, hard limit
|
||||
soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine
|
||||
raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300
|
||||
=================== =========== ==========================================================================
|
||||
|
||||
|
||||
.. _engine results:
|
||||
.. _engine media types:
|
||||
|
||||
Media Types
|
||||
===========
|
||||
|
||||
Each result item of an engine can be of different media-types. Currently the
|
||||
following media-types are supported. To set another media-type as ``default``,
|
||||
the parameter ``template`` must be set to the desired type.
|
||||
|
||||
.. table:: Parameter of the **default** media type:
|
||||
:width: 100%
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
========================= =====================================================
|
||||
|
||||
|
||||
.. table:: Parameter of the **images** media type:
|
||||
:width: 100%
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
------------------------- -----------------------------------------------------
|
||||
template is set to ``images.html``
|
||||
========================= =====================================================
|
||||
url string, url to the result site
|
||||
title string, title of the result *(partly implemented)*
|
||||
content *(partly implemented)*
|
||||
publishedDate :py:class:`datetime.datetime`,
|
||||
time of publish *(partly implemented)*
|
||||
img\_src string, url to the result image
|
||||
thumbnail\_src string, url to a small-preview image
|
||||
========================= =====================================================
|
||||
|
||||
|
||||
.. table:: Parameter of the **videos** media type:
|
||||
:width: 100%
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
------------------------- -----------------------------------------------------
|
||||
template is set to ``videos.html``
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content *(not implemented yet)*
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
thumbnail string, url to a small-preview image
|
||||
========================= =====================================================
|
||||
Searx is able to return results of different media-types. Currently the
|
||||
following media-types are supported:
|
||||
|
||||
- default_
|
||||
- images_
|
||||
- videos_
|
||||
- torrent_
|
||||
- map_
|
||||
|
||||
To set another media-type as default, the parameter ``template`` must be set to
|
||||
the desired type.
|
||||
|
||||
default
|
||||
-------
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
========================= =====================================================
|
||||
|
||||
images
|
||||
------
|
||||
|
||||
To use this template, the parameter:
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
template is set to ``images.html``
|
||||
url string, url to the result site
|
||||
title string, title of the result *(partly implemented)*
|
||||
content *(partly implemented)*
|
||||
publishedDate :py:class:`datetime.datetime`,
|
||||
time of publish *(partly implemented)*
|
||||
img\_src string, url to the result image
|
||||
thumbnail\_src string, url to a small-preview image
|
||||
========================= =====================================================
|
||||
|
||||
videos
|
||||
------
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
template is set to ``videos.html``
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content *(not implemented yet)*
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
thumbnail string, url to a small-preview image
|
||||
========================= =====================================================
|
||||
|
||||
torrent
|
||||
-------
|
||||
|
||||
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
|
||||
|
||||
.. table:: Parameter of the **torrent** media type:
|
||||
:width: 100%
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
------------------------- -----------------------------------------------------
|
||||
template is set to ``torrent.html``
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`,
|
||||
time of publish *(not implemented yet)*
|
||||
seed int, number of seeder
|
||||
leech int, number of leecher
|
||||
filesize int, size of file in bytes
|
||||
files int, number of files
|
||||
magnetlink string, magnetlink_ of the result
|
||||
torrentfile string, torrentfile of the result
|
||||
========================= =====================================================
|
||||
|
||||
.. table:: Parameter of the **map** media type:
|
||||
:width: 100%
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
------------------------- -----------------------------------------------------
|
||||
template is set to ``map.html``
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
latitude latitude of result (in decimal format)
|
||||
longitude longitude of result (in decimal format)
|
||||
boundingbox boundingbox of result (array of 4. values
|
||||
``[lat-min, lat-max, lon-min, lon-max]``)
|
||||
geojson geojson of result (https://geojson.org/)
|
||||
osm.type type of osm-object (if OSM-Result)
|
||||
osm.id id of osm-object (if OSM-Result)
|
||||
address.name name of object
|
||||
address.road street name of object
|
||||
address.house_number house number of object
|
||||
address.locality city, place of object
|
||||
address.postcode postcode of object
|
||||
address.country country of object
|
||||
========================= =====================================================
|
||||
|
||||
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
|
||||
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
|
||||
|
||||
.. list-table:: Parameter of the **paper** media type /
|
||||
see `BibTeX field types`_ and `BibTeX format`_
|
||||
:header-rows: 2
|
||||
:width: 100%
|
||||
|
||||
* - result-parameter
|
||||
- Python type
|
||||
- information
|
||||
|
||||
* - template
|
||||
- :py:class:`str`
|
||||
- is set to ``paper.html``
|
||||
|
||||
* - title
|
||||
- :py:class:`str`
|
||||
- title of the result
|
||||
|
||||
* - content
|
||||
- :py:class:`str`
|
||||
- abstract
|
||||
|
||||
* - comments
|
||||
- :py:class:`str`
|
||||
- free text display in italic below the content
|
||||
|
||||
* - tags
|
||||
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
|
||||
- free tag list
|
||||
|
||||
* - publishedDate
|
||||
- :py:class:`datetime <datetime.datetime>`
|
||||
- last publication date
|
||||
|
||||
* - type
|
||||
- :py:class:`str`
|
||||
- short description of medium type, e.g. *book*, *pdf* or *html* ...
|
||||
|
||||
* - authors
|
||||
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
|
||||
- list of authors of the work (authors with a "s")
|
||||
|
||||
* - editor
|
||||
- :py:class:`str`
|
||||
- list of editors of a book
|
||||
|
||||
* - publisher
|
||||
- :py:class:`str`
|
||||
- name of the publisher
|
||||
|
||||
* - journal
|
||||
- :py:class:`str`
|
||||
- name of the journal or magazine the article was
|
||||
published in
|
||||
|
||||
* - volume
|
||||
- :py:class:`str`
|
||||
- volume number
|
||||
|
||||
* - pages
|
||||
- :py:class:`str`
|
||||
- page range where the article is
|
||||
|
||||
* - number
|
||||
- :py:class:`str`
|
||||
- number of the report or the issue number for a journal article
|
||||
|
||||
* - doi
|
||||
- :py:class:`str`
|
||||
- DOI number (like ``10.1038/d41586-018-07848-2``)
|
||||
|
||||
* - issn
|
||||
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
|
||||
- ISSN number like ``1476-4687``
|
||||
|
||||
* - isbn
|
||||
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
|
||||
- ISBN number like ``9780201896831``
|
||||
|
||||
* - pdf_url
|
||||
- :py:class:`str`
|
||||
- URL to the full article, the PDF version
|
||||
|
||||
* - html_url
|
||||
- :py:class:`str`
|
||||
- URL to full article, HTML version
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
template is set to ``torrent.html``
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`,
|
||||
time of publish *(not implemented yet)*
|
||||
seed int, number of seeder
|
||||
leech int, number of leecher
|
||||
filesize int, size of file in bytes
|
||||
files int, number of files
|
||||
magnetlink string, magnetlink_ of the result
|
||||
torrentfile string, torrentfile of the result
|
||||
========================= =====================================================
|
||||
|
||||
|
||||
map
|
||||
---
|
||||
|
||||
========================= =====================================================
|
||||
result-parameter information
|
||||
========================= =====================================================
|
||||
url string, url of the result
|
||||
title string, title of the result
|
||||
content string, general result-text
|
||||
publishedDate :py:class:`datetime.datetime`, time of publish
|
||||
latitude latitude of result (in decimal format)
|
||||
longitude longitude of result (in decimal format)
|
||||
boundingbox boundingbox of result (array of 4. values
|
||||
``[lat-min, lat-max, lon-min, lon-max]``)
|
||||
geojson geojson of result (http://geojson.org)
|
||||
osm.type type of osm-object (if OSM-Result)
|
||||
osm.id id of osm-object (if OSM-Result)
|
||||
address.name name of object
|
||||
address.road street name of object
|
||||
address.house_number house number of object
|
||||
address.locality city, place of object
|
||||
address.postcode postcode of object
|
||||
address.country country of object
|
||||
========================= =====================================================
|
||||
|
||||
@ -1,403 +0,0 @@
|
||||
.. _lxcdev:
|
||||
|
||||
==============================
|
||||
Developing in Linux Containers
|
||||
==============================
|
||||
|
||||
.. _LXC: https://linuxcontainers.org/lxc/introduction/
|
||||
|
||||
In this article we will show, how you can make use of Linux Containers (LXC_) in
|
||||
*distributed and heterogeneous development cycles* (TL;DR; jump to the
|
||||
:ref:`lxcdev summary`).
|
||||
|
||||
.. sidebar:: Audience
|
||||
|
||||
This blog post is written for experienced admins and developers. Readers
|
||||
should have a serious meaning about the terms: *distributed*, *merge* and
|
||||
*linux container*.
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
Usually in our development cycle, we edit the sources and run some test and/or
|
||||
builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This cycle
|
||||
is simple and perfect but might fail in some aspects we should not overlook.
|
||||
|
||||
**The environment in which we run all our development processes matters!**
|
||||
|
||||
The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but
|
||||
they do not have access to all prerequisites. For example, there may have
|
||||
dependencies on packages that are installed on the developer's desktop, but
|
||||
usually are not preinstalled on a server or client system. Another example is;
|
||||
settings have been made to the software on developer's desktop that would never
|
||||
be set on a *production* system.
|
||||
|
||||
**Linux Containers are isolate environments and not to mix up all the
|
||||
prerequisites from various projects on developer's desktop is always a good
|
||||
choice.**
|
||||
|
||||
The scripts from :ref:`searx_utils` can divide in those to install and maintain
|
||||
software:
|
||||
|
||||
- :ref:`searxng.sh`
|
||||
|
||||
and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or
|
||||
even development tasks over a stack of isolated containers / what we call the:
|
||||
|
||||
**SearXNG LXC suite**
|
||||
|
||||
.. hint::
|
||||
|
||||
If you see any problems with the internet connectivity of your
|
||||
containers read section :ref:`internet connectivity docker`.
|
||||
|
||||
|
||||
Gentlemen, start your engines!
|
||||
==============================
|
||||
|
||||
.. _LXD: https://linuxcontainers.org/lxd/introduction/
|
||||
.. _archlinux: https://www.archlinux.org/
|
||||
|
||||
Before you can start with containers, you need to install and initiate LXD_
|
||||
once:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ snap install lxd
|
||||
$ lxd init --auto
|
||||
|
||||
And you need to clone from origin or if you have your own fork, clone from your
|
||||
fork:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ cd ~/Downloads
|
||||
$ git clone https://github.com/searxng/searxng.git searxng
|
||||
$ cd searxng
|
||||
|
||||
The :ref:`lxc-searxng.env` consists of several images, see ``export
|
||||
LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. For this blog post
|
||||
we exercise on a archlinux_ image. The container of this image is named
|
||||
``searxng-archlinux``. Lets build the container, but be sure that this container
|
||||
does not already exists, so first lets remove possible old one:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh remove searxng-archlinux
|
||||
$ sudo -H ./utils/lxc.sh build searxng-archlinux
|
||||
|
||||
.. sidebar:: The ``searxng-archlinux`` container
|
||||
|
||||
is the base of all our exercises here.
|
||||
|
||||
In this container we install all services :ref:`including searx, morty & filtron
|
||||
<lxc.sh install suite>` in once:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
|
||||
|
||||
To proxy HTTP from filtron and morty in the container to the outside of the
|
||||
container, install nginx into the container. Once for the bot blocker filtron:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
./utils/filtron.sh nginx install
|
||||
...
|
||||
INFO: got 429 from http://10.174.184.156/searx
|
||||
|
||||
and once for the content sanitizer (content proxy morty):
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
./utils/morty.sh nginx install
|
||||
...
|
||||
INFO: got 200 from http://10.174.184.156/morty/
|
||||
|
||||
.. sidebar:: Fully functional SearXNG suite
|
||||
|
||||
From here on you have a fully functional SearXNG suite running with bot
|
||||
blocker (filtron) and WEB content sanitizer (content proxy morty), both are
|
||||
needed for a *privacy protecting* search engine.
|
||||
|
||||
On your system, the IP of your ``searxng-archlinux`` container differs from
|
||||
http://10.174.184.156/searx, just open the URL reported in your installation
|
||||
protocol in your WEB browser from the desktop to test the instance from outside
|
||||
of the container.
|
||||
|
||||
In such a earXNG suite admins can maintain and access the debug log of the
|
||||
different services quite easy.
|
||||
|
||||
.. _working in containers:
|
||||
|
||||
In containers, work as usual
|
||||
============================
|
||||
|
||||
Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers
|
||||
open the root-bash in the container using ``./utils/lxc.sh cmd
|
||||
searxng-archlinux``:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash
|
||||
INFO: [searxng-archlinux] bash
|
||||
[root@searxng-archlinux searx]# pwd
|
||||
/share/searxng
|
||||
|
||||
The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user in
|
||||
the searxng-container. To debug the running SearXNG instance use:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: root@searxng-archlinux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./utils/searx.sh inspect service
|
||||
...
|
||||
use [CTRL-C] to stop monitoring the log
|
||||
...
|
||||
|
||||
Back in the browser on your desktop open the service http://10.174.184.156/searx
|
||||
and run your application tests while the debug log is shown in the terminal from
|
||||
above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug
|
||||
option"* in SearXNG's settings file and restarts the SearXNG uwsgi application.
|
||||
To debug services from filtron and morty analogous use:
|
||||
|
||||
Another point we have to notice is that the service (:ref:`SearXNG <searxng.sh>`
|
||||
runs under dedicated system user account with the same name (compare
|
||||
:ref:`create searxng user`). To get a shell from these accounts, simply call:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: root@searxng-archlinux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./utils/searxng.sh instance cmd bash
|
||||
|
||||
To get in touch, open a shell from the service user (searxng@searxng-archlinux):
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash
|
||||
INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash
|
||||
[searxng@searxng-archlinux ~]$
|
||||
|
||||
The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system
|
||||
user ``searx`` in the ``searxng-archlinux`` container and the python *virtualenv*
|
||||
``(searxng-pyenv)`` environment is activated.
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: searxng@searxng-archlinux
|
||||
|
||||
.. code:: bash
|
||||
|
||||
(searxng-pyenv) [searxng@searxng-archlinux ~]$ pwd
|
||||
/usr/local/searxng
|
||||
|
||||
|
||||
Wrap production into developer suite
|
||||
====================================
|
||||
|
||||
In this section we will see how to change the *"Fully functional SearXNG suite"*
|
||||
from a LXC container (which is quite ready for production) into a developer
|
||||
suite. For this, we have to keep an eye on the :ref:`installation basic`:
|
||||
|
||||
- SearXNG setup in: ``/etc/searxng/settings.yml``
|
||||
- SearXNG user's home: ``/usr/local/searxng``
|
||||
- virtualenv in: ``/usr/local/searxng/searxng-pyenv``
|
||||
- SearXNG software in: ``/usr/local/searxng/searxng-src``
|
||||
|
||||
With the use of the :ref:`searxng.sh` the SearXNG service was installed as
|
||||
:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use
|
||||
``systemctl`` (compare :ref:`uWSGI maintenance`).
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
systemctl stop uwsgi@searxng
|
||||
|
||||
With the command above, we stopped the SearXNG uWSGI-App in the archlinux
|
||||
container.
|
||||
|
||||
The uWSGI-App for the archlinux dsitros is configured in
|
||||
:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at
|
||||
least you should attend the settings of ``uid``, ``chdir``, ``env`` and
|
||||
``http``::
|
||||
|
||||
env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
|
||||
http = 127.0.0.1:8888
|
||||
|
||||
chdir = /usr/local/searxng/searxng-src/searx
|
||||
virtualenv = /usr/local/searxng/searxng-pyenv
|
||||
pythonpath = /usr/local/searxng/searxng-src
|
||||
|
||||
If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that
|
||||
each container shares the root folder of the repository and the command
|
||||
``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the
|
||||
SearXNG installation into a developer one, we simple have to create a smylink to
|
||||
the **transparent** reposetory from the desktop. Now lets replace the
|
||||
repository at ``searxng-src`` in the container with the working tree from outside
|
||||
of the container:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: container becomes a developer suite
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
ln -s /share/searx/ /usr/local/searxng/searxng-src
|
||||
|
||||
Now we can develop as usual in the working tree of our desktop system. Every
|
||||
time the software was changed, you have to restart the SearXNG service (in the
|
||||
container):
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
systemctl restart uwsgi@searx
|
||||
|
||||
|
||||
Remember: :ref:`working in containers` .. here are just some examples from my
|
||||
daily usage:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
To *inspect* the SearXNG instance (already described above):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
./utils/searx.sh inspect service
|
||||
|
||||
Run :ref:`makefile`, e.g. to test inside the container:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
make test
|
||||
|
||||
To install all prerequisites needed for a :ref:`buildhosts`:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
./utils/searxng.sh install buildhost
|
||||
|
||||
To build the docs on a buildhost :ref:`buildhosts`:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
||||
make docs.html
|
||||
|
||||
.. _lxcdev summary:
|
||||
|
||||
Summary
|
||||
=======
|
||||
|
||||
We build up a fully functional SearXNG suite in a archlinux container:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
|
||||
|
||||
To access HTTP from the desktop we installed nginx for the services inside the
|
||||
container:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: [root@searxng-archlinux]
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./utils/filtron.sh nginx install
|
||||
$ ./utils/morty.sh nginx install
|
||||
|
||||
To wrap the suite into a developer one, we created a symbolic link to the
|
||||
repository which is shared **transparent** from the desktop's file system into
|
||||
the container :
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: [root@searxng-archlinux]
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old
|
||||
$ ln -s /share/searx/ /usr/local/searxng/searxng-src
|
||||
$ systemctl restart uwsgi@searx
|
||||
|
||||
To get information about the searxNG suite in the archlinux container we can
|
||||
use:
|
||||
|
||||
.. tabs::
|
||||
|
||||
.. group-tab:: desktop
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo -H ./utils/lxc.sh show suite searxng-archlinux
|
||||
...
|
||||
[searxng-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx
|
||||
[searxng-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/
|
||||
[searxng-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/
|
||||
[searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65]
|
||||
...
|
||||
|
||||
@ -1,78 +0,0 @@
|
||||
.. _offline engines:
|
||||
|
||||
===============
|
||||
Offline Engines
|
||||
===============
|
||||
|
||||
.. sidebar:: offline engines
|
||||
|
||||
- :ref:`demo offline engine`
|
||||
- :ref:`sql engines`
|
||||
- :ref:`engine command`
|
||||
- :origin:`Redis <searx/engines/redis_server.py>`
|
||||
|
||||
To extend the functionality of SearXNG, offline engines are going to be
|
||||
introduced. An offline engine is an engine which does not need Internet
|
||||
connection to perform a search and does not use HTTP to communicate.
|
||||
|
||||
Offline engines can be configured, by adding those to the `engines` list of
|
||||
:origin:`settings.yml <searx/settings.yml>`. An example skeleton for offline
|
||||
engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py
|
||||
<searx/engines/demo_offline.py>`).
|
||||
|
||||
|
||||
Programming Interface
|
||||
=====================
|
||||
|
||||
:py:func:`init(engine_settings=None) <searx.engines.demo_offline.init>`
|
||||
All offline engines can have their own init function to setup the engine before
|
||||
accepting requests. The function gets the settings from settings.yml as a
|
||||
parameter. This function can be omitted, if there is no need to setup anything
|
||||
in advance.
|
||||
|
||||
:py:func:`search(query, params) <searx.engines.demo_offline.searc>`
|
||||
|
||||
Each offline engine has a function named ``search``. This function is
|
||||
responsible to perform a search and return the results in a presentable
|
||||
format. (Where *presentable* means presentable by the selected result
|
||||
template.)
|
||||
|
||||
The return value is a list of results retrieved by the engine.
|
||||
|
||||
Engine representation in ``/config``
|
||||
If an engine is offline, the attribute ``offline`` is set to ``True``.
|
||||
|
||||
.. _offline requirements:
|
||||
|
||||
Extra Dependencies
|
||||
==================
|
||||
|
||||
If an offline engine depends on an external tool, SearXNG does not install it by
|
||||
default. When an administrator configures such engine and starts the instance,
|
||||
the process returns an error with the list of missing dependencies. Also,
|
||||
required dependencies will be added to the comment/description of the engine, so
|
||||
admins can install packages in advance.
|
||||
|
||||
If there is a need to install additional packages in *Python's Virtual
|
||||
Environment* of your SearXNG instance you need to switch into the environment
|
||||
(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`::
|
||||
|
||||
$ sudo utils/searxng.sh instance cmd bash
|
||||
(searxng-pyenv)$ pip install ...
|
||||
|
||||
|
||||
Private engines (Security)
|
||||
==========================
|
||||
|
||||
To limit the access to offline engines, if an instance is available publicly,
|
||||
administrators can set token(s) for each of the :ref:`private engines`. If a
|
||||
query contains a valid token, then SearXNG performs the requested private
|
||||
search. If not, requests from an offline engines return errors.
|
||||
|
||||
|
||||
Acknowledgement
|
||||
===============
|
||||
|
||||
This development was sponsored by `Search and Discovery Fund
|
||||
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .
|
||||
|
||||
@ -1,15 +0,0 @@
|
||||
.. _searxng_extra:
|
||||
|
||||
=============================
|
||||
Tooling box ``searxng_extra``
|
||||
=============================
|
||||
|
||||
In the folder :origin:`searxng_extra/` we maintain some tools useful for CI and
|
||||
developers.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:caption: Contents
|
||||
|
||||
update
|
||||
standalone_searx.py
|
||||
@ -1,9 +0,0 @@
|
||||
|
||||
.. _standalone_searx.py:
|
||||
|
||||
=====================================
|
||||
``searxng_extra/standalone_searx.py``
|
||||
=====================================
|
||||
|
||||
.. automodule:: searxng_extra.standalone_searx
|
||||
:members:
|
||||
@ -1,88 +0,0 @@
|
||||
=========================
|
||||
``searxng_extra/update/``
|
||||
=========================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/__init__.py>`
|
||||
|
||||
Scripts to update static data in :origin:`searx/data/`
|
||||
|
||||
.. _update_ahmia_blacklist.py:
|
||||
|
||||
``update_ahmia_blacklist.py``
|
||||
=============================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_ahmia_blacklist.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_ahmia_blacklist
|
||||
:members:
|
||||
|
||||
|
||||
``update_currencies.py``
|
||||
========================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_currencies.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_currencies
|
||||
:members:
|
||||
|
||||
``update_engine_descriptions.py``
|
||||
=================================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_engine_descriptions.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_engine_descriptions
|
||||
:members:
|
||||
|
||||
|
||||
``update_external_bangs.py``
|
||||
============================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_external_bangs.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_external_bangs
|
||||
:members:
|
||||
|
||||
|
||||
``update_firefox_version.py``
|
||||
=============================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_firefox_version.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_firefox_version
|
||||
:members:
|
||||
|
||||
|
||||
``update_languages.py``
|
||||
=======================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_languages.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_languages
|
||||
:members:
|
||||
|
||||
|
||||
``update_osm_keys_tags.py``
|
||||
===========================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_osm_keys_tags.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_osm_keys_tags
|
||||
:members:
|
||||
|
||||
|
||||
``update_pygments.py``
|
||||
======================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_pygments.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_pygments
|
||||
:members:
|
||||
|
||||
|
||||
``update_wikidata_units.py``
|
||||
============================
|
||||
|
||||
:origin:`[source] <searxng_extra/update/update_wikidata_units.py>`
|
||||
|
||||
.. automodule:: searxng_extra.update.update_wikidata_units
|
||||
:members:
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 18 KiB |
@ -1,40 +0,0 @@
|
||||
Donate to searxng.org
|
||||
=====================
|
||||
|
||||
Why donating?
|
||||
-------------
|
||||
|
||||
If you want to support the SearXNG team you can make a donation.
|
||||
|
||||
This will help us to pay the costs for:
|
||||
|
||||
- the two VPS servers for searx.space
|
||||
- the domain names (searxng.org and searx.space)
|
||||
- the protonmail account
|
||||
|
||||
If there is enough fund we can ask for a security audit or pay an User Experience (UX) designer.
|
||||
|
||||
Payment methods
|
||||
---------------
|
||||
|
||||
- Credit/debit card and bank transfer
|
||||
|
||||
- `Liberapay`_ (recurrent donation)
|
||||
- `Buy Me a Coffee`_ (one time donation)
|
||||
|
||||
- Cryptocurrency
|
||||
|
||||
- Bitcoin: `bc1qn3rw8t86h05cs3grx2kmwmptw9k4kt4hyzktqj`_ (Segwit
|
||||
compatible)
|
||||
- Bitcoin cash: `qpead2yu482e3h9amy5zk45l8qrfhk59jcpw3cth9e`_
|
||||
- Ethereum: `0xCf82c7eb915Ee70b5B69C1bBB5525e157F35FA43`_
|
||||
- Dogecoin: `DBCYS9issTt84pddXSsTHpQxyQDTFp1TE4`_
|
||||
- Litecoin: `ltc1q5j6x6f4f2htldhq570e353clc8fmw44ra5er5q`_
|
||||
|
||||
.. _Liberapay: https://liberapay.com/SearXNG/
|
||||
.. _Buy Me a Coffee: https://buymeacoffee.com/searxng
|
||||
.. _bc1qn3rw8t86h05cs3grx2kmwmptw9k4kt4hyzktqj: bitcoin:bc1qn3rw8t86h05cs3grx2kmwmptw9k4kt4hyzktqj
|
||||
.. _qpead2yu482e3h9amy5zk45l8qrfhk59jcpw3cth9e: bitcoincash:qpead2yu482e3h9amy5zk45l8qrfhk59jcpw3cth9e
|
||||
.. _0xCf82c7eb915Ee70b5B69C1bBB5525e157F35FA43: ethereum:0xCf82c7eb915Ee70b5B69C1bBB5525e157F35FA43
|
||||
.. _DBCYS9issTt84pddXSsTHpQxyQDTFp1TE4: dogecoin:DBCYS9issTt84pddXSsTHpQxyQDTFp1TE4
|
||||
.. _ltc1q5j6x6f4f2htldhq570e353clc8fmw44ra5er5q: litecoin:ltc1q5j6x6f4f2htldhq570e353clc8fmw44ra5er5q
|
||||
@ -1,14 +0,0 @@
|
||||
===========
|
||||
Source-Code
|
||||
===========
|
||||
|
||||
This is a partial documentation of our source code. We are not aiming to document
|
||||
every item from the source code, but we will add documentation when requested.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:caption: Contents
|
||||
:glob:
|
||||
|
||||
searx.*
|
||||
@ -1,8 +0,0 @@
|
||||
.. _searx.babel_extract:
|
||||
|
||||
===============================
|
||||
Custom message extractor (i18n)
|
||||
===============================
|
||||
|
||||
.. automodule:: searx.babel_extract
|
||||
:members:
|
||||
@ -1,9 +0,0 @@
|
||||
.. _demo offline engine:
|
||||
|
||||
===================
|
||||
Demo Offline Engine
|
||||
===================
|
||||
|
||||
.. automodule:: searx.engines.demo_offline
|
||||
:members:
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
.. _demo online engine:
|
||||
|
||||
==================
|
||||
Demo Online Engine
|
||||
==================
|
||||
|
||||
.. automodule:: searx.engines.demo_online
|
||||
:members:
|
||||
|
||||
@ -1,55 +0,0 @@
|
||||
.. _google engines:
|
||||
|
||||
==============
|
||||
Google Engines
|
||||
==============
|
||||
|
||||
.. contents:: Contents
|
||||
:depth: 2
|
||||
:local:
|
||||
:backlinks: entry
|
||||
|
||||
|
||||
.. _google API:
|
||||
|
||||
google API
|
||||
==========
|
||||
|
||||
.. _Query Parameter Definitions:
|
||||
https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions
|
||||
|
||||
For detailed description of the *REST-full* API see: `Query Parameter
|
||||
Definitions`_. Not all parameters can be appied and some engines are *special*
|
||||
(e.g. :ref:`google news engine`).
|
||||
|
||||
.. _google web engine:
|
||||
|
||||
Google WEB
|
||||
==========
|
||||
|
||||
.. automodule:: searx.engines.google
|
||||
:members:
|
||||
|
||||
.. _google images engine:
|
||||
|
||||
Google Images
|
||||
=============
|
||||
|
||||
.. automodule:: searx.engines.google_images
|
||||
:members:
|
||||
|
||||
.. _google videos engine:
|
||||
|
||||
Google Videos
|
||||
=============
|
||||
|
||||
.. automodule:: searx.engines.google_videos
|
||||
:members:
|
||||
|
||||
.. _google news engine:
|
||||
|
||||
Google News
|
||||
===========
|
||||
|
||||
.. automodule:: searx.engines.google_news
|
||||
:members:
|
||||
@ -1,8 +0,0 @@
|
||||
.. _load_engines:
|
||||
|
||||
============
|
||||
Load Engines
|
||||
============
|
||||
|
||||
.. automodule:: searx.engines
|
||||
:members:
|
||||
@ -1,9 +0,0 @@
|
||||
.. _tineye engine:
|
||||
|
||||
======
|
||||
Tineye
|
||||
======
|
||||
|
||||
.. automodule:: searx.engines.tineye
|
||||
:members:
|
||||
|
||||
@ -1,8 +0,0 @@
|
||||
.. _yahoo engine:
|
||||
|
||||
============
|
||||
Yahoo Engine
|
||||
============
|
||||
|
||||
.. automodule:: searx.engines.yahoo
|
||||
:members:
|
||||
@ -1,8 +0,0 @@
|
||||
.. _searx.infopage:
|
||||
|
||||
================
|
||||
Online ``/info``
|
||||
================
|
||||
|
||||
.. automodule:: searx.infopage
|
||||
:members:
|
||||
@ -1,8 +0,0 @@
|
||||
.. _searx.locales:
|
||||
|
||||
=======
|
||||
Locales
|
||||
=======
|
||||
|
||||
.. automodule:: searx.locales
|
||||
:members:
|
||||
@ -1,8 +0,0 @@
|
||||
.. _autodetect search language:
|
||||
|
||||
======================
|
||||
Search language plugin
|
||||
======================
|
||||
|
||||
.. automodule:: searx.plugins.autodetect_search_language
|
||||
:members:
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue