forked from Archives/searxng
Merge branch 'master' into bug/oscar-theme
commit
1d86d0054f
@ -0,0 +1,33 @@
|
|||||||
|
digraph G {
|
||||||
|
|
||||||
|
node [style=filled, shape=box, fillcolor="#ffffcc", fontname="Sans"];
|
||||||
|
edge [fontname="Sans"];
|
||||||
|
|
||||||
|
browser [label="Browser", shape=Mdiamond];
|
||||||
|
rp [label="Reverse Proxy", href="url to configure reverse proxy"];
|
||||||
|
filtron [label="Filtron", href="https://github.com/asciimoo/filtron"];
|
||||||
|
morty [label="Morty", href="https://github.com/asciimoo/morty"];
|
||||||
|
static [label="Static files", href="url to configure static files"];
|
||||||
|
uwsgi [label="uwsgi", href="url to configure uwsgi"]
|
||||||
|
searx1 [label="Searx #1"];
|
||||||
|
searx2 [label="Searx #2"];
|
||||||
|
searx3 [label="Searx #3"];
|
||||||
|
searx4 [label="Searx #4"];
|
||||||
|
|
||||||
|
browser -> rp [label="HTTPS"]
|
||||||
|
|
||||||
|
subgraph cluster_searx {
|
||||||
|
label = "Searx instance" fontname="Sans";
|
||||||
|
bgcolor="#fafafa";
|
||||||
|
{ rank=same; static rp };
|
||||||
|
rp -> morty [label="optional: images and HTML pages proxy"];
|
||||||
|
rp -> static [label="optional: reverse proxy serves directly static files"];
|
||||||
|
rp -> filtron [label="HTTP"];
|
||||||
|
filtron -> uwsgi [label="HTTP"];
|
||||||
|
uwsgi -> searx1;
|
||||||
|
uwsgi -> searx2;
|
||||||
|
uwsgi -> searx3;
|
||||||
|
uwsgi -> searx4;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
@ -0,0 +1,24 @@
|
|||||||
|
.. _architecture:
|
||||||
|
|
||||||
|
============
|
||||||
|
Architecture
|
||||||
|
============
|
||||||
|
|
||||||
|
.. sidebar:: Needs work!
|
||||||
|
|
||||||
|
This article needs some work / Searx is a collaborative effort. If you have
|
||||||
|
any contribution, feel welcome to send us your :pull:`PR <../pulls>`, see
|
||||||
|
:ref:`how to contribute`.
|
||||||
|
|
||||||
|
Herein you will find some hints and suggestions about typical architectures of
|
||||||
|
searx infrastructures.
|
||||||
|
|
||||||
|
We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`.
|
||||||
|
It shows a *reference* setup for public searx instances.
|
||||||
|
|
||||||
|
.. _arch public:
|
||||||
|
|
||||||
|
.. kernel-figure:: arch_public.dot
|
||||||
|
:alt: arch_public.dot
|
||||||
|
|
||||||
|
Reference architecture of a public searx setup.
|
@ -0,0 +1,103 @@
|
|||||||
|
.. _buildhosts:
|
||||||
|
|
||||||
|
==========
|
||||||
|
Buildhosts
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. sidebar:: This article needs some work
|
||||||
|
|
||||||
|
If you have any contribution send us your :pull:`PR <../pulls>`, see
|
||||||
|
:ref:`how to contribute`.
|
||||||
|
|
||||||
|
To get best results from build, its recommend to install additional packages
|
||||||
|
on build hosts.
|
||||||
|
|
||||||
|
.. _docs build:
|
||||||
|
|
||||||
|
Build docs
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. _Graphviz: https://graphviz.gitlab.io
|
||||||
|
.. _ImageMagick: https://www.imagemagick.org
|
||||||
|
.. _XeTeX: https://tug.org/xetex/
|
||||||
|
.. _dvisvgm: https://dvisvgm.de/
|
||||||
|
|
||||||
|
.. sidebar:: Sphinx build needs
|
||||||
|
|
||||||
|
- ImageMagick_
|
||||||
|
- Graphviz_
|
||||||
|
- XeTeX_
|
||||||
|
- dvisvgm_
|
||||||
|
|
||||||
|
Most of the sphinx requirements are installed from :origin:`setup.py` and the
|
||||||
|
docs can be build from scratch with ``make docs``. For better math and image
|
||||||
|
processing additional packages are needed. The XeTeX_ needed not only for PDF
|
||||||
|
creation, its also needed for :ref:`math` when HTML output is build.
|
||||||
|
|
||||||
|
To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered
|
||||||
|
as images (``sphinx.ext.imgmath`` extension). If your docs build (``make
|
||||||
|
docs``) shows warnings like this::
|
||||||
|
|
||||||
|
WARNING: dot(1) not found, for better output quality install \
|
||||||
|
graphviz from http://www.graphviz.org
|
||||||
|
..
|
||||||
|
WARNING: LaTeX command 'latex' cannot be run (needed for math \
|
||||||
|
display), check the imgmath_latex setting
|
||||||
|
|
||||||
|
you need to install additional packages on your build host, to get better HTML
|
||||||
|
output.
|
||||||
|
|
||||||
|
.. _system requirements:
|
||||||
|
|
||||||
|
.. tabs::
|
||||||
|
|
||||||
|
.. group-tab:: Ubuntu / debian
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
$ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin
|
||||||
|
|
||||||
|
.. group-tab:: Arch Linux
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
$ sudo pacman -S graphviz imagemagick texlive-bin extra/librsvg
|
||||||
|
|
||||||
|
.. group-tab:: Fedora / RHEL
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
$ sudo dnf install graphviz graphviz-gd texlive-xetex-bin librsvg2-tools
|
||||||
|
|
||||||
|
|
||||||
|
For PDF output you also need:
|
||||||
|
|
||||||
|
.. tabs::
|
||||||
|
|
||||||
|
.. group-tab:: Ubuntu / debian
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu
|
||||||
|
|
||||||
|
.. group-tab:: Arch Linux
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ sudo pacman -S texlive-core texlive-latexextra ttf-dejavu
|
||||||
|
|
||||||
|
.. group-tab:: Fedora / RHEL
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ sudo dnf install \
|
||||||
|
texlive-collection-fontsrecommended texlive-collection-latex \
|
||||||
|
dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts
|
||||||
|
|
||||||
|
.. _system requirements END:
|
||||||
|
|
||||||
|
.. literalinclude:: ../conf.py
|
||||||
|
:language: python
|
||||||
|
:start-after: # sphinx.ext.imgmath setup
|
||||||
|
:end-before: # sphinx.ext.imgmath setup END
|
||||||
|
|
@ -0,0 +1,68 @@
|
|||||||
|
.. _engines generic:
|
||||||
|
|
||||||
|
=======
|
||||||
|
engines
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. sidebar:: Further reading ..
|
||||||
|
|
||||||
|
- :ref:`engines generic`
|
||||||
|
- :ref:`configured engines`
|
||||||
|
- :ref:`engine settings`
|
||||||
|
- :ref:`engine file`
|
||||||
|
|
||||||
|
============= =========== ==================== ============
|
||||||
|
:ref:`engine settings` :ref:`engine file`
|
||||||
|
------------------------- ---------------------------------
|
||||||
|
Name (cfg) Categories
|
||||||
|
------------------------- ---------------------------------
|
||||||
|
Engine .. Paging support **P**
|
||||||
|
------------------------- -------------------- ------------
|
||||||
|
Shortcut **S** Language support **L**
|
||||||
|
Timeout **TO** Time range support **TR**
|
||||||
|
Disabled **D** Offline **O**
|
||||||
|
------------- ----------- -------------------- ------------
|
||||||
|
Suspend end **SE**
|
||||||
|
------------- ----------- ---------------------------------
|
||||||
|
Safe search **SS**
|
||||||
|
============= =========== =================================
|
||||||
|
|
||||||
|
Configuration defaults (at built time):
|
||||||
|
|
||||||
|
.. _configured engines:
|
||||||
|
|
||||||
|
.. jinja:: webapp
|
||||||
|
|
||||||
|
.. flat-table:: Engines configured at built time (defaults)
|
||||||
|
:header-rows: 1
|
||||||
|
:stub-columns: 2
|
||||||
|
|
||||||
|
* - Name (cfg)
|
||||||
|
- S
|
||||||
|
- Engine
|
||||||
|
- TO
|
||||||
|
- Categories
|
||||||
|
- P
|
||||||
|
- L
|
||||||
|
- SS
|
||||||
|
- D
|
||||||
|
- TR
|
||||||
|
- O
|
||||||
|
- SE
|
||||||
|
|
||||||
|
{% for name, mod in engines.items() %}
|
||||||
|
|
||||||
|
* - {{name}}
|
||||||
|
- !{{mod.shortcut}}
|
||||||
|
- {{mod.__name__}}
|
||||||
|
- {{mod.timeout}}
|
||||||
|
- {{", ".join(mod.categories)}}
|
||||||
|
- {{(mod.paging and "y") or ""}}
|
||||||
|
- {{(mod.language_support and "y") or ""}}
|
||||||
|
- {{(mod.safesearch and "y") or ""}}
|
||||||
|
- {{(mod.disabled and "y") or ""}}
|
||||||
|
- {{(mod.time_range_support and "y") or ""}}
|
||||||
|
- {{(mod.offline and "y") or ""}}
|
||||||
|
- {{mod.suspend_end_time}}
|
||||||
|
|
||||||
|
{% endfor %}
|
@ -0,0 +1,39 @@
|
|||||||
|
.. _plugins generic:
|
||||||
|
|
||||||
|
===============
|
||||||
|
Plugins builtin
|
||||||
|
===============
|
||||||
|
|
||||||
|
.. sidebar:: Further reading ..
|
||||||
|
|
||||||
|
- :ref:`dev plugin`
|
||||||
|
|
||||||
|
Configuration defaults (at built time):
|
||||||
|
|
||||||
|
:DO: Default on
|
||||||
|
|
||||||
|
.. _configured plugins:
|
||||||
|
|
||||||
|
.. jinja:: webapp
|
||||||
|
|
||||||
|
.. flat-table:: Plugins configured at built time (defaults)
|
||||||
|
:header-rows: 1
|
||||||
|
:stub-columns: 1
|
||||||
|
:widths: 3 1 9
|
||||||
|
|
||||||
|
* - Name
|
||||||
|
- DO
|
||||||
|
- Description
|
||||||
|
|
||||||
|
JS & CSS dependencies
|
||||||
|
|
||||||
|
{% for plgin in plugins %}
|
||||||
|
|
||||||
|
* - {{plgin.name}}
|
||||||
|
- {{(plgin.default_on and "y") or ""}}
|
||||||
|
- {{plgin.description}}
|
||||||
|
|
||||||
|
{% for dep in (plgin.js_dependencies + plgin.css_dependencies) %}
|
||||||
|
| ``{{dep}}`` {% endfor %}
|
||||||
|
|
||||||
|
{% endfor %}
|
@ -0,0 +1,6 @@
|
|||||||
|
stub col row 1, column, "loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
|
||||||
|
eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam
|
||||||
|
voluptua."
|
||||||
|
stub col row 1, "At vero eos et accusam et justo duo dolores et ea rebum. Stet clita
|
||||||
|
kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.", column
|
||||||
|
stub col row 1, column, column
|
@ -0,0 +1,3 @@
|
|||||||
|
graph G {
|
||||||
|
Hello -- World
|
||||||
|
}
|
@ -0,0 +1,221 @@
|
|||||||
|
.. _makefile:
|
||||||
|
|
||||||
|
================
|
||||||
|
Makefile Targets
|
||||||
|
================
|
||||||
|
|
||||||
|
.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
|
||||||
|
|
||||||
|
.. sidebar:: build environment
|
||||||
|
|
||||||
|
Before looking deeper at the targets, first read about :ref:`makefile setup`
|
||||||
|
and :ref:`make pyenv`.
|
||||||
|
|
||||||
|
With the aim to simplify development cycles, started with :pull:`1756` a
|
||||||
|
``Makefile`` based boilerplate was added. If you are not familiar with
|
||||||
|
Makefiles, we recommend to read gnu-make_ introduction.
|
||||||
|
|
||||||
|
The usage is simple, just type ``make {target-name}`` to *build* a target.
|
||||||
|
Calling the ``help`` target gives a first overview::
|
||||||
|
|
||||||
|
$ make help
|
||||||
|
test - run developer tests
|
||||||
|
docs - build documentation
|
||||||
|
docs-live - autobuild HTML documentation while editing
|
||||||
|
run - run developer instance
|
||||||
|
install - developer install (./local)
|
||||||
|
uninstall - uninstall (./local)
|
||||||
|
gh-pages - build docs & deploy on gh-pages branch
|
||||||
|
clean - drop builds and environments
|
||||||
|
...
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:depth: 2
|
||||||
|
:local:
|
||||||
|
:backlinks: entry
|
||||||
|
|
||||||
|
|
||||||
|
.. _makefile setup:
|
||||||
|
|
||||||
|
Setup
|
||||||
|
=====
|
||||||
|
|
||||||
|
.. _git stash: https://git-scm.com/docs/git-stash
|
||||||
|
|
||||||
|
The main setup is done in the :origin:`Makefile`::
|
||||||
|
|
||||||
|
export GIT_URL=https://github.com/asciimoo/searx
|
||||||
|
export SEARX_URL=https://searx.me
|
||||||
|
export DOCS_URL=https://asciimoo.github.io/searx
|
||||||
|
|
||||||
|
.. sidebar:: fork & upstream
|
||||||
|
|
||||||
|
Commit changes in your (local) branch, fork or whatever, but do not push them
|
||||||
|
upstream / `git stash`_ is your friend.
|
||||||
|
|
||||||
|
:GIT_URL: Changes this, to point to your searx fork.
|
||||||
|
|
||||||
|
:SEARX_URL: Changes this, to point to your searx instance.
|
||||||
|
|
||||||
|
:DOCS_URL: If you host your own (branded) documentation, change this URL.
|
||||||
|
|
||||||
|
.. _make pyenv:
|
||||||
|
|
||||||
|
Python environment
|
||||||
|
==================
|
||||||
|
|
||||||
|
.. sidebar:: activate environment
|
||||||
|
|
||||||
|
``source ./local/py3/bin/activate``
|
||||||
|
|
||||||
|
With Makefile we do no longer need to build up the virualenv manually (as
|
||||||
|
described in the :ref:`devquickstart` guide). Jump into your git working tree
|
||||||
|
and release a ``make pyenv``:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ cd ~/searx-clone
|
||||||
|
$ make pyenv
|
||||||
|
PYENV usage: source ./local/py3/bin/activate
|
||||||
|
...
|
||||||
|
|
||||||
|
With target ``pyenv`` a development environment (aka virtualenv) was build up in
|
||||||
|
``./local/py3/``. To make a *developer install* of searx (:origin:`setup.py`)
|
||||||
|
into this environment, use make target ``install``:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ make install
|
||||||
|
PYENV usage: source ./local/py3/bin/activate
|
||||||
|
PYENV using virtualenv from ./local/py3
|
||||||
|
PYENV install .
|
||||||
|
|
||||||
|
You have never to think about intermediate targets like ``pyenv`` or
|
||||||
|
``install``, the ``Makefile`` chains them as requisites. Just run your main
|
||||||
|
target.
|
||||||
|
|
||||||
|
.. sidebar:: drop environment
|
||||||
|
|
||||||
|
To get rid of the existing environment before re-build use :ref:`clean target
|
||||||
|
<make clean>` first.
|
||||||
|
|
||||||
|
If you think, something goes wrong with your ./local environment or you change
|
||||||
|
the :origin:`setup.py` file (or the requirements listed in
|
||||||
|
:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call
|
||||||
|
:ref:`make clean`.
|
||||||
|
|
||||||
|
|
||||||
|
.. _make run:
|
||||||
|
|
||||||
|
``make run``
|
||||||
|
============
|
||||||
|
|
||||||
|
To get up a running a developer instance simply call ``make run``. This enables
|
||||||
|
*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py``
|
||||||
|
instance, disables *debug* option again and opens the URL in your favorite WEB
|
||||||
|
browser (:man:`xdg-open`):
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ make run
|
||||||
|
PYENV usage: source ./local/py3/bin/activate
|
||||||
|
PYENV install .
|
||||||
|
./local/py3/bin/python ./searx/webapp.py
|
||||||
|
...
|
||||||
|
INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
|
||||||
|
...
|
||||||
|
|
||||||
|
.. _make clean:
|
||||||
|
|
||||||
|
``make clean``
|
||||||
|
==============
|
||||||
|
|
||||||
|
Drop all intermediate files, all builds, but keep sources untouched. Includes
|
||||||
|
target ``pyclean`` which drops ./local environment. Before calling ``make
|
||||||
|
clean`` stop all processes using :ref:`make pyenv`.
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ make clean
|
||||||
|
CLEAN pyclean
|
||||||
|
CLEAN clean
|
||||||
|
|
||||||
|
.. _make docs:
|
||||||
|
|
||||||
|
``make docs docs-live docs-clean``
|
||||||
|
==================================
|
||||||
|
|
||||||
|
We describe the usage of the ``doc*`` targets in the :ref:`How to contribute /
|
||||||
|
Documentation <contrib docs>` section. If you want to edit the documentation
|
||||||
|
read our :ref:`make docs-live` section. If you are working in your own brand,
|
||||||
|
adjust your :ref:`Makefile setup <makefile setup>`.
|
||||||
|
|
||||||
|
|
||||||
|
.. _make gh-pages:
|
||||||
|
|
||||||
|
``make gh-pages``
|
||||||
|
=================
|
||||||
|
|
||||||
|
To deploy on github.io first adjust your :ref:`Makefile setup <makefile
|
||||||
|
setup>`. For any further read :ref:`deploy on github.io`.
|
||||||
|
|
||||||
|
.. _make test:
|
||||||
|
|
||||||
|
``make test``
|
||||||
|
=============
|
||||||
|
|
||||||
|
Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does
|
||||||
|
additional :ref:`pylint checks <make pylint>`. You can run tests selective,
|
||||||
|
e.g.:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ make test.pep8 test.unit
|
||||||
|
. ./local/py3/bin/activate; ./manage.sh pep8_check
|
||||||
|
[!] Running pep8 check
|
||||||
|
. ./local/py3/bin/activate; ./manage.sh unit_tests
|
||||||
|
[!] Running unit tests
|
||||||
|
|
||||||
|
.. _make pylint:
|
||||||
|
|
||||||
|
``make pylint``
|
||||||
|
===============
|
||||||
|
|
||||||
|
.. _Pylint: https://www.pylint.org/
|
||||||
|
|
||||||
|
Before commiting its recommend to do some (more) linting. Pylint_ is known as
|
||||||
|
one of the best source-code, bug and quality checker for the Python programming
|
||||||
|
language. Pylint_ is not yet a quality gate within our searx project (like
|
||||||
|
:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code
|
||||||
|
quality anyway. The pylint profile we use at searx project is found in
|
||||||
|
project's root folder :origin:`.pylintrc`.
|
||||||
|
|
||||||
|
Code quality is a ongoing process. Don't try to fix all messages from Pylint,
|
||||||
|
run Pylint and check if your changed lines are bringing up new messages. If so,
|
||||||
|
fix it. By this, code quality gets incremental better and if there comes the
|
||||||
|
day, the linting is balanced out, we might decide to add Pylint as a quality
|
||||||
|
gate.
|
||||||
|
|
||||||
|
|
||||||
|
``make pybuild``
|
||||||
|
================
|
||||||
|
|
||||||
|
.. _PyPi: https://pypi.org/
|
||||||
|
.. _twine: https://twine.readthedocs.io/en/latest/
|
||||||
|
|
||||||
|
Build Python packages in ``./dist/py``.
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$ make pybuild
|
||||||
|
...
|
||||||
|
BUILD pybuild
|
||||||
|
running sdist
|
||||||
|
running egg_info
|
||||||
|
...
|
||||||
|
$ ls ./dist/py/
|
||||||
|
searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz
|
||||||
|
|
||||||
|
To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs
|
||||||
|
twine_ to be installed. Since you are not the owner of :pypi:`searx` you will
|
||||||
|
never need the latter.
|
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,10 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg -->
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg"
|
||||||
|
version="1.1" baseProfile="full"
|
||||||
|
width="70px" height="40px" viewBox="0 0 700 400">
|
||||||
|
<line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/>
|
||||||
|
<rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/>
|
||||||
|
<line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
|
||||||
|
<polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
|
||||||
|
</svg>
|
After Width: | Height: | Size: 580 B |
Loading…
Reference in New Issue