Commit Graph

389 Commits

Author SHA1 Message Date
Markus Heiser
1c79c1a3d2 [LXC] fedora33 EOL [1], upgrade to fedora35 [2]
[1] https://docs.fedoraproject.org/en-US/releases/eol/
[2] https://docs.fedoraproject.org/en-US/releases/f35/

Tested by::

    # build the container ..
    $ sudo -H ./utils/lxc.sh build searx-fedora35

    # open a shell in the container
    $ sudo -H ./utils/lxc.sh cmd searx-fedora35 bash
    [root@searx-fedora35 SearXNG]#

    # install a complete SearXNG suite ..
    [root@searx-fedora35 SearXNG]# ./utils/searx.sh install all
    ...

    # install apache to export the SearXNG instance by HTTP
    [root@searx-fedora35 SearXNG]# ./utils/searx.sh apache install
    ...
    INFO:  got 200 from http://10.174.184.94/searx

To build wheel `python3-devel` needs to be added to SEARX_PACKAGES_fedora::

    |searx|   × Building wheel for setproctitle (pyproject.toml) did not run successfully.
    |searx|   │ exit code: 1
    ...
    |searx|       In file included from src/spt.h:15,
    |searx|                        from src/setproctitle.c:14:
    |searx|       src/spt_python.h:16:10: fatal error: Python.h: No such file or directory
    |searx|          16 | #include <Python.h>
    |searx|             |          ^~~~~~~~~~
    |searx|       compilation terminated.
    |searx|       error: command '/usr/bin/gcc' failed with exit code 1
    |searx|       [end of output]
    ...
    |searx|   ERROR: Failed building wheel for setproctitle

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 17:44:00 +01:00
Markus Heiser
8d69ee5e7f [mod] Serving static files with uWSGI (searxng.ini)
1. Serving static files with uWSGI by using static file mount points [1].
2. Expires set to one year since there are hashes [2]

[1] https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html#mode-3-using-static-file-mount-points
[2] https://github.com/searxng/searxng/pull/932

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-08 18:13:13 +01:00
Markus Heiser
5eedd5b72a [fix] socket in SearXNG's uWSGI app (searxng.ini)
Use SEARX_UWSGI_SOCKET in uWSGI systemd service

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-08 17:44:27 +01:00
Markus Heiser
e48e2a9914 [fix] APACHE_SEARX_SITE=searxng.conf renamed in 2b125214
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-08 17:18:02 +01:00
Martin Fischer
7ce7625117
Merge pull request #804 from return42/minor-fix
[mod] lib_nvm.sh: minor improvements / no functional change
2022-01-24 17:58:16 +01:00
Markus Heiser
309147d86f [mod] introduce node.env.devtools function 2022-01-24 07:51:31 +01:00
Martin Fischer
cac0352986 [mod] remove .nvm_packages, add eslint to package.json 2022-01-23 08:00:39 +01:00
Markus Heiser
a163385474 [mod] lib_nvm.sh: minor improvements / no functional change
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-22 19:57:25 +01:00
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Markus Heiser
fe8b88c8a4 [mod] script to build & install a redis instance
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.

    $ ./manage redis.help
    redis.:
      devpkg    : install essential packages to compile redis
      build     : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
      install   : create user (searxng-redis) and install systemd service (searxng-redis)
      remove    : delete user (searxng-redis) and remove service (searxng-redis)
      shell     : start bash interpreter from user searxng-redis
      src       : clone redis source code to <path> and checkput 6.2.6
      useradd   : create user (searxng-redis) at /usr/local/searxng-redis
      userdel   : delete user (searxng-redis)
      addgrp    : add <user> to group (searxng-redis)
      rmgrp     : remove <user> from group (searxng-redis)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 14:34:26 +01:00
Alexandre Flament
76cbfbbdda reference docs.searxng.org 2022-01-02 21:18:29 +01:00
Markus Heiser
c89d2aab3f [mod] make static.build.commit more robust
- use single quote in the STATIC_BUILT_PATHS to avoid bash globbing
- don't try to commit if no files have been changed

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-01 20:25:39 +01:00
Markus Heiser
9349c71c54 [mod] themes/simple/img/searxng.svg -> src/brand/searxng.svg
* move `searx/static/themes/simple/img/searxng.svg` to `src/brand/searxng.svg`

* README.rst can use it without a reference to a theme.

* the simple theme can create `searx/static/themes/simple/img/searxng.png` using
  the svg2png task

Suggested-by: @dalf https://github.com/searxng/searxng/pull/561#issuecomment-981747902
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-29 19:53:28 +01:00
Markus Heiser
ee8b6aac36 [fix] searx.sh update: replace git pull by a hard reset
If the fetched branch has been rebased a 'git pull' will fails.  To get fetched
branch in the working tree, a git reset is needed.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-26 15:15:17 +01:00
Markus Heiser
ef3f084c49
Merge pull request #516 from return42/searxng/filtron
[mod] filtron.sh - install github.com/searxng/filtron
2021-11-23 00:23:23 +01:00
Markus Heiser
93e933278a [fix] filtron.sh & morty.sh: quote "$GO_ENV"
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-21 17:13:34 +01:00
Markus Heiser
c1bee0d96f [mod] filtron.sh - install github.com/searxng/filtron
SearXNG maintains a fork of filtron at [1]

[1] https://github.com/searxng/filtron

Related: https://github.com/searxng/searxng/issues/507#issuecomment-970254027
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-21 17:13:34 +01:00
Markus Heiser
5242a841a5 [mod] NVM: dev-tools pre-installed in NVM's node installation
The Node.js installation in the NVM environment can be used by IDEs and other
developer tasks.  The required developer packagaes are added to the file
./.nvm_packages and will be installed when Node.js is installed.  Initial we
start with:

- eslint

Having a dedicated developer enviroment, provided by nvm makes it easy to
integrate Node.js packages into various IDEs.  One example is shown in the
.dir-locals.el which is used by emacs.

[1] https://github.com/nvm-sh/nvm#default-global-packages-from-file-while-installing
[2] https://eslint.org

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 17:08:22 +01:00
Markus Heiser
69dd025da9 [mod] nvm.cmd ... : run command ... in NVM environment
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 15:40:18 +01:00
Markus Heiser
6b4cad768d [mod] NVM: pin Node.js to v16.13.0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 15:19:17 +01:00
Markus Heiser
3e2f3900b8 [fix] nvm.install: add missing popd / node.env exit with error (254)
Issue::

    $ make clean node.env
    ...
    CLEAN     [NVM] drop .nvm/
    ...
    INFO:  install Node.js by NVM
    ...
    Now using node v16.13.0 (npm v8.1.0)
    ...
    INSTALL   searx/static/themes/oscar/package.json
    npm ERR! code ENOENT
    npm ERR! syscall open
        # Here now comes the issue, caused by the missing 'popd' ..
    npm ERR! path SearXNG/.nvm/searx/static/themes/oscar/package.json
    npm ERR! errno -2
    npm ERR! enoent ENOENT: no such file or directory, open 'SearXNG/.nvm/searx/static/themes/oscar/package.json'
    ERROR: node.env exit with error (254)
    make: *** [Makefile:99: node.env] Error 254

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 15:16:14 +01:00
Markus Heiser
c681570bb9 [fix] brand - partial revert of e82e37ce6
In commit e82e37ce6 accidentally the brand file was touched / partial
revert::

  export GIT_URL='https://github.com/searxng/searxng'
  export GIT_BRANCH='master'

Related: https://github.com/searxng/searxng/pull/515

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 14:01:58 +01:00
Markus Heiser
48752815ed [fix] make nvm.clean - don't raise error when NVM is not installed
Issue was::

    $ LANG=C make nvm.clean
    INFO:  NVM is not installed
    make: *** [Makefile:99: nvm.clean] Error 42

Now::

    $ LANG=C make nvm.clean
    CLEAN     [NVM] not installed

BTW: change info_msg to build_msg

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-19 13:19:54 +01:00
Markus Heiser
e82e37ce6d
Merge pull request #515 from return42/doc-theme
Various modification of the documentation and the theme sphinx-theme
2021-11-19 11:46:29 +01:00
Markus Heiser
1e997386e9
Merge pull request #508 from return42/arm64
[mod] Tools to install and maintain golang binaries & packages.
2021-11-18 19:49:01 +00:00
Markus Heiser
1dae0c0be0 [brand] SearXNG - docs rename links and fix documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-18 18:27:26 +01:00
Markus Heiser
dc1442a2d1 [mod] Tools to install and maintain NVM versions manager for Node.js
[1] https://github.com/nvm-sh/nvm

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 23:31:11 +01:00
Markus Heiser
79b41478ac [fix] filtron & morty - install golang binary that fits to arch & os
Closes: https://github.com/searxng/searxng/issues/507
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 20:44:53 +01:00
Markus Heiser
abd5eee392 [mod] Tools to install and maintain golang [1] binaries & packages.
[1] https://golang.org/doc/devel/release#policy

A simple *helloworld* test with user 'my_user' :

  sudo -H adduser my_user
  ./manage go.golang go1.17.3 my_user
  ./manage go.install github.com/go-training/helloworld@latest my_user
  ./manage go.bash my_user
  $ helloword
  Hello World!!

Don't forget to remove 'my_user':  sudo -H deluser --remove-home my_user

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 18:00:47 +01:00
MrPaulBlack
566dfe3330 [simple theme] update to ion icons 5 and drop webfont
* update from ionicons-npm to ionicons ver.5
* drop the webfont built by grunt for icons
* built icons.html template for inlining icons with jinja2 into html
* update icon to use mostly the outline version
* add icons to categories and do not display them on mobile to save space
* remove all legacy ion icon font files from simple theme
* icons.html is added in this commit since make statc.build.restore requires git to know the file already
* cleanup error-dialog
2021-11-13 10:42:07 +01:00
Markus Heiser
3a981257de [simple theme] create favicon.png from src/svg/searxng-wordmark.svg
new bash function convert_if_newer() / usage::

    convert_if_newer <origfile> <outfile> [<options>, ...]
    convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64

Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>.  The command line is to convert is::

    convert <origfile> [<options>, ...] <outfile>

PNG 'searx/static/themes/simple/img/favicon.png' has been created by::

  $ make themes.simple
  CONVERT   searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser
bf52353a5c [enh] minify searxng-wordmark.svg (HTMLMinifier)
Command::

  ./node_modules/.bin/html-minifier \
      --remove-comments \
      --collapse-whitespace \
      src/svg/searxng-wordmark.svg \
      -o ../../../templates/__common__/searxng-wordmark.min.svg

- html-minifier: https://github.com/kangax/html-minifier
  - onilne: https://kangax.github.io/html-minifier
  - grunt: https://www.npmjs.com/package/grunt-contrib-htmlmin
  - grunt-contrib-htmlmin: https://github.com/gruntjs/grunt-contrib-htmlmin
  - npm: https://www.npmjs.com/package/html-minifier

To test, rebuild your node environment::

  make node.env

Alternatives:

- pretty-data: https://github.com/vkiryukhin/pretty-data
  - grunt: https://www.npmjs.com/package/grunt-xmlmin
  - grunt-xmlming: https://github.com/dtrunk90/grunt-xmlmin
  - npm: https://www.npmjs.com/package/grunt-xmlmin

- minify-xml: https://github.com/kristian/minify-xml
  - no grunt package available
  - npm: https://www.npmjs.com/package/minify-xml

src/svg/searxng-wordmark.svg':
'../../../templates/__common__/searxng-wordmark.min.svg'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser
9f30d5b1f8 [fix] morty & filtron: don't use golang installed by package manager
When golang is installed via a package manager the local version, which filtron
downloads, is ignored.

BTW: With the new go1.17.2 (cfea51f4), 'go get' is depricated::

  go get: installing executables with 'go get' in module mode is deprecated.
    Use 'go install pkg@version' instead.
    For more information, see https://golang.org/doc/go-get-install-deprecation
    or run 'go help get' or 'go help install'.

Reported-by: @tiekoetter https://github.com/searxng/searxng/pull/455#issuecomment-954918411
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 19:18:32 +02:00
Markus Heiser
db6a4d13e7 [mod] searx, filtron & morty scripts: add 'reinstall all' command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser
4167bf15dc [mod] utils/searx.sh - apache, nginx & uwsgi check old 'searx' installations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser
38482739de [mod] utils/searx.sh - check a SearXNG installation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser
2b1252148d [brand] SearXNG - nginx & apache searxng.conf, uwsgi searxng.conf
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser
a9fc4885f2 [brand] SearXNG - bash env SEARXNG_URL
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser
cfea51f40f [mod] utils/morty.sh - check golang version is go1.17.2
Related-to: 5c4afdd7 https://github.com/searxng/searxng/issues/455

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-29 15:51:30 +00:00
Markus Heiser
f3c161f72a [mod] utils/morty.sh - check a morty installation
Impplement a script to check a morty installation::

   ./utils/morty.sh install check

Related-to: https://github.com/searxng/searxng/issues/450
Related-to: https://github.com/searxng/searxng/pull/446#issuecomment-954599668

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-29 15:51:30 +00:00
Markus Heiser
d8d6c3d4ee [mod] utils/filtron.sh - check golang version is go1.17.2
Related-to: 5c4afdd7 https://github.com/searxng/searxng/issues/455

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-29 15:51:30 +00:00
Markus Heiser
064b545f14 [mod] utils/filtron.sh - check a filtron installation
Impplement a script to check a filtron installation::

   ./utils/filtron.sh install check

Related-to: https://github.com/searxng/searxng/issues/450
Related-to: https://github.com/searxng/searxng/pull/446#issuecomment-954599668

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-29 15:51:30 +00:00
Markus Heiser
f46508b35f [mod] utils/searxng_check.py - check a SearXNG installation
Impplement a script to check a SearXNG installation::

   ./utils/searx.sh install check

Related-to: https://github.com/searxng/searxng/issues/450
Related-to: https://github.com/searxng/searxng/pull/446#issuecomment-954599668
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-29 15:51:30 +00:00
Alexandre Flament
5c4afdd737 [fix] filtron.sh: fix filtron proxy error
morty & filtron upgrade golang from v1.13.5 to v1.17.2 (#stable go[1])

Closes: #443

[1] https://golang.org/dl/#stable
2021-10-29 14:09:58 +00:00
Markus Heiser
60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament
3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Alexandre Flament
253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Alexandre Flament
9e266ecad3 SearXNG: SEARX_BIND_ADDRESS 2021-10-02 16:58:09 +02:00
Alexandre Flament
6443ed7562 SearXNG: SEARXNG_PORT 2021-10-02 16:56:24 +02:00
Alexandre Flament
e5e322e8ed [mod] utils/lib.sh: add pyenv.activate
* initialize virtualenv (pyenv.install)
* activate virtualenv

close #310
2021-09-17 14:15:35 +02:00
Allen
08ac175236 [fix] Wording for Morty & Filtron error message 2021-09-04 10:22:52 +02:00
Markus Heiser
8ba4dc26d1 [fix] utils/searx.sh install settings - do not eval the template
The settings template in utils/templates/etc/searx/settings.yml
does not include any replacements (shell environment variables).

BTW: utils/filtron.sh & utils/lib.sh
  Add some more messages to installation procedures, to be more clear.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-20 14:21:54 +02:00
Alexandre Flament
a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Markus Heiser
5b3eeb6e54 [mod] searx.sh - improve some messages prompted while installation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser
85e0908e4f [fix] searx.sh - get_installed_version_variables()
Fix the quotation marks in the command line to run

    python -m searx.version

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser
64d64535a2 [fix] buildenv: first unset environment / then import searx package
The first import of names from the searx package implies loading the
settings.yml.  Before this is done, the enviroment variables must be unset to
not overwrite the values from the settings.yml

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Alexandre Flament
4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Alexandre Flament
24fcd7475a Revert "[brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42"
This reverts commit e946c3462c.
2021-07-27 08:40:53 +02:00
Markus Heiser
e946c3462c [brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42
Do not merge this patch in master branch of SearXNG!  This branch exists only
for testing the feature branch fix-searx.sh @return42.

This patch changes the buildenv to::

    GIT_URL='https://github.com/return42/searxng'
    GIT_BRANCH='fix-searx.sh'
    SEARX_PORT='7777'
    SEARX_BIND_ADDRESS='127.0.0.12'

To test installation procedure, clone feature branch (fix-searx.sh)::

    $ cd ~/Downloads
    $ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
    $ cd searxng

    $ ./utils/searx.sh install all
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
7196a9b5dd [fix] ./utils/filtron.sh - FILTRON_TARGET from YAML settings
The filtron target is the SearXNG installation and the default of FILTRON_TARGET
is taken from the YAML configuration ('server.port' & 'server.bind_address').

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
57b8f340a6 [mod] utils/searx.sh - add command 'inspect settings'
Inspect YAML setting <key> from SearXNG instance (${SEARX_SRC})::

  utils/searx.sh inspect settings server.base_url

utils/lib_install.sh
  should not log on stdout which is used for the output of function
  prompt_installation_setting().  Turned build_msg (stdout) into
  info_msg (stderr).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:24:22 +02:00
Markus Heiser
28c874bf3f [brand] searx.sh & morty.sh - renamed 'searx' into 'SearXNG'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:39:02 +02:00
Markus Heiser
7167a7ee36 [mod] utils/.searx.sh: add commands: install [dot-config|init-src]
This patch implements two new commands:

install dot-config :
  - copy ./config.sh to ${SEARX_SRC}
  - implemented in new function install_DOT_CONFIG

install init-src:
  - sync files (SEARX_SRC_INIT_FILES) with ${SEARX_SRC}
  - implemented in new function init_SEARX_SRC()

new functions: verify_continue_install() and prompt_installation_status()
  These functions are used in installation procedures to get the status of the
  installation procedure.  The status is based on the existing function:

    usage: install_searx_get_state

        Prompts a string indicating the status of the installation procedure

    missing-searx-clone:
       There is no clone at ${SEARX_SRC}
    missing-searx-pyenv:
       There is no pyenv in ${SEARX_PYENV}
    installer-modified:
       There are files modified locally in the installer (clone),
       see ${SEARX_SRC_INIT_FILES} description.
    python-installed:
       Scripts can be executed in instance's environment
       - user:  ${SERVICE_USER}
       - pyenv: ${SEARX_PYENV}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:59 +02:00
Markus Heiser
f61c918dd4 [mod] normalize .config.sh with settings.yml
In commit 94851790 we have centralized all SearXNG setups in the settings.yml
file:

  94851790 [mod] move brand options from Makefile to settings.yml

This step has not yet been completed for the installation procedures!  Since all
SearXNG setups are done in the settings.yml these environment variables needs to
be removed from the ./conf.sh file.  Scripts and other tasks running outside of
an instance got the needed values from the ./utils/brand.env file.

By example: ATM the environment variables of the ./config.sh file are in
conflict with them from settings.yml:

  - PUBLIC_URL          --> {server:base_url}
  - SEARX_INTERNAL_HTTP --> {server:bind_address}.{server:port}
  - GIT_BRANCH          --> {brand:GIT_URL}

These environment variable of a SearXNG instance and additional

  - SEARX_SETTINGS_TEMPLATE

has been remove from the '.config.sh' file.  With this patch, the main focus of
./conf.sh resists on environment variables needed for the installation of morty,
filtron software.

modified  .config.sh:
  - removed no longer supported variables (see above)
  - add comment about: SearXNG setup in settings.yml

modified utils/searx.sh:
  - SEARX_INTERNAL_HTTP no longer take from .config.sh
  - SEARX_SETTINGS_PATH /etc/searx/settings.yml
  - SEARX_SETTINGS_TEMPLATE obsolete

modified utils/lib_install.sh:
  Initialize environment variables SEARX_PYENV, SEARX_SETTINGS_PATH and
  PUBLIC_URL.

modified:  utils/morty.sh
  Add missing hint about SEARX_SETTINGS_PATH and move PUBLIC_URL to
  utils/lib_install.sh

modified:  utils/morty.sh
  Move PUBLIC_URL to utils/lib_install.sh

Renamed utils/templates/etc/searx/use_default_settings.yml -> settings.yml
  - removed option which can't be modified after installation
  - add some comments with examples

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:57 +02:00
Markus Heiser
2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser
414a6105e7 [mod] load .config.sh from installation (utils/lib_install.sh)
**new** utils/lib_install.sh:
  Used to initialize installation procedures

  - Modified source_dot_config function that
    - loads .config.sh from an existing installation (at SEARX_SRC).
    - initialize **SEARX_SRC_INIT_FILES**
  - functions like:
    - install_log_searx_instance()
    - install_searx_get_state()

**modified** utils/searx.sh
    - obsolete environment SEARX_INSTANCE_NAME has been replaced
      by string 'SearXNG'.

**modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh
  - source utils/lib_install.sh
  - normalize logging of environment variables using new function
    install_log_searx_instance()

**modified** utils/lib.sh
  - fix marginal typos

**Installation scripts**
  The utils/lib_install.sh is sourced by the installations scripts:

  - utils/searx.sh
  - utils/morty.sh
  - utils/filtron.sh

  If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function
  loads this configuration (instead of './.config.sh').

**SEARX_SRC_INIT_FILES**
  Array of file names to sync into a installation at $SEARX_SRC.  The file names
  are relative to the $REPO_ROOT.  Set by function init_SEARX_SRC_INIT_FILES().
  Most often theses are files like:

  - .config.sh
  - searx/settings.yml
  - utils/brand.env
  - ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:33 +02:00
Markus Heiser
19abaf272d [mod] make pyenv.OK() less verbose when pyenv is OK
Prompt the verbose messages ...

  PYENV     [check] import yaml --> OK
  PYENV     OK

only when environment $VERBOSE is active.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
c9220de690 [mod] drop unused setting option brand:twitter_url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
3e50e8de3e [mod] drop usage of the searx.brand namespace (python procs)
Added function searx.get_setting(name, default=_unset):
  Returns the value to which ``name`` point.  If there is no such name in the
  settings and the ``default`` is unset, a KeyError exception is raised.

In all the python processes ..

- make docs
- make buildenv
- make install (setup.py)

the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.

- brand.SEARX_URL        --> get_setting('server.base_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.GIT_BRANCH'      --> get_setting('server.base_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')
- brand.WIKI_URL         --> get_setting('brand.wiki_url')
- brand.TWITTER_URL      --> get_setting('brand.twitter_url', '')

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18 15:55:42 +02:00
Markus Heiser
b3ff510e10 [fix] pacman --noprogressbar in shell scripts
From pacman's man page:

  --noprogressbar
    Do not show a progress bar when downloading files. This can be useful for
    scripts that call pacman and capture the output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:55 +02:00
Markus Heiser
73fc6499d8 [fix] utils/lib.sh: in a pipe, do not print ASCII escape code
In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:49 +02:00
Alexandre Flament
30e4a2a224
Merge pull request #174 from searxng/static_script
[mod] add make targets to manage the build files in the /searx/static directory
2021-06-28 10:53:32 +02:00
Markus Heiser
2a12637f59 [fix] utils/lib.sh: prefix_stdout show cursor (ANSI escape \e[?25h)
Some piped commands hide the cursor, show cursory when the stream ends.

Most often this is a bug of the command which piped.  The command should not
hide the cursor when it writes to a pipe.  I have seen this bug with the package
manager (pacman) from ArchLinux.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-27 11:48:57 +02:00
Markus Heiser
25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00
Markus Heiser
03d5d14d98 [mod] utils/manage_static.sh: remove git_log_current_branch
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
80710c666f [mod] utils/manage_static.sh: simplify git restore actions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
2cb1f35086 [mod] utils/manage_static.sh: add script to shellcheck
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)

No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Alexandre Flament
0606cde105 [mod] add make targets to manage the build files in the /searx/static directory
The idea is to avoid conflict when there are differents branches with changes are made on the static files.
A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies.
So the solution in this commit keep the sources and the build files in the same git repository.

In one branch, the modification of the source (*.less, *.js) are commited without the built files.
The built files are commited in a uniq commit, with a commit message "Static build"
In case of merge or rebase, this commit can be dropped.

New make targets:
* static.build.commit.drop:
    drop the last "Static build".
    The command checks that there are only build files in the commit.
* static.build.commit :
    call "make static.build.commit.drop"
    call "make themes.all"
    commit the files
* static.git.restore.staged:
    git restore --staged <build files>
* static.git.restore:
    git restore <build files>

Related to https://github.com/searxng/searxng/issues/137
2021-06-26 13:13:31 +02:00
Markus Heiser
d19869b9f7
Merge pull request #173 from return42/fix-lxc-iptables
[mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
2021-06-25 08:23:40 +00:00
Markus Heiser
c8a6548592 [mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC.  With this patch:

- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
  suggestion for a solution a solution

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 12:47:52 +02:00
Markus Heiser
b5f5ddd5a6 [fix] utils/searx.sh: signature of YAML settings has been changed
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this
into account.

Related-to: 5a7b12ee [yamllint] searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 09:56:59 +02:00
Alexandre Flament
dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Markus Heiser
ae677cb64b [enh] ./manage node.env - check build tools first
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS.  These tools can be installed by::

    sudo -H ./utils/searx.sh install buildhost

If one of the tools is not installed, the script node.env stops with a
appropriate message.

BTW: We ignore CentOS-7 as developer & build platform

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-17 09:23:02 +02:00
Alexandre Flament
7137d2893f [mod] simple theme: remove src/less/ion.less
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file

Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
2021-06-16 14:30:09 +02:00
Markus Heiser
1f91d85506 [fix] utils/morty.sh - PUBLIC_URL_MORTY is based on PUBLIC_URL
The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx"
The default is SEARX_URL which is taken from ./utils/brand.env.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-13 11:55:32 +02:00
Markus Heiser
4a814dabf3 [yamllint] ./utils/templates/etc/searx/*.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12 20:46:07 +02:00
Markus Heiser
6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Markus Heiser
666bf1d32b [fix] typo: 'interactiv' --> 'interactive'
Reported-by: https://github.com/searxng/searxng/pull/38#pullrequestreview-646940531
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 16:29:03 +02:00
Markus Heiser
d9ffb09011 [filtron.sh] make filtron rule file configurable
To select a different file with filtron rules, set environment

    FILTRON_RULES_TEMPLATE

the default is

    utils/templates/etc/filtron/rules.json

The installation is done by the new function install_rules() which offers a
multiple choice in case of collisions (known from searx.sh install setup).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 12:48:49 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
438d35c386 [fix] do not spit out error messages for utils/lxc.sh --help
Error::

  $ utils/lxc.sh --help
  ERROR: missing command lxc
  ...

This breaks also docs build::

  $ make docs
  ...
  WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help'
  ...

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
d5eaa3fb31 [enh] utils/lib.sh - commands to build Sphinx-doc & deploy gh-pages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
036933599b [enh] utils/lib.sh - commands pyenv, pyenv.drop pyenv.(un)install
Implement a boilerplate to manage performance optimized virtualenv builds.
Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv
without having to worry about whether and how the environment is provided. ::

  pyenv.cmd which python
  ..../local/py3/bin/python

  pyenv.cmd which pip
  ..../local/py3/bin/pip

If pyenv.cmd released multiple times the installation will only rebuild if the
function 'pyenv.OK' fails.  Function 'pyenv.OK' make some test to validate that
the virtualenv exists and works as expected.  The check also fails if
requirements listed requirements-dev.txt and requirements.txt has been edited.
Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python
script that validate the python installation.  Here is an example how a
'pyenv.check' implementation could look like::

    pyenv.check() {
       cat  <<EOF
    import yaml
    print('import yaml --> OK')
    EOF
    }

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
f9b05a6c44 [fix] utils/searx.sh - Double quote to prevent globbing
Fix issues reported by `make test.sh`

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
3nprob
0fb423ea59 Allow overriding env vars SEARX_SETTINGS_PATH,UWSGI_SETTINGS_PATH 2021-04-08 12:56:15 +09:00