[mod] get rid of ./utils/brand.env and its workflow

All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2024-01-07 18:33:34 +01:00 committed by Markus Heiser
parent 60bc5baea3
commit f9c5727ddc
16 changed files with 53 additions and 201 deletions

View File

@ -163,7 +163,6 @@ jobs:
- name: Update transations
id: update
run: |
git restore utils/brand.env
make V=1 weblate.push.translations
dockers:

View File

@ -56,7 +56,7 @@ test.shell:
$(Q)shellcheck -x -s dash \
dockerfiles/docker-entrypoint.sh
$(Q)shellcheck -x -s bash \
utils/brand.env \
utils/brand.sh \
$(MTOOLS) \
utils/lib.sh \
utils/lib_sxng*.sh \
@ -74,7 +74,6 @@ test.shell:
# wrap ./manage script
MANAGE += buildenv
MANAGE += weblate.translations.commit weblate.push.translations
MANAGE += data.all data.traits data.useragents
MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean

View File

@ -21,17 +21,13 @@
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
``base_url`` : ``$SEARXNG_URL``
The base URL where SearXNG is deployed. Used to create correct inbound links.
If you change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS``
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications. If you
change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
services running behind a proxy and using socket communications.
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.

View File

@ -61,13 +61,9 @@ working tree and release a ``make install`` to get a virtualenv with a
$ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
Successfully installed searxng-2023.7.19+a446dea1b
If you release ``make install`` multiple times the installation will only
rebuild if the sha256 sum of the *requirement files* fails. With other words:
@ -82,13 +78,9 @@ the check fails if you edit the requirements listed in
...
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
Successfully installed searxng-2023.7.19+a446dea1b
.. sidebar:: drop environment
@ -98,67 +90,6 @@ the check fails if you edit the requirements listed in
If you think, something goes wrong with your ./local environment or you change
the :origin:`setup.py` file, you have to call :ref:`make clean`.
.. _make buildenv:
``make buildenv``
=================
Rebuild instance's environment with the modified settings from the
:ref:`settings brand` and :ref:`settings server` section of your
:ref:`settings.yml <settings location>`.
What is the :origin:`utils/brand.env` needed for and why do you need to rebuild
it if necessary?
Short answer: :ref:`installation and maintenance <searxng maintenance>`
scripts are running outside of instance's runtime environment and need some
values defined in the runtime environment.
All the SearXNG setups are centralized in the :ref:`settings.yml` file. This
setup is available as long we are in a *installed instance*. E.g. the
*installed instance* on the server or the *installed developer instance* at
``./local`` (the later one is created by a :ref:`make install <make install>` or
:ref:`make run <make run>`).
Tasks running outside of an *installed instance*, especially :ref:`installation
and maintenance <searxng maintenance>` tasks running at (pre-) installation time
do not have access to the SearXNG setup (from a *installed instance*). Those
tasks need a *build environment*.
The ``make buildenv`` target will update the *build environment* in:
- :origin:`utils/brand.env`
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- ``SEARXNG_URL`` from :ref:`server.base_url <settings server>` (aka
``PUBLIC_URL``)
- ``SEARXNG_BIND_ADDRESS`` from :ref:`server.bind_address <settings server>`
- ``SEARXNG_PORT`` from :ref:`server.port <settings server>`
The ``GIT_URL`` and ``GIT_BRANCH`` in the origin:`utils/brand.env` file, are
read from the git VCS and the branch that is checked out when ``make
buildenv`` command runs.
.. _brand:
**I would like to create my own brand, how should I proceed?**
Create a remote branch (``example.org``), checkout the remote branch (on your
local developer desktop) and in the :origin:`searx/settings.yml` file in the
:ref:`settings server` section set ``base_url``. Run ``make buildenv`` and
create a commit for your brand.
On your server you clone the branch (``example.org``) into your HOME folder
``~`` from where you run the :ref:`installation <installation>` and
:ref:`maintenance <searxng maintenance>` task.
To upgrade you brand, rebase on SearXNG's master branch (on your local
developer desktop), force push it to your remote branch. Go to your server, do
a force pull and run :ref:`sudo -H ./utils/searxng.sh instance update <update
searxng>`.
.. _make node.env:
Node.js environment (``make node.env``)

26
manage
View File

@ -83,8 +83,6 @@ PYLINT_OPTIONS="-m pylint -j 0 --rcfile .pylintrc"
help() {
nvm.help
cat <<EOF
buildenv:
rebuild ./utils/brand.env
webapp.:
run : run developer instance
docs.:
@ -155,29 +153,6 @@ webapp.run() {
SEARXNG_DEBUG=1 pyenv.cmd python -m searx.webapp
}
buildenv() {
# settings file from repository's working tree are used by default
SEARXNG_SETTINGS_PATH="${REPO_ROOT}/searx/settings.yml"
if [ -f /etc/searx/settings.yml ]; then
err_msg "settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/"
fi
if [ -r '/etc/searxng/settings.yml' ]; then
if ask_yn "should settings read from: /etc/searxng/settings.yml"; then
SEARXNG_SETTINGS_PATH='/etc/searxng/settings.yml'
fi
fi
export SEARXNG_SETTINGS_PATH
(
set -e
SEARXNG_DEBUG=1 pyenv.cmd python utils/build_env.py 2>&1 \
| prefix_stdout "${_Blue}BUILDENV${_creset} "
)
return "${PIPESTATUS[0]}"
}
docker.push() {
docker.build push
}
@ -345,7 +320,6 @@ pyenv.install() {
pyenv
build_msg PYENV "[install] pip install -e 'searx${PY_SETUP_EXTRAS}'"
"${PY_ENV_BIN}/python" -m pip install -e ".${PY_SETUP_EXTRAS}"
buildenv
)
local exit_val=$?
if [ ! $exit_val -eq 0 ]; then

View File

@ -66,9 +66,7 @@ search:
- html
server:
# If you change port, bind_address or base_url don't forget to rebuild
# instance's environment (make buildenv). Is overwritten by ${SEARXNG_PORT}
# and ${SEARXNG_BIND_ADDRESS}
# Is overwritten by ${SEARXNG_PORT} and ${SEARXNG_BIND_ADDRESS}
port: 8888
bind_address: "127.0.0.1"
# public URL of the instance, to ensure correct inbound links. Is overwritten

View File

@ -1,5 +0,0 @@
export SEARXNG_URL=''
export SEARXNG_PORT='8888'
export SEARXNG_BIND_ADDRESS='127.0.0.1'
export GIT_URL='https://github.com/searxng/searxng'
export GIT_BRANCH='master'

32
utils/brand.sh Executable file
View File

@ -0,0 +1,32 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# shellcheck source=utils/lib.sh
. /dev/null
build.env.export() {
GIT_BRANCH="$(git branch | grep '\*' | cut -d' ' -f2-)"
GIT_REMOTE="$(git config "branch.${GIT_BRANCH}.remote")"
GIT_URL="$(git config --get "remote.${GIT_REMOTE}.url")"
if [[ "${GIT_URL}" == git@* ]]; then
GIT_URL="${GIT_URL/://}"
GIT_URL="${GIT_URL/git@/https://}"
fi
if [[ "${GIT_URL}" == *.git ]]; then
GIT_URL="${GIT_URL%.git}"
fi
SEARXNG_URL="$(python "${REPO_ROOT}/utils/get_setting.py" server.base_url)"
SEARXNG_PORT="$(python "${REPO_ROOT}/utils/get_setting.py" server.port)"
SEARXNG_BIND_ADDRESS="$(python "${REPO_ROOT}/utils/get_setting.py" server.bind_address)"
export GIT_URL
export GIT_BRANCH
export SEARXNG_URL
export SEARXNG_PORT
export SEARXNG_BIND_ADDRESS
}
pushd "${REPO_ROOT}" &> /dev/null
build.env.export
popd &> /dev/null

View File

@ -1,61 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""build environment used by shell scripts
"""
# set path
import sys
import os
from os.path import realpath, dirname, join, sep, abspath
repo_root = realpath(dirname(realpath(__file__)) + sep + '..')
sys.path.insert(0, repo_root)
# Assure that the settings file from repository's working tree is used to
# generate the build_env, not from /etc/searxng/settings.yml.
os.environ['SEARXNG_SETTINGS_PATH'] = join(repo_root, 'etc', 'settings.yml')
def _env(*arg, **kwargs):
val = get_setting(*arg, **kwargs)
if val is True:
val = '1'
elif val is False:
val = ''
return val
# If you add or remove variables here, do not forget to update:
# - ./docs/admin/engines/settings.rst
# - ./docs/dev/makefile.rst (section make buildenv)
name_val = [
('SEARXNG_URL' , 'server.base_url'),
('SEARXNG_PORT' , 'server.port'),
('SEARXNG_BIND_ADDRESS' , 'server.bind_address'),
]
brand_env = 'utils' + sep + 'brand.env'
# Some defaults in the settings.yml are taken from the environment,
# e.g. SEARXNG_BIND_ADDRESS (:py:obj:`searx.settings_defaults.SHEMA`). When the
# 'brand.env' file is created these environment variables should be unset first::
_unset = object()
for name, option in name_val:
if not os.environ.get(name, _unset) is _unset:
del os.environ[name]
# After the variables are unset in the environ, we can import from the searx
# package (what will read the values from the settings.yml).
from searx.version import GIT_URL, GIT_BRANCH
from searx import get_setting
print('build %s (settings from: %s)' % (brand_env, os.environ['SEARXNG_SETTINGS_PATH']))
sys.path.insert(0, repo_root)
with open(repo_root + sep + brand_env, 'w', encoding='utf-8') as f:
for name, option in name_val:
print("export %s='%s'" % (name, _env(option)), file=f)
print(f"export GIT_URL='{GIT_URL}'", file=f)
print(f"export GIT_BRANCH='{GIT_BRANCH}'", file=f)

View File

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

View File

@ -12,17 +12,6 @@ from pathlib import Path
repo_root = Path(__file__).resolve().parent.parent
# If you add or remove variables here, do not forget to update:
# - ./docs/admin/engines/settings.rst
# - ./docs/dev/makefile.rst (section make buildenv)
name_val = [
("SEARXNG_URL", "server.base_url"),
("SEARXNG_PORT", "server.port"),
("SEARXNG_BIND_ADDRESS", "server.bind_address"),
]
def main(setting_name):
settings_path = repo_root / "searx" / "settings.yml"

View File

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# load environment of the LXC suite
LXC_ENV="${LXC_ENV:-${REPO_ROOT}/utils/lxc-searxng.env}"

View File

@ -3,8 +3,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

View File

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

View File

@ -9,8 +9,8 @@ SEARXNG_UWSGI_USE_SOCKET="${SEARXNG_UWSGI_USE_SOCKET:-true}"
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/lib_redis.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib_redis.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# shellcheck source=utils/brand.sh
source "${REPO_ROOT}/utils/brand.sh"
SERVICE_NAME="searxng"
SERVICE_USER="searxng"
@ -159,7 +159,7 @@ searxng.instance.env() {
echo " SEARXNG_INTERNAL_HTTP: ${SEARXNG_INTERNAL_HTTP}"
fi
cat <<EOF
environment ${SEARXNG_SRC}/utils/brand.env:
environment:
GIT_URL : ${GIT_URL}
GIT_BRANCH : ${GIT_BRANCH}
SEARXNG_URL : ${SEARXNG_URL}
@ -527,7 +527,6 @@ searxng.install.settings() {
if ! [[ -f "${SEARXNG_SRC}/.git/config" ]]; then
die "Before install settings, first install SearXNG."
exit 42
fi
mkdir -p "$(dirname "${SEARXNG_SETTINGS_PATH}")"
@ -608,8 +607,8 @@ searxng.install.uwsgi.http() {
searxng.install.uwsgi.socket() {
rst_para "Install ${SEARXNG_UWSGI_APP} using socket at: ${SEARXNG_UWSGI_SOCKET}"
mkdir -p "$(dirname ${SEARXNG_UWSGI_SOCKET})"
chown -R "${SERVICE_USER}:${SERVICE_GROUP}" "$(dirname ${SEARXNG_UWSGI_SOCKET})"
mkdir -p "$(dirname "${SEARXNG_UWSGI_SOCKET}")"
chown -R "${SERVICE_USER}:${SERVICE_GROUP}" "$(dirname "${SEARXNG_UWSGI_SOCKET}")"
case $DIST_ID-$DIST_VERS in
fedora-*)

View File

@ -7,6 +7,9 @@ import os
import sys
import logging
import warnings
from pathlib import Path
repo_root = Path(__file__).resolve().parent.parent
LOG_FORMAT_DEBUG = '%(levelname)-7s %(name)-30.30s: %(message)s'
logging.basicConfig(level=logging.getLevelName('DEBUG'), format=LOG_FORMAT_DEBUG)
@ -26,6 +29,12 @@ if os.path.isfile(OLD_SETTING):
))
warnings.warn(msg, DeprecationWarning)
OLD_BRAND_ENV = repo_root / 'utils' / 'brand.env'
if os.path.isfile(OLD_BRAND_ENV):
msg = ('%s is no longer needed, remove the file' % (OLD_BRAND_ENV))
warnings.warn(msg, DeprecationWarning)
from searx import redisdb, get_setting
if not redisdb.initialize():