[fix] utils/build_env.py and add documentation

modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

docs/admin/engines/settings.rst

@@ -37,15 +37,13 @@ see how you can simplify your *user defined* ``settings.yml``.
 Global Settings
 ===============
 
-``general:``
+.. _settings global brand:
-------------
 
-.. code:: yaml
+``brand:``
+------------
 
-   general:
+If you change a value in this section, don't forget to rebuild instance's
-     debug: false               # Debug mode, only for development
+environment (:ref:`utils/brand.env <make buildenv>`)
-     instance_name:  "searxng"  # displayed name
-     contact_url: false         # mailto:contact@example.com
 
 .. code:: yaml
 
@@ -57,22 +55,42 @@ Global Settings
      public_instances: https://searx.space
      wiki_url: https://github.com/searxng/searxng/wiki
 
-``debug`` :
+``git_url`` & ``git_branch`` : :ref:`buildenv GIT_URL & GIT_BRANCH<make buildenv>`
+  Changes this, to point to your searx fork (branch).
+
+``issue_url`` : :ref:`buildenv ISSUE_URL<make buildenv>`
+  If you host your own issue tracker change this URL.
+
+``docs_url`` : :ref:`buildenv DOCS_URL<make buildenv>`
+  If you host your own documentation change this URL.
+
+``public_instances`` : :ref:`buildenv PUBLIC_INSTANCES<make buildenv>`
+  If you host your own https://searx.space change this URL.
+
+``wiki_url`` : :ref:`buildenv WIKI_URL<make buildenv>`
+  Link to your wiki (or ``false``)
+
+.. _settings global general:
+
+``general:``
+------------
+
+.. code:: yaml
+
+   general:
+     debug: false               # Debug mode, only for development
+     instance_name:  "searxng"  # displayed name
+     contact_url: false         # mailto:contact@example.com
+
+``debug`` : ``$SEARX_DEBUG``
   Allow a more detailed log if you run searx directly. Display *detailed* error
   messages in the browser too, so this must be deactivated in production.
 
 ``contact_url``:
   Contact ``mailto:`` address or WEB form.
 
-``git_url`` and ``git_branch``:
-  Changes this, to point to your searx fork (branch).
-
-``docs_url``
-  If you host your own documentation, change this URL.
-
-``wiki_url``:
-  Link to your wiki (or ``false``)
 
+.. _settings global server:
 
 ``server:``
 -----------
@@ -94,16 +112,18 @@ Global Settings
          X-Robots-Tag : noindex, nofollow
          Referrer-Policy : no-referrer
 
-``port`` & ``bind_address``:
+``port`` & ``bind_address``: ``$SEARX_PORT`` & ``$SEARX_BIND_ADDRESS``
   Port number and *bind address* of the searx web application if you run it
   directly using ``python searx/webapp.py``.  Doesn't apply to searx running on
   Apache or Nginx.
 
-``secret_key`` :
+``secret_key`` : ``$SEARX_SECRET``
   Used for cryptography purpose.
 
-``base_url`` :
+``base_url`` : :ref:`buildenv SEARX_URL <make buildenv>`
   The base URL where searx is deployed.  Used to create correct inbound links.
+  If you change the value, don't forget to rebuild instance's environment
+  (:ref:`utils/brand.env <make buildenv>`)
 
 ``image_proxy`` :
   Allow your instance of searx of being able to proxy images.  Uses memory space.

docs/admin/installation-searx.rst

@@ -94,7 +94,7 @@ For a *minimal setup*, configure like shown below – replace ``searx@$(uname
 
   .. group-tab:: Use default settings
 
-    .. literalinclude:: ../../utils/templates/etc/searx/use_default_settings.yml
+    .. literalinclude:: ../../utils/templates/etc/searx/settings.yml
        :language: yaml
 
   .. group-tab:: searx/settings.yml

docs/build-templates/searx.rst

@@ -134,7 +134,7 @@ ${fedora_build}
     .. code-block:: sh
 
        $ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\"
-       $ sudo -H cp \"$SEARX_SRC/utils/templates/etc/searx/use_default_settings.yml\" \\
+       $ sudo -H cp \"$SEARX_SRC/utils/templates/etc/searx/settings.yml\" \\
                     \"${SEARX_SETTINGS_PATH}\"
 
   .. group-tab:: searx/settings.yml
@@ -152,7 +152,6 @@ ${fedora_build}
     .. code-block:: sh
 
        $ sudo -H sed -i -e \"s/ultrasecretkey/\$(openssl rand -hex 16)/g\" \"$SEARX_SETTINGS_PATH\"
-       $ sudo -H sed -i -e \"s/{instance_name}/searx@\$(uname -n)/g\" \"$SEARX_SETTINGS_PATH\"
 
 .. END searx config
 

docs/dev/makefile.rst

@@ -81,6 +81,30 @@ the check fails if you edit the requirements listed in
 If you think, something goes wrong with your ./local environment or you change
 the :origin:`setup.py` file, you have to call :ref:`make clean`.
 
+.. _make buildenv:
+
+``make buildenv``
+=================
+
+Rebuild instance's environment with the modified settings from the
+:ref:`settings global brand` and :ref:`settings global server` section of your
+:ref:`settings.yml <settings location>`.
+
+We have all SearXNG setups are centralized in the :ref:`settings.yml` file.
+This setup is available as long we are in a *installed instance*.  E.g. the
+*installed instance* on the server or the *installed developer instance* at
+``./local`` (the later one is created by a :ref:`make install <make
+install>` or :ref:`make run <make run>`).
+
+Tasks running outside of an *installed instance*, especially those tasks and
+scripts running at (pre-) installation time do not have access to the SearXNG
+setup (from a *installed instance*).  Those tasks need a *build environment*.
+
+The ``make buildenv`` target will update the *build environment* in:
+
+- :origin:`utils/brand.env`
+
+
 .. _make run:
 
 ``make run``

manage

@@ -105,6 +105,16 @@ fi
 export DOCS_BUILD
 
 buildenv() {
+
+    # settings file from repository's working tree are used by default
+    SEARX_SETTINGS_PATH="${REPO_ROOT}/searx/settings.yml"
+
+    if [ -r '/etc/searx/settings.yml' ]; then
+        if ask_yn "should settings read from: /etc/searx/settings.yml"; then
+            SEARX_SETTINGS_PATH='/etc/searx/settings.yml'
+        fi
+    fi
+    export SEARX_SETTINGS_PATH
     SEARX_DEBUG=1 pyenv.cmd python utils/build_env.py 2>&1 \
         | prefix_stdout "${_Blue}BUILDENV${_creset}  "
     return "${PIPESTATUS[0]}"

searx/settings.yml

@@ -4,6 +4,8 @@ general:
   contact_url: false        # mailto:contact@example.com
 
 brand:
+  # If you change a value in this section don't forget to rebuild instance's
+  # enviroment (make buildenv)
   git_url: https://github.com/searxng/searxng
   git_branch: master
   issue_url: https://github.com/searxng/searxng/issues
@@ -31,12 +33,14 @@ search:
 
 server:
   port: 8888
-  # address to listen on
   bind_address: "127.0.0.1"
-  secret_key: "ultrasecretkey"  # change this!
+
-  # Set custom base_url. Possible values:
+  # Set custom base_url. Possible values: false or "https://example.org/location"
-  #   false or "https://your.custom.host/location/"
+  # If you change base_url don't forget to rebuild instance's enviroment (make
+  # buildenv) port & address to listen on
   base_url: false
+
+  secret_key: "ultrasecretkey"  # change this!
   # Proxying image results through searx
   image_proxy: false
   # 1.0 and 1.1 are supported

searx/settings_defaults.py

@@ -147,7 +147,7 @@ SCHEMA = {
         'formats': SettingsValue(list, OUTPUT_FORMATS),
     },
     'server': {
-        'port': SettingsValue(int, 8888),
+        'port': SettingsValue((int,str), 8888, 'SEARX_PORT'),
         'bind_address': SettingsValue(str, '127.0.0.1', 'SEARX_BIND_ADDRESS'),
         'secret_key': SettingsValue(str, environ_name='SEARX_SECRET'),
         'base_url': SettingsValue((False, str), False),

utils/brand.env

@@ -4,5 +4,4 @@ export GIT_BRANCH='master'
 export ISSUE_URL='https://github.com/searxng/searxng/issues'
 export DOCS_URL='https://searxng.github.io/searxng'
 export PUBLIC_INSTANCES='https://searx.space'
-export CONTACT_URL=''
 export WIKI_URL='https://github.com/searxng/searxng/wiki'

utils/build_env.py

@@ -9,15 +9,12 @@ from os.path import realpath, dirname, join, sep, abspath
 
 repo_root = realpath(dirname(realpath(__file__)) + sep + '..')
 sys.path.insert(0, repo_root)
-os.environ['SEARX_SETTINGS_PATH'] = abspath(dirname(__file__) + '/settings.yml')
 
 # Under the assumption that a brand is always a fork assure that the settings
 # file from reposetorie's working tree is used to generate the build_env, not
 # from /etc/searx/settings.yml.
 os.environ['SEARX_SETTINGS_PATH'] = abspath(dirname(__file__) + sep + 'settings.yml')
 
-from searx import get_setting
-
 def _env(*arg, **kwargs):
     val = get_setting(*arg, **kwargs)
     if val is True:
@@ -27,19 +24,35 @@ def _env(*arg, **kwargs):
     return val
 
 name_val = [
-    ('SEARX_URL'              , _env('server.base_url','')),
+    ('SEARX_URL'              , 'server.base_url'),
-    ('GIT_URL'                , _env('brand.git_url', '')),
+    ('GIT_URL'                , 'brand.git_url'),
-    ('GIT_BRANCH'             , _env('brand.git_branch', '')),
+    ('GIT_BRANCH'             , 'brand.git_branch'),
-    ('ISSUE_URL'              , _env('brand.issue_url', '')),
+    ('ISSUE_URL'              , 'brand.issue_url'),
-    ('DOCS_URL'               , _env('brand.docs_url', '')),
+    ('DOCS_URL'               , 'brand.docs_url'),
-    ('PUBLIC_INSTANCES'       , _env('brand.public_instances', '')),
+    ('PUBLIC_INSTANCES'       , 'brand.public_instances'),
-    ('CONTACT_URL'            , _env('general.contact_url', '')),
+    ('WIKI_URL'               , 'brand.wiki_url'),
-    ('WIKI_URL'               , _env('brand.wiki_url', '')),
 ]
 
 brand_env = 'utils' + sep + 'brand.env'
 
-print('build %s' % brand_env)
+# Some defaults in the settings.yml are taken from the environment,
+# e.g. SEARX_BIND_ADDRESS (:py:obj:`searx.settings_defaults.SHEMA`).  When the
+# 'brand.env' file is created these enviroment variables should be unset first::
+
+_unset = object()
+for name, option in name_val:
+    if not os.environ.get(name, _unset) is _unset:
+        del os.environ[name]
+
+# After the variables are unset in the environ, we can import settings
+# (get_setting) from searx module.
+
+from searx import get_setting
+
+print('build %s (settings from: %s)' % (brand_env, os.environ['SEARX_SETTINGS_PATH']))
+sys.path.insert(0, repo_root)
+from searx import settings
+
 with open(repo_root + sep + brand_env, 'w', encoding='utf-8') as f:
-    for name, val in name_val:
+    for name, option in name_val:
-        print("export %s='%s'" % (name, val), file=f)
+        print("export %s='%s'" % (name, _env(option)), file=f)