zaclys/searxng
1
0
Fork 1
mirror of https://github.com/searxng/searxng synced 2024-01-01 19:24:07 +01:00
Code Issues Projects Releases Packages Wiki Activity

[mod] get rid of ./utils/brand.env and its workflow

Browse source 
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2024-01-07 18:33:34 +01:00 • committed by Markus Heiser
parent 60bc5baea3
commit f9c5727ddc
16 changed files with 53 additions and 201 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.github/workflows/integration.yml vendored
View file

@ -163,7 +163,6 @@ jobs:
- name: Update transations
id: update
run: |
git restore utils/brand.env
make V=1 weblate.push.translations
dockers:

3
Makefile
View file

@ -56,7 +56,7 @@ test.shell:
$(Q)shellcheck -x -s dash \
dockerfiles/docker-entrypoint.sh
$(Q)shellcheck -x -s bash \
utils/brand.env \
utils/brand.sh \
$(MTOOLS) \
utils/lib.sh \
utils/lib_sxng*.sh \
@ -74,7 +74,6 @@ test.shell:
# wrap ./manage script
MANAGE += buildenv
MANAGE += weblate.translations.commit weblate.push.translations
MANAGE += data.all data.traits data.useragents
MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean

10
docs/admin/settings/settings_server.rst
View file

@ -21,17 +21,13 @@
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
``base_url`` : ``$SEARXNG_URL``
The base URL where SearXNG is deployed. Used to create correct inbound links.
If you change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS``
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications. If you
change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
services running behind a proxy and using socket communications.
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.

73
docs/dev/makefile.rst
View file

@ -61,13 +61,9 @@ working tree and release a ``make install`` to get a virtualenv with a
$ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
Successfully installed searxng-2023.7.19+a446dea1b
If you release ``make install`` multiple times the installation will only
rebuild if the sha256 sum of the *requirement files* fails. With other words:
@ -82,13 +78,9 @@ the check fails if you edit the requirements listed in
...
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
Successfully installed searxng-2023.7.19+a446dea1b
.. sidebar:: drop environment
@ -98,67 +90,6 @@ the check fails if you edit the requirements listed in
If you think, something goes wrong with your ./local environment or you change
the :origin:`setup.py` file, you have to call :ref:`make clean`.
.. _make buildenv:
``make buildenv``
=================
Rebuild instance's environment with the modified settings from the
:ref:`settings brand` and :ref:`settings server` section of your
:ref:`settings.yml <settings location>`.
What is the :origin:`utils/brand.env` needed for and why do you need to rebuild
it if necessary?
Short answer: :ref:`installation and maintenance <searxng maintenance>`
scripts are running outside of instance's runtime environment and need some
values defined in the runtime environment.
All the SearXNG setups are centralized in the :ref:`settings.yml` file. This
setup is available as long we are in a *installed instance*. E.g. the
*installed instance* on the server or the *installed developer instance* at
``./local`` (the later one is created by a :ref:`make install <make install>` or
:ref:`make run <make run>`).
Tasks running outside of an *installed instance*, especially :ref:`installation
and maintenance <searxng maintenance>` tasks running at (pre-) installation time
do not have access to the SearXNG setup (from a *installed instance*). Those
tasks need a *build environment*.
The ``make buildenv`` target will update the *build environment* in:
- :origin:`utils/brand.env`
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- ``SEARXNG_URL`` from :ref:`server.base_url <settings server>` (aka
``PUBLIC_URL``)
- ``SEARXNG_BIND_ADDRESS`` from :ref:`server.bind_address <settings server>`
- ``SEARXNG_PORT`` from :ref:`server.port <settings server>`
The ``GIT_URL`` and ``GIT_BRANCH`` in the origin:`utils/brand.env` file, are
read from the git VCS and the branch that is checked out when ``make
buildenv`` command runs.
.. _brand:
**I would like to create my own brand, how should I proceed?**
Create a remote branch (``example.org``), checkout the remote branch (on your
local developer desktop) and in the :origin:`searx/settings.yml` file in the
:ref:`settings server` section set ``base_url``. Run ``make buildenv`` and
create a commit for your brand.
On your server you clone the branch (``example.org``) into your HOME folder
``~`` from where you run the :ref:`installation <installation>` and
:ref:`maintenance <searxng maintenance>` task.
To upgrade you brand, rebase on SearXNG's master branch (on your local
developer desktop), force push it to your remote branch. Go to your server, do
a force pull and run :ref:`sudo -H ./utils/searxng.sh instance update <update
searxng>`.
.. _make node.env:
Node.js environment (``make node.env``)

26
manage
View file

@ -83,8 +83,6 @@ PYLINT_OPTIONS="-m pylint -j 0 --rcfile .pylintrc"
help() {
nvm.help
cat <<EOF
buildenv:
rebuild ./utils/brand.env
webapp.:
run : run developer instance
docs.:
@ -155,29 +153,6 @@ webapp.run() {
SEARXNG_DEBUG=1 pyenv.cmd python -m searx.webapp
}
buildenv() {
# settings file from repository's working tree are used by default
SEARXNG_SETTINGS_PATH="${REPO_ROOT}/searx/settings.yml"
if [ -f /etc/searx/settings.yml ]; then
err_msg "settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/"
fi
if [ -r '/etc/searxng/settings.yml' ]; then
if ask_yn "should settings read from: /etc/searxng/settings.yml"; then
SEARXNG_SETTINGS_PATH='/etc/searxng/settings.yml'
fi
fi
export SEARXNG_SETTINGS_PATH
(
set -e
SEARXNG_DEBUG=1 pyenv.cmd python utils/build_env.py 2>&1 \
| prefix_stdout "${_Blue}BUILDENV${_creset} "
)
return "${PIPESTATUS[0]}"
}
docker.push() {
docker.build push
}
@ -345,7 +320,6 @@ pyenv.install() {
pyenv
build_msg PYENV "[install] pip install -e 'searx${PY_SETUP_EXTRAS}'"
"${PY_ENV_BIN}/python" -m pip install -e ".${PY_SETUP_EXTRAS}"
buildenv
)
local exit_val=$?
if [ ! $exit_val -eq 0 ]; then

4
searx/settings.yml
View file

@ -66,9 +66,7 @@ search:
- html
server:
# If you change port, bind_address or base_url don't forget to rebuild
# instance's environment (make buildenv). Is overwritten by ${SEARXNG_PORT}
# and ${SEARXNG_BIND_ADDRESS}
# Is overwritten by ${SEARXNG_PORT} and ${SEARXNG_BIND_ADDRESS}
port: 8888
bind_address: "127.0.0.1"
# public URL of the instance, to ensure correct inbound links. Is overwritten

5
utils/brand.env
View file

@ -1,5 +0,0 @@
export SEARXNG_URL=''
export SEARXNG_PORT='8888'
export SEARXNG_BIND_ADDRESS='127.0.0.1'
export GIT_URL='https://github.com/searxng/searxng'
export GIT_BRANCH='master'

32
utils/brand.sh Executable file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-or-later
# shellcheck source=utils/lib.sh
. /dev/null
build.env.export() {
GIT_BRANCH="$(git branch | grep '\*' | cut -d' ' -f2-)"
GIT_REMOTE="$(git config "branch.${GIT_BRANCH}.remote")"
GIT_URL="$(git config --get "remote.${GIT_REMOTE}.url")"
if [[ "${GIT_URL}" == git@* ]]; then
GIT_URL="${GIT_URL/://}"
GIT_URL="${GIT_URL/git@/https://}"
fi
if [[ "${GIT_URL}" == *.git ]]; then
GIT_URL="${GIT_URL%.git}"
fi
SEARXNG_URL="$(python "${REPO_ROOT}/utils/get_setting.py" server.base_url)"
SEARXNG_PORT="$(python "${REPO_ROOT}/utils/get_setting.py" server.port)"
SEARXNG_BIND_ADDRESS="$(python "${REPO_ROOT}/utils/get_setting.py" server.bind_address)"
export GIT_URL
export GIT_BRANCH
export SEARXNG_URL
export SEARXNG_PORT
export SEARXNG_BIND_ADDRESS
}
pushd "${REPO_ROOT}" &> /dev/null
build.env.export
popd &> /dev/null

61
utils/build_env.py
View file

@ -1,61 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""build environment used by shell scripts
"""
# set path
import sys
import os
from os.path import realpath, dirname, join, sep, abspath
repo_root = realpath(dirname(realpath(__file__)) + sep + '..')
sys.path.insert(0, repo_root)
# Assure that the settings file from repository's working tree is used to
# generate the build_env, not from /etc/searxng/settings.yml.
os.environ['SEARXNG_SETTINGS_PATH'] = join(repo_root, 'etc', 'settings.yml')
def _env(*arg, **kwargs):
val = get_setting(*arg, **kwargs)
if val is True:
val = '1'
elif val is False:
val = ''
return val
# If you add or remove variables here, do not forget to update:
# - ./docs/admin/engines/settings.rst
# - ./docs/dev/makefile.rst (section make buildenv)
name_val = [
('SEARXNG_URL' , 'server.base_url'),
('SEARXNG_PORT' , 'server.port'),
('SEARXNG_BIND_ADDRESS' , 'server.bind_address'),
]
brand_env = 'utils' + sep + 'brand.env'
# Some defaults in the settings.yml are taken from the environment,
# e.g. SEARXNG_BIND_ADDRESS (:py:obj:`searx.settings_defaults.SHEMA`). When the
# 'brand.env' file is created these environment variables should be unset first::
_unset = object()
for name, option in name_val:
if not os.environ.get(name, _unset) is _unset:
del os.environ[name]
# After the variables are unset in the environ, we can import from the searx
# package (what will read the values from the settings.yml).
from searx.version import GIT_URL, GIT_BRANCH
from searx import get_setting
print('build %s (settings from: %s)' % (brand_env, os.environ['SEARXNG_SETTINGS_PATH']))
sys.path.insert(0, repo_root)
with open(repo_root + sep + brand_env, 'w', encoding='utf-8') as f:
for name, option in name_val:
print("export %s='%s'" % (name, _env(option)), file=f)
print(f"export GIT_URL='{GIT_URL}'", file=f)
print(f"export GIT_BRANCH='{GIT_BRANCH}'", file=f)

2
utils/filtron.sh
View file

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

11
utils/get_setting.py
View file

@ -12,17 +12,6 @@ from pathlib import Path
repo_root = Path(__file__).resolve().parent.parent
# If you add or remove variables here, do not forget to update:
# - ./docs/admin/engines/settings.rst
# - ./docs/dev/makefile.rst (section make buildenv)
name_val = [
("SEARXNG_URL", "server.base_url"),
("SEARXNG_PORT", "server.port"),
("SEARXNG_BIND_ADDRESS", "server.bind_address"),
]
def main(setting_name):
settings_path = repo_root / "searx" / "settings.yml"

2
utils/lxc.sh
View file

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# load environment of the LXC suite
LXC_ENV="${LXC_ENV:-${REPO_ROOT}/utils/lxc-searxng.env}"

2
utils/morty.sh
View file

@ -3,8 +3,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

2
utils/searx.sh
View file

@ -4,8 +4,6 @@
# shellcheck source=utils/lib.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# ----------------------------------------------------------------------------
# config

11
utils/searxng.sh
View file

@ -9,8 +9,8 @@ SEARXNG_UWSGI_USE_SOCKET="${SEARXNG_UWSGI_USE_SOCKET:-true}"
source "$(dirname "${BASH_SOURCE[0]}")/lib.sh"
# shellcheck source=utils/lib_redis.sh
source "$(dirname "${BASH_SOURCE[0]}")/lib_redis.sh"
# shellcheck source=utils/brand.env
source "${REPO_ROOT}/utils/brand.env"
# shellcheck source=utils/brand.sh
source "${REPO_ROOT}/utils/brand.sh"
SERVICE_NAME="searxng"
SERVICE_USER="searxng"
@ -159,7 +159,7 @@ searxng.instance.env() {
echo " SEARXNG_INTERNAL_HTTP: ${SEARXNG_INTERNAL_HTTP}"
fi
cat <<EOF
environment ${SEARXNG_SRC}/utils/brand.env:
environment:
GIT_URL : ${GIT_URL}
GIT_BRANCH : ${GIT_BRANCH}
SEARXNG_URL : ${SEARXNG_URL}
@ -527,7 +527,6 @@ searxng.install.settings() {
if ! [[ -f "${SEARXNG_SRC}/.git/config" ]]; then
die "Before install settings, first install SearXNG."
exit 42
fi
mkdir -p "$(dirname "${SEARXNG_SETTINGS_PATH}")"
@ -608,8 +607,8 @@ searxng.install.uwsgi.http() {
searxng.install.uwsgi.socket() {
rst_para "Install ${SEARXNG_UWSGI_APP} using socket at: ${SEARXNG_UWSGI_SOCKET}"
mkdir -p "$(dirname ${SEARXNG_UWSGI_SOCKET})"
chown -R "${SERVICE_USER}:${SERVICE_GROUP}" "$(dirname ${SEARXNG_UWSGI_SOCKET})"
mkdir -p "$(dirname "${SEARXNG_UWSGI_SOCKET}")"
chown -R "${SERVICE_USER}:${SERVICE_GROUP}" "$(dirname "${SEARXNG_UWSGI_SOCKET}")"
case $DIST_ID-$DIST_VERS in
fedora-*)

9
utils/searxng_check.py
View file

@ -7,6 +7,9 @@ import os
import sys
import logging
import warnings
from pathlib import Path
repo_root = Path(__file__).resolve().parent.parent
LOG_FORMAT_DEBUG = '%(levelname)-7s %(name)-30.30s: %(message)s'
logging.basicConfig(level=logging.getLevelName('DEBUG'), format=LOG_FORMAT_DEBUG)
@ -26,6 +29,12 @@ if os.path.isfile(OLD_SETTING):
))
warnings.warn(msg, DeprecationWarning)
OLD_BRAND_ENV = repo_root / 'utils' / 'brand.env'
if os.path.isfile(OLD_BRAND_ENV):
msg = ('%s is no longer needed, remove the file' % (OLD_BRAND_ENV))
warnings.warn(msg, DeprecationWarning)
from searx import redisdb, get_setting
if not redisdb.initialize():