searxng/docs/dev/makefile.rst

Makefile & ./manage

All relevant build and development tasks are implemented in the ./manage <manage> script and for CI or IDE integration a small Makefile wrapper is available. If you are not familiar with Makefiles, we recommend to read gnu-make introduction.

build environment

Before looking deeper at the targets, first read about make install.

To install developer requirements follow buildhosts.

The usage is simple, just type make {target-name} to build a target. Calling the help target gives a first overview (make help):

make

bash -c "cd ..; make --no-print-directory help"

./manage

The Makefile targets are implemented for comfort, if you can do without tab-completion and need to have a more granular control, use manage without the Makefile wrappers.

$ ./manage help

Python environment (make install)

activate environment

source ./local/py3/bin/activate

We do no longer need to build up the virtualenv manually. Jump into your git working tree and release a make install to get a virtualenv with a developer install of SearXNG (setup.py). :

$ cd ~/searxng-clone
$ make install
PYENV     [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV     OK
PYENV     [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV  INFO:searx:Initialisation done
BUILDENV  build utils/brand.env

If you release make install multiple times the installation will only rebuild if the sha256 sum of the requirement files fails. With other words: the check fails if you edit the requirements listed in requirements-dev.txt and requirements.txt). :

$ make install
PYENV     OK
PYENV     [virtualenv] requirements.sha256 failed
          [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e...  ./requirements-dev.txt
          [virtualenv] - 471efef6c73558e391c3adb35f4...  ./requirements.txt
...
PYENV     [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV     OK
PYENV     [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV  INFO:searx:Initialisation done
BUILDENV  build utils/brand.env

drop environment

To get rid of the existing environment before re-build use clean target <make clean> first.

If you think, something goes wrong with your ./local environment or you change the setup.py file, you have to call make clean.

make buildenv

Rebuild instance's environment with the modified settings from the settings brand and settings server section of your settings.yml <settings location>.

What is the utils/brand.env needed for and why do you need to rebuild it if necessary?

Short answer: installation and maintenance <searxng maintenance> scripts are running outside of instance's runtime environment and need some values defined in the runtime environment.

All the SearXNG setups are centralized in the settings.yml file. This setup is available as long we are in a installed instance. E.g. the installed instance on the server or the installed developer instance at ./local (the later one is created by a make install <make install> or make run <make run>).

Tasks running outside of an installed instance, especially installation and maintenance <searxng maintenance> tasks running at (pre-) installation time do not have access to the SearXNG setup (from a installed instance). Those tasks need a build environment.

The make buildenv target will update the build environment in:

• utils/brand.env

Tasks running outside of an installed instance, need the following settings from the YAML configuration:

• SEARXNG_URL from server.base_url <settings server> (aka PUBLIC_URL)
• SEARXNG_BIND_ADDRESS from server.bind_address <settings server>
• SEARXNG_PORT from server.port <settings server>

The GIT_URL and GIT_BRANCH in the origin:utils/brand.env file, are read from the git VCS and the branch that is checked out when make buildenv command runs.

I would like to create my own brand, how should I proceed?

Create a remote branch (example.org), checkout the remote branch (on your local developer desktop) and in the searx/settings.yml file in the settings server section set base_url. Run make buildenv and create a commit for your brand.

On your server you clone the branch (example.org) into your HOME folder ~ from where you run the installation <installation> and maintenance <searxng maintenance> task.

To upgrade you brand, rebase on SearXNG's master branch (on your local developer desktop), force push it to your remote branch. Go to your server, do a force pull and run sudo -H ./utils/searxng.sh instance update <update searxng>.

Node.js environment (make node.env)

searx

Node.js version {{version.node}} or higher is required to build the themes. If the requirement is not met, the build chain uses nvm (Node Version Manager) to install latest LTS of Node.js locally: there is no need to install nvm or npm on your system.

To install NVM and Node.js in once you can use make nvm.nodejs.

NVM make nvm.install nvm.status

Use make nvm.status to get the current status of your Node.js and nvm setup.

nvm.install

$ LANG=C make nvm.install
INFO:  install (update) NVM at ./searxng/.nvm
INFO:  clone: https://github.com/nvm-sh/nvm.git
  || Cloning into './searxng/.nvm'...
INFO:  checkout v0.39.4
  || HEAD is now at 8fbf8ab v0.39.4

nvm.status (ubu2004)

Here is the output you will typically get on a Ubuntu 20.04 system which serves only a no longer active Release Node.js v10.19.0.

$ make nvm.status
INFO:  Node.js is installed at /usr/bin/node
INFO:  Node.js is version v10.19.0
WARN:  minimal Node.js version is 16.13.0
INFO:  npm is installed at /usr/bin/npm
INFO:  npm is version 6.14.4
WARN:  NVM is not installed

make nvm.nodejs

Install latest Node.js LTS locally (uses nvm):

$ make nvm.nodejs
INFO:  install (update) NVM at /share/searxng/.nvm
INFO:  clone: https://github.com/nvm-sh/nvm.git
...
Downloading and installing node v16.13.0...
...
INFO:  Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node
INFO:  Node.js is version v16.13.0
INFO:  npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm
INFO:  npm is version 8.1.0
INFO:  NVM is installed at searxng/.nvm

make run

To get up a running a developer instance simply call make run. This enables debug option in searx/settings.yml, starts a ./searx/webapp.py instance and opens the URL in your favorite WEB browser (xdg-open):

$ make run

Changes to theme's HTML templates (jinja2) are instant. Changes to the CSS & JS sources of the theme need to be rebuild. You can do that by running:

$ make themes.all

Alternatively to themes.all you can run live builds of the theme you are modify (make themes):

$ LIVE_THEME=simple make run

make format.python

Format Python source code using Black code style. See $BLACK_OPTIONS and $BLACK_TARGETS in Makefile.

Attention

We stuck at Black 22.12.0, please read comment in PR Bump black from 22.12.0 to 23.1.0

make clean

Drops all intermediate files, all builds, but keep sources untouched. Before calling make clean stop all processes using the make install or make node.env. :

$ make clean
CLEAN     pyenv
PYENV     [virtualenv] drop local/py3
CLEAN     docs -- build/docs dist/docs
CLEAN     themes -- locally installed npm dependencies
...
CLEAN     test stuff
CLEAN     common files

make docs

Target docs builds the documentation:

$ make docs
HTML ./docs --> file://
DOCS      build build/docs/includes
...
The HTML pages are in dist/docs.

make docs.clean docs.live

We describe the usage of the doc.* targets in the How to contribute / Documentation <contrib docs> section. If you want to edit the documentation read our make docs.live section. If you are working in your own brand, adjust your settings brand.

make docs.gh-pages

To deploy on github.io first adjust your settings brand. For any further read deploy on github.io.

make test

Runs a series of tests: make test.pylint, test.pep8, test.unit and test.robot. You can run tests selective, e.g.:

$ make test.pep8 test.unit test.shell
TEST      test.pep8 OK
...
TEST      test.unit OK
...
TEST      test.shell OK

make test.shell

sh lint / if you have changed some bash scripting run this test before commit.

make test.pylint

Pylint is known as one of the best source-code, bug and quality checker for the Python programming language. The pylint profile used in the SearXNG project is found in project's root folder .pylintrc.

make search.checker.{engine name}

To check all engines:

make search.checker

To check a engine with whitespace in the name like google news replace space by underline:

make search.checker.google_news

To see HTTP requests and more use SEARXNG_DEBUG:

make SEARXNG_DEBUG=1 search.checker.google_news

To filter out HTTP redirects (3xx):

make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news                   Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--

make themes.*

further read

• devquickstart

The Makefile targets make theme.* cover common tasks to build the theme(s). The ./manage themes.* command line can be used to convenient run common theme build tasks.

bash -c "cd ..; ./manage themes.help"

To get live builds while modifying CSS & JS use (make run):

$ LIVE_THEME=simple make run

make static.build.*

further read

• devquickstart

The Makefile targets static.build.* cover common tasks to build (a commit of) the static files. The ./manage static.build..* command line can be used to convenient run common build tasks of the static files.

bash -c "cd ..; ./manage static.help"

./manage redis.help

The ./manage redis.* command line can be used to convenient run common Redis tasks (Redis developer notes).

bash -c "cd ..; ./manage redis.help"

./manage go.help

The ./manage go.* command line can be used to convenient run common go (wiki) tasks.

bash -c "cd ..; ./manage go.help"