forked from zaclys/searxng
439 lines
13 KiB
ReStructuredText
439 lines
13 KiB
ReStructuredText
.. _lxcdev:
|
|
|
|
==============================
|
|
Developing in Linux Containers
|
|
==============================
|
|
|
|
.. _LXC: https://linuxcontainers.org/lxc/introduction/
|
|
|
|
In this article we will show, how you can make use of Linux Containers (LXC_) in
|
|
*distributed and heterogeneous development cycles* (TL;DR; jump to the
|
|
:ref:`lxcdev summary`).
|
|
|
|
.. sidebar:: Audience
|
|
|
|
This blog post is written for experienced admins and developers. Readers
|
|
should have a serious meaning about the terms: *distributed*, *merge* and
|
|
*linux container*.
|
|
|
|
**hint**
|
|
|
|
If you have issues with the internet connectivity of your containers read
|
|
section :ref:`internet connectivity docker`.
|
|
|
|
|
|
.. contents:: Contents
|
|
:depth: 2
|
|
:local:
|
|
:backlinks: entry
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
Most often in our development cycle, we edit the sources and run some test
|
|
and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This
|
|
cycle is simple and perfect but might fail in some aspects we should not
|
|
overlook.
|
|
|
|
**The environment in which we run all our development processes matters!**
|
|
|
|
The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but
|
|
these tools do not have access to all prerequisites. For example, there may
|
|
have dependencies on packages that are installed on developer's desktop, but
|
|
usually are not preinstalled on a server or client system. Another example is;
|
|
settings have been made to the software on developer's desktop that would never
|
|
be set on a *production* system.
|
|
|
|
**Linux Containers are isolate environments**, we use them to not mix up all
|
|
the prerequisites from various projects on developer's desktop.
|
|
|
|
The scripts from :ref:`searx_utils` can divide in those to install and maintain
|
|
software
|
|
|
|
- :ref:`searxng.sh`
|
|
|
|
and the script
|
|
|
|
- :ref:`lxc.sh`
|
|
|
|
with we can scale our installation, maintenance or even development tasks over a
|
|
stack of isolated containers / what we call the:
|
|
|
|
- :ref:`searxng lxc suite`
|
|
|
|
.. _lxcdev install searxng:
|
|
|
|
Gentlemen, start your engines!
|
|
==============================
|
|
|
|
.. _LXD: https://linuxcontainers.org/lxd/introduction/
|
|
.. _archlinux: https://www.archlinux.org/
|
|
|
|
Before you can start with containers, you need to install and initiate LXD_
|
|
once:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ snap install lxd
|
|
$ lxd init --auto
|
|
|
|
And you need to clone from origin or if you have your own fork, clone from your
|
|
fork:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ cd ~/Downloads
|
|
$ git clone https://github.com/searxng/searxng.git searxng
|
|
$ cd searxng
|
|
|
|
.. sidebar:: The ``searxng-archlinux`` container
|
|
|
|
is the base of all our exercises here.
|
|
|
|
The :ref:`lxc-searxng.env` consists of several images, see ``export
|
|
LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`.
|
|
For this blog post we exercise on a archlinux_ image. The container of this
|
|
image is named ``searxng-archlinux``.
|
|
|
|
Lets build the container, but be sure that this container does not already
|
|
exists, so first lets remove possible old one:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh remove searxng-archlinux
|
|
$ sudo -H ./utils/lxc.sh build searxng-archlinux
|
|
|
|
|
|
.. sidebar:: further read
|
|
|
|
- :ref:`lxc.sh install suite`
|
|
- :ref:`installation nginx`
|
|
|
|
To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP
|
|
proxy :ref:`installation nginx` into the archlinux container run:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
|
|
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
|
|
$ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL
|
|
...
|
|
[searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng
|
|
|
|
.. sidebar:: Fully functional SearXNG suite
|
|
|
|
From here on you have a fully functional SearXNG suite (including a
|
|
:ref:`redis db`).
|
|
|
|
In such a SearXNG suite admins can maintain and access the debug log of the
|
|
services quite easy.
|
|
|
|
In the example above the SearXNG instance in the container is wrapped to
|
|
``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the
|
|
IP of your ``searxng-archlinux`` container is different to this example. To
|
|
test the instance in the conatiner from outside of the container, in your WEB
|
|
browser on your desktop just open the URL reported in your installation
|
|
|
|
.. _working in containers:
|
|
|
|
In containers, work as usual
|
|
============================
|
|
|
|
Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers
|
|
open the root-bash in the container is done by the ``./utils/lxc.sh cmd
|
|
searxng-archlinux`` command:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash
|
|
INFO: [searxng-archlinux] bash
|
|
[root@searxng-archlinux SearXNG]$
|
|
|
|
The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user
|
|
in the container (GUEST). To debug the running SearXNG instance use:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ ./utils/searxng.sh instance inspect
|
|
...
|
|
use [CTRL-C] to stop monitoring the log
|
|
...
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect
|
|
...
|
|
use [CTRL-C] to stop monitoring the log
|
|
...
|
|
|
|
|
|
Back in the browser on your desktop open the service http://n.n.n.140/searxng
|
|
and run your application tests while the debug log is shown in the terminal from
|
|
above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug
|
|
option"* in SearXNG's settings file and restarts the SearXNG uwsgi application.
|
|
|
|
Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>`
|
|
runs under dedicated system user account with the same name (compare
|
|
:ref:`create searxng user`). To get a login shell from these accounts, simply
|
|
call:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ ./utils/searxng.sh instance cmd bash -l
|
|
(searx-pyenv) [searxng@searxng-archlinux ~]$ pwd
|
|
/usr/local/searxng
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l
|
|
INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l
|
|
(searx-pyenv) [searxng@searxng-archlinux ~]$ pwd
|
|
/usr/local/searxng
|
|
|
|
The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system
|
|
user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv*
|
|
``(searxng-pyenv)`` environment is activated.
|
|
|
|
|
|
Wrap production into developer suite
|
|
====================================
|
|
|
|
In this section we will see how to change the *"Fully functional SearXNG suite"*
|
|
from a LXC container (which is quite ready for production) into a developer
|
|
suite. For this, we have to keep an eye on the :ref:`installation basic`:
|
|
|
|
- SearXNG setup in: ``/etc/searxng/settings.yml``
|
|
- SearXNG user's home: ``/usr/local/searxng``
|
|
- virtualenv in: ``/usr/local/searxng/searxng-pyenv``
|
|
- SearXNG software in: ``/usr/local/searxng/searxng-src``
|
|
|
|
With the use of the :ref:`searxng.sh` the SearXNG service was installed as
|
|
:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use
|
|
``systemctl`` (compare :ref:`uWSGI maintenance`).
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: uwsgi@searxng
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng
|
|
|
|
With the command above, we stopped the SearXNG uWSGI-App in the archlinux
|
|
container.
|
|
|
|
The uWSGI-App for the archlinux dsitros is configured in
|
|
:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at
|
|
least you should attend the settings of ``uid``, ``chdir``, ``env`` and
|
|
``http``::
|
|
|
|
env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
|
|
http = 127.0.0.1:8888
|
|
|
|
chdir = /usr/local/searxng/searxng-src/searx
|
|
virtualenv = /usr/local/searxng/searxng-pyenv
|
|
pythonpath = /usr/local/searxng/searxng-src
|
|
|
|
If you have read the :ref:`Good to know` you remember, that each container
|
|
shares the root folder of the repository and the command ``utils/lxc.sh cmd``
|
|
handles relative path names **transparent**.
|
|
|
|
To wrap the SearXNG installation in the container into a developer one, we
|
|
simple have to create a smylink to the **transparent** reposetory from the
|
|
desktop. Now lets replace the repository at ``searxng-src`` in the container
|
|
with the working tree from outside of the container:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old
|
|
$ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
|
mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux \
|
|
ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
|
|
|
|
Now we can develop as usual in the working tree of our desktop system. Every
|
|
time the software was changed, you have to restart the SearXNG service (in the
|
|
container):
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: uwsgi@searxng
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
|
|
|
|
|
|
Remember: :ref:`working in containers` .. here are just some examples from my
|
|
daily usage:
|
|
|
|
To *inspect* the SearXNG instance (already described above):
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ ./utils/searx.sh inspect service
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service
|
|
|
|
Run :ref:`makefile`, e.g. to test inside the container:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ make test
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test
|
|
|
|
|
|
|
|
To install all prerequisites needed for a :ref:`buildhosts`:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ ./utils/searxng.sh install buildhost
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost
|
|
|
|
|
|
To build the docs on a buildhost :ref:`buildhosts`:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST)
|
|
|
|
.. code:: bash
|
|
|
|
$ make docs.html
|
|
|
|
.. group-tab:: desktop (HOST)
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html
|
|
|
|
|
|
.. _lxcdev summary:
|
|
|
|
Summary
|
|
=======
|
|
|
|
We build up a fully functional SearXNG suite in a archlinux container:
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh build searxng-archlinux
|
|
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
|
|
...
|
|
Developer install? (wraps source from HOST into the running instance) [YES/no]
|
|
|
|
To wrap the suite into a developer one answer ``YES`` (or press Enter).
|
|
|
|
.. code:: text
|
|
|
|
link SearXNG's sources to: /share/SearXNG
|
|
=========================================
|
|
|
|
mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup"
|
|
ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src"
|
|
ls -ld /usr/local/searxng/searxng-src
|
|
|searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG
|
|
|
|
On code modification the instance has to be restarted (see :ref:`uWSGI
|
|
maintenance`):
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
|
|
|
|
To access HTTP from the desktop we installed nginx for the services inside the
|
|
container:
|
|
|
|
.. code:: bash
|
|
|
|
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
|
|
|
|
To get information about the SearxNG suite in the archlinux container we can
|
|
use:
|
|
|
|
.. code:: text
|
|
|
|
$ sudo -H ./utils/lxc.sh show suite searxng-archlinux
|
|
[searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/
|
|
[searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744]
|
|
[searxng-archlinux] uWSGI:
|
|
[searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket
|
|
[searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env:
|
|
[searxng-archlinux] GIT_URL : https://github.com/searxng/searxng
|
|
[searxng-archlinux] GIT_BRANCH : master
|
|
[searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng
|
|
[searxng-archlinux] SEARXNG_PORT : 8888
|
|
[searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1
|
|
|