Commit Graph

64 Commits

Author SHA1 Message Date
Alexandre Flament 3c77412d33 [mod] replace pycodestyle by black
"make test.black" checks for the code style
"make format.python" format the python code
2021-12-27 08:58:59 +01:00
Alexandre Flament 81abc5f4c2 [fix] ./manage: upgrade geckodriver when GECKODRIVER_VERSION change 2021-12-24 09:14:15 +01:00
Markus Heiser 69af219d39 [test.robot] update gecko driver / required by selenium 4.1.0
Update gecko driver to v0.30.0 [1]

[1] https://github.com/mozilla/geckodriver/releases/tag/v0.30.0

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-23 13:09:48 +01:00
Markus Heiser 8069bc83e6 [fix] weblate: separate commit description from commit body
It is a common convention to separate commit description from commit body by a
empty line [1].

[1] https://www.conventionalcommits.org/en/v1.0.0/#summary

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-17 10:48:12 +01:00
Markus Heiser f287787c44 [simple theme] activate - SearXNG JavaScript Style Guide
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::

    make themes.simple
    make themes.simple.test

In the github CI the `themes.all` target enforce a `themes.simple`.

BTW: Remove 'jshint' left overs from 0ee316f3d

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-14 07:50:53 +01:00
Alexandre Flament 59f4c792b4 [mod] simple theme: use sharp instead of convert to create .png from .svg
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-28 20:05:37 +01:00
Markus Heiser c68c53acd8 [brand] initial - folder src/brand for searxng-wordmark
Folder src/brand is intended to place logo, wordmark and more in.  This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder

About the new created src/ folder:

On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 10:14:55 +01:00
Markus Heiser 1e997386e9
Merge pull request #508 from return42/arm64
[mod] Tools to install and maintain golang binaries & packages.
2021-11-18 19:49:01 +00:00
Markus Heiser dc1442a2d1 [mod] Tools to install and maintain NVM versions manager for Node.js
[1] https://github.com/nvm-sh/nvm

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 23:31:11 +01:00
Markus Heiser abd5eee392 [mod] Tools to install and maintain golang [1] binaries & packages.
[1] https://golang.org/doc/devel/release#policy

A simple *helloworld* test with user 'my_user' :

  sudo -H adduser my_user
  ./manage go.golang go1.17.3 my_user
  ./manage go.install github.com/go-training/helloworld@latest my_user
  ./manage go.bash my_user
  $ helloword
  Hello World!!

Don't forget to remove 'my_user':  sudo -H deluser --remove-home my_user

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 18:00:47 +01:00
Markus Heiser 5db24d4818 [fix] make test.shell - ./manage line 80
make test.shell reports an issue that has been added in [PR-500]::

    In ./manage line 80:
      yamllint  : lint YAML files: $YAMLLINT_FILES
                                   ^-------------^
                                   SC2128: Expanding an array without an
                                   index only gives the first element.

[PR-500] https://github.com/searxng/searxng/pull/500

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-15 17:02:44 +01:00
Markus Heiser e7a9cd7a34 [fix] manage help - add description of test.yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 15:35:08 +01:00
Markus Heiser 3a981257de [simple theme] create favicon.png from src/svg/searxng-wordmark.svg
new bash function convert_if_newer() / usage::

    convert_if_newer <origfile> <outfile> [<options>, ...]
    convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64

Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>.  The command line is to convert is::

    convert <origfile> [<options>, ...] <outfile>

PNG 'searx/static/themes/simple/img/favicon.png' has been created by::

  $ make themes.simple
  CONVERT   searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser 68effd5eda [brand] SearXNG - bash env PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Alexandre Flament 9e03823b98 SearXNG: docker 2021-10-12 20:23:57 +02:00
Markus Heiser 60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament 3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Markus Heiser afc5c9870e [fix] commit message of 'make weblate.push.translations'
Fix `\n` issue in the commit message [1] by using multiple `-m` options [2]::

    7d9ffd680 translations     [translations] update messages.pot and messages.po files\nFrom cebc0e39 - 2021-10-04 - Markus Heiser <markus.heiser@darmarIT.de>

[1] https://github.com/searxng/searxng/pull/379#issuecomment-933242702
[2] https://git-scm.com/docs/git-commit#Documentation/git-commit.txt--mltmsggt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-04 11:09:30 +02:00
Alexandre Flament f4dd5b226d
[fix] make weblate.push.translations
Even when there are changes, the function exit without pushing them.
2021-10-03 21:11:32 +02:00
Alexandre Flament 47eb836c65
Merge pull request #375 from dalf/searxng_extra
SearXNG: searx_extra
2021-10-03 19:09:07 +02:00
Alexandre Flament 1bb82a6b54 SearXNG: searxng_extra 2021-10-02 17:30:39 +02:00
Alexandre Flament 253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Alexandre Flament 430babca25 SearXNG: environment variables 2021-10-02 16:54:11 +02:00
Markus Heiser 7a3a1cd115 [mod] manage - use pyenv.activate where it makes sense
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 15:31:09 +02:00
Alexandre Flament e5e322e8ed [mod] utils/lib.sh: add pyenv.activate
* initialize virtualenv (pyenv.install)
* activate virtualenv

close #310
2021-09-17 14:15:35 +02:00
Markus Heiser 3f2c10ef9c [fix] manage help(): commands babel.* has been renamed to weblate.*
In commit 97355672c the functions named babel.* has been renamed to weblate.*
but it was forgotten to change it also in the help().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-12 15:46:00 +02:00
Markus Heiser 82847df300 [fix] add 'categories' to PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES
androp no longer needed (see line 591 in 7b235a1)::

    # pylint: disable=undefined-variable

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 10:29:38 +02:00
Alexandre Flament b57d776edb
Merge pull request #293 from dalf/fix-290
[fix] translation: lock weblate only when there is an actual change of messages.pot
2021-09-06 20:00:19 +02:00
Markus Heiser 7b235a1c36 [mod] one logger per engine
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/98#issuecomment-849013518
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-06 17:47:28 +02:00
Alexandre Flament 2a5f690a71 [fix] translation: lock weblate only when there is an actual change of messages.pot
Close #290
2021-09-06 17:16:51 +02:00
Alexandre Flament b9c73fb697 [mod] move searx/testing.py to the tests directory
move robot tests to tests.robot
manage calls "python -m tests.robot"
2021-09-02 16:01:34 +02:00
Markus Heiser 97355672cd [translations] ./manage rename shell functions to integrate weblate
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':

- babel.setup.translations.worktree --> weblate.translations.worktree

  Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
  'translations' from Weblate's counterpart (weblate) of the SearXNG
   (origin)::

       remote weblate https://weblate.bubu1.eu/git/searxng/searxng/

- babel.weblate.to.translations     --> weblate.to.translations

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.

- babel.translations.to.master      --> weblate.translations.commit

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.  Copy the changes to the master branch, compile translations and
  create a commit in the local branch (master)

- babel.master.to.translations      --> weblate.push.translations

  Push *translation changes* from SearXNG (origin) to Weblate's
  counterpart (weblate).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:06:12 +02:00
Markus Heiser 0dc8d244c5 [translation] ./manage: add comments to new functions, rename job
./mange

  Comment functions:
  - babel.setup.translations.worktree
  - babel.weblate.to.translations
  - babel.translations.to.master
  - babel.master.to.translations
  BTW:
  - changed some cd into pushd
  - removed obsolete `set -x`
  - remove not needed echo from `last_commit_message=$(echo ..`

.github/workflows/translations-update.yml
docs/dev/translation.rst

  Rename job to: "create PR for additons from weblate"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:00:16 +02:00
Alexandre Flament 901ab87717 [translations] web integration
* make babel.translations.to.master: pull weblate updates
* make babel.master.to.translations: push .pot and .po files to weblate
2021-08-07 15:06:06 +02:00
Markus Heiser 900baf2eae [mod] manage - implement babel commands, drop update_translations.sh
In ./manage implement babel.*:

-  extract   : extract messages from source files and generate POT file
-  update    : update existing message catalogs from POT file
-  compile   : compile translation catalogs into binary MO files

Replace searx_extra/update/update_translations.sh by command:

- ci.babel.update

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-02 10:06:27 +02:00
Alexandre Flament a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Alexandre Flament 4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Markus Heiser 61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser 2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser 51c06722c3 [mod] ./manage docker.buildx make variables local
The environment variables ..

- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD

are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser e179dba7e4 [fix] ./manage script, prompt the unknown command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17 17:15:28 +02:00
Markus Heiser 25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00
Alexandre Flament 2530576f24
Merge pull request #157 from searxng/mod_static_build
static build: move generated files to searx/static/themes/*/src/generated
2021-06-22 10:36:19 +02:00
Markus Heiser 1559ed386e [fix] pygments.less - remove pyenv_OK condition
BTW: add pygments.less to the help message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:22:33 +02:00
Alexandre Flament dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Alexandre Flament d20f6a1f19 [mod] make themes.all update pygments*.less
Add a searx/static/themes/*/src/generated empty folder
2021-06-22 08:15:11 +02:00
Markus Heiser f41b7a7204 [fix] py.build - fix small typo in the build message
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 12:52:26 +02:00
Markus Heiser a62e1123e2 [fix] node.clean - ignore npm dependencies when npm is not installed
error pattern::

    $ make clean
    CLEAN     pyenv
    PYENV     [virtualenv] drop local/py3
    CLEAN     docs -- build/docs dist/docs
    CLEAN     locally installed npm dependencies
    ./manage: line 318: npm: command not found
    ERROR: node.clean exit with error (127)
    make: *** [Makefile:90: node.clean] Error 127

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-19 16:58:18 +02:00
Markus Heiser ae677cb64b [enh] ./manage node.env - check build tools first
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS.  These tools can be installed by::

    sudo -H ./utils/searx.sh install buildhost

If one of the tools is not installed, the script node.env stops with a
appropriate message.

BTW: We ignore CentOS-7 as developer & build platform

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-17 09:23:02 +02:00
Alexandre Flament 14ba56fd1a [mod] make node.clean: call the "clean" script from the packages.json
The dependencies and build process are described in the gruntfiles.js and packages.json files.
It makes sense to move the clean up to these files too.
2021-06-16 16:04:58 +02:00