Commit Graph

79 Commits

Author SHA1 Message Date
Markus Heiser 309147d86f [mod] introduce node.env.devtools function 2022-01-24 07:51:31 +01:00
Martin Fischer 6d701d2fea [fix] ensure that test.pyright installs pyright
nodejs.ensure only sets up NVM if there isn't a system-wide
installation of Node that matches our NODE_MINIMUM_VERSION.

The ubuntu image in the CI comes with an up to date node version,
so pyright from .nvm_packages is never installed.

This commit fixes this by introducing a package.json file.
2022-01-23 08:00:39 +01:00
Martin Fischer 96450b17d4 [mod] add test.pyright to test & ci.test targets
Since we currently have many type checking errors,
we for now only test with typeCheckingMode: off
which makes pyright only check files that contain a comment:

    # pyright: basic

to enable basic type checking, or

    # pyright: strict

to enable strict type checking.
2022-01-23 08:00:39 +01:00
Martin Fischer 3cd5ce55e8 [enh] clarify why we run pyright in the pyenv 2022-01-23 08:00:39 +01:00
Markus Heiser 8342773216 [mod] manage test.pyright: a commandline to run pyright tests
This patch implements the command and Makefile target::

    ./manage test.pyright
    make test.pyright

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-23 08:00:39 +01:00
Markus Heiser fea9ad7927 [fix] issue reported by: make test.shell
$ make test.shell
    ./manage line 716:
    build_msg TEST "[reST markup] ${RST_FILES[@]}""
                                  ^-------------^ SC2145: Argument mixes string and array. Use * or separate argument.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-18 12:27:42 +01:00
Markus Heiser 27e3e56800 [mod] CI target test.rst: test reST markup of README.rst
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-17 09:35:42 +01:00
Markus Heiser 977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Markus Heiser c9cfa6181b [fix] debug console (pdb) in a session started by `make run`
Commit c7f27404 moves the `python -m searx.webapp` process to the background.  A
background job can't open a simple python-debugger (pdb) console.

This patch moves all processes to the background except the searx.webapp.

Insert a break point somewhere in the webapp application::

    import pdb
    pdb.set_trace()

And start a debug session by::

    make run

and test you break point.

To test that the entire 'make run' stops in the case of an error in the
themes.live [1] background process try:

    make LIVE_THEME=typo-theme-name run

[1] https://github.com/searxng/searxng/pull/664#discussion_r776419585

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-05 10:09:17 +01:00
Markus Heiser 00d3a7477c
Merge pull request #664 from return42/themes.live
[mod] manage: add themes.live command (rebuild on modification)
2022-01-04 19:44:51 +01:00
Markus Heiser fe8b88c8a4 [mod] script to build & install a redis instance
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.

    $ ./manage redis.help
    redis.:
      devpkg    : install essential packages to compile redis
      build     : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
      install   : create user (searxng-redis) and install systemd service (searxng-redis)
      remove    : delete user (searxng-redis) and remove service (searxng-redis)
      shell     : start bash interpreter from user searxng-redis
      src       : clone redis source code to <path> and checkput 6.2.6
      useradd   : create user (searxng-redis) at /usr/local/searxng-redis
      userdel   : delete user (searxng-redis)
      addgrp    : add <user> to group (searxng-redis)
      rmgrp     : remove <user> from group (searxng-redis)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 14:34:26 +01:00
Markus Heiser c7f2740414 [mod] live build of a theme: LIVE_THEME=simple make run
Environment variable to get live builds while modifying CSS & JS of a theme::

    LIVE_THEME=simple make run

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser 1706b130a4 [fix] remove leftover from 59f4c792
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser edfd069d16 [mod] manage: add themes.live command (rebuild on modification)
To have a live build while editing the HTML client of a theme.  The first
argument of the command is the theme name:

    ./manage themes.live simple

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser c849731eb1 [mod] test.black: show issues (diff) reported by python-black
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 08:59:47 +01:00
Alexandre Flament 3c77412d33 [mod] replace pycodestyle by black
"make test.black" checks for the code style
"make format.python" format the python code
2021-12-27 08:58:59 +01:00
Alexandre Flament 81abc5f4c2 [fix] ./manage: upgrade geckodriver when GECKODRIVER_VERSION change 2021-12-24 09:14:15 +01:00
Markus Heiser 69af219d39 [test.robot] update gecko driver / required by selenium 4.1.0
Update gecko driver to v0.30.0 [1]

[1] https://github.com/mozilla/geckodriver/releases/tag/v0.30.0

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-23 13:09:48 +01:00
Markus Heiser 8069bc83e6 [fix] weblate: separate commit description from commit body
It is a common convention to separate commit description from commit body by a
empty line [1].

[1] https://www.conventionalcommits.org/en/v1.0.0/#summary

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-17 10:48:12 +01:00
Markus Heiser f287787c44 [simple theme] activate - SearXNG JavaScript Style Guide
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::

    make themes.simple
    make themes.simple.test

In the github CI the `themes.all` target enforce a `themes.simple`.

BTW: Remove 'jshint' left overs from 0ee316f3d

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-14 07:50:53 +01:00
Alexandre Flament 59f4c792b4 [mod] simple theme: use sharp instead of convert to create .png from .svg
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-28 20:05:37 +01:00
Markus Heiser c68c53acd8 [brand] initial - folder src/brand for searxng-wordmark
Folder src/brand is intended to place logo, wordmark and more in.  This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder

About the new created src/ folder:

On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 10:14:55 +01:00
Markus Heiser 1e997386e9
Merge pull request #508 from return42/arm64
[mod] Tools to install and maintain golang binaries & packages.
2021-11-18 19:49:01 +00:00
Markus Heiser dc1442a2d1 [mod] Tools to install and maintain NVM versions manager for Node.js
[1] https://github.com/nvm-sh/nvm

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 23:31:11 +01:00
Markus Heiser abd5eee392 [mod] Tools to install and maintain golang [1] binaries & packages.
[1] https://golang.org/doc/devel/release#policy

A simple *helloworld* test with user 'my_user' :

  sudo -H adduser my_user
  ./manage go.golang go1.17.3 my_user
  ./manage go.install github.com/go-training/helloworld@latest my_user
  ./manage go.bash my_user
  $ helloword
  Hello World!!

Don't forget to remove 'my_user':  sudo -H deluser --remove-home my_user

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 18:00:47 +01:00
Markus Heiser 5db24d4818 [fix] make test.shell - ./manage line 80
make test.shell reports an issue that has been added in [PR-500]::

    In ./manage line 80:
      yamllint  : lint YAML files: $YAMLLINT_FILES
                                   ^-------------^
                                   SC2128: Expanding an array without an
                                   index only gives the first element.

[PR-500] https://github.com/searxng/searxng/pull/500

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-15 17:02:44 +01:00
Markus Heiser e7a9cd7a34 [fix] manage help - add description of test.yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 15:35:08 +01:00
Markus Heiser 3a981257de [simple theme] create favicon.png from src/svg/searxng-wordmark.svg
new bash function convert_if_newer() / usage::

    convert_if_newer <origfile> <outfile> [<options>, ...]
    convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64

Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>.  The command line is to convert is::

    convert <origfile> [<options>, ...] <outfile>

PNG 'searx/static/themes/simple/img/favicon.png' has been created by::

  $ make themes.simple
  CONVERT   searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser 68effd5eda [brand] SearXNG - bash env PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Alexandre Flament 9e03823b98 SearXNG: docker 2021-10-12 20:23:57 +02:00
Markus Heiser 60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament 3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Markus Heiser afc5c9870e [fix] commit message of 'make weblate.push.translations'
Fix `\n` issue in the commit message [1] by using multiple `-m` options [2]::

    7d9ffd680 translations     [translations] update messages.pot and messages.po files\nFrom cebc0e39 - 2021-10-04 - Markus Heiser <markus.heiser@darmarIT.de>

[1] https://github.com/searxng/searxng/pull/379#issuecomment-933242702
[2] https://git-scm.com/docs/git-commit#Documentation/git-commit.txt--mltmsggt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-04 11:09:30 +02:00
Alexandre Flament f4dd5b226d
[fix] make weblate.push.translations
Even when there are changes, the function exit without pushing them.
2021-10-03 21:11:32 +02:00
Alexandre Flament 47eb836c65
Merge pull request #375 from dalf/searxng_extra
SearXNG: searx_extra
2021-10-03 19:09:07 +02:00
Alexandre Flament 1bb82a6b54 SearXNG: searxng_extra 2021-10-02 17:30:39 +02:00
Alexandre Flament 253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Alexandre Flament 430babca25 SearXNG: environment variables 2021-10-02 16:54:11 +02:00
Markus Heiser 7a3a1cd115 [mod] manage - use pyenv.activate where it makes sense
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 15:31:09 +02:00
Alexandre Flament e5e322e8ed [mod] utils/lib.sh: add pyenv.activate
* initialize virtualenv (pyenv.install)
* activate virtualenv

close #310
2021-09-17 14:15:35 +02:00
Markus Heiser 3f2c10ef9c [fix] manage help(): commands babel.* has been renamed to weblate.*
In commit 97355672c the functions named babel.* has been renamed to weblate.*
but it was forgotten to change it also in the help().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-12 15:46:00 +02:00
Markus Heiser 82847df300 [fix] add 'categories' to PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES
androp no longer needed (see line 591 in 7b235a1)::

    # pylint: disable=undefined-variable

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 10:29:38 +02:00
Alexandre Flament b57d776edb
Merge pull request #293 from dalf/fix-290
[fix] translation: lock weblate only when there is an actual change of messages.pot
2021-09-06 20:00:19 +02:00
Markus Heiser 7b235a1c36 [mod] one logger per engine
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/98#issuecomment-849013518
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-06 17:47:28 +02:00
Alexandre Flament 2a5f690a71 [fix] translation: lock weblate only when there is an actual change of messages.pot
Close #290
2021-09-06 17:16:51 +02:00
Alexandre Flament b9c73fb697 [mod] move searx/testing.py to the tests directory
move robot tests to tests.robot
manage calls "python -m tests.robot"
2021-09-02 16:01:34 +02:00
Markus Heiser 97355672cd [translations] ./manage rename shell functions to integrate weblate
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':

- babel.setup.translations.worktree --> weblate.translations.worktree

  Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
  'translations' from Weblate's counterpart (weblate) of the SearXNG
   (origin)::

       remote weblate https://weblate.bubu1.eu/git/searxng/searxng/

- babel.weblate.to.translations     --> weblate.to.translations

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.

- babel.translations.to.master      --> weblate.translations.commit

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.  Copy the changes to the master branch, compile translations and
  create a commit in the local branch (master)

- babel.master.to.translations      --> weblate.push.translations

  Push *translation changes* from SearXNG (origin) to Weblate's
  counterpart (weblate).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:06:12 +02:00
Markus Heiser 0dc8d244c5 [translation] ./manage: add comments to new functions, rename job
./mange

  Comment functions:
  - babel.setup.translations.worktree
  - babel.weblate.to.translations
  - babel.translations.to.master
  - babel.master.to.translations
  BTW:
  - changed some cd into pushd
  - removed obsolete `set -x`
  - remove not needed echo from `last_commit_message=$(echo ..`

.github/workflows/translations-update.yml
docs/dev/translation.rst

  Rename job to: "create PR for additons from weblate"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:00:16 +02:00
Alexandre Flament 901ab87717 [translations] web integration
* make babel.translations.to.master: pull weblate updates
* make babel.master.to.translations: push .pot and .po files to weblate
2021-08-07 15:06:06 +02:00
Markus Heiser 900baf2eae [mod] manage - implement babel commands, drop update_translations.sh
In ./manage implement babel.*:

-  extract   : extract messages from source files and generate POT file
-  update    : update existing message catalogs from POT file
-  compile   : compile translation catalogs into binary MO files

Replace searx_extra/update/update_translations.sh by command:

- ci.babel.update

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-02 10:06:27 +02:00