Commit Graph

231 Commits

Author SHA1 Message Date
Alexandre Flament d26261c5ab
Merge pull request #2619 from return42/drop-ubu1604
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04 11:18:21 +01:00
Alexandre Flament 4c2a8aea39
Merge pull request #2620 from return42/fix-git
[fix] support git versions <v2.22
2021-03-04 11:17:24 +01:00
Markus Heiser 2e58988191 [fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:

    ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx

The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:57:20 +01:00
Markus Heiser 237b1cfdb3 [fix] support git versions <v2.22
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::

    $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
    ...
    git version 2.17.1

The option `--show-current` was added in git v2.22, the alternative to this
option is::

    git rev-parse --abbrev-ref HEAD

Issue when using option `--show-current`::

    [searx-ubu1804] Clone searx sources
    [searx-ubu1804] -------------------
    [searx-ubu1804]
    [searx-ubu1804] error: unknown option `show-current'
    [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
    ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:22:41 +01:00
Markus Heiser daa4b676df [mod] Drop Ubuntu 16.04 (Xenial Xerus) support
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time.  Issues are comming from dependencies (e.g. pip)::

    [searx-ubu1604]   |searx| SyntaxError: invalid syntax
    [searx-ubu1604]   |searx| Traceback (most recent call last):
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
    [searx-ubu1604]   |searx|     from pip._internal.cli.main import main
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
    [searx-ubu1604]   |searx|     sys.stderr.write(f"ERROR: {exc}")
    [searx-ubu1604]   |searx|                                    ^
    [searx-ubu1604]   |searx| SyntaxError: invalid syntax

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 17:52:50 +01:00
Marc Abonce Seguin d6681fd33b remove articles number from engines_languages.json 2021-02-25 23:54:21 -07:00
Alexandre Flament 7c1847d5f2 [mod] add utils/fetch_external_bangs.py
Based on duckduckgo bangs
Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-24 18:48:36 +01:00
Alexandre Flament 5f4a085fc4
Merge pull request #2595 from dalf/update-wikidata-units
[mod] update wikidata_units.json and fetch_wikidata_units.py
2021-02-23 17:22:37 +01:00
Alexandre Flament 46ca32c3cc [mod] update currencies.json and fetch_currencies.py
use a sparql request on wikidata to get the list of currencies.

currencies.json contains the translation for all supported searx languages.

Supersede #993
2021-02-23 16:42:28 +01:00
Alexandre Flament 93d1da4906 [mod] update wikidata_units.json and fetch_wikidata_units.py
The fetch_wikidata_units.py result won't change randomly.
See comments in the script.
2021-02-23 13:10:38 +01:00
Alexandre Flament 90b9d0d6a8 [mod] CI: minor changes
* utils/makefile.python: travis-gh-pages renamed ci-gh-pages
2021-02-02 08:53:57 +01:00
Alexandre Flament 34de715e62
Merge pull request #2500 from dalf/github-action-data
[enh] every Sunday, call utils/fetch_*.py scripts and create a PR automatically
2021-02-01 17:16:58 +01:00
Alexandre Flament 99244440e4
Merge pull request #2514 from return42/fix-gh-pages
[fix] Makefile target gh-pages & flatten history of branch gh.pages
2021-02-01 17:07:08 +01:00
Markus Heiser 38b39ef0ae [fix] re-add 'pip-exe' target - partial revert 9b48ae47
Target pip-exe is a prerequisite of the targets:

  - pyinstall
  - pyuninstall

and was accidentally deleted in commit 9b48ae47.

HINT:
  do not confuse pyinstall with penvinstall

pyinstall & pyuninstall
    Installing into user's HOME using pip from OS,
    therefore the message is needed.

pyenvinstall & pyenvuninstall
    Installing into virtualenv (./local) using pip which is provided by
    prerequisite 'pyenv' in the virtualenv.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-01 16:58:04 +01:00
Alexandre Flament d70c5a621a [mod] more robust make pyenv / make pyenvinstall
"make pyenv" ensures that ./local/py3/bin/python is an executable
2021-02-01 16:58:04 +01:00
Alexandre Flament 806af50738
Merge pull request #2494 from return42/rm-fabfile
[fix] remove Fabric file
2021-02-01 15:09:35 +01:00
Markus Heiser 40d2a116e1 [fix] Makefile target gh-pages & flatten history of branch gh.pages
1. This patch fixes error:

    rm -rf gh-pages/
    make V=1 gh-pages
    make[1]: Leaving directory '/800GBPCIex4/share/searx'
    [ -d "gh-pages/.git" ] || git clone  gh-pages
    fatal: repository 'gh-pages' does not exist

2. The gh-page build has been moved to ./build/gh-pages this also affects
   'travis-gh-pages'

3. The gh-pages commit messages now includes a ref to the repository and commit

4. Since a gh-pages history has only the drawback that the reposetory grows
   fast, this patch also flattens the history:

    cd build/gh-pages/; git log --oneline
    bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
    026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-29 11:41:48 +01:00
Alexandre Flament 6047087aac [mod] utils/fetch_languages.py: write files at the right location 2021-01-24 14:25:27 +01:00
Markus Heiser ea5c992d4f [fix] lxc.sh - SC2034: ubu2010_boilerplate appears unused.
$ make test.sh
  In utils/lxc.sh line 42:
  ubu2010_boilerplate="$ubu1904_boilerplate"
  ^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-24 08:29:13 +01:00
Markus Heiser a8544798ec [fix] remove Fabric file
The fabfile.py has not been updated since 5 years.  I also asked [1] if someone
still use Fabric wtihout any response.  Lets drop outdated Fabric file.

[1] https://github.com/searx/searx/discussions/2400

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-22 17:57:55 +01:00
Alexandre Flament aa887eb375 [mod] checker : replace pycld3 by langdetect
pycld3 requires the native library cld3
langdetect is a pure python package
2021-01-19 21:26:04 +01:00
Alexandre Flament 709dd960f1
Merge pull request #2473 from return42/fix-setup.py
[fix] setup.py requires pyyaml installed
2021-01-16 19:05:36 +01:00
Markus Heiser c4a98862bf [fix] setup.py requires pyyaml installed
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
    ERROR: Command errored out with exit status 1:
     command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
         cwd: /usr/local/searx/searx-src/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
        from searx.version import VERSION_STRING
      File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
        import searx.settings_loader
      File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
        import yaml
    ModuleNotFoundError: No module named 'yaml'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-16 08:58:13 +01:00
Alexandre Flament a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
2021-01-14 20:57:17 +01:00
Alexandre Flament 6e2872f436 [enh] add searx.shared
shared dictionary between the workers (UWSGI or werkzeug)
scheduler: run a task once every x seconds (UWSGI or werkzeug)
2021-01-12 11:47:17 +01:00
Alexandre Flament 8cbc9f2d58 [enh] add checker 2021-01-12 11:47:17 +01:00
Alexandre Flament d5882147e0 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH
2021-01-11 22:59:35 +01:00
Markus Heiser a31cb68668 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH

[1] https://github.com/searx/searx/pull/2408#pullrequestreview-565247895

Suggested-by: @dalf [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:56:15 +01:00
Markus Heiser 9e53470b4c [mod] get rid of searx/brand.py
Removes module searx/brand.py and creates a namespace at searx.brand.

This patch is a first 'proof of concept'.  Later we can decide to remove the
brand namespace entirely or not.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser 9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser dc544498d0 [mod] add target upload-pypi-test https://test.pypi.org/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-10 17:05:37 +01:00
Markus Heiser 89e6ba5af7 [mod] add Ubuntu 20.10 image to the LXC suite
Ubuntu 20.10 image [1] is avialable from https://images.linuxcontainers.org

BTW: replace builhost package 'ttf-dejavu' by more common meta package
'fonts-dejavu' ('ttf-dejavu' is no longer available in 20.10).

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:27:16 +01:00
Markus Heiser c7d92d6475 [fix] remove Ubuntu 19.10 from the LXC suite (EOL)
Official support for Ubuntu 19.10 'Eoan Ermine' ended on July 17, 2020.

The image has been dropped [1] from https://images.linuxcontainers.org

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:23:18 +01:00
Alexandre Flament 5fb9a5c7c1
Merge pull request #2411 from dalf/update-secret-key-check
Update secret key check
2020-12-27 18:04:53 +01:00
Markus Heiser af584f2c40 [perf] pylint: Use multiple processes to speed up Pylint.
Specifying '-j 0' will auto-detect the number of processors available to use.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2421#issuecomment-751444257
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-27 12:04:39 +01:00
Alexandre Flament 568b9465e9 [mod] check secret_key when searx.webapp is imported
Without this commit the module searx checks the secret_key value.

With this commit, make docs, utils/standalone_searx.py,
utils/fetch_firefox_version.py works without SEARX_DEBUG=1

For reference see https://github.com/searx/searx/pull/2386
2020-12-27 10:30:20 +01:00
Alexandre Flament 1ba53e6129
Merge pull request #2420 from return42/fix-sphinx-book-targets
fix makefile `books/{name}.*` targets and add documentation about these targets
2020-12-27 09:47:49 +01:00
Alexandre Flament 1956ab4b50
Merge pull request #2412 from dalf/update-buildenv
[fix] update buildenv
2020-12-27 08:31:23 +01:00
Markus Heiser d175a0fb7b [mod] LXC_ENV_FOLDER moved from ./lxc to ./lxc-env
BTW:

- add target $(LXC_ENV_FOLDER)
- fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:43:18 +01:00
Markus Heiser bc7bd1d80a [fix] utils/lib.sh: various typos in messages and comments
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:38:47 +01:00
Markus Heiser 6316a8d49a [fix] message to install different python versions from apt PPA
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 23:17:21 +01:00
Markus Heiser 809bf85426 [perf] pyenvinstall: reinstall only when requirements are change
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 16:25:09 +01:00
Markus Heiser d85012b49b [perf] virtualenv: install boilerplate only once
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 15:59:57 +01:00
Markus Heiser aace837665 [fix] makefile.sphinx: additions to commit c2a6f145
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.

This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 13:54:07 +01:00
Alexandre Flament 2fc2d17a27
Merge pull request #2418 from return42/fix-pyenv-install
[fix] makefile.python: remove duplicate pyenv-(un)install targets
2020-12-24 10:23:09 +01:00
Markus Heiser af49bd2bd2 [fix] makefile.python: remove duplicate pyenv-(un)install targets
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 09:18:22 +01:00
Markus Heiser 5fc9273e23 [mod] utils/searx.sh install: create branch if not already exists
This patch keeps an eye on:

1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 16:06:06 +01:00
Markus Heiser 0fc3759d74 [fix] use SEARX_SETTINGS_TEMPLATE from .config environment
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 14:10:07 +01:00
Alexandre Flament db5b060455 [fix] update buildenv
CONTACT_URL is unset in Makefile, but searx/brand.py and
utils/brand.env are not updated.

This commit fixes this issue.
2020-12-21 10:55:28 +01:00
Alexandre Flament 72d2bb2289
Merge pull request #2399 from return42/fix-lxc
[fix] determine path to makefile.lxc in a LXC
2020-12-20 09:32:00 +01:00