Commit Graph

165 Commits

Author SHA1 Message Date
Alexandre Flament
485a502b88 [mod] add typing and __slots__ 2020-09-22 19:05:05 +02:00
Alexandre Flament
691d12726b [mod] check the engine tokens in searx/webadapter.py instead of searx/search.py 2020-09-22 18:59:51 +02:00
Alexandre Flament
2dbc0de0cd [mod] add searx/webadapter.py
* move searx.search.get_search_query_from_webapp to searx.webadapter
* move searx.query.SearchQuery to searx.search
2020-09-22 18:59:51 +02:00
Marc Abonce Seguin
c86504b47a fix utils/fetch_languages to work with new languages 2020-09-22 11:37:44 +02:00
Markus Heiser
69a61ce333 [fix] remove doc-build for centos-7 from './utils/searx.sh doc'
Since there is no centos-7 documentation 'searx.sh doc' produce an error
message::

    ERROR: centos-: uWSGI not yet implemented

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-09-20 11:11:14 +02:00
Dalf
1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser
623fb87065
[enh] add centos-7 to LXC images (#2118)
## What does this PR do?

To test centos-7 on non centos systems add centos-7 image to the LXC suite.

## Why is this change important?

PR #2112 adds centos-7 support to the install scripts

## How to test this PR locally?

`sudo ./utils/lxc.sh build searx-centos7`
2020-08-08 19:50:11 +02:00
Eliesemoule
1986b5ecac
Utility scripts adapted to run on CentOS 7 (#2112) 2020-08-01 22:12:44 +02:00
Markus Heiser
f9faafa896
[fix] external_bang - UnicodeDecodeError: 'ascii' codec can't decode (#2043)
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:

    LANG=C.UTF-8
    LANGUAGE=C.UTF-8
    LC_ALL=C.UTF-8

Error pattern:

    Traceback (most recent call last):
      File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
        from searx.search import SearchWithPlugins, get_search_query_from_webapp
      File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
        from searx.external_bang import get_bang_url
      File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
        for bang in json.load(json_file)['bang']:
      File "/usr/lib/python3.8/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
      UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)

close: https://github.com/asciimoo/searx/issues/2041

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-07-11 12:17:06 +02:00
Markus Heiser
8d318ee142
Merge branch 'master' into gigablast 2020-06-29 16:09:59 +00:00
Markus Heiser
18805ee352 [fix] markup of auto generated installation instruction (uWSGI part)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-24 15:15:15 +02:00
Markus Heiser
4eab5cf854
Merge branch 'master' into gigablast 2020-06-19 04:51:48 +00:00
Markus Heiser
15924bcb67 [fix] buildprocess of the docs by adding sphinx-doc-prebuilds target
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs

BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:44:22 +02:00
Markus Heiser
51428de0fe [fix] add missing latexmk package to debian's BUILD_PACKAGES
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:35:59 +02:00
Markus Heiser
6ff20cef73 [fix] indentation of filtron's rules (json)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:31:46 +02:00
Markus Heiser
1510382558 utils/fetch_languages.py: print more meaningfull messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-28 15:18:47 +02:00
Markus Heiser
f576cadfba [fix] installation instructions: pip install -e searx (setup.py)
In the past we did not really install searx into the virtualenv.  This is a
problem, since entry-points and other python installation stuff is not trigger.

See discussion: https://github.com/asciimoo/searx/pull/1938#issuecomment-632056508

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-21 15:46:43 +02:00
Markus Heiser
e50ccc99c0 archlinux: add package 'inetutils' to boilerplate
Even it is a core component, some hoster do not have pre installed the
'inetutils' package.  We do need this package for the 'hostname' command.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-04 16:15:23 +02:00
Markus Heiser
0973255aff [fix] fatal: repository '/root/searx' does not exist
The $SERVICE_USER (searx) needs read permission to clone repository.  The
installation script stops if permission is not granted.

- https://github.com/asciimoo/searx/pull/1803#issuecomment-616406528

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-30 17:49:26 +02:00
Markus Heiser
8829796f76 utils/morty.sh option new-key: set new morty key
A command to set new morty key in morty & searx service (settings.yml) and to
restart both services with new key::

  utils/morty.sh option new-key

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-28 16:54:42 +02:00
Markus Heiser
ea3255835a utils/morty.sh: set morty key to avoid service abuse
- https://github.com/asciimoo/searx/issues/1871#issuecomment-592459798

    make install all

generates random MORTY_KEY, install service with that key and sets option in the
searx settingy.yml file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-28 16:21:45 +02:00
Markus Heiser
9504acaca6 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-23 12:49:56 +02:00
Markus Heiser
d21e0b0ef8 [fix] old images need an initial 'apt update'
see https://github.com/asciimoo/searx/pull/1803#issuecomment-616684353

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-22 01:11:25 +02:00
Markus Heiser
697dd6c6d3 [fix] ffox useragent: use Windows NT 10.0 and not Windows NT 10
This is an addittion to PR #1934:

  The .0 change really counts on some engines like Bing which seems to allow
  Windows NT 10.0 but not Windows NT 10.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-22 00:43:42 +02:00
Markus Heiser
99ff16c465 tooling box: added nginx + polished bash scripts and environment
- add installation method for nginx sites, morty and filtron
- clean up PUBLIC_URL environment in and outside of containers
- clean up comand lines
- handle uWSGI quirks on fedora (emperor mode)
- handle Python quirks on debian (there is no 'python' command anymore)
- lib.sh: add die and die_caller functions
- lxc_suite_install_info is now a function
- lint: shellcheck

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-13 11:34:28 +02:00
Markus Heiser
58d5da8b57 nginx: normalize installation (docs and script)s over all distros
This is the revision of the documentation about the varous nginx installation
variants.  It also implements the nginx installation scripts for morty and
filtron.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-11 13:19:11 +02:00
Markus Heiser
67afa6b1fb bash scripts: source code cleaned up (shellcheck)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-10 14:00:52 +02:00
Markus Heiser
44edc92569 [fix] morty: PUBLIC_URL_PATH_MORTY needs trailing slash
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-09 14:28:34 +02:00
Markus Heiser
f693149cde Changes from the installation tests on (all) LXC containers.
Tested and fixed HTTP & uWSGI installation on:

  ubu1604 ubu1804 ubu1910 ubu2004 fedora31 archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-08 18:38:36 +02:00
Markus Heiser
ee39a098ac apache: normalize installation (docs and script)s over all distros
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-07 18:31:51 +02:00
Markus Heiser
eb0d4646d8 docs: rework of chapter "Install with apache"
BTW: normalize installation-nginx.rst
2020-04-06 17:59:06 +02:00
Markus Heiser
c81849cb5a filtron.sh & morty.sh: improve usage message (if used in containers)
BTW: normalize soma variable names

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-05 17:40:37 +02:00
Markus Heiser
c2caf9569c [fix] creation of /.lxcenv in containers
In utils/lib.sh there are two functions with the same name, but different tasks.
Rename one of them from lxc_init_container() into lxc_init_container_env().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-05 15:15:28 +02:00
Markus Heiser
c748047e92 utils/searx.sh: add cmd to (un)set option image_proxy and result_proxy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 17:54:20 +02:00
Markus Heiser
e530e20ae6 misc: fix variuous marginals
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 17:53:16 +02:00
Markus Heiser
3c19f2f1ce utils/lxc.sh install: support installation on selected containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 12:40:14 +02:00
Markus Heiser
e32b08f616 utils/morty.sh: add generic documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 20:26:20 +02:00
Markus Heiser
853be4e11c utils/filtron.sh: add generic documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 20:24:40 +02:00
Markus Heiser
af988dbf71 utils/lxc.sh: support build of selected containers
$ sudo -H ./utils/lxc.sh build  <name>
  $ sudo -H ./utils/lxc.sh show   [images|suite|info|config [<name>]]

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 17:08:42 +02:00
Markus Heiser
f32b4fcedd LXC: add virtualenv to LXC_BASE_PACKAGES
Commit 09a40625 adds virtualenv dependency.  BTW remove deprecated
--no-site-packages.  Not having access to global site-packages is now the
default behavior.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-02 20:27:44 +02:00
Markus Heiser
2441e24288 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-02 18:59:43 +02:00
Markus Heiser
09a4062505 makefile.python: partial revert off 9b48ae47, adds py2 support back
revert "makefile.python: ... drop build support py2" to get back Py2 support.
TPy2 support need virtualenv installed by the OS.

BTW: log environment and python version in travis's install phase

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 17:21:26 +02:00
Markus Heiser
31437d04e8 makefile.python: hardening - start pylint by module
- replace: $(PY_ENV_BIN)/pylint --> $(PY_ENV_BIN)/python -m pylint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 16:07:59 +02:00
Markus Heiser
9b48ae47ec makefile.python: hardening - drop virtualenv, drop build support py2
From py3.3 on a subset of virtualenv is built-in, so you can run '-m venv' ot of
the box.

- replace: $(PY_ENV_BIN)/pip --> $(PY_ENV_BIN)/python -m pip
- remove obsolete virtualenv-exe target and adjust VTENV_OPTS
- remove obsolete msg-pip-exe target
- print list of py launchers available from $(PY_ENV_BIN) to the log
- fix hard coded ./local

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 14:16:49 +02:00
Markus Heiser
3bbcfe6ab3 travis: make travis build more verbose
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 11:57:48 +02:00
Markus Heiser
6007bd7fc8 makefile.python: fix (old) typo
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 10:59:51 +02:00
Markus Heiser
f27f6c6c5d docs: use make and uitls/searx.sh in containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-31 18:25:40 +02:00
Markus Heiser
dd53c45a2c docs: add utils/lxc.sh docs, normalize filtron, morty & searx docs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-30 18:47:01 +02:00
Markus Heiser
3e4d022d04 [fix] make test.sh: fix various shellcheck error messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-30 11:03:46 +02:00
Markus Heiser
5e2b161379 Makefile: marginal fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 16:46:07 +02:00