Commit Graph

5365 Commits

Author SHA1 Message Date
Markus Heiser 900baf2eae [mod] manage - implement babel commands, drop update_translations.sh
In ./manage implement babel.*:

-  extract   : extract messages from source files and generate POT file
-  update    : update existing message catalogs from POT file
-  compile   : compile translation catalogs into binary MO files

Replace searx_extra/update/update_translations.sh by command:

- ci.babel.update

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-02 10:06:27 +02:00
Alexandre Flament 7fa7177759 Update translations (pot, po) 2021-08-02 09:44:13 +02:00
Alexandre Flament 49aa7822d9 github workflow: babel jobs to update messages.pot 2021-08-02 09:44:13 +02:00
Alexandre Flament 32263ea637 remove .tx directory (transifex) 2021-08-02 09:44:13 +02:00
Alexandre Flament d1fb3c2592 babel: update searx_extra/update/update_translations.sh
the script expects searx/translations/messages.pot to be commited

it calls "pybabel extract", and if there is a meaningful change,
it calls "pybabel update"

exit code 0 when there is a change in messages.pot
2021-08-02 09:44:13 +02:00
Alexandre Flament efc81ca3dc babel.cfg: ignore node_modules directories
The NPM packages contains python scripts: they should be included by "pybabel extract"
2021-08-02 09:44:13 +02:00
Markus Heiser e0ca749ff5
Merge pull request #242 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-08-02 07:40:19 +00:00
Alexandre Flament b6711ba0ad
Merge pull request #241 from searxng/update_data_update_languages.py
Update searx.data - update_languages.py
2021-08-02 09:01:57 +02:00
Markus Heiser 274edd523c [fix] get common used languages back (min_engines_per_lang = 13)
In commit 4d3f2f48d common used languages has been droped.  By reducing the
number of `min_engines_per_lang` from 15 to 13 we get theses languages back.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-01 13:04:24 +02:00
Markus Heiser 0fc1dcbfa5
Merge pull request #243 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-08-01 09:42:01 +00:00
Markus Heiser 33ba700c3b
Merge pull request #240 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2021-08-01 09:22:38 +00:00
dalf 7fae5647d3 Update searx.data - update_wikidata_units.py 2021-08-01 06:24:23 +00:00
dalf 2963c83daa Update searx.data - update_currencies.py 2021-08-01 06:24:13 +00:00
dalf 4d3f2f48d2 Update searx.data - update_languages.py 2021-08-01 06:24:12 +00:00
dalf f5389fd722 Update searx.data - update_firefox_version.py 2021-08-01 06:24:01 +00:00
Markus Heiser d2d6234f92
Merge pull request #239 from return42/fix-doc-git
[fix] documentation - remove brand.git_url & brand.git_branch
2021-08-01 05:29:41 +00:00
Markus Heiser 5e3525a298
Merge pull request #236 from dalf/manage_without_brand
manage script: remove unused reference to utils/brand.env and .config.sh
2021-08-01 05:29:24 +00:00
Alexandre Flament a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Markus Heiser aee613d256 [fix] documentation - remove brand.git_url & brand.git_branch
In commit 4b43775c9 the brand.git_url & brand.git_branch has been removed, with
this patch these settings are removed from the documentation.

docs/admin/engines/settings.rst
  - Remove brand.git_url & brand.git_branch

docs/dev/makefile.rst
  - Remove brand.git_url & brand.git_branch

modified   docs/dev/lxcdev.rst
  - Remove brand.git_url & brand.git_branch
  - fix searXNG to SearXNG

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-01 06:57:14 +02:00
Alexandre Flament 4ca318aad2
dependabot.yml: check npm packages 2021-07-31 09:56:57 +02:00
Alexandre Flament 38ee88b390
Merge pull request #229 from dalf/version-from-git
version based on git
2021-07-30 18:57:00 +02:00
Markus Heiser 3b97657f3b
Merge pull request #233 from searxng/dependabot/pip/master/sphinx-4.1.2
Bump sphinx from 4.1.1 to 4.1.2
2021-07-30 16:27:44 +00:00
Markus Heiser a2fc031e7a
Merge pull request #232 from searxng/dependabot/pip/master/pylint-2.9.6
Bump pylint from 2.9.5 to 2.9.6
2021-07-30 16:22:01 +00:00
Markus Heiser 5b3eeb6e54 [mod] searx.sh - improve some messages prompted while installation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser 85e0908e4f [fix] searx.sh - get_installed_version_variables()
Fix the quotation marks in the command line to run

    python -m searx.version

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser 64d64535a2 [fix] buildenv: first unset environment / then import searx package
The first import of names from the searx package implies loading the
settings.yml.  Before this is done, the enviroment variables must be unset to
not overwrite the values from the settings.yml

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Alexandre Flament 4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
dependabot[bot] d95720b0dc
Bump sphinx from 4.1.1 to 4.1.2
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.1.1 to 4.1.2.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.1.1...v4.1.2)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-30 07:04:19 +00:00
dependabot[bot] 146b4f4b42
Bump pylint from 2.9.5 to 2.9.6
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.5 to 2.9.6.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.5...v2.9.6)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-30 07:04:12 +00:00
Alexandre Flament feef577f99
Merge pull request #230 from dalf/remove-zh-hant-tw
Remove zh_Hant_TW translation
2021-07-28 08:34:09 +02:00
Alexandre Flament 0b2dca088f Remove zh_Hant_TW translation
See https://github.com/searxng/searxng/issues/195#issuecomment-887518808
2021-07-27 18:54:25 +02:00
Alexandre Flament ed0db4d61a
Merge pull request #226 from return42/hardening
[mod] hardening SearXNG instances by default (formats)
2021-07-27 11:22:18 +02:00
Alexandre Flament 24fcd7475a Revert "[brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42"
This reverts commit e946c3462c.
2021-07-27 08:40:53 +02:00
Markus Heiser beff764d74
Merge pull request #216 from return42/fix-searx.sh
normalize environment of installation tasks (shell) with YAML settings
2021-07-27 05:32:57 +00:00
Markus Heiser 80f5241387
Merge pull request #225 from return42/docs-nosql
[docs] document NoSLQ engines MongoDB and redis
2021-07-27 05:32:26 +00:00
Markus Heiser c5cfc24467
Merge pull request #227 from return42/fix-seznam
[fix] Seznam engine - some XPath selectors has been changed
2021-07-27 05:20:39 +00:00
Samuel Dudik 7a7ef9cea6 [fix] Seznam engine - some XPath selectors has been changed
Merged from https://github.com/dudik/searx/commit/5a4207759

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-27 07:13:41 +02:00
Markus Heiser 7b40c736e6 [mod] hardening SearXNG instances by default (formats)
Deny formats has been implemented in 6ed4616d.

To harden SearXNG instances by default, other formats than HTML should be
denied.  Most of JSON, RSS and CSV requests are bots [1]::

    Bots are the only users of this feature on a public instance, and they abuse
    it too much that the engines rate limit pretty quickly the IP address of the
    instance.

[1] https://github.com/searxng/searxng/issues/95

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Markus Heiser e02b5469f0 [mod] use tests/unit/settings/test_settings.yml in unit tests
In unit tests settings from

    searx/settings.yml

and the user settings from:

    unit/settings/test_settings.yml

are used.  In the latter, settings can be activated that are needed in the unit
test but should not activated by default in production.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Noémi Ványi 5e648e77ab [docs] document NoSLQ engines MongoDB and redis
Slightly modified merge of commit [3b192e638]

[3b192e638] https://github.com/searx/searx/commit/3b192e638

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 11:39:11 +02:00
Alexandre Flament 48fe83b901
Merge pull request #221 from dalf/fix-peertube_fetch_supported_languages
[fix] peertube: update _fetch_supported_languages
2021-07-25 10:30:53 +02:00
Alexandre Flament cb4220657c
Merge pull request #223 from return42/fix-qwant
[fix] qwant engine - hardening the engine
2021-07-25 10:30:41 +02:00
Markus Heiser e946c3462c [brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42
Do not merge this patch in master branch of SearXNG!  This branch exists only
for testing the feature branch fix-searx.sh @return42.

This patch changes the buildenv to::

    GIT_URL='https://github.com/return42/searxng'
    GIT_BRANCH='fix-searx.sh'
    SEARX_PORT='7777'
    SEARX_BIND_ADDRESS='127.0.0.12'

To test installation procedure, clone feature branch (fix-searx.sh)::

    $ cd ~/Downloads
    $ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
    $ cd searxng

    $ ./utils/searx.sh install all
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser 7196a9b5dd [fix] ./utils/filtron.sh - FILTRON_TARGET from YAML settings
The filtron target is the SearXNG installation and the default of FILTRON_TARGET
is taken from the YAML configuration ('server.port' & 'server.bind_address').

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser 61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser 57b8f340a6 [mod] utils/searx.sh - add command 'inspect settings'
Inspect YAML setting <key> from SearXNG instance (${SEARX_SRC})::

  utils/searx.sh inspect settings server.base_url

utils/lib_install.sh
  should not log on stdout which is used for the output of function
  prompt_installation_setting().  Turned build_msg (stdout) into
  info_msg (stderr).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:24:22 +02:00
Markus Heiser fe67f1478f [fix] qwant engine - prevent API locale exception on lang 'all'
Has been reported in [1], error message::

    Error
        Error: searx.exceptions.SearxEngineAPIException
        Percentage: 0
        Parameters: ('API error::locale must be a string,locale must be one of
        the following values: en_gb, en_ie, en_us, en_ca, en_in, en_my, en_au,
        en_nz, cy_gb, gd_gb, de_de, de_ch, de_at, fr_fr, br_fr, fr_be, fr_ch,
        fr_ca, fr_ad, fc_ca, ec_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx,
        es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_br,
        pt_pt, pt_ad, nl_be, nl_nl, pl_pl, zh_hk, zh_cn, fi_fi, bg_bg, et_ee,
        hu_hu, da_dk, nb_no, sv_se, ko_kr, th_th, cs_cz, ro_ro, el_gr',)
        File name: searx/engines/qwant.py:114
        Function: response
        Code: raise SearxEngineAPIException('API error::' + msg)

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:27 +02:00
Markus Heiser ca57c7421b [fix] qwant engine - prevent exception on date/time value is None
Has been reported in [1], error messages::

  Error
       Error: ValueError
       Percentage: 0
       Parameters: ()
       File name: searx/engines/qwant.py:159
       Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'], None)

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('an integer is required (got type NoneType)',)
        File name: searx/engines/qwant.py:196
        Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'])

Fix timedelta from seconds to milliseconds [1], error message::

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('unsupported type for timedelta seconds component: NoneType',)
        File name: searx/engines/qwant.py:195
        Function: response
        Code: length = timedelta(seconds=item['duration'])

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:14 +02:00
Alexandre Flament b0a12924a0 [fix] peertube: update _fetch_supported_languages
update the regex to match the changes in peertube source code
fix "make data.languages"
2021-07-23 12:03:16 +02:00