Commit Graph

4666 Commits

Author SHA1 Message Date
Noémi Ványi 5e648e77ab [docs] document NoSLQ engines MongoDB and redis
Slightly modified merge of commit [3b192e638]

[3b192e638] https://github.com/searx/searx/commit/3b192e638

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 11:39:11 +02:00
Alexandre Flament 48fe83b901
Merge pull request #221 from dalf/fix-peertube_fetch_supported_languages
[fix] peertube: update _fetch_supported_languages
2021-07-25 10:30:53 +02:00
Alexandre Flament cb4220657c
Merge pull request #223 from return42/fix-qwant
[fix] qwant engine - hardening the engine
2021-07-25 10:30:41 +02:00
Markus Heiser fe67f1478f [fix] qwant engine - prevent API locale exception on lang 'all'
Has been reported in [1], error message::

    Error
        Error: searx.exceptions.SearxEngineAPIException
        Percentage: 0
        Parameters: ('API error::locale must be a string,locale must be one of
        the following values: en_gb, en_ie, en_us, en_ca, en_in, en_my, en_au,
        en_nz, cy_gb, gd_gb, de_de, de_ch, de_at, fr_fr, br_fr, fr_be, fr_ch,
        fr_ca, fr_ad, fc_ca, ec_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx,
        es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_br,
        pt_pt, pt_ad, nl_be, nl_nl, pl_pl, zh_hk, zh_cn, fi_fi, bg_bg, et_ee,
        hu_hu, da_dk, nb_no, sv_se, ko_kr, th_th, cs_cz, ro_ro, el_gr',)
        File name: searx/engines/qwant.py:114
        Function: response
        Code: raise SearxEngineAPIException('API error::' + msg)

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:27 +02:00
Markus Heiser ca57c7421b [fix] qwant engine - prevent exception on date/time value is None
Has been reported in [1], error messages::

  Error
       Error: ValueError
       Percentage: 0
       Parameters: ()
       File name: searx/engines/qwant.py:159
       Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'], None)

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('an integer is required (got type NoneType)',)
        File name: searx/engines/qwant.py:196
        Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'])

Fix timedelta from seconds to milliseconds [1], error message::

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('unsupported type for timedelta seconds component: NoneType',)
        File name: searx/engines/qwant.py:195
        Function: response
        Code: length = timedelta(seconds=item['duration'])

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:14 +02:00
Alexandre Flament b0a12924a0 [fix] peertube: update _fetch_supported_languages
update the regex to match the changes in peertube source code
fix "make data.languages"
2021-07-23 12:03:16 +02:00
Markus Heiser ea0f4346b1
Merge pull request #220 from dalf/fix-category-order
[fix] /preferences: remove the empty engine category "social medias"(it is "social media")
2021-07-23 08:52:39 +00:00
Alexandre Flament b1100e2b7d [fix] /preferences: remove the empty engine category "social medias" (it is "social media") 2021-07-23 10:43:16 +02:00
Markus Heiser 6869119f5e
Merge pull request #218 from searxng/dependabot/pip/master/twine-3.4.2
Bump twine from 3.4.1 to 3.4.2
2021-07-23 07:14:19 +00:00
Markus Heiser c25df4bb17
Merge pull request #219 from searxng/dependabot/pip/master/pylint-2.9.5
Bump pylint from 2.9.3 to 2.9.5
2021-07-23 07:12:13 +00:00
dependabot[bot] 6f084db41b
Bump pylint from 2.9.3 to 2.9.5
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.3 to 2.9.5.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.3...v2.9.5)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-23 07:04:41 +00:00
dependabot[bot] e289ae277b
Bump twine from 3.4.1 to 3.4.2
Bumps [twine](https://github.com/pypa/twine) from 3.4.1 to 3.4.2.
- [Release notes](https://github.com/pypa/twine/releases)
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pypa/twine/compare/3.4.1...3.4.2)

---
updated-dependencies:
- dependency-name: twine
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-23 07:04:37 +00:00
Alexandre Flament 4f0d232a3d
Merge pull request #213 from return42/drop-brand
[mod] drop searx.brand namespace
2021-07-21 14:27:49 +02:00
Markus Heiser 19abaf272d [mod] make pyenv.OK() less verbose when pyenv is OK
Prompt the verbose messages ...

  PYENV     [check] import yaml --> OK
  PYENV     OK

only when environment $VERBOSE is active.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 51c06722c3 [mod] ./manage docker.buildx make variables local
The environment variables ..

- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD

are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 6fbf5180bf [mod] drop obsolete searx.brand namespace
The usages of the searx.brand namespace has been removed, the searx.brand
namespace is now longer needed.

The searx.brand namespace was an interim solution which has been added in commit
9e53470b4, see commit message there ...

    This patch is a first 'proof of concept'.  Later we can decide to remove the
    brand namespace entirely or not.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 3f638ed196 [mod] drop usage of the searx.brand namespace (templates & /config)
In the templates and the /config (JSON) the usage of the 'brand.*' name
space is replaced by 'searx.get_setting' function.

- new_issue_url          --> get_setting('brand.new_issue_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')

The macro 'new_issue' from searx/templates/*/messages/no_results.html
is now imported with context::

    {% from '__common__/new_issue.html' import new_issue with context %}

To get *public instances URL* from context's 'get_setting()' function::

    get_setting('brand.public_instances','')

Macro's prototype does no longer need the 'new_issue_url' argument and has been
changed to::

    macro new_issue(engine_name, engine_reliability)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser c9220de690 [mod] drop unused setting option brand:twitter_url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 2a2b6c4d57 [fix] add missing brand options to the settings SCHEMA
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Alexandre Flament 17e028e579
Merge pull request #106 from searxng/dependabot/pip/master/flask-2.0.1
Bump flask from 1.1.2 to 2.0.1
2021-07-19 10:41:10 +02:00
Markus Heiser 3e50e8de3e [mod] drop usage of the searx.brand namespace (python procs)
Added function searx.get_setting(name, default=_unset):
  Returns the value to which ``name`` point.  If there is no such name in the
  settings and the ``default`` is unset, a KeyError exception is raised.

In all the python processes ..

- make docs
- make buildenv
- make install (setup.py)

the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.

- brand.SEARX_URL        --> get_setting('server.base_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.GIT_BRANCH'      --> get_setting('server.base_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')
- brand.WIKI_URL         --> get_setting('brand.wiki_url')
- brand.TWITTER_URL      --> get_setting('brand.twitter_url', '')

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18 15:55:42 +02:00
Markus Heiser 197fa188c0
Merge pull request #212 from return42/fix-manage
[fix] ./manage script, prompt the unknown command
2021-07-17 15:18:47 +00:00
Markus Heiser e179dba7e4 [fix] ./manage script, prompt the unknown command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17 17:15:28 +02:00
dependabot[bot] 594e549839
Bump flask from 1.1.2 to 2.0.1
Bumps [flask](https://github.com/pallets/flask) from 1.1.2 to 2.0.1.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/1.1.2...2.0.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-17 12:57:51 +00:00
Alexandre Flament 555bea6f24
Merge pull request #89 from searxng/dependabot/pip/master/jinja2-3.0.1
Bump jinja2 from 2.11.3 to 3.0.1
2021-07-17 14:57:05 +02:00
Alexandre Flament f523fd3ea7
Merge pull request #211 from MarcAbonce/onions_v3_fix_searxng
Update onion engines to v3
2021-07-16 17:25:37 +02:00
Alexandre Flament 5f4d05db33
Merge pull request #210 from searxng/dependabot/pip/master/sphinx-4.1.1
Bump sphinx from 4.0.2 to 4.1.1
2021-07-16 16:16:05 +02:00
Alexandre Flament d47b8e36cf
Merge pull request #207 from return42/mongodb
[enh] add mongodb offline engine
2021-07-16 16:15:01 +02:00
dependabot[bot] 048100be3e
Bump jinja2 from 2.11.3 to 3.0.1
Bumps [jinja2](https://github.com/pallets/jinja) from 2.11.3 to 3.0.1.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/2.11.3...3.0.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16 13:59:22 +00:00
Alexandre Flament 401f7d38f6
Merge pull request #209 from searxng/dependabot/pip/master/python-dateutil-2.8.2
Bump python-dateutil from 2.8.1 to 2.8.2
2021-07-16 15:58:33 +02:00
Alexandre Flament 8237cb557e
Merge pull request #206 from return42/improve-qwant
[mod] improve video results of the qwant engine
2021-07-16 15:33:56 +02:00
Alexandre Flament 0d65a81b1c [mod] qwant engine: fix typos / minor change
minor modification of commit 628b5703f3
(no functionnal change)
2021-07-16 15:32:12 +02:00
Marc Abonce Seguin 1b05ea6a6b update onion engines to v3
remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-07-16 01:36:34 -07:00
dependabot[bot] 7d3320974d
Bump python-dateutil from 2.8.1 to 2.8.2
Bumps [python-dateutil](https://github.com/dateutil/dateutil) from 2.8.1 to 2.8.2.
- [Release notes](https://github.com/dateutil/dateutil/releases)
- [Changelog](https://github.com/dateutil/dateutil/blob/master/NEWS)
- [Commits](https://github.com/dateutil/dateutil/compare/2.8.1...2.8.2)

---
updated-dependencies:
- dependency-name: python-dateutil
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16 08:05:05 +00:00
Alexandre Flament 1ba3e5afa0
Merge pull request #208 from searxng/dependabot/pip/master/uvloop-0.15.3
Bump uvloop from 0.15.2 to 0.15.3
2021-07-16 10:03:56 +02:00
dependabot[bot] 595ad77b8d
Bump sphinx from 4.0.2 to 4.1.1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.0.2 to 4.1.1.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.0.2...v4.1.1)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16 07:05:27 +00:00
dependabot[bot] 41dda87531
Bump uvloop from 0.15.2 to 0.15.3
Bumps [uvloop](https://github.com/MagicStack/uvloop) from 0.15.2 to 0.15.3.
- [Release notes](https://github.com/MagicStack/uvloop/releases)
- [Commits](https://github.com/MagicStack/uvloop/compare/v0.15.2...v0.15.3)

---
updated-dependencies:
- dependency-name: uvloop
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16 07:05:10 +00:00
Markus Heiser 0a9cd08bf1 [enh] add mongodb offline engine
Cherry-Pick: https://github.com/searx/searx/commit/198aad43
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15 21:35:33 +02:00
Markus Heiser 628b5703f3 [mod] improve video results of the qwant engine
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15 20:10:37 +02:00
Alexandre Flament f376b4ed3e
Merge pull request #205 from unixfox/patch-2
Add missing parameter for mobile UI search
2021-07-15 17:19:12 +02:00
Alexandre Flament 7efb527719
Merge pull request #204 from return42/fix-qwant
[fix] Qwant engine - implement API v3 and add engine "qwant videos"
2021-07-15 17:19:06 +02:00
Émilien Devos 6c9f276571
Add missing parameter for mobile UI search 2021-07-15 13:00:32 +00:00
Markus Heiser 420e3caba3 [mod] enable Qwant engines (web, news, images, video) by default
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA.  Let
us enable Qwant engines by default.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser ef6e1bd6b9 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Alexandre Flament ff180cf2db
Merge pull request #203 from return42/drop-dogpile
[fix] drop engine dogpile
2021-07-13 15:58:37 +02:00
Markus Heiser 2767046065 [fix] drop engine dogpile
The engine was added in commit a4b07460 but now it shows new issues [1].

In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]

  Powered by technology, Dogpile returns all the best results from leading
  search engines including Google and Yahoo!

Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough.   It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.

With this patch the dogpile engine is removed

[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-13 14:42:09 +02:00
Alexandre Flament 4f09c7ba0c
Merge pull request #201 from return42/drop-torrentz
[drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
2021-07-12 08:22:49 +02:00
Markus Heiser 513c73a309 [drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/

Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-11 13:24:33 +02:00
Markus Heiser 438357a5b8
Merge pull request #200 from unixfox/patch-1
Fix google images
2021-07-10 16:17:21 +00:00
Émilien Devos d9d9bd720d
Fix google images
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010
2021-07-10 14:09:29 +00:00