Commit Graph

6605 Commits

Author SHA1 Message Date
Markus Heiser eb02cc77c5 [fix] google - simplify XPath selectors to fetch more results
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 18:55:31 +02:00
Markus Heiser 5706d110f2
Merge pull request #1650 from joxcat/eng-librs
Add lib.rs engine
2022-08-10 17:45:28 +02:00
Johan Planchon 90bdb23756 add lib.rs engine 2022-08-10 12:17:55 +02:00
Émilien Devos b9f16a77db output format protobuf to HTML for google mobile 2022-08-10 09:36:06 +00:00
Thomas Renard d4acbcfe63 [mod] add deepl translation engine
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API.  This api only needs a to-lang, from-lang is a fake
by now.

There is a free option to use [1].

[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
2022-08-10 09:14:36 +02:00
Léon Tiekötter 07342899e5
Merge pull request #1643 from vojkovic/master
Revert Google XPath.
2022-08-10 04:18:42 +02:00
Brock Vojković 24210fb10b
Revert PR #1633
This reverts the changes made to the Google results XPath in PR #1633.
2022-08-10 03:41:39 +02:00
Markus Heiser 3d6b098a5f
Merge pull request #1633 from tiekoetter/fix-google-results-xpath
[fix] google engine: results XPath
2022-08-09 06:28:03 +02:00
Léon Tiekötter 94b3656b4a [fix] google engine: results XPath
Seems google rolls out changes first on the `google.com` domain and later on the
"language" domains.  By example: yesterday [1] `google.com` did not work but
`google.de` and `google.fr` did work, today they do not work any longer and this
fix is needed on all domains.

Closes: https://github.com/searxng/searxng/issues/1628
[1] https://github.com/searxng/searxng/issues/1628#issuecomment-1208191816
2022-08-09 06:23:59 +02:00
Markus Heiser 925f3ff487
Merge pull request #1627 from allendema/add-marginalia
[enh] Initial Marginalia.nu support (foss)
2022-08-08 06:53:02 +02:00
Markus Heiser 58f630d312
Merge pull request #1623 from return42/mod-yep.com
[mod] engine yep.com: show all 100 results yep.com has
2022-08-08 06:52:15 +02:00
Allen 5b815b3966 [enh] initial Marginalia.nu support (foss)
Currently it uses a public api_key `/public/` [1]

The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.

    0: popular
    1: blogs
    2: big_sites
    3: default
    4: experimental

'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.

[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626

Closes: https://github.com/searxng/searxng/issues/1620
2022-08-08 06:31:04 +02:00
Markus Heiser 04e3427022
Merge pull request #1626 from return42/fix-minor
[fix] typo in uWSGI cache: searxcache --> searxngcache
2022-08-07 12:53:33 +02:00
Markus Heiser 8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
Markus Heiser 647e6187d7 [mod] engine yep.com: show all 100 results yep.com has
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-06 13:21:28 +02:00
Markus Heiser 63f5121b38 [fix] pycodestyle has been replaced by black in 3c77412d3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-05 10:29:58 +02:00
Markus Heiser cee586029c
Merge pull request #1614 from searxng/dependabot/pip/master/setproctitle-1.3.1
Bump setproctitle from 1.3.0 to 1.3.1
2022-08-05 10:03:35 +02:00
Markus Heiser 3d0bf840b9
Merge pull request #1617 from searxng/translations_update
Update translations
2022-08-05 09:44:57 +02:00
Markus Heiser 051cf5bc61
Merge pull request #1615 from searxng/dependabot/pip/master/pycodestyle-2.9.1
Bump pycodestyle from 2.8.0 to 2.9.1
2022-08-05 09:33:05 +02:00
searxng-bot 0f02fcc836 [translations] update from Weblate
acf8bd39 - 2022-08-05 - Markus Heiser <markus.heiser@darmarit.de>
4ad75b6e - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
ee8cbee6 - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
87c19313 - 2022-08-01 - Academic tyro <y13593582403@gmail.com>
cbe0de32 - 2022-07-30 - Markus Heiser <markus.heiser@darmarit.de>
45029a17 - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
6eec3795 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f8d8f31f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
b3fb365f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
aaeabbc9 - 2022-08-03 - Lakatos Tamás <tomimost@gmail.com>
6c71c501 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f7b5ba19 - 2022-08-01 - Markus Heiser <markus.heiser@darmarit.de>
850e7fa0 - 2022-08-04 - Mico Hautaluoma <m@mha.fi>
0cb696fc - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
04c3785f - 2022-08-02 - Markus Heiser <markus.heiser@darmarit.de>
b500f2ad - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b576b83 - 2022-08-01 - GooGuJiang <gu@gmoe.cc>
0adeb6e2 - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b025f17 - 2022-07-31 - PRATYAY MUSTAFI <pratyaymustafi@gmail.com>
2022-08-05 07:18:24 +00:00
dependabot[bot] 27cdbeb9c4
Bump pycodestyle from 2.8.0 to 2.9.1
Bumps [pycodestyle](https://github.com/PyCQA/pycodestyle) from 2.8.0 to 2.9.1.
- [Release notes](https://github.com/PyCQA/pycodestyle/releases)
- [Changelog](https://github.com/PyCQA/pycodestyle/blob/main/CHANGES.txt)
- [Commits](https://github.com/PyCQA/pycodestyle/compare/2.8.0...2.9.1)

---
updated-dependencies:
- dependency-name: pycodestyle
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:16 +00:00
dependabot[bot] c6b5bc5c69
Bump setproctitle from 1.3.0 to 1.3.1
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.3.0...version-1.3.1)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:12 +00:00
Markus Heiser 225f8caaf9
Merge pull request #1608 from allendema/brave-time_range
[enh] time range support for brave
2022-08-04 06:48:50 +02:00
Allen 58369e41d6 [wip] brave time range support 2022-08-04 04:47:25 +02:00
Markus Heiser 7c9c112484
Merge pull request #1560 from return42/http-accept-language
[mod] add 'Accept-Language' HTTP header to online processores
2022-08-01 17:59:49 +02:00
Markus Heiser 8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
Markus Heiser a2badb4fe4 [doc] add description of method EngineProcessor.get_params()
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 16:42:33 +02:00
Markus Heiser 1fbb514a4e
Merge pull request #1595 from tiekoetter/fix-uwsgi-buffer-size
[fix] uWSGI: increase buffer-size
2022-07-31 13:13:54 +02:00
Markus Heiser 48968bf46a [doc] list of changes that affect the infrastructure
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-31 13:08:06 +02:00
Léon Tiekötter 2b94fef7ec [fix] uWSGI: increase buffer-size
Increase max size of a request, by default it is 4k [1].  4096 as buffer-size is
too small and will result in the preference urls not working.

[1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
2022-07-31 12:40:06 +02:00
Markus Heiser 88733c0ce6
Merge pull request #1569 from dalf/fix-pyright-reported-errors
Fix pyright errors
2022-07-30 18:40:16 +02:00
Alexandre Flament 2babf59adc [fix] pyright repported errors
The errors make pyright usage useless since a new error won't be seen [1].

[1] https://github.com/searxng/searxng/pull/1569

```
  searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]"
    "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]"
    Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues)
  searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str"
    Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
  searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int"
    Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues)
  searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join"
    Type "str" cannot be assigned to type "BytesPath"
      "str" is incompatible with "bytes"
      "str" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['themes']" cannot be assigned to type "BytesPath"
      "Literal['themes']" is incompatible with "bytes"
      "Literal['themes']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "str | Any | None" cannot be assigned to type "BytesPath"
      Type "str" cannot be assigned to type "BytesPath"
        "str" is incompatible with "bytes"
        "str" is incompatible with protocol "PathLike[bytes]"
          "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['img']" cannot be assigned to type "BytesPath"
      "Literal['img']" is incompatible with "bytes"
      "Literal['img']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports)
  searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports)
  searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource)
  searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable)
  searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable)
  searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown
    Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType)
  searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
  searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
```
2022-07-30 18:04:44 +02:00
Markus Heiser ededaab807
Merge pull request #1589 from return42/searxng-install-additions
[mod] fix minor leftovers from PR #1332
2022-07-30 16:25:32 +02:00
Markus Heiser bded8750d5 [mod] fix minor leftovers from PR #1332
Related: https://github.com/searxng/searxng/pull/1332
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 16:13:47 +02:00
Markus Heiser 645c2a2ca1
Merge pull request #1332 from return42/searxng-install
Upgrade installation scripts and documentation
2022-07-30 13:45:12 +02:00
Léon Tiekötter e5323b8aa2 [docs] corrections from @tiekoetter's review 2022-07-30 13:39:35 +02:00
Markus Heiser 6fbffe9d20 [docs] add section "Migrate and stay tuned!"
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser 692708aa77 [clean up] drop obsolete searx, filtron and morty install scripts
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.

For users who want to upgrade, the procedures for removing old installations
have still been retained.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser ed8a169029 [doc] update documentation of the installation procedures
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser 782f73540e [utils/searxng.sh] implement new script to install SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser 81bba44869 [install scripts] rename SEARX_<name> variables to SEARXNG_<name>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Alexandre Flament 0cf5310933
Merge pull request #1581 from searxng/dependabot/pip/master/setproctitle-1.3.0
Bump setproctitle from 1.2.3 to 1.3.0
2022-07-29 10:35:49 +02:00
Markus Heiser ca2ddc615f
Merge pull request #1583 from searxng/translations_update
Update translations
2022-07-29 09:31:45 +02:00
searxng-bot 953aaec6a3 [translations] update from Weblate
c0c9107c - 2022-07-27 - Sangha Lee <totoriato@gmail.com>
5b48bce6 - 2022-07-24 - Linerly <linerly@protonmail.com>
79669e65 - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
520e9284 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
7cf52ff5 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
9d3ebe72 - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
2d03c097 - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
388af012 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
a4bcf098 - 2022-07-25 - Miguel Silva <miguelcabeca.dev@gmail.com>
93fd0b72 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
8f68b206 - 2022-07-26 - tents <remendne@pentrens.jp>
9007c99c - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
aeec96f2 - 2022-07-26 - Matija Kromar <matija.kromar@gmail.com>
69084863 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
b48190ab - 2022-07-24 - alexfs2015 <alex04fs@gmail.com>
b6bbc0a5 - 2022-07-23 - Markus Heiser <markus.heiser@darmarit.de>
1a503806 - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
c960cb93 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
8a2bd34b - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
1064cea0 - 2022-07-23 - LagManCZ <lagmen@post.cz>
67423045 - 2022-07-24 - alexfs2015 <alex04fs@gmail.com>
56c87fda - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
36a64f1c - 2022-07-24 - Ankit Gupta <guptaa.ankitt@gmail.com>
2022-07-29 07:17:55 +00:00
Markus Heiser e08072def5
Merge pull request #1582 from searxng/dependabot/pip/master/sphinx-5.1.1
Bump sphinx from 5.0.2 to 5.1.1
2022-07-29 09:13:44 +02:00
dependabot[bot] f0f195c043
Bump sphinx from 5.0.2 to 5.1.1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 5.0.2 to 5.1.1.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/5.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v5.0.2...v5.1.1)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-29 07:04:21 +00:00
dependabot[bot] 9e3ae571d0
Bump setproctitle from 1.2.3 to 1.3.0
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.2.3 to 1.3.0.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.2.3...version-1.3.0)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-29 07:04:16 +00:00
Markus Heiser 14085cc0a7
Merge pull request #1580 from searxng/update_data_update_engine_descriptions.py
Update searx.data - update_engine_descriptions.py
2022-07-29 07:23:02 +02:00
Markus Heiser 7602c23d71
Merge pull request #1579 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2022-07-29 07:22:12 +02:00
Markus Heiser 0bad4fb931
Merge pull request #1578 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2022-07-29 07:21:06 +02:00