Commit Graph

7608 Commits

Author SHA1 Message Date
Markus Heiser 4cb8cbc6c3 [fix] duckduckgo lite engine: set HTTP header 'Referer'
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081

Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:53 +02:00
Bnyro 16ce63b8a2 [mod] yacy: use official instance by default and fix crashes 2023-10-19 18:04:53 +02:00
Alex Balgavy c2f26a560a [mod] add hotkeys option to settings.yml
The change in the hotkey mechanism introduced in 317db5b04 does not allow
configuration via `settings.yml`.  This commit adds that functionality.

Closes: #2898
2023-10-19 18:04:53 +02:00
Hackurei ce6b7c9f75 [feat] implement hackernews engine - news.ycombinator.com 2023-10-19 18:04:53 +02:00
Aine fa830b375b [fix] matrixrooms add proper MRS integration
Related:

- https://github.com/searxng/searxng/issues/2918
2023-10-19 18:04:53 +02:00
Bnyro 34d0d2c9ab [feat] duckduckgo: support for videos and news 2023-10-19 18:04:53 +02:00
Bnyro 84b4932e21 [fix] kickass: crash when no results 2023-10-19 18:04:53 +02:00
Bnyro 057ae1767b [mod] piped: always show video length if available 2023-10-19 18:04:53 +02:00
Bnyro a742cfe2e7 [feat] engine: implementation of mastodon 2023-10-19 18:04:53 +02:00
dependabot[bot] 89ed17a0b3 Bump nose2[coverage_plugin] from 0.13.0 to 0.14.0
Bumps [nose2[coverage_plugin]](https://github.com/nose-devs/nose2) from 0.13.0 to 0.14.0.
- [Changelog](https://github.com/nose-devs/nose2/blob/main/docs/changelog.rst)
- [Commits](https://github.com/nose-devs/nose2/compare/0.13.0...0.14.0)

---
updated-dependencies:
- dependency-name: nose2[coverage_plugin]
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:53 +02:00
dependabot[bot] d71c7d7d9f Bump setproctitle from 1.3.2 to 1.3.3
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.2 to 1.3.3.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.3.2...version-1.3.3)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:53 +02:00
dependabot[bot] 0b6cb92f5c Bump pylint from 2.17.6 to 3.0.1
Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.17.6 to 3.0.1.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.17.6...v3.0.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:53 +02:00
dependabot[bot] 2df2eee361 Bump babel from 2.12.1 to 2.13.0
Bumps [babel](https://github.com/python-babel/babel) from 2.12.1 to 2.13.0.
- [Release notes](https://github.com/python-babel/babel/releases)
- [Changelog](https://github.com/python-babel/babel/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-babel/babel/compare/v2.12.1...v2.13.0)

---
updated-dependencies:
- dependency-name: babel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:53 +02:00
searxng-bot b6cae7d53f [translations] update from Weblate
68d743281 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
42f091b7f - 2023-10-05 - return42 <markus.heiser@darmarit.de>
2479c0d7b - 2023-10-05 - ghose <correo@xmgz.eu>
a4e6cd592 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
9d4e5f5c3 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
b79d44775 - 2023-10-05 - gallegonovato <fran-carro@hotmail.es>
746291184 - 2023-10-06 - return42 <markus.heiser@darmarit.de>
f24d7e8b1 - 2023-10-05 - return42 <markus.heiser@darmarit.de>
6140911f9 - 2023-10-05 - Fjuro <ifjuro@proton.me>
2023-10-19 18:04:53 +02:00
Markus Heiser ba5f9bd4ac [mod] engine - simplify region & lang handling, make filters configurable
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:53 +02:00
Bnyro 70f6a84aff [feat] engine: implementation of radio-browser.info 2023-10-19 18:04:53 +02:00
Markus Heiser 4de856bf19 [fix] limiter / botdetection: remove http_connection method
Related:

- https://github.com/searxng/searxng/issues/2892#issuecomment-1742153932

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:53 +02:00
Jinyuan Huang 993ea9c1ec [typo] solved a typo in yahoo error message. 2023-10-19 18:04:53 +02:00
Jinyuan Huang 7de4e3b48f [fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866
Co-authored-by: Blair Noctis <n@sail.ng>
2023-10-19 18:04:53 +02:00
Alexandre Flament bb433a3873 Upgrade dependencies
* geckodriver: 0.33.0
* node 16.20.2
* pyright 1.1.329
* eslint 8.50.0
2023-10-19 18:04:53 +02:00
Bnyro f6ee574508 [fix] emojipedia: fix engine 2023-10-19 18:04:52 +02:00
Markus Heiser f4f2af7db6 [fix] Revision of the Bing engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00
jazzzooo ae91b993b1 [fix] engine - bing fix search, pagination, remove safesearch 2023-10-19 18:04:52 +02:00
Bnyro b732fbfe5b [feat] engine: implementation of pinterest 2023-10-19 18:04:52 +02:00
Bnyro 32e04f05bf [fix] matrixrooms.info: pagination not working properly 2023-10-19 18:04:52 +02:00
Markus Heiser cef109e0e0 [fix] engine - moviepilot instead of thumbnail use img_src
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.

Related:

- b4e0d2eedc (r128785388)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00
Bnyro 3b51a0bf0f [mod] tagesschau: add option to only use tagesschau urls 2023-10-19 18:04:52 +02:00
Bnyro 07d932fba0 [feat] engine: implementation of matrixrooms.info 2023-10-19 18:04:52 +02:00
Bnyro 3974a88dae [feat] engine: implementation of tootfinder 2023-10-19 18:04:52 +02:00
Bnyro aad75ae867 [mod] add movies category for tmdb, imdb and moviepilot 2023-10-19 18:04:52 +02:00
Émilien (perso) e0bab6aa63 [mod] warn to info for failed ping limiter 2023-10-19 18:04:52 +02:00
Bnyro da82436054 [mod] settings.yml: remove plugin settings for plugins that don't exist anymore 2023-10-19 18:04:52 +02:00
dependabot[bot] 12ca43fd90 Bump selenium from 4.12.0 to 4.13.0
Bumps [selenium](https://github.com/SeleniumHQ/Selenium) from 4.12.0 to 4.13.0.
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.12.0...selenium-4.13.0)

---
updated-dependencies:
- dependency-name: selenium
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:52 +02:00
dependabot[bot] 9c9ea60e86 Bump pylint from 2.17.5 to 2.17.6
Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.17.5 to 2.17.6.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.17.5...v2.17.6)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-19 18:04:52 +02:00
searxng-bot eb5bcc523c [translations] update from Weblate
80bbf74ef - 2023-09-27 - return42 <markus.heiser@darmarit.de>
5d554ef71 - 2023-09-27 - BernieHuang2008 <berniehuang2008@163.com>
771fd1b90 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
d412012c8 - 2023-09-27 - y0nei <y0nei@proton.me>
1ac171d0a - 2023-09-27 - return42 <markus.heiser@darmarit.de>
a004c40cc - 2023-09-27 - return42 <markus.heiser@darmarit.de>
47e729f89 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
f85b28cc4 - 2023-09-26 - Fjuro <ifjuro@proton.me>
75ce5d581 - 2023-09-27 - return42 <markus.heiser@darmarit.de>
40ce06009 - 2023-09-24 - nonepork <ptohrek@gmail.com>
21f343f71 - 2023-09-25 - tentsbet <remendne@pentrens.jp>
4c80f80fd - 2023-09-22 - return42 <markus.heiser@darmarit.de>
ef3c859e4 - 2023-09-22 - artnay <jiri.gronroos@iki.fi>
2023-10-19 18:04:52 +02:00
dalf 0620c43e87 Update searx.data - update_firefox_version.py 2023-10-19 18:04:52 +02:00
dalf 5e984a2856 Update searx.data - update_ahmia_blacklist.py 2023-10-19 18:04:52 +02:00
dalf cfe1974b65 Update searx.data - update_wikidata_units.py 2023-10-19 18:04:52 +02:00
dalf 442f3f0c06 Update searx.data - update_engine_traits.py 2023-10-19 18:04:52 +02:00
dalf 22822fc179 Update searx.data - update_currencies.py 2023-10-19 18:04:52 +02:00
dalf 46eb440057 Update searx.data - update_engine_descriptions.py 2023-10-19 18:04:52 +02:00
Bnyro e17575112a [fix] categories.html: can't select social media category using search on category select 2023-10-19 18:04:52 +02:00
jazzzooo c4d280d890 [fix] engine - currency fix and simplify 2023-10-19 18:04:52 +02:00
Jinyuan Huang 3c8ad561ff [fix] bilibili new api used 2023-10-19 18:04:52 +02:00
jazzzooo adaf30a5ff [fix] engine - kickass update url, fix parsing, use multiple mirrors 2023-10-19 18:04:52 +02:00
Markus Heiser ba0d5f78b9 Update searx.data - update_engine_traits.py
Update searx.data - update_engine_traits.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00
Markus Heiser c61c7f3416 [fix] engine brave - fetch traits (modified settings menu)
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00
Markus Heiser ddc1735d0f [fix] engine annas archive - fetch traits (modified xpath selectors)
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00
Jinyuan Huang fb225803f2 solve #2839 2023-10-19 18:04:52 +02:00
Markus Heiser a72138b0f3 [fix] harmonize default settings for caching of /static files
Caching files on the client side for more than a day can confuse the end user
when updating static files[1].

Depending on the way of providing a SearXNG instance via HTTP, there are several
ways to optimize the access to the /static files.  However, since we don't know
what optimization an admin has provided for his static files, we should have
moderate settings in the defaults that run robustly in a wide variety of
installations.

In this sense, all caches on the client side should be cleared after one day at
the latest.  So far the files were cached for one year on client side; as soon
as changes are made to the static files (with the option `static_use_hash:
true`) the old static files are kept for one year on the CLient side / which can
also be evaluated as unnecessary caching.

[1] https://github.com/searxng/searxng/discussions/2821

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-19 18:04:52 +02:00