searxngRebrandZaclys/searx
Markus Heiser 80aaef6c95
Merge pull request #2357 / limiter -> botdetection
The monolithic implementation of the limiter was divided into methods and
implemented in the Python package searx.botdetection.  Detailed documentation on
the methods has been added.

The methods are divided into two groups:

1. Probe HTTP headers

- Method http_accept
- Method http_accept_encoding
- Method http_accept_language
- Method http_connection
- Method http_user_agent

2. Rate limit:

- Method ip_limit
- Method link_token (new)

The (reduced) implementation of the limiter is now in the module
searx.botdetection.limiter.  The first group was transferred unchanged to this
module.  The ip_limit contains the sliding windows implemented by the limiter so
far.

This merge also fixes some long outstandig issue:

- limiter does not evaluate the Accept-Language correct [1]
- limiter needs a IPv6 prefix to block networks instead of IPs [2]

Without additional configuration the limiter works as before (apart from the
bugfixes).  For the commissioning of additional methods (link_toke), a
configuration must be made in an additional configuration file.  Without this
configuration, the limiter runs as before (zero configuration).

The ip_limit Method implements the sliding windows of the vanilla limiter,
additionally the link_token method can be used in this method.  The link_token
method can be used to investigate whether a request is suspicious. To activate
the link_token method in the ip_limit method add the following to your
/etc/searxng/limiter.toml::

    [botdetection.ip_limit]
    link_token = true


[1] https://github.com/searxng/searxng/issues/2455
[2] https://github.com/searxng/searxng/issues/2477
2023-06-03 06:00:15 +02:00
..
answerers [fix] issue reported by pylint 22.1.0 2022-02-04 09:45:35 +01:00
botdetection [mod] increase SUSPICIOUS_IP_WINDOW from one day to 30 days 2023-06-01 16:00:49 +02:00
data Update searx.data - update_engine_descriptions.py 2023-05-29 07:28:50 +02:00
enginelib [fix] searxng_extra/update/update_engine_descriptions.py (part 1) 2023-04-15 16:03:59 +02:00
engines use logger.warning 2023-05-19 19:35:29 +05:30
infopage [fix] minor typo in de/search-syntax page 2023-04-08 10:25:52 +02:00
metrics [fix] prepare for pylint 2.14.0 2022-06-03 15:41:52 +02:00
network [mod] Shuffle httpx's default ciphers of a SSL context randomly. 2023-03-19 13:40:31 +01:00
plugins [fix] limiter: replace real_ip by IPv4/v6 network 2023-06-01 15:51:14 +02:00
search use logger.warning 2023-05-19 19:35:29 +05:30
static [build] /static 2023-06-02 19:05:43 +02:00
templates/simple Merge pull request #2357 / limiter -> botdetection 2023-06-03 06:00:15 +02:00
tools [mod] limiter: add config file /etc/searxng/limiter.toml 2023-06-01 14:38:53 +02:00
translations [translations] update from Weblate 2023-06-02 09:34:36 +02:00
__init__.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
autocomplete.py [mod] remove obsolete EngineTraits.supported_languages 2023-03-24 10:37:42 +01:00
babel_extract.py [mod] move category and names of constants to searx/searxng.msg 2022-03-16 09:55:53 +01:00
compat.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
exceptions.py [doc] Add doc-strings to searx.exceptions 2023-01-29 19:06:19 +01:00
external_bang.py [mod] external bang: go to main instead of search page when query is empty 2023-04-25 15:02:34 +02:00
external_urls.py [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
flaskfix.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
locales.py use logger.warning 2023-05-19 19:35:29 +05:30
preferences.py [fix] ClientPref - don't raise exception if Accept-Language is invalid 2023-05-22 12:38:59 +02:00
query.py [mod] replace searx.languages by searx.sxng_locales 2023-03-24 10:37:42 +01:00
redisdb.py move searx.shared.redisdb to searx.redisdb 2022-12-10 09:26:38 +01:00
redislib.py [mod] redislib - optimize LUA script registration. 2022-05-30 11:21:09 +02:00
results.py [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
searxng.msg [mod] clarify the difference of the default category and subgrouping 2023-04-07 11:03:25 +02:00
settings_defaults.py [mod] replace searx.languages by searx.sxng_locales 2023-03-24 10:37:42 +01:00
settings_loader.py [fix] categories can't be removed from UI (categories_as_tabs) 2023-04-03 19:08:27 +02:00
settings.yml [mod] donation_url: disable by default 2023-05-15 09:19:17 +02:00
sxng_locales.py Update searx.data - update_engine_traits.py 2023-04-16 08:40:44 +02:00
unixthreadname.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
utils.py [mod] replace utils.match_language by locales.match_locale 2023-03-24 10:37:42 +01:00
version.py [fix] version format string generated by 'git show' 2023-05-10 18:33:45 +02:00
webadapter.py [mod] move language recognition to get_search_query_from_webapp 2023-04-15 22:23:33 +02:00
webapp.py [mod] limiter -> botdetection: modularization and documentation 2023-05-29 14:54:56 +02:00
webutils.py [mod] in the preference page, show !bang of subgrouping categories 2023-04-08 11:10:14 +02:00