zaclys/searxng
1
0
Fork 1
mirror of https://github.com/searxng/searxng synced 2024-01-01 19:24:07 +01:00
Code Issues Projects Releases Packages Wiki Activity
searxng/searx
History
czaky 122a9568de [network]: Add redundant parallel proxy requests. 
Anecdotally, using SearX over unreliable proxies,
like tor, seems to be quite error prone.
SearX puts quite an effort to measure the
performance and reliability of engines, most
likely owning to those aspects being of
significant concern.

The patch here proposes to mitigate related
problems, by issuing concurrent redundant requests
through the specified proxies at once, returning
the first response that is not an error.
The functionality is enabled using the:
`proxy_request_redundancy` parameter within the
outgoing network settings or the engine settings.

Example:

```yaml

outgoing:
    request_timeout: 8.0
    proxies:
        "all://":
            - socks5h://tor:9050
            - socks5h://tor1:9050
            - socks5h://tor2:9050
            - socks5h://tor3:9050
    proxy_request_redundancy: 4
```

In this example, each network request will be
send 4 times, once through every proxy. The
first (non-error) response wins.

In my testing environment using several tor proxy
end-points, this approach almost entirely
removes engine errors related to timeouts
and denied requests. The latency of the
network system is also improved.

The implementation, uses a
`AsyncParallelTransport(httpx.AsyncBaseTransport)`
wrapper to wrap multiple sub-trasports,
and `asyncio.wait` to wait on the first completed
request.

The existing implementation of the network
proxy cycling has also been moved into the
`AsyncParallelTransport` class, which should
improve network client memoization and
performance.

TESTED:
- unit tests for the new functions and classes.
- tested on desktop PC with 10+ upstream proxies
    and comparable request redundancy.
2024-05-17 02:09:29 +00:00
..
answerers [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
botdetection Drop pytomlpp dependency for Python >= 3.11 2024-05-05 17:35:29 +02:00
data [fix] update wikidata units - remove URL prefix from Q-name 2024-05-01 18:58:28 +02:00
enginelib [network]: Add redundant parallel proxy requests. 2024-05-17 02:09:29 +00:00
engines [mod] simple theme: drop img_src from default results 2024-05-16 07:30:38 +02:00
infopage [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
metrics [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
network [network]: Add redundant parallel proxy requests. 2024-05-17 02:09:29 +00:00
plugins [fix] unit converter plugin: can't be disabled in settings 2024-05-09 17:40:37 +02:00
search [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
static [build] /static 2024-05-16 07:30:38 +02:00
templates/simple [mod] simple theme: drop img_src from default results 2024-05-16 07:30:38 +02:00
translations [l10n] update translations from Weblate 2024-05-10 09:29:22 +02:00
__init__.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
autocomplete.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
babel_extract.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
exceptions.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
external_bang.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
external_urls.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
flaskfix.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
limiter.py [fix] even if limiter not activated, botdetection must be activated 2024-04-08 07:53:04 +02:00
limiter.toml [mod] isolation of botdetection from the limiter 2023-11-01 06:44:56 +01:00
locales.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
preferences.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
query.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
redisdb.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
redislib.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
results.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
searxng.msg [fix] add missing localizable (gettext) messages to searxng.msg 2024-04-26 07:34:32 +02:00
settings.yml [network]: Add redundant parallel proxy requests. 2024-05-17 02:09:29 +00:00
settings_defaults.py [network]: Add redundant parallel proxy requests. 2024-05-17 02:09:29 +00:00
settings_loader.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
sxng_locales.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
unixthreadname.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
utils.py [fix] remove usage of no longer existing names from lxml 2024-04-08 10:55:23 +02:00
version.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
webadapter.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00
webapp.py improve "search existing issues from github" link 2024-04-27 20:01:27 +02:00
webutils.py [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 2024-03-11 14:55:38 +01:00