The monolithic implementation of the limiter was divided into methods and
implemented in the Python package searx.botdetection. Detailed documentation on
the methods has been added.
The methods are divided into two groups:
1. Probe HTTP headers
- Method http_accept
- Method http_accept_encoding
- Method http_accept_language
- Method http_connection
- Method http_user_agent
2. Rate limit:
- Method ip_limit
- Method link_token (new)
The (reduced) implementation of the limiter is now in the module
searx.botdetection.limiter. The first group was transferred unchanged to this
module. The ip_limit contains the sliding windows implemented by the limiter so
far.
This merge also fixes some long outstandig issue:
- limiter does not evaluate the Accept-Language correct [1]
- limiter needs a IPv6 prefix to block networks instead of IPs [2]
Without additional configuration the limiter works as before (apart from the
bugfixes). For the commissioning of additional methods (link_toke), a
configuration must be made in an additional configuration file. Without this
configuration, the limiter runs as before (zero configuration).
The ip_limit Method implements the sliding windows of the vanilla limiter,
additionally the link_token method can be used in this method. The link_token
method can be used to investigate whether a request is suspicious. To activate
the link_token method in the ip_limit method add the following to your
/etc/searxng/limiter.toml::
[botdetection.ip_limit]
link_token = true
[1] https://github.com/searxng/searxng/issues/2455
[2] https://github.com/searxng/searxng/issues/2477
HINT: this patch has no functional change / it is the preparation for following
changes and bugfixes
Over the years, the preferences template became an unmanageable beast. To make
the source code more readable the monolith is splitted into elements. The
splitting into elements also has the advantage that a new template can make use
of them.
The reversed checkbox is a quirk that is only used in the prefereces and must be
eliminated in the long term. For this the macro 'checkbox_onoff_reversed' was
added to the preferences.html template. The 'checkbox' macro is also a quirk of
the preferences.html we don't want to use in other templates (it is an
input-checkbox in a HTML form that was misused for status display).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.
This patch does not contain functional change, except it fixes issue #2455
----
Aktivate limiter in the settings.yml and simulate a bot request by::
curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
-H 'Accept: text/html'
-H 'User-Agent: xyz' \
-H 'Accept-Encoding: gzip' \
'http://127.0.0.1:8888/search?q=foo'
In the LOG:
DEBUG searx.botdetection.link_token : missing ping for this request: .....
Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.
Closes: https://github.com/searxng/searxng/issues/2455
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
If there were no results but errors in the engines then the error dialogs of the
engines was displayed in the result list.
With the new design errors of the engines should only be displayed in the
sidebar and at the same time duplications of the (template) code will be
avoided.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* set border top and bottom on sidebar collasables
* inrease peading on summary so its easier to click on mobile
* remove margins and add flex wrapper to normalize elements in sidebar
Make elements in the sidebar collapse able. Except infoboxes all elements in
the sidebar are collapsed by default.
By folding out the sidebar elements, the UI looks less cluttered. Especially on
small devices like smartphones, where the sidebar is above the results list, the
UX should be improved [1].
[1] https://github.com/searxng/searxng/issues/2140
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This PR does no functional change it is just an attempt to make more clear in
the code, what a default category is and what a subcategory is. The previous
name 'others' leads to confusion with the **category 'other'**.
If a engine is not assigned to a category, the default is assigned::
DEFAULT_CATEGORY = 'other'
If an engine has only one category and this category is shown as tab in the user
interface, this engine has no further subgrouping::
NO_SUBGROUPING = 'without further subgrouping'
Related:
- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
With the language and region tags from the EngineTraitsMap the handling of
SearXNG's tags of languages and regions has been normalized and is no longer
a *mystery*. The "languages" became "locales" that are supported by babel and
by this, the update_engine_traits.py can be simplified a lot.
Other code places can be simplified as well, but these simplifications
should (respectively can) only be done when none of the engines work with the
deprecated EngineTraits.supported_languages interface anymore.
This commit replaces searx.languages by searx.sxng_locales and fix the naming of
some names from "language" to "locale" (e.g. language_codes --> sxng_locales).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
When the user choose "Auto-detected", the choice remains on the following queries.
The detected language is displayed.
For example "Auto-detected (en)":
* the next query language is going to be auto detected
* for the current query, the detected language is English.
This replace the autodetect_search_language plugin.
* use html input elements instead of buttons for the pagination forms at the bottom of the result page
* move the less section that hides the pagination number widget on mobile to the mobile section
* clean up the less code for styling the numbers of the pagination widget
* fix: add the margin for box results (like in news category) to the bottom of the result to have a margin between pagination widget and article result
Adds to the navigation widget, preserving forward/backward nav, and
inserting a list of clickable page numbers between them.
Phone sized devices continue without this widget as deterministic
display under small screen sizes has not been solved.
The widget is agnostic to the actual amount of pages there are that one
can navigate to and as such shows all plausible, albeit not necessarilly
valid, possibilities.
This widget does not interfere with infinite scroll in any fashion.
The word "hackable" may arouse interest in programmers to participate in the
development, but it scares the ordinary user.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG
(related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work
flawles.
Inspired by the OpenSearch description from qwant and with informations from the
specification[4] the ``opensearch.xml`` has been *improved*.
- convert `<Url>` methods from lower case to upper case (`POST`|`GET`)
- add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"`
- add `<Query role="example" searchTerms="SearXNG" />` [4]
OpenSearch description documents should include at least one Query element of
`role="example"` that is expected to return search results. Search clients may
use this example query to validate that the search engine is working properly.
- modified `<LongName>` to SearXNG
- modified `<Description>` the word 'hackable' scares uninitiated users and was removed
- add the `type="image/png"` to `<Image>`
Test can be done by::
make run
Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser /
test with different WEB-Browser from desktop and Smartphones (are there any iOS
user here, please test on Safari and Chrome).
[1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw
[2] https://github.com/searxng/searxng/issues/431
[3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch
[4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some administrators may want to hide the link to public_instances: If you run a
private instance for a company that wants there employees to use the private
instance instead of any of the public instances.
Add a new setting: general.donation_url
By default the value is https://docs.searxng.org/donate.html
When the value is false, the link is hidden
When the value is true, the link goes to the infopage donation,
the administrator can create a custom page.
Related to https://github.com/searxng/searxng/issues/1326
URL for the logo is referenced twice:
* in index.html for preloading: it contains the hash for cache busting (when static_use_hash: true)
* in searxng.min.css: to actually display the image. The URL doesn't contain the hash.
So the image preload actually loads twice the same image.
This commit removed the image preloading.
This is workaround: the real fix is to be able to use the URL with the hash in the CSS.
MatthieuBarbu
Bnyro