In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.
This patch does not contain functional change, except it fixes issue #2455
----
Aktivate limiter in the settings.yml and simulate a bot request by::
curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
-H 'Accept: text/html'
-H 'User-Agent: xyz' \
-H 'Accept-Encoding: gzip' \
'http://127.0.0.1:8888/search?q=foo'
In the LOG:
DEBUG searx.botdetection.link_token : missing ping for this request: .....
Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.
Closes: https://github.com/searxng/searxng/issues/2455
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Implementations of the *traits* of the engines.
Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*. Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.
To load traits from the persistence::
searx.enginelib.traits.EngineTraitsMap.from_data()
For new traits new properties can be added to the class::
searx.enginelib.traits.EngineTraits
.. hint::
Implementation is downward compatible to the deprecated *supported_languages
method* from the vintage implementation.
The vintage code is tagged as *deprecated* an can be removed when all engines
has been ported to the *traits method*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"
The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.
It worked before because SearXNG initialized the Redis connection when the limiter started.
In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).
- add new engine option: send_accept_language_header
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The ? search operator has been broken for some time and
currently only raises the question why it's still there.
## Context ##
The query "Paris !images" searches for "Paris" in the "images" category.
Once upon a time Searx supported "Paris ?images" to search for "Paris"
in the currently enabled categories and the "images" category.
The feature makes sense ... the ? syntax does not.
We will hopefully introduce a +!images syntax in the future.
Fixes#702.
* allow not to record metrics (response time, etc...)
* this commit doesn't change the UI. If the metrics are disabled
/stats and /stats/errors will return empty response.
in /preferences, the columns response time and reliability will be empty.
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:
categories_as_tabs:
general:
icon: search-outline
Add a redis connector, the default DB connector is a socket at::
unix:///usr/local/searxng-redis/run/redis.sock?db=0
To set up a redis instance simply use::
$ ./manage redis.build
$ sudo -H ./manage redis.install
A hint for developers:
To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.
This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs can still be searched with !bangs.
In commit 4b43775c9 the brand.git_url & brand.git_branch has been removed, with
this patch these settings are removed from the documentation.
docs/admin/engines/settings.rst
- Remove brand.git_url & brand.git_branch
docs/dev/makefile.rst
- Remove brand.git_url & brand.git_branch
modified docs/dev/lxcdev.rst
- Remove brand.git_url & brand.git_branch
- fix searXNG to SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts. This patch reduce the variables in ./utils/brand.env to what is
needed. The following ('brand:' settings) can be removed from this file:
- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- GIT_URL <--> brand.git_url
- GIT_BRANCH <--> brand.git_branch
- SEARX_URL <--> server.base_url (aka PUBLIC_URL)
- SEARX_PORT <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
modified docs/admin/engines/settings.rst
- Fix documentation and add section 'brand'.
- Add remarks about **buildenv** variables.
- Add remarks about settings from environment variables $SEARX_DEBUG,
$SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET
modified docs/admin/installation-searx.rst & docs/build-templates/searx.rst
Fix template location /templates/etc/searx/settings.yml
modified docs/dev/makefile.rst
Add description of the 'make buildenv' target and describe
- we have all SearXNG setups are centralized in the settings.yml file
- why some tasks need a utils/brand.env (aka instance's buildenv)
modified manage
Settings file from repository's working tree are used by default and
ask user if a /etc/searx/settings.yml file exists.
modified searx/settings.yml
Add comments about when it is needed to run 'make buildenv'
modified searx/settings_defaults.py
Default for server:port is taken from enviroment variable SEARX_PORT.
modified utils/build_env.py
- Some defaults in the settings.yml are taken from the environment,
e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA). When the
'brand.env' file is created these enviroment variables should be
unset first.
- The CONTACT_URL enviroment is not needed in the utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser