searxng/searx
Markus Heiser 744d96a16c [fix] startpage engine: language/region & time support & fix CAPTCHA
One reason for the often seen CAPTCHA of the startpage requests are the
incomplete requests SearXNG sends to startpage.com.  To avoid CAPTCHA we need to
send a well formed HTTP POST request with a cookie, we need to form a request
that is identical to the request build by startpage.com itself:

- in the cookie the **region** is selected
- in the POST arguments the **language** is selected

Based on the *engine_properties* boilerplate, SearXNG's startpage engine now
implements a `_fetch_engine_properties()` function to fetch regions & languages
from startpage.com.

This patch is a complete new implementation of the request() function, reversed
engineered from the startpage.com page.  The new implementation adds

- time-range support
- save-search support

to the startpage engine which has been missed in the past.

The locale code 'no_NO' from startpage does not exists and is mapped to nb-NO.
For reference see languages-subtag at iana [1], `no` is the macrolanguage::

     type: language
     Subtag: nb
     Description: Norwegian Bokmål
     Added: 2005-10-16
     Suppress-Script: Latn
     Macrolanguage: no

Additional hints:

- To fetch languages from startpage, this patch makes use of the
  EngineProperties implemented in 7bf0d46c

- Te get Startpage's locale & language, the function get_engine_locale from
  9ae409a is used.

[1] https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
[2] https://www.w3.org/International/questions/qa-choosing-language-tags#langsubtag

Closes: https://github.com/searxng/searxng/issues/1081
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-29 19:27:50 +02:00
..
answerers [fix] issue reported by pylint 22.1.0 2022-02-04 09:45:35 +01:00
data [fix] startpage engine: language/region & time support & fix CAPTCHA 2022-08-29 19:27:50 +02:00
engines [fix] startpage engine: language/region & time support & fix CAPTCHA 2022-08-29 19:27:50 +02:00
infopage Donation link: default value to searxng.org, can be hidden or custom 2022-07-02 11:29:21 +02:00
metrics [fix] prepare for pylint 2.14.0 2022-06-03 15:41:52 +02:00
network bing.py: resolve bing.com/ck/a redirections 2022-07-08 22:02:21 +02:00
plugins [mod] limiter plugin: Accept-Encoding handling 2022-08-25 23:21:30 +02:00
search [fix] replace language_support by a language/region view 2022-08-29 18:13:46 +02:00
shared [fix] typo in uWSGI cache: searxcache --> searxngcache 2022-08-07 12:31:09 +02:00
static [build] /static 2022-08-27 06:52:20 +00:00
templates/simple [fix] replace language_support by a language/region view 2022-08-29 18:13:46 +02:00
translations [translations] update from Weblate 2022-08-26 07:24:01 +00:00
__init__.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
autocomplete.py [fix] startpage engine: language/region & time support & fix CAPTCHA 2022-08-29 19:27:50 +02:00
babel_extract.py [mod] move category and names of constants to searx/searxng.msg 2022-03-16 09:55:53 +01:00
compat.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
exceptions.py [pyright:strict] searx.search.checker.background 2022-01-27 22:07:12 +01:00
external_bang.py External bang: bug fix: URL encode the query so "!!g 1+1" works as intended 2022-08-27 07:10:26 +00:00
external_urls.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
flaskfix.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
languages.py [fix] add back missing languages & regions (followup of PR #1071) 2022-04-22 12:09:42 +02:00
locales.py [fix] get_engine_locale: better approximation of 'en' is 'en-US' 2022-08-14 15:45:07 +02:00
preferences.py [fix] typo: add missing 'n' in center_aligment --> center_alignment 2022-07-03 17:35:54 +02:00
query.py [mod] add flags to the languages filter 2022-03-19 15:09:13 +01:00
redislib.py [mod] redislib - optimize LUA script registration. 2022-05-30 11:21:09 +02:00
results.py [simple theme] replace Image_layout.js with flexbox CS impl. 2022-02-26 22:31:47 +01:00
searxng.msg [mod] move category and names of constants to searx/searxng.msg 2022-03-16 09:55:53 +01:00
settings.yml Merge pull request #1699 from liimee/eng-app-store 2022-08-27 07:43:23 +02:00
settings_defaults.py Merge pull request #1428 from return42/fix-center_aligment 2022-07-07 09:43:12 +02:00
settings_loader.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
unixthreadname.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
utils.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
version.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
webadapter.py [fix] disabling all engines in a category makes the bang search in general 2022-01-03 20:23:46 +01:00
webapp.py [fix] replace language_support by a language/region view 2022-08-29 18:13:46 +02:00
webutils.py theme: remove __common__ 2022-05-07 19:40:48 +02:00