searxngRebrandZaclys/searx
Markus Heiser 9ae409a05a [mod] add locale.get_engine_locale to get predictable results
The match_language function sometimes returns incorrect results which is why a
new function get_engine_locale is required.

A bugfix of the match_language is not easily possible, because there is almost
no documentation for it and already the call parameters are undefined.  E.g. the
function processes values like the ones from yahoo::

    "yahoo": [
        "ar",
        ...
        "zh_chs",
        "zh_cht"
     ]

The get_engine_locale has been documented in detail, there is a clear
description of the assumptions as well as the requirements and approximation
rules (read doc-string for more details)::

    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
    corresponding *engine locales*:

      <engine>: {
          # SearXNG string : engine-string
          'ca-ES'          : 'ca_ES',
          'fr-BE'          : 'fr_BE',
          'fr-CA'          : 'fr_CA',
          'fr-CH'          : 'fr_CH',
          'fr'             : 'fr_FR',
          ...
          'pl-PL'          : 'pl_PL',
          'pt-PT'          : 'pt_PT'
      }

    .. hint::

       The *SearXNG locale* string has to be known by babel!

In the following you will find a comparison:

>>> import babel.languages
>>> from searx.utils import match_language
>>> from searx.locales import get_engine_locale

Assume we have an engine that supports the follwoing locales:

>>> lang_list = {
...     "zh-CN": "zh_CN",
...     "zh-HK": "zh_HK",
...     "nl-BE": "nl_BE",
...     "fr-CA": "fr_CA",
... }

Assumption:

  A. When a user selects a language the results should be optimized according to
     the selected language.

  B. When user selects a language and a territory the results should be
     optimized with first priority on territory and second on language.

----

Example: (Assumption A.)

  A user selects region 'zh-TW' which should end in zh_HK

hint:
  CN is 'Hans' and HK ('Hant') fits better to TW ('Hant')

>>> get_engine_locale('zh-TW', lang_list)
'zh_HK'
>>> lang_list[match_language('zh-TW', lang_list)]
'zh_CN'

----

Example: (Assumption A.)

  A user selects only the language 'zh' which should end in CN

>>> get_engine_locale('zh', lang_list)
'zh_CN'
>>> lang_list[match_language('zh', lang_list)]
'zh_CN'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

hint:
  priority should be on the territory the user selected.  If the user
  prefers 'fr' he will select 'fr' without a region tag.

>>> get_engine_locale('fr-BE', lang_list, default='unknown')
'nl_BE'
>>> match_language('fr-BE', lang_list, fallback='unknown')
'fr-CA'

----

Example: (Assumption A.)

  A user selects only the language 'fr' which should end in fr_CA

>>> get_engine_locale('fr', lang_list)
'fr_CA'
>>> lang_list[match_language('fr', lang_list)]
'fr_CA'

----

The difference in priority on the territory is best shown with a engine that
supports the following locales:

>>> lang_list = {
...     "fr-FR": "fr_FR",
...     "fr-CA": "fr_CA",
...     "en-GB": "en_GB",
...     "nl-BE": "nl_BE",
... }

----

Example: (Assumption A.)

   A user selects only a language

>>> get_engine_locale('en', lang_list)
'en_GB'
>>> match_language('en', lang_list)
'en-GB'

hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..

>>> get_engine_locale('fr', lang_list)
'fr_FR'
>>> lang_list[match_language('fr', lang_list)]
'fr_FR'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

>>> get_engine_locale('fr-BE', lang_list)
'nl_BE'
>>> lang_list[match_language('fr-BE', lang_list)]
'fr_FR'

----

If the user selects a language and there are two locales like the following:

>>> lang_list = {
...      "fr-BE": "fr_BE",
...      "fr-CH": "fr_CH",
...  }
>>>

>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_BE'

Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):

>>> lang_list = {
...      "fr-CH": "fr_CH",
...      "fr-BE": "fr_BE",
...  }
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_CH'
>>>

The get_engine_locale selects the locale by looking at the "population percent"
and this percentage has an higher amount in BE (68.%) compared to CH (21%)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:35:55 +02:00
..
answerers [fix] issue reported by pylint 22.1.0 2022-02-04 09:45:35 +01:00
data Update searx.data - update_engine_descriptions.py 2022-08-10 21:48:34 +02:00
engines [mod] decouple qwant's categories from SearXNG's categories 2022-08-14 10:26:54 +02:00
infopage Donation link: default value to searxng.org, can be hidden or custom 2022-07-02 11:29:21 +02:00
metrics [fix] prepare for pylint 2.14.0 2022-06-03 15:41:52 +02:00
network bing.py: resolve bing.com/ck/a redirections 2022-07-08 22:02:21 +02:00
plugins [mod] add tor_check plugin - convenient tor checking trough searxng 2022-07-19 07:34:54 +02:00
search [mod] add 'Accept-Language' HTTP header to online processores 2022-08-01 17:01:59 +02:00
shared [fix] typo in uWSGI cache: searxcache --> searxngcache 2022-08-07 12:31:09 +02:00
static [build] /static 2022-07-08 09:04:57 +02:00
templates/simple [fix] improve OpenSearch description 2022-08-11 19:04:36 +02:00
translations [translations] update from Weblate 2022-08-12 07:18:14 +00:00
__init__.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
autocomplete.py [format.python] based on bugfix in 9ed626130 2022-05-07 18:23:10 +02:00
babel_extract.py [mod] move category and names of constants to searx/searxng.msg 2022-03-16 09:55:53 +01:00
compat.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
exceptions.py [pyright:strict] searx.search.checker.background 2022-01-27 22:07:12 +01:00
external_bang.py [fix] external bangs: don't overwrite Bangs in data trie 2022-01-12 19:37:13 +01:00
external_urls.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
flaskfix.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
languages.py [fix] add back missing languages & regions (followup of PR #1071) 2022-04-22 12:09:42 +02:00
locales.py [mod] add locale.get_engine_locale to get predictable results 2022-08-14 10:35:55 +02:00
preferences.py [fix] typo: add missing 'n' in center_aligment --> center_alignment 2022-07-03 17:35:54 +02:00
query.py [mod] add flags to the languages filter 2022-03-19 15:09:13 +01:00
redislib.py [mod] redislib - optimize LUA script registration. 2022-05-30 11:21:09 +02:00
results.py [simple theme] replace Image_layout.js with flexbox CS impl. 2022-02-26 22:31:47 +01:00
searxng.msg [mod] move category and names of constants to searx/searxng.msg 2022-03-16 09:55:53 +01:00
settings_defaults.py Merge pull request #1428 from return42/fix-center_aligment 2022-07-07 09:43:12 +02:00
settings_loader.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
settings.yml [mod] decouple qwant's categories from SearXNG's categories 2022-08-14 10:26:54 +02:00
unixthreadname.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
utils.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
version.py [fix] pyright repported errors 2022-07-30 18:04:44 +02:00
webadapter.py [fix] disabling all engines in a category makes the bang search in general 2022-01-03 20:23:46 +01:00
webapp.py [fix] improve OpenSearch description 2022-08-11 19:04:36 +02:00
webutils.py theme: remove __common__ 2022-05-07 19:40:48 +02:00