[mod] add locale.get_engine_locale to get predictable results

The match_language function sometimes returns incorrect results which is why a
new function get_engine_locale is required.

A bugfix of the match_language is not easily possible, because there is almost
no documentation for it and already the call parameters are undefined.  E.g. the
function processes values like the ones from yahoo::

    "yahoo": [
        "ar",
        ...
        "zh_chs",
        "zh_cht"
     ]

The get_engine_locale has been documented in detail, there is a clear
description of the assumptions as well as the requirements and approximation
rules (read doc-string for more details)::

    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
    corresponding *engine locales*:

      <engine>: {
          # SearXNG string : engine-string
          'ca-ES'          : 'ca_ES',
          'fr-BE'          : 'fr_BE',
          'fr-CA'          : 'fr_CA',
          'fr-CH'          : 'fr_CH',
          'fr'             : 'fr_FR',
          ...
          'pl-PL'          : 'pl_PL',
          'pt-PT'          : 'pt_PT'
      }

    .. hint::

       The *SearXNG locale* string has to be known by babel!

In the following you will find a comparison:

>>> import babel.languages
>>> from searx.utils import match_language
>>> from searx.locales import get_engine_locale

Assume we have an engine that supports the follwoing locales:

>>> lang_list = {
...     "zh-CN": "zh_CN",
...     "zh-HK": "zh_HK",
...     "nl-BE": "nl_BE",
...     "fr-CA": "fr_CA",
... }

Assumption:

  A. When a user selects a language the results should be optimized according to
     the selected language.

  B. When user selects a language and a territory the results should be
     optimized with first priority on territory and second on language.

----

Example: (Assumption A.)

  A user selects region 'zh-TW' which should end in zh_HK

hint:
  CN is 'Hans' and HK ('Hant') fits better to TW ('Hant')

>>> get_engine_locale('zh-TW', lang_list)
'zh_HK'
>>> lang_list[match_language('zh-TW', lang_list)]
'zh_CN'

----

Example: (Assumption A.)

  A user selects only the language 'zh' which should end in CN

>>> get_engine_locale('zh', lang_list)
'zh_CN'
>>> lang_list[match_language('zh', lang_list)]
'zh_CN'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

hint:
  priority should be on the territory the user selected.  If the user
  prefers 'fr' he will select 'fr' without a region tag.

>>> get_engine_locale('fr-BE', lang_list, default='unknown')
'nl_BE'
>>> match_language('fr-BE', lang_list, fallback='unknown')
'fr-CA'

----

Example: (Assumption A.)

  A user selects only the language 'fr' which should end in fr_CA

>>> get_engine_locale('fr', lang_list)
'fr_CA'
>>> lang_list[match_language('fr', lang_list)]
'fr_CA'

----

The difference in priority on the territory is best shown with a engine that
supports the following locales:

>>> lang_list = {
...     "fr-FR": "fr_FR",
...     "fr-CA": "fr_CA",
...     "en-GB": "en_GB",
...     "nl-BE": "nl_BE",
... }

----

Example: (Assumption A.)

   A user selects only a language

>>> get_engine_locale('en', lang_list)
'en_GB'
>>> match_language('en', lang_list)
'en-GB'

hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..

>>> get_engine_locale('fr', lang_list)
'fr_FR'
>>> lang_list[match_language('fr', lang_list)]
'fr_FR'

----

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

>>> get_engine_locale('fr-BE', lang_list)
'nl_BE'
>>> lang_list[match_language('fr-BE', lang_list)]
'fr_FR'

----

If the user selects a language and there are two locales like the following:

>>> lang_list = {
...      "fr-BE": "fr_BE",
...      "fr-CH": "fr_CH",
...  }
>>>

>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_BE'

Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):

>>> lang_list = {
...      "fr-CH": "fr_CH",
...      "fr-BE": "fr_BE",
...  }
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_CH'
>>>

The get_engine_locale selects the locale by looking at the "population percent"
and this percentage has an higher amount in BE (68.%) compared to CH (21%)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

searx/locales.py

@@ -10,6 +10,8 @@ import pathlib
 
 from babel import Locale
 from babel.support import Translations
+import babel.languages
+import babel.core
 import flask_babel
 import flask
 from flask.ctx import has_request_context
@@ -150,3 +152,138 @@ def locales_initialize(directory=None):
             LOCALE_NAMES[tag] = get_locale_descr(locale, dirname)
             if locale.text_direction == 'rtl':
                 RTL_LOCALES.add(tag)
+
+
+def get_engine_locale(searxng_locale, engine_locales, default=None):
+    """Return engine's language (aka locale) string that best fits to argument
+    ``searxng_locale``.
+
+    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
+    corresponding *engine locales*:
+
+      <engine>: {
+          # SearXNG string : engine-string
+          'ca-ES'          : 'ca_ES',
+          'fr-BE'          : 'fr_BE',
+          'fr-CA'          : 'fr_CA',
+          'fr-CH'          : 'fr_CH',
+          'fr'             : 'fr_FR',
+          ...
+          'pl-PL'          : 'pl_PL',
+          'pt-PT'          : 'pt_PT'
+      }
+
+    .. hint::
+
+       The *SearXNG locale* string has to be known by babel!
+
+    If there is no direct 1:1 mapping, this functions tries to narrow down
+    engine's language (locale).  If no value can be determined by these
+    approximation attempts the ``default`` value is returned.
+
+    Assumptions:
+
+    A. When user select a language the results should be optimized according to
+       the selected language.
+
+    B. When user select a language and a territory the results should be
+       optimized with first priority on terrirtory and second on language.
+
+    First approximation rule (*by territory*):
+
+      When the user selects a locale with terrirtory (and a language), the
+      territory has priority over the language.  If any of the offical languages
+      in the terrirtory is supported by the engine (``engine_locales``) it will
+      be used.
+
+    Second approximation rule (*by language*):
+
+      If "First approximation rule" brings no result or the user selects only a
+      language without a terrirtory.  Check in which territories the language
+      has an offical status and if one of these territories is supported by the
+      engine.
+
+    """
+    # pylint: disable=too-many-branches
+
+    engine_locale = engine_locales.get(searxng_locale)
+
+    if engine_locale is not None:
+        # There was a 1:1 mapping (e.g. "fr-BE --> fr_BE" or "fr --> fr_FR"), no
+        # need to narrow language nor territory.
+        return engine_locale
+
+    locale = babel.Locale.parse(searxng_locale, sep='-')
+
+    # SearXNG's selected locale is not supported by the engine ..
+
+    if locale.territory:
+        # Try to narrow by *offical* languages in the territory (??-XX).
+
+        for official_language in babel.languages.get_official_languages(locale.territory, de_facto=True):
+            searxng_locale = official_language + '-' + locale.territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+    # Engine does not support one of the offical languages in the territory or
+    # there is only a language selected without a territory.
+
+    # Now lets have a look if the searxng_lang (the language selected by the
+    # user) is a offical language in other territories.  If so, check if
+    # engine does support the searxng_lang in this other territory.
+
+    if locale.language:
+
+        searxng_lang = locale.language
+        if locale.script:
+            searxng_lang += '_' + locale.script
+
+        terr_lang_dict = {}
+        for territory, langs in babel.core.get_global("territory_languages").items():
+            if not langs.get(searxng_lang, {}).get('official_status'):
+                continue
+            terr_lang_dict[territory] = langs.get(searxng_lang)
+
+        # first: check fr-FR, de-DE .. is supported by the engine
+
+        territory = locale.language.upper()
+        if terr_lang_dict.get(territory):
+            searxng_locale = locale.language + '-' + territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+        # second: sort by population_percent and take first match
+
+        # drawback of "population percent": if there is a terrirtory with a
+        #   small number of people (e.g 100) but the majority speaks the
+        #   language, then the percentage migth be 100% (--> 100 people) but in
+        #   a different terrirtory with more people (e.g. 10.000) where only 10%
+        #   speak the language the total amount of speaker is higher (--> 200
+        #   people).
+        #
+        #   By example: The population of Saint-Martin is 33.000, of which 100%
+        #   speak French, but this is less than the 30% of the approximately 2.5
+        #   million Belgian citizens
+        #
+        #   - 'fr-MF', 'population_percent': 100.0, 'official_status': 'official'
+        #   - 'fr-BE', 'population_percent': 38.0, 'official_status': 'official'
+
+        terr_lang_list = []
+        for k, v in terr_lang_dict.items():
+            terr_lang_list.append((k, v))
+
+        for territory, _lang in sorted(terr_lang_list, key=lambda item: item[1]['population_percent'], reverse=True):
+            searxng_locale = locale.language + '-' + territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+    # No luck: narrow by "language from territory" and "territory from language"
+    # does not fit to a locale supported by the engine.
+
+    if engine_locale is None:
+        engine_locale = default
+
+    return default