forked from zaclys/searxng
9ae409a05a
The match_language function sometimes returns incorrect results which is why a new function get_engine_locale is required. A bugfix of the match_language is not easily possible, because there is almost no documentation for it and already the call parameters are undefined. E.g. the function processes values like the ones from yahoo:: "yahoo": [ "ar", ... "zh_chs", "zh_cht" ] The get_engine_locale has been documented in detail, there is a clear description of the assumptions as well as the requirements and approximation rules (read doc-string for more details):: Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to corresponding *engine locales*: <engine>: { # SearXNG string : engine-string 'ca-ES' : 'ca_ES', 'fr-BE' : 'fr_BE', 'fr-CA' : 'fr_CA', 'fr-CH' : 'fr_CH', 'fr' : 'fr_FR', ... 'pl-PL' : 'pl_PL', 'pt-PT' : 'pt_PT' } .. hint:: The *SearXNG locale* string has to be known by babel! In the following you will find a comparison: >>> import babel.languages >>> from searx.utils import match_language >>> from searx.locales import get_engine_locale Assume we have an engine that supports the follwoing locales: >>> lang_list = { ... "zh-CN": "zh_CN", ... "zh-HK": "zh_HK", ... "nl-BE": "nl_BE", ... "fr-CA": "fr_CA", ... } Assumption: A. When a user selects a language the results should be optimized according to the selected language. B. When user selects a language and a territory the results should be optimized with first priority on territory and second on language. ---- Example: (Assumption A.) A user selects region 'zh-TW' which should end in zh_HK hint: CN is 'Hans' and HK ('Hant') fits better to TW ('Hant') >>> get_engine_locale('zh-TW', lang_list) 'zh_HK' >>> lang_list[match_language('zh-TW', lang_list)] 'zh_CN' ---- Example: (Assumption A.) A user selects only the language 'zh' which should end in CN >>> get_engine_locale('zh', lang_list) 'zh_CN' >>> lang_list[match_language('zh', lang_list)] 'zh_CN' ---- Example: (Assumption B.) A user selects region 'fr-BE' which should end in nl-BE hint: priority should be on the territory the user selected. If the user prefers 'fr' he will select 'fr' without a region tag. >>> get_engine_locale('fr-BE', lang_list, default='unknown') 'nl_BE' >>> match_language('fr-BE', lang_list, fallback='unknown') 'fr-CA' ---- Example: (Assumption A.) A user selects only the language 'fr' which should end in fr_CA >>> get_engine_locale('fr', lang_list) 'fr_CA' >>> lang_list[match_language('fr', lang_list)] 'fr_CA' ---- The difference in priority on the territory is best shown with a engine that supports the following locales: >>> lang_list = { ... "fr-FR": "fr_FR", ... "fr-CA": "fr_CA", ... "en-GB": "en_GB", ... "nl-BE": "nl_BE", ... } ---- Example: (Assumption A.) A user selects only a language >>> get_engine_locale('en', lang_list) 'en_GB' >>> match_language('en', lang_list) 'en-GB' hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR takes priority .. >>> get_engine_locale('fr', lang_list) 'fr_FR' >>> lang_list[match_language('fr', lang_list)] 'fr_FR' ---- Example: (Assumption B.) A user selects region 'fr-BE' which should end in nl-BE >>> get_engine_locale('fr-BE', lang_list) 'nl_BE' >>> lang_list[match_language('fr-BE', lang_list)] 'fr_FR' ---- If the user selects a language and there are two locales like the following: >>> lang_list = { ... "fr-BE": "fr_BE", ... "fr-CH": "fr_CH", ... } >>> >>> get_engine_locale('fr', lang_list) 'fr_BE' >>> lang_list[match_language('fr', lang_list)] 'fr_BE' Looks like both functions return the same value, but match_language depends on the order of the dictionary (which is not predictable): >>> lang_list = { ... "fr-CH": "fr_CH", ... "fr-BE": "fr_BE", ... } >>> get_engine_locale('fr', lang_list) 'fr_BE' >>> lang_list[match_language('fr', lang_list)] 'fr_CH' >>> The get_engine_locale selects the locale by looking at the "population percent" and this percentage has an higher amount in BE (68.%) compared to CH (21%) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> |
||
---|---|---|
.. | ||
answerers | ||
data | ||
engines | ||
infopage | ||
metrics | ||
network | ||
plugins | ||
search | ||
shared | ||
static | ||
templates/simple | ||
translations | ||
__init__.py | ||
autocomplete.py | ||
babel_extract.py | ||
compat.py | ||
exceptions.py | ||
external_bang.py | ||
external_urls.py | ||
flaskfix.py | ||
languages.py | ||
locales.py | ||
preferences.py | ||
query.py | ||
redislib.py | ||
results.py | ||
searxng.msg | ||
settings_defaults.py | ||
settings_loader.py | ||
settings.yml | ||
unixthreadname.py | ||
utils.py | ||
version.py | ||
webadapter.py | ||
webapp.py | ||
webutils.py |