Alexandre Flament 931e1f6503 searx.utils.detect_language : add model from fastlangid ...

The project fastlangid has a FastText model to achieve higher accuracy
for Japanese, Korean, Chinese languages.

This commits includes the model from this project,
and change searx.utils.detect_language to rely on that model
as a second stage.

The model is lazy loaded when the base model detect Japanese, Korean or Chinese.

https://github.com/currentslab/fastlangid

2023-09-17 09:31:47 +00:00

386 KiB

Raw History

View raw