zaclys/searxng
1
0
Fork 1
mirror of https://github.com/searxng/searxng synced 2024-01-01 19:24:07 +01:00
Code Issues Projects Releases Packages Wiki Activity

[refactor] typification of SearXNG (initial) / result items (part 1)

Browse source 
Typification of SearXNG
=======================

This patch introduces the typing of the results.  The why and how is described
in the documentation, please generate the documentation ..

    $ make docs.clean docs.live

and read the following articles in the "Developer documentation":

- result types --> http://0.0.0.0:8000/dev/result_types/index.html

The result types are available from the `searx.result_types` module.  The
following have been implemented so far:

- base result type: `searx.result_type.Result`
  --> http://0.0.0.0:8000/dev/result_types/base_result.html

- answer results
  --> http://0.0.0.0:8000/dev/result_types/answer.html

including the type for translations (inspired by #3925).  For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.

Doc of the fields used in Templates
===================================

The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development).  It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.

- https://github.com/searxng/searxng/issues/357

Answer Templates
================

With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).

    !en-de dog

Plugins & Answerer
==================

The implementation for `Plugin` and `Answer` has been revised, see
documentation:

- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html

With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)

Autocomplete
============

The autocompletion had a bug where the results from `Answer` had not been shown
in the past.  To test activate autocompletion and try search terms for which we
have answerers

- statistics: type `min 1 2 3` .. in the completion list you should find an
  entry like `[de] min(1, 2, 3) = 1`

- random: type `random uuid` .. in the completion list, the first item is a
  random UUID

Extended Types
==============

SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:

- Extended Types
  --> http://0.0.0.0:8000/dev/extended_types.html

Unit-Tests
==========

The unit tests have been completely revised.  In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it.  This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:

- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
  issues, with non-deterministic behavior ..

- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005

Why msgspec.Struct
==================

We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:

- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356

In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...

The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.

Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!

Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.

HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec.  They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.

Why just one commit?
====================

I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2024-12-15 09:59:50 +01:00
parent f29b2b113a
commit e4e7d73e8e
143 changed files with 3846 additions and 2100 deletions
Download patch file Download diff file Expand all files Collapse all files

77
searx/__init__.py
View file

@ -9,8 +9,7 @@ import logging
import searx.unixthreadname
import searx.settings_loader
from searx.settings_defaults import settings_set_defaults
from searx.settings_defaults import SCHEMA, apply_schema
# Debug
LOG_FORMAT_DEBUG = '%(levelname)-7s %(name)-30.30s: %(message)s'
@ -21,14 +20,52 @@ LOG_LEVEL_PROD = logging.WARNING
searx_dir = abspath(dirname(__file__))
searx_parent_dir = abspath(dirname(dirname(__file__)))
settings, settings_load_message = searx.settings_loader.load_settings()
if settings is not None:
settings = settings_set_defaults(settings)
settings = {}
searx_debug = False
logger = logging.getLogger('searx')
_unset = object()
def init_settings():
"""Initialize global ``settings`` and ``searx_debug`` variables and
``logger`` from ``SEARXNG_SETTINGS_PATH``.
"""
global settings, searx_debug # pylint: disable=global-variable-not-assigned
cfg, msg = searx.settings_loader.load_settings(load_user_settings=True)
cfg = cfg or {}
apply_schema(cfg, SCHEMA, [])
settings.clear()
settings.update(cfg)
searx_debug = settings['general']['debug']
if searx_debug:
_logging_config_debug()
else:
logging.basicConfig(level=LOG_LEVEL_PROD, format=LOG_FORMAT_PROD)
logging.root.setLevel(level=LOG_LEVEL_PROD)
logging.getLogger('werkzeug').setLevel(level=LOG_LEVEL_PROD)
logger.info(msg)
# log max_request_timeout
max_request_timeout = settings['outgoing']['max_request_timeout']
if max_request_timeout is None:
logger.info('max_request_timeout=%s', repr(max_request_timeout))
else:
logger.info('max_request_timeout=%i second(s)', max_request_timeout)
if settings['server']['public_instance']:
logger.warning(
"Be aware you have activated features intended only for public instances. "
"This force the usage of the limiter and link_token / "
"see https://docs.searxng.org/admin/searx.limiter.html"
)
def get_setting(name, default=_unset):
"""Returns the value to which ``name`` point. If there is no such name in the
settings and the ``default`` is unset, a :py:obj:`KeyError` is raised.
@ -50,20 +87,20 @@ def get_setting(name, default=_unset):
return value
def is_color_terminal():
def _is_color_terminal():
if os.getenv('TERM') in ('dumb', 'unknown'):
return False
return sys.stdout.isatty()
def logging_config_debug():
def _logging_config_debug():
try:
import coloredlogs # pylint: disable=import-outside-toplevel
except ImportError:
coloredlogs = None
log_level = os.environ.get('SEARXNG_DEBUG_LOG_LEVEL', 'DEBUG')
if coloredlogs and is_color_terminal():
if coloredlogs and _is_color_terminal():
level_styles = {
'spam': {'color': 'green', 'faint': True},
'debug': {},
@ -87,26 +124,4 @@ def logging_config_debug():
logging.basicConfig(level=logging.getLevelName(log_level), format=LOG_FORMAT_DEBUG)
searx_debug = settings['general']['debug']
if searx_debug:
logging_config_debug()
else:
logging.basicConfig(level=LOG_LEVEL_PROD, format=LOG_FORMAT_PROD)
logging.root.setLevel(level=LOG_LEVEL_PROD)
logging.getLogger('werkzeug').setLevel(level=LOG_LEVEL_PROD)
logger = logging.getLogger('searx')
logger.info(settings_load_message)
# log max_request_timeout
max_request_timeout = settings['outgoing']['max_request_timeout']
if max_request_timeout is None:
logger.info('max_request_timeout=%s', repr(max_request_timeout))
else:
logger.info('max_request_timeout=%i second(s)', max_request_timeout)
if settings['server']['public_instance']:
logger.warning(
"Be aware you have activated features intended only for public instances. "
"This force the usage of the limiter and link_token / "
"see https://docs.searxng.org/admin/searx.limiter.html"
)
init_settings()

86
searx/answerers/__init__.py
View file

@ -1,51 +1,49 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
"""The *answerers* give instant answers related to the search query, they
usually provide answers of type :py:obj:`Answer <searx.result_types.Answer>`.
import sys
from os import listdir
from os.path import realpath, dirname, join, isdir
from collections import defaultdict
Here is an example of a very simple answerer that adds a "Hello" into the answer
area:
from searx.utils import load_module
.. code::
answerers_dir = dirname(realpath(__file__))
from flask_babel import gettext as _
from searx.answerers import Answerer
from searx.result_types import Answer
class MyAnswerer(Answerer):
keywords = [ "hello", "hello world" ]
def info(self):
return AnswererInfo(name=_("Hello"), description=_("lorem .."), keywords=self.keywords)
def answer(self, request, search):
return [ Answer(answer="Hello") ]
----
.. autoclass:: Answerer
:members:
.. autoclass:: AnswererInfo
:members:
.. autoclass:: AnswerStorage
:members:
.. autoclass:: searx.answerers._core.ModuleAnswerer
:members:
:show-inheritance:
"""
from __future__ import annotations
__all__ = ["AnswererInfo", "Answerer", "AnswerStorage"]
def load_answerers():
answerers = [] # pylint: disable=redefined-outer-name
from ._core import AnswererInfo, Answerer, AnswerStorage
for filename in listdir(answerers_dir):
if not isdir(join(answerers_dir, filename)) or filename.startswith('_'):
continue
module = load_module('answerer.py', join(answerers_dir, filename))
if not hasattr(module, 'keywords') or not isinstance(module.keywords, tuple) or not module.keywords:
sys.exit(2)
answerers.append(module)
return answerers
def get_answerers_by_keywords(answerers): # pylint:disable=redefined-outer-name
by_keyword = defaultdict(list)
for answerer in answerers:
for keyword in answerer.keywords:
for keyword in answerer.keywords:
by_keyword[keyword].append(answerer.answer)
return by_keyword
def ask(query):
results = []
query_parts = list(filter(None, query.query.split()))
if not query_parts or query_parts[0] not in answerers_by_keywords:
return results
for answerer in answerers_by_keywords[query_parts[0]]:
result = answerer(query)
if result:
results.append(result)
return results
answerers = load_answerers()
answerers_by_keywords = get_answerers_by_keywords(answerers)
STORAGE: AnswerStorage = AnswerStorage()
STORAGE.load_builtins()

169
searx/answerers/_core.py Normal file
View file

@ -0,0 +1,169 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=too-few-public-methods, missing-module-docstring
from __future__ import annotations
import abc
import importlib
import logging
import pathlib
import warnings
from dataclasses import dataclass
from searx.utils import load_module
from searx.result_types.answer import BaseAnswer
_default = pathlib.Path(__file__).parent
log: logging.Logger = logging.getLogger("searx.answerers")
@dataclass
class AnswererInfo:
"""Object that holds informations about an answerer, these infos are shown
to the user in the Preferences menu.
To be able to translate the information into other languages, the text must
be written in English and translated with :py:obj:`flask_babel.gettext`.
"""
name: str
"""Name of the *answerer*."""
description: str
"""Short description of the *answerer*."""
examples: list[str]
"""List of short examples of the usage / of query terms."""
keywords: list[str]
"""See :py:obj:`Answerer.keywords`"""
class Answerer(abc.ABC):
"""Abstract base class of answerers."""
keywords: list[str]
"""Keywords to which the answerer has *answers*."""
@abc.abstractmethod
def answer(self, query: str) -> list[BaseAnswer]:
"""Function that returns a list of answers to the question/query."""
@abc.abstractmethod
def info(self) -> AnswererInfo:
"""Informations about the *answerer*, see :py:obj:`AnswererInfo`."""
class ModuleAnswerer(Answerer):
"""A wrapper class for legacy *answerers* where the names (keywords, answer,
info) are implemented on the module level (not in a class).
.. note::
For internal use only!
"""
def __init__(self, mod):
for name in ["keywords", "self_info", "answer"]:
if not getattr(mod, name, None):
raise SystemExit(2)
if not isinstance(mod.keywords, tuple):
raise SystemExit(2)
self.module = mod
self.keywords = mod.keywords # type: ignore
def answer(self, query: str) -> list[BaseAnswer]:
return self.module.answer(query)
def info(self) -> AnswererInfo:
kwargs = self.module.self_info()
kwargs["keywords"] = self.keywords
return AnswererInfo(**kwargs)
class AnswerStorage(dict):
"""A storage for managing the *answerers* of SearXNG. With the
:py:obj:`AnswerStorage.ask`” method, a caller can ask questions to all
*answerers* and receives a list of the results."""
answerer_list: set[Answerer]
"""The list of :py:obj:`Answerer` in this storage."""
def __init__(self):
super().__init__()
self.answerer_list = set()
def load_builtins(self):
"""Loads ``answerer.py`` modules from the python packages in
:origin:`searx/answerers`. The python modules are wrapped by
:py:obj:`ModuleAnswerer`."""
for f in _default.iterdir():
if f.name.startswith("_"):
continue
if f.is_file() and f.suffix == ".py":
self.register_by_fqn(f"searx.answerers.{f.stem}.SXNGAnswerer")
continue
# for backward compatibility (if a fork has additional answerers)
if f.is_dir() and (f / "answerer.py").exists():
warnings.warn(
f"answerer module {f} is deprecated / migrate to searx.answerers.Answerer", DeprecationWarning
)
mod = load_module("answerer.py", str(f))
self.register(ModuleAnswerer(mod))
def register_by_fqn(self, fqn: str):
"""Register a :py:obj:`Answerer` via its fully qualified class namen(FQN)."""
mod_name, _, obj_name = fqn.rpartition('.')
mod = importlib.import_module(mod_name)
code_obj = getattr(mod, obj_name, None)
if code_obj is None:
msg = f"answerer {fqn} is not implemented"
log.critical(msg)
raise ValueError(msg)
self.register(code_obj())
def register(self, answerer: Answerer):
"""Register a :py:obj:`Answerer`."""
self.answerer_list.add(answerer)
for _kw in answerer.keywords:
self[_kw] = self.get(_kw, [])
self[_kw].append(answerer)
def ask(self, query: str) -> list[BaseAnswer]:
"""An answerer is identified via keywords, if there is a keyword at the
first position in the ``query`` for which there is one or more
answerers, then these are called, whereby the entire ``query`` is passed
as argument to the answerer function."""
results = []
keyword = None
for keyword in query.split():
if keyword:
break
if not keyword or keyword not in self:
return results
for answerer in self[keyword]:
for answer in answerer.answer(query):
# In case of *answers* prefix ``answerer:`` is set, see searx.result_types.Result
answer.engine = f"answerer: {keyword}"
results.append(answer)
return results
@property
def info(self) -> list[AnswererInfo]:
return [a.info() for a in self.answerer_list]

80
searx/answerers/random.py Normal file
View file

@ -0,0 +1,80 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
import hashlib
import random
import string
import uuid
from flask_babel import gettext
from searx.result_types import Answer
from searx.result_types.answer import BaseAnswer
from . import Answerer, AnswererInfo
def random_characters():
random_string_letters = string.ascii_lowercase + string.digits + string.ascii_uppercase
return [random.choice(random_string_letters) for _ in range(random.randint(8, 32))]
def random_string():
return ''.join(random_characters())
def random_float():
return str(random.random())
def random_int():
random_int_max = 2**31
return str(random.randint(-random_int_max, random_int_max))
def random_sha256():
m = hashlib.sha256()
m.update(''.join(random_characters()).encode())
return str(m.hexdigest())
def random_uuid():
return str(uuid.uuid4())
def random_color():
color = "%06x" % random.randint(0, 0xFFFFFF)
return f"#{color.upper()}"
class SXNGAnswerer(Answerer):
"""Random value generator"""
keywords = ["random"]
random_types = {
"string": random_string,
"int": random_int,
"float": random_float,
"sha256": random_sha256,
"uuid": random_uuid,
"color": random_color,
}
def info(self):
return AnswererInfo(
name=gettext(self.__doc__),
description=gettext("Generate different random values"),
keywords=self.keywords,
examples=[f"random {x}" for x in self.random_types],
)
def answer(self, query: str) -> list[BaseAnswer]:
parts = query.split()
if len(parts) != 2 or parts[1] not in self.random_types:
return []
return [Answer(answer=self.random_types[parts[1]]())]

2
searx/answerers/random/__init__.py
View file

@ -1,2 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring

79
searx/answerers/random/answerer.py
View file

@ -1,79 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
import hashlib
import random
import string
import uuid
from flask_babel import gettext
# required answerer attribute
# specifies which search query keywords triggers this answerer
keywords = ('random',)
random_int_max = 2**31
random_string_letters = string.ascii_lowercase + string.digits + string.ascii_uppercase
def random_characters():
return [random.choice(random_string_letters) for _ in range(random.randint(8, 32))]
def random_string():
return ''.join(random_characters())
def random_float():
return str(random.random())
def random_int():
return str(random.randint(-random_int_max, random_int_max))
def random_sha256():
m = hashlib.sha256()
m.update(''.join(random_characters()).encode())
return str(m.hexdigest())
def random_uuid():
return str(uuid.uuid4())
def random_color():
color = "%06x" % random.randint(0, 0xFFFFFF)
return f"#{color.upper()}"
random_types = {
'string': random_string,
'int': random_int,
'float': random_float,
'sha256': random_sha256,
'uuid': random_uuid,
'color': random_color,
}
# required answerer function
# can return a list of results (any result type) for a given query
def answer(query):
parts = query.query.split()
if len(parts) != 2:
return []
if parts[1] not in random_types:
return []
return [{'answer': random_types[parts[1]]()}]
# required answerer function
# returns information about the answerer
def self_info():
return {
'name': gettext('Random value generator'),
'description': gettext('Generate different random values'),
'examples': ['random {}'.format(x) for x in random_types],
}

64
searx/answerers/statistics.py Normal file
View file

@ -0,0 +1,64 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
from functools import reduce
from operator import mul
import babel
import babel.numbers
from flask_babel import gettext
from searx.extended_types import sxng_request
from searx.result_types import Answer
from searx.result_types.answer import BaseAnswer
from . import Answerer, AnswererInfo
kw2func = [
("min", min),
("max", max),
("avg", lambda args: sum(args) / len(args)),
("sum", sum),
("prod", lambda args: reduce(mul, args, 1)),
]
class SXNGAnswerer(Answerer):
"""Statistics functions"""
keywords = [kw for kw, _ in kw2func]
def info(self):
return AnswererInfo(
name=gettext(self.__doc__),
description=gettext(f"Compute {'/'.join(self.keywords)} of the arguments"),
keywords=self.keywords,
examples=["avg 123 548 2.04 24.2"],
)
def answer(self, query: str) -> list[BaseAnswer]:
results = []
parts = query.split()
if len(parts) < 2:
return results
ui_locale = babel.Locale.parse(sxng_request.preferences.get_value('locale'), sep='-')
try:
args = [babel.numbers.parse_decimal(num, ui_locale, numbering_system="latn") for num in parts[1:]]
except: # pylint: disable=bare-except
# seems one of the args is not a float type, can't be converted to float
return results
for k, func in kw2func:
if k == parts[0]:
res = func(args)
res = babel.numbers.format_decimal(res, locale=ui_locale)
f_str = ', '.join(babel.numbers.format_decimal(arg, locale=ui_locale) for arg in args)
results.append(Answer(answer=f"[{ui_locale}] {k}({f_str}) = {res} "))
break
return results

2
searx/answerers/statistics/__init__.py
View file

@ -1,2 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring

53
searx/answerers/statistics/answerer.py
View file

@ -1,53 +0,0 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from functools import reduce
from operator import mul
from flask_babel import gettext
keywords = ('min', 'max', 'avg', 'sum', 'prod')
# required answerer function
# can return a list of results (any result type) for a given query
def answer(query):
parts = query.query.split()
if len(parts) < 2:
return []
try:
args = list(map(float, parts[1:]))
except: # pylint: disable=bare-except
return []
func = parts[0]
_answer = None
if func == 'min':
_answer = min(args)
elif func == 'max':
_answer = max(args)
elif func == 'avg':
_answer = sum(args) / len(args)
elif func == 'sum':
_answer = sum(args)
elif func == 'prod':
_answer = reduce(mul, args, 1)
if _answer is None:
return []
return [{'answer': str(_answer)}]
# required answerer function
# returns information about the answerer
def self_info():
return {
'name': gettext('Statistics functions'),
'description': gettext('Compute {functions} of the arguments').format(functions='/'.join(keywords)),
'examples': ['avg 123 548 2.04 24.2'],
}

12
searx/autocomplete.py
View file

@ -8,9 +8,11 @@ import json
import html
from urllib.parse import urlencode, quote_plus
import lxml
import lxml.etree
import lxml.html
from httpx import HTTPError
from searx.extended_types import SXNG_Response
from searx import settings
from searx.engines import (
engines,
@ -26,12 +28,12 @@ def update_kwargs(**kwargs):
kwargs['raise_for_httperror'] = True
def get(*args, **kwargs):
def get(*args, **kwargs) -> SXNG_Response:
update_kwargs(**kwargs)
return http_get(*args, **kwargs)
def post(*args, **kwargs):
def post(*args, **kwargs) -> SXNG_Response:
update_kwargs(**kwargs)
return http_post(*args, **kwargs)
@ -111,7 +113,7 @@ def google_complete(query, sxng_locale):
)
results = []
resp = get(url.format(subdomain=google_info['subdomain'], args=args))
if resp.ok:
if resp and resp.ok:
json_txt = resp.text[resp.text.find('[') : resp.text.find(']', -3) + 1]
data = json.loads(json_txt)
for item in data[0]:
@ -205,7 +207,7 @@ def wikipedia(query, sxng_locale):
results = []
eng_traits = engines['wikipedia'].traits
wiki_lang = eng_traits.get_language(sxng_locale, 'en')
wiki_netloc = eng_traits.custom['wiki_netloc'].get(wiki_lang, 'en.wikipedia.org')
wiki_netloc = eng_traits.custom['wiki_netloc'].get(wiki_lang, 'en.wikipedia.org') # type: ignore
url = 'https://{wiki_netloc}/w/api.php?{args}'
args = urlencode(

6
searx/botdetection/_helpers.py
View file

@ -13,12 +13,14 @@ import flask
import werkzeug
from searx import logger
from searx.extended_types import SXNG_Request
from . import config
logger = logger.getChild('botdetection')
def dump_request(request: flask.Request):
def dump_request(request: SXNG_Request):
return (
request.path
+ " || X-Forwarded-For: %s" % request.headers.get('X-Forwarded-For')
@ -66,7 +68,7 @@ def _log_error_only_once(err_msg):
_logged_errors.append(err_msg)
def get_real_ip(request: flask.Request) -> str:
def get_real_ip(request: SXNG_Request) -> str:
"""Returns real IP of the request. Since not all proxies set all the HTTP
headers and incoming headers can be faked it may happen that the IP cannot
be determined correctly.

8
searx/botdetection/http_accept.py
View file

@ -12,7 +12,6 @@ Accept_ header ..
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
"""
# pylint: disable=unused-argument
from __future__ import annotations
from ipaddress import (
@ -20,17 +19,18 @@ from ipaddress import (
IPv6Network,
)
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from . import config
from ._helpers import too_many_requests
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
cfg: config.Config,
request: SXNG_Request,
cfg: config.Config, # pylint: disable=unused-argument
) -> werkzeug.Response | None:
if 'text/html' not in request.accept_mimetypes:

8
searx/botdetection/http_accept_encoding.py
View file

@ -13,7 +13,6 @@ bot if the Accept-Encoding_ header ..
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
"""
# pylint: disable=unused-argument
from __future__ import annotations
from ipaddress import (
@ -21,17 +20,18 @@ from ipaddress import (
IPv6Network,
)
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from . import config
from ._helpers import too_many_requests
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
cfg: config.Config,
request: SXNG_Request,
cfg: config.Config, # pylint: disable=unused-argument
) -> werkzeug.Response | None:
accept_list = [l.strip() for l in request.headers.get('Accept-Encoding', '').split(',')]

9
searx/botdetection/http_accept_language.py
View file

@ -10,24 +10,25 @@ if the Accept-Language_ header is unset.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
"""
# pylint: disable=unused-argument
from __future__ import annotations
from ipaddress import (
IPv4Network,
IPv6Network,
)
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from . import config
from ._helpers import too_many_requests
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
cfg: config.Config,
request: SXNG_Request,
cfg: config.Config, # pylint: disable=unused-argument
) -> werkzeug.Response | None:
if request.headers.get('Accept-Language', '').strip() == '':
return too_many_requests(network, "missing HTTP header Accept-Language")

8
searx/botdetection/http_connection.py
View file

@ -10,7 +10,6 @@ the Connection_ header is set to ``close``.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection
"""
# pylint: disable=unused-argument
from __future__ import annotations
from ipaddress import (
@ -18,17 +17,18 @@ from ipaddress import (
IPv6Network,
)
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from . import config
from ._helpers import too_many_requests
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
cfg: config.Config,
request: SXNG_Request,
cfg: config.Config, # pylint: disable=unused-argument
) -> werkzeug.Response | None:
if request.headers.get('Connection', '').strip() == 'close':

8
searx/botdetection/http_user_agent.py
View file

@ -11,7 +11,6 @@ the User-Agent_ header is unset or matches the regular expression
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
"""
# pylint: disable=unused-argument
from __future__ import annotations
import re
@ -20,9 +19,10 @@ from ipaddress import (
IPv6Network,
)
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from . import config
from ._helpers import too_many_requests
@ -56,8 +56,8 @@ def regexp_user_agent():
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
cfg: config.Config,
request: SXNG_Request,
cfg: config.Config, # pylint: disable=unused-argument
) -> werkzeug.Response | None:
user_agent = request.headers.get('User-Agent', 'unknown')

3
searx/botdetection/ip_limit.py
View file

@ -45,6 +45,7 @@ from ipaddress import (
import flask
import werkzeug
from searx.extended_types import SXNG_Request
from searx import redisdb
from searx.redislib import incr_sliding_window, drop_counter
@ -91,7 +92,7 @@ SUSPICIOUS_IP_MAX = 3
def filter_request(
network: IPv4Network | IPv6Network,
request: flask.Request,
request: SXNG_Request,
cfg: config.Config,
) -> werkzeug.Response | None:

8
searx/botdetection/link_token.py
View file

@ -43,11 +43,11 @@ from ipaddress import (
import string
import random
import flask
from searx import logger
from searx import redisdb
from searx.redislib import secret_hash
from searx.extended_types import SXNG_Request
from ._helpers import (
get_network,
@ -69,7 +69,7 @@ TOKEN_KEY = 'SearXNG_limiter.token'
logger = logger.getChild('botdetection.link_token')
def is_suspicious(network: IPv4Network | IPv6Network, request: flask.Request, renew: bool = False):
def is_suspicious(network: IPv4Network | IPv6Network, request: SXNG_Request, renew: bool = False):
"""Checks whether a valid ping is exists for this (client) network, if not
this request is rated as *suspicious*. If a valid ping exists and argument
``renew`` is ``True`` the expire time of this ping is reset to
@ -92,7 +92,7 @@ def is_suspicious(network: IPv4Network | IPv6Network, request: flask.Request, re
return False
def ping(request: flask.Request, token: str):
def ping(request: SXNG_Request, token: str):
"""This function is called by a request to URL ``/client<token>.css``. If
``token`` is valid a :py:obj:`PING_KEY` for the client is stored in the DB.
The expire time of this ping-key is :py:obj:`PING_LIVE_TIME`.
@ -113,7 +113,7 @@ def ping(request: flask.Request, token: str):
redis_client.set(ping_key, 1, ex=PING_LIVE_TIME)
def get_ping_key(network: IPv4Network | IPv6Network, request: flask.Request) -> str:
def get_ping_key(network: IPv4Network | IPv6Network, request: SXNG_Request) -> str:
"""Generates a hashed key that fits (more or less) to a *WEB-browser
session* in a network."""
return (

7
searx/engines/brave.py
View file

@ -139,6 +139,7 @@ from searx.utils import (
get_embeded_stream_url,
)
from searx.enginelib.traits import EngineTraits
from searx.result_types import Answer
if TYPE_CHECKING:
import logging
@ -274,10 +275,14 @@ def _parse_search(resp):
result_list = []
dom = html.fromstring(resp.text)
# I doubt that Brave is still providing the "answer" class / I haven't seen
# answers in brave for a long time.
answer_tag = eval_xpath_getindex(dom, '//div[@class="answer"]', 0, default=None)
if answer_tag:
url = eval_xpath_getindex(dom, '//div[@id="featured_snippet"]/a[@class="result-header"]/@href', 0, default=None)
result_list.append({'answer': extract_text(answer_tag), 'url': url})
answer = extract_text(answer_tag)
if answer is not None:
Answer(results=result_list, answer=answer, url=url)
# xpath_results = '//div[contains(@class, "snippet fdb") and @data-type="web"]'
xpath_results = '//div[contains(@class, "snippet ")]'

7
searx/engines/deepl.py
View file

@ -1,6 +1,8 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""Deepl translation engine"""
from searx.result_types import Translations
about = {
"website": 'https://deepl.com',
"wikidata_id": 'Q43968444',
@ -45,8 +47,7 @@ def response(resp):
if not result.get('translations'):
return results
translations = [{'text': translation['text']} for translation in result['translations']]
results.append({'answer': translations[0]['text'], 'answer_type': 'translations', 'translations': translations})
translations = [Translations.Item(text=t['text']) for t in result['translations']]
Translations(results=results, translations=translations)
return results

95
searx/engines/dictzone.py
View file

@ -3,8 +3,12 @@
Dictzone
"""
import urllib.parse
from lxml import html
from searx.utils import eval_xpath
from searx.utils import eval_xpath, extract_text
from searx.result_types import Translations
from searx.network import get as http_get # https://github.com/searxng/searxng/issues/762
# about
about = {
@ -18,46 +22,83 @@ about = {
engine_type = 'online_dictionary'
categories = ['general', 'translate']
url = 'https://dictzone.com/{from_lang}-{to_lang}-dictionary/{query}'
base_url = "https://dictzone.com"
weight = 100
results_xpath = './/table[@id="r"]/tr'
https_support = True
def request(query, params): # pylint: disable=unused-argument
params['url'] = url.format(from_lang=params['from_lang'][2], to_lang=params['to_lang'][2], query=params['query'])
from_lang = params["from_lang"][2] # "english"
to_lang = params["to_lang"][2] # "german"
query = params["query"]
params["url"] = f"{base_url}/{from_lang}-{to_lang}-dictionary/{urllib.parse.quote_plus(query)}"
return params
def _clean_up_node(node):
for x in ["./i", "./span", "./button"]:
for n in node.xpath(x):
n.getparent().remove(n)
def response(resp):
results = []
item_list = []
if not resp.ok:
return results
dom = html.fromstring(resp.text)
translations = []
for result in eval_xpath(dom, results_xpath)[1:]:
try:
from_result, to_results_raw = eval_xpath(result, './td')
except: # pylint: disable=bare-except
for result in eval_xpath(dom, ".//table[@id='r']//tr"):
# each row is an Translations.Item
td_list = result.xpath("./td")
if len(td_list) != 2:
# ignore header columns "tr/th"
continue
to_results = []
for to_result in eval_xpath(to_results_raw, './p/a'):
t = to_result.text_content()
if t.strip():
to_results.append(to_result.text_content())
col_from, col_to = td_list
_clean_up_node(col_from)
translations.append(
{
'text': f"{from_result.text_content()} - {'; '.join(to_results)}",
}
)
text = f"{extract_text(col_from)}"
if translations:
result = {
'answer': translations[0]['text'],
'translations': translations,
'answer_type': 'translations',
}
synonyms = []
p_list = col_to.xpath(".//p")
return [result]
for i, p_item in enumerate(p_list):
smpl: str = extract_text(p_list[i].xpath("./i[@class='smpl']")) # type: ignore
_clean_up_node(p_item)
p_text: str = extract_text(p_item) # type: ignore
if smpl:
p_text += " // " + smpl
if i == 0:
text += f" : {p_text}"
continue
synonyms.append(p_text)
item = Translations.Item(text=text, synonyms=synonyms)
item_list.append(item)
# the "autotranslate" of dictzone is loaded by the JS from URL:
# https://dictzone.com/trans/hello%20world/en_de
from_lang = resp.search_params["from_lang"][1] # "en"
to_lang = resp.search_params["to_lang"][1] # "de"
query = resp.search_params["query"]
# works only sometimes?
autotranslate = http_get(f"{base_url}/trans/{query}/{from_lang}_{to_lang}", timeout=1.0)
if autotranslate.ok and autotranslate.text:
item_list.insert(0, Translations.Item(text=autotranslate.text))
Translations(results=results, translations=item_list, url=resp.search_params["url"])
return results

8
searx/engines/duckduckgo.py
View file

@ -27,6 +27,7 @@ from searx.network import get # see https://github.com/searxng/searxng/issues/7
from searx import redisdb
from searx.enginelib.traits import EngineTraits
from searx.exceptions import SearxEngineCaptchaException
from searx.result_types import Answer
if TYPE_CHECKING:
import logging
@ -398,12 +399,7 @@ def response(resp):
):
current_query = resp.search_params["data"].get("q")
results.append(
{
'answer': zero_click,
'url': "https://duckduckgo.com/?" + urlencode({"q": current_query}),
}
)
Answer(results=results, answer=zero_click, url="https://duckduckgo.com/?" + urlencode({"q": current_query}))
return results

8
searx/engines/duckduckgo_definitions.py
View file

@ -21,6 +21,7 @@ from lxml import html
from searx.data import WIKIDATA_UNITS
from searx.utils import extract_text, html_to_text, get_string_replaces_function
from searx.external_urls import get_external_url, get_earth_coordinates_url, area_to_osm_zoom
from searx.result_types import Answer
if TYPE_CHECKING:
import logging
@ -99,9 +100,10 @@ def response(resp):
# add answer if there is one
answer = search_res.get('Answer', '')
if answer:
logger.debug('AnswerType="%s" Answer="%s"', search_res.get('AnswerType'), answer)
if search_res.get('AnswerType') not in ['calc', 'ip']:
results.append({'answer': html_to_text(answer), 'url': search_res.get('AbstractURL', '')})
answer_type = search_res.get('AnswerType')
logger.debug('AnswerType="%s" Answer="%s"', answer_type, answer)
if isinstance(answer, str) and answer_type not in ['calc', 'ip']:
Answer(results=results, answer=html_to_text(answer), url=search_res.get('AbstractURL', ''))
# add infobox
if 'Definition' in search_res:

8
searx/engines/google.py
View file

@ -25,6 +25,7 @@ from searx.locales import language_tag, region_tag, get_official_locales
from searx.network import get # see https://github.com/searxng/searxng/issues/762
from searx.exceptions import SearxEngineCaptchaException
from searx.enginelib.traits import EngineTraits
from searx.result_types import Answer
if TYPE_CHECKING:
import logging
@ -331,12 +332,7 @@ def response(resp):
for item in answer_list:
for bubble in eval_xpath(item, './/div[@class="nnFGuf"]'):
bubble.drop_tree()
results.append(
{
'answer': extract_text(item),
'url': (eval_xpath(item, '../..//a/@href') + [None])[0],
}
)
Answer(results=results, answer=extract_text(item), url=(eval_xpath(item, '../..//a/@href') + [None])[0])
# parse results

25
searx/engines/libretranslate.py
View file

@ -2,7 +2,8 @@
"""LibreTranslate (Free and Open Source Machine Translation API)"""
import random
from json import dumps
import json
from searx.result_types import Translations
about = {
"website": 'https://libretranslate.com',
@ -16,19 +17,27 @@ about = {
engine_type = 'online_dictionary'
categories = ['general', 'translate']
base_url = "https://translate.terraprint.co"
api_key = ''
base_url = "https://libretranslate.com/translate"
api_key = ""
def request(_query, params):
request_url = random.choice(base_url) if isinstance(base_url, list) else base_url
if request_url.startswith("https://libretranslate.com") and not api_key:
return None
params['url'] = f"{request_url}/translate"
args = {'source': params['from_lang'][1], 'target': params['to_lang'][1], 'q': params['query'], 'alternatives': 3}
args = {
'q': params['query'],
'source': params['from_lang'][1],
'target': params['to_lang'][1],
'alternatives': 3,
}
if api_key:
args['api_key'] = api_key
params['data'] = dumps(args)
params['data'] = json.dumps(args)
params['method'] = 'POST'
params['headers'] = {'Content-Type': 'application/json'}
params['req_url'] = request_url
@ -41,12 +50,10 @@ def response(resp):
json_resp = resp.json()
text = json_resp.get('translatedText')
if not text:
return results
translations = [{'text': text}] + [{'text': alternative} for alternative in json_resp.get('alternatives', [])]
results.append({'answer': text, 'answer_type': 'translations', 'translations': translations})
item = Translations.Item(text=text, examples=json_resp.get('alternatives', []))
Translations(results=results, translations=[item])
return results

41
searx/engines/lingva.py
View file

@ -1,6 +1,8 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""Lingva (alternative Google Translate frontend)"""
from searx.result_types import Translations
about = {
"website": 'https://lingva.ml',
"wikidata_id": None,
@ -14,13 +16,10 @@ engine_type = 'online_dictionary'
categories = ['general', 'translate']
url = "https://lingva.thedaviddelta.com"
search_url = "{url}/api/v1/{from_lang}/{to_lang}/{query}"
def request(_query, params):
params['url'] = search_url.format(
url=url, from_lang=params['from_lang'][1], to_lang=params['to_lang'][1], query=params['query']
)
params['url'] = f"{url}/api/v1/{params['from_lang'][1]}/{params['to_lang'][1]}/{params['query']}"
return params
@ -45,32 +44,30 @@ def response(resp):
for definition in info['definitions']:
for translation in definition['list']:
data.append(
{
'text': result['translation'],
'definitions': [translation['definition']] if translation['definition'] else [],
'examples': [translation['example']] if translation['example'] else [],
'synonyms': translation['synonyms'],
}
Translations.Item(
text=result['translation'],
definitions=[translation['definition']] if translation['definition'] else [],
examples=[translation['example']] if translation['example'] else [],
synonyms=translation['synonyms'],
)
)
for translation in info["extraTranslations"]:
for word in translation["list"]:
data.append(
{
'text': word['word'],
'definitions': word['meanings'],
}
Translations.Item(
text=word['word'],
definitions=word['meanings'],
)
)
if not data and result['translation']:
data.append({'text': result['translation']})
data.append(Translations.Item(text=result['translation']))
results.append(
{
'answer': data[0]['text'],
'answer_type': 'translations',
'translations': data,
}
params = resp.search_params
Translations(
results=results,
translations=data,
url=f"{url}/{params['from_lang'][1]}/{params['to_lang'][1]}/{params['query']}",
)
return results

29
searx/engines/mozhi.py
View file

@ -3,7 +3,9 @@
import random
import re
from urllib.parse import urlencode
import urllib.parse
from searx.result_types import Translations
about = {
"website": 'https://codeberg.org/aryak/mozhi',
@ -27,34 +29,33 @@ def request(_query, params):
request_url = random.choice(base_url) if isinstance(base_url, list) else base_url
args = {'from': params['from_lang'][1], 'to': params['to_lang'][1], 'text': params['query'], 'engine': mozhi_engine}
params['url'] = f"{request_url}/api/translate?{urlencode(args)}"
params['url'] = f"{request_url}/api/translate?{urllib.parse.urlencode(args)}"
return params
def response(resp):
results = []
translation = resp.json()
data = {'text': translation['translated-text'], 'definitions': [], 'examples': []}
item = Translations.Item(text=translation['translated-text'])
if translation['target_transliteration'] and not re.match(
re_transliteration_unsupported, translation['target_transliteration']
):
data['transliteration'] = translation['target_transliteration']
item.transliteration = translation['target_transliteration']
if translation['word_choices']:
for word in translation['word_choices']:
if word.get('definition'):
data['definitions'].append(word['definition'])
item.definitions.append(word['definition'])
for example in word.get('examples_target', []):
data['examples'].append(re.sub(r"<|>", "", example).lstrip('- '))
item.examples.append(re.sub(r"<|>", "", example).lstrip('- '))
data['synonyms'] = translation.get('source_synonyms', [])
item.synonyms = translation.get('source_synonyms', [])
result = {
'answer': translation['translated-text'],
'answer_type': 'translations',
'translations': [data],
}
return [result]
url = urllib.parse.urlparse(resp.search_params["url"])
# remove the api path
url = url._replace(path="", fragment="").geturl()
Translations(results=results, translations=[item], url=url)
return results

35
searx/engines/openstreetmap.py
View file

@ -4,16 +4,16 @@
"""
import re
from json import loads
from urllib.parse import urlencode
import urllib.parse
from functools import partial
from flask_babel import gettext
from searx.data import OSM_KEYS_TAGS, CURRENCIES
from searx.utils import searx_useragent
from searx.external_urls import get_external_url
from searx.engines.wikidata import send_wikidata_query, sparql_string_escape, get_thumbnail
from searx.result_types import Answer
# about
about = {
@ -37,8 +37,7 @@ search_string = 'search?{query}&polygon_geojson=1&format=jsonv2&addressdetails=1
result_id_url = 'https://openstreetmap.org/{osm_type}/{osm_id}'
result_lat_lon_url = 'https://www.openstreetmap.org/?mlat={lat}&mlon={lon}&zoom={zoom}&layers=M'
route_url = 'https://graphhopper.com/maps/?point={}&point={}&locale=en-US&vehicle=car&weighting=fastest&turn_costs=true&use_miles=false&layer=Omniscale' # pylint: disable=line-too-long
route_re = re.compile('(?:from )?(.+) to (.+)')
route_url = 'https://graphhopper.com/maps'
wikidata_image_sparql = """
select ?item ?itemLabel ?image ?sign ?symbol ?website ?wikipediaName
@ -138,27 +137,25 @@ KEY_RANKS = {k: i for i, k in enumerate(KEY_ORDER)}
def request(query, params):
"""do search-request"""
params['url'] = base_url + search_string.format(query=urlencode({'q': query}))
params['route'] = route_re.match(query)
params['headers']['User-Agent'] = searx_useragent()
if 'Accept-Language' not in params['headers']:
params['headers']['Accept-Language'] = 'en'
params['url'] = base_url + search_string.format(query=urllib.parse.urlencode({'q': query}))
return params
def response(resp):
"""get response from search-request"""
results = []
nominatim_json = loads(resp.text)
nominatim_json = resp.json()
user_language = resp.search_params['language']
if resp.search_params['route']:
results.append(
{
'answer': gettext('Get directions'),
'url': route_url.format(*resp.search_params['route'].groups()),
}
l = re.findall(r"from\s+(.*)\s+to\s+(.+)", resp.search_params["query"])
if not l:
l = re.findall(r"\s*(.*)\s+to\s+(.+)", resp.search_params["query"])
if l:
point1, point2 = [urllib.parse.quote_plus(p) for p in l[0]]
Answer(
results=results,
answer=gettext('Show route in map ..'),
url=f"{route_url}/?point={point1}&point={point2}",
)
# simplify the code below: make sure extratags is a dictionary

9
searx/engines/tineye.py
View file

@ -156,6 +156,7 @@ def parse_tineye_match(match_json):
def response(resp):
"""Parse HTTP response from TinEye."""
results = []
# handle the 422 client side errors, and the possible 400 status code error
if resp.status_code in (400, 422):
@ -182,14 +183,14 @@ def response(resp):
message = ','.join(description)
# see https://github.com/searxng/searxng/pull/1456#issuecomment-1193105023
# results.append({'answer': message})
logger.error(message)
return []
# from searx.result_types import Answer
# Answer(results=results, answer=message)
logger.info(message)
return results
# Raise for all other responses
resp.raise_for_status()
results = []
json_data = resp.json()
for match_json in json_data['matches']:

45
searx/engines/translated.py
View file

@ -3,6 +3,10 @@
"""
import urllib.parse
from searx.result_types import Translations
# about
about = {
"website": 'https://mymemory.translated.net/',
@ -15,8 +19,8 @@ about = {
engine_type = 'online_dictionary'
categories = ['general', 'translate']
url = 'https://api.mymemory.translated.net/get?q={query}&langpair={from_lang}|{to_lang}{key}'
web_url = 'https://mymemory.translated.net/en/{from_lang}/{to_lang}/{query}'
api_url = "https://api.mymemory.translated.net"
web_url = "https://mymemory.translated.net"
weight = 100
https_support = True
@ -24,27 +28,32 @@ api_key = ''
def request(query, params): # pylint: disable=unused-argument
args = {"q": params["query"], "langpair": f"{params['from_lang'][1]}|{params['to_lang'][1]}"}
if api_key:
key_form = '&key=' + api_key
else:
key_form = ''
params['url'] = url.format(
from_lang=params['from_lang'][1], to_lang=params['to_lang'][1], query=params['query'], key=key_form
)
args["key"] = api_key
params['url'] = f"{api_url}/get?{urllib.parse.urlencode(args)}"
return params
def response(resp):
json_resp = resp.json()
text = json_resp['responseData']['translatedText']
results = []
data = resp.json()
alternatives = [match['translation'] for match in json_resp['matches'] if match['translation'] != text]
translations = [{'text': translation} for translation in [text] + alternatives]
result = {
'answer': translations[0]['text'],
'answer_type': 'translations',
'translations': translations,
args = {
"q": resp.search_params["query"],
"lang": resp.search_params.get("searxng_locale", "en"), # ui language
"sl": resp.search_params['from_lang'][1],
"tl": resp.search_params['to_lang'][1],
}
return [result]
link = f"{web_url}/search.php?{urllib.parse.urlencode(args)}"
text = data['responseData']['translatedText']
examples = [f"{m['segment']} : {m['translation']}" for m in data['matches'] if m['translation'] != text]
item = Translations.Item(text=text, examples=examples)
Translations(results=results, translations=[item], url=link)
return results

2
searx/engines/xpath.py
View file

@ -262,7 +262,7 @@ def request(query, params):
def response(resp): # pylint: disable=too-many-branches
'''Scrap *results* from the response (see :ref:`engine results`).'''
'''Scrap *results* from the response (see :ref:`result types`).'''
if no_result_for_http_status and resp.status_code in no_result_for_http_status:
return []

82
searx/extended_types.py Normal file
View file

@ -0,0 +1,82 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""This module implements the type extensions applied by SearXNG.
- :py:obj:`flask.request` is replaced by :py:obj:`sxng_request`
- :py:obj:`flask.Request` is replaced by :py:obj:`SXNG_Request`
- :py:obj:`httpx.response` is replaced by :py:obj:`SXNG_Response`
----
.. py:attribute:: sxng_request
:type: SXNG_Request
A replacement for :py:obj:`flask.request` with type cast :py:obj:`SXNG_Request`.
.. autoclass:: SXNG_Request
:members:
.. autoclass:: SXNG_Response
:members:
"""
# pylint: disable=invalid-name
from __future__ import annotations
__all__ = ["SXNG_Request", "sxng_request", "SXNG_Response"]
import typing
import flask
import httpx
if typing.TYPE_CHECKING:
import searx.preferences
import searx.results
class SXNG_Request(flask.Request):
"""SearXNG extends the class :py:obj:`flask.Request` with properties from
*this* class definition, see type cast :py:obj:`sxng_request`.
"""
user_plugins: list[str]
"""list of searx.plugins.Plugin.id (the id of the plugins)"""
preferences: "searx.preferences.Preferences"
"""The prefernces of the request."""
errors: list[str]
"""A list of errors (translated text) added by :py:obj:`searx.webapp` in
case of errors."""
# request.form is of type werkzeug.datastructures.ImmutableMultiDict
# form: dict[str, str]
start_time: float
"""Start time of the request, :py:obj:`timeit.default_timer` added by
:py:obj:`searx.webapp` to calculate the total time of the request."""
render_time: float
"""Duration of the rendering, calculated and added by
:py:obj:`searx.webapp`."""
timings: list["searx.results.Timing"]
"""A list of :py:obj:`searx.results.Timing` of the engines, calculatid in
and hold by :py:obj:`searx.results.ResultContainer.timings`."""
#: A replacement for :py:obj:`flask.request` with type cast :py:`SXNG_Request`.
sxng_request = typing.cast(SXNG_Request, flask.request)
class SXNG_Response(httpx.Response):
"""SearXNG extends the class :py:obj:`httpx.Response` with properties from
*this* class (type cast of :py:obj:`httpx.Response`).
.. code:: python
response = httpx.get("https://example.org")
response = typing.cast(SXNG_Response, response)
if response.ok:
...
"""
ok: bool

13
searx/favicons/proxy.py
View file

@ -18,6 +18,7 @@ from searx import get_setting
from searx.webutils import new_hmac, is_hmac_of
from searx.exceptions import SearxEngineResponseException
from searx.extended_types import sxng_request
from .resolvers import DEFAULT_RESOLVER_MAP
from . import cache
@ -124,7 +125,7 @@ def favicon_proxy():
server>` setting.
"""
authority = flask.request.args.get('authority')
authority = sxng_request.args.get('authority')
# malformed request or RFC 3986 authority
if not authority or "/" in authority:
@ -134,11 +135,11 @@ def favicon_proxy():
if not is_hmac_of(
CFG.secret_key,
authority.encode(),
flask.request.args.get('h', ''),
sxng_request.args.get('h', ''),
):
return '', 400
resolver = flask.request.preferences.get_value('favicon_resolver') # type: ignore
resolver = sxng_request.preferences.get_value('favicon_resolver') # type: ignore
# if resolver is empty or not valid, just return HTTP 400.
if not resolver or resolver not in CFG.resolver_map.keys():
return "", 400
@ -151,7 +152,7 @@ def favicon_proxy():
return resp
# return default favicon from static path
theme = flask.request.preferences.get_value("theme") # type: ignore
theme = sxng_request.preferences.get_value("theme") # type: ignore
fav, mimetype = CFG.favicon(theme=theme)
return flask.send_from_directory(fav.parent, fav.name, mimetype=mimetype)
@ -215,7 +216,7 @@ def favicon_url(authority: str) -> str:
"""
resolver = flask.request.preferences.get_value('favicon_resolver') # type: ignore
resolver = sxng_request.preferences.get_value('favicon_resolver') # type: ignore
# if resolver is empty or not valid, just return nothing.
if not resolver or resolver not in CFG.resolver_map.keys():
return ""
@ -224,7 +225,7 @@ def favicon_url(authority: str) -> str:
if data_mime == (None, None):
# we have already checked, the resolver does not have a favicon
theme = flask.request.preferences.get_value("theme") # type: ignore
theme = sxng_request.preferences.get_value("theme") # type: ignore
return CFG.favicon_data_url(theme=theme)
if data_mime is not None:

3
searx/infopage/__init__.py
View file

@ -6,13 +6,14 @@ Usage in a Flask app route:
.. code:: python
from searx import infopage
from searx.extended_types import sxng_request
_INFO_PAGES = infopage.InfoPageSet(infopage.MistletoePage)
@app.route('/info/<pagename>', methods=['GET'])
def info(pagename):
locale = request.preferences.get_value('locale')
locale = sxng_request.preferences.get_value('locale')
page = _INFO_PAGES.get_page(pagename, locale)
"""

7
searx/limiter.py
View file

@ -105,6 +105,7 @@ from searx import (
redisdb,
)
from searx import botdetection
from searx.extended_types import SXNG_Request, sxng_request
from searx.botdetection import (
config,
http_accept,
@ -144,7 +145,7 @@ def get_cfg() -> config.Config:
return CFG
def filter_request(request: flask.Request) -> werkzeug.Response | None:
def filter_request(request: SXNG_Request) -> werkzeug.Response | None:
# pylint: disable=too-many-return-statements
cfg = get_cfg()
@ -201,13 +202,13 @@ def filter_request(request: flask.Request) -> werkzeug.Response | None:
val = func.filter_request(network, request, cfg)
if val is not None:
return val
logger.debug(f"OK {network}: %s", dump_request(flask.request))
logger.debug(f"OK {network}: %s", dump_request(sxng_request))
return None
def pre_request():
"""See :py:obj:`flask.Flask.before_request`"""
return filter_request(flask.request)
return filter_request(sxng_request)
def is_installed():

9
searx/locales.py
View file

@ -35,13 +35,14 @@ from babel.support import Translations
import babel.languages
import babel.core
import flask_babel
import flask
from flask.ctx import has_request_context
from searx import (
data,
logger,
searx_dir,
)
from searx.extended_types import sxng_request
logger = logger.getChild('locales')
@ -85,13 +86,13 @@ Kong."""
def localeselector():
locale = 'en'
if has_request_context():
value = flask.request.preferences.get_value('locale')
value = sxng_request.preferences.get_value('locale')
if value:
locale = value
# first, set the language that is not supported by babel
if locale in ADDITIONAL_TRANSLATIONS:
flask.request.form['use-translation'] = locale
sxng_request.form['use-translation'] = locale
# second, map locale to a value python-babel supports
locale = LOCALE_BEST_MATCH.get(locale, locale)
@ -109,7 +110,7 @@ def localeselector():
def get_translations():
"""Monkey patch of :py:obj:`flask_babel.get_translations`"""
if has_request_context():
use_translation = flask.request.form.get('use-translation')
use_translation = sxng_request.form.get('use-translation')
if use_translation in ADDITIONAL_TRANSLATIONS:
babel_ext = flask_babel.current_app.extensions['babel']
return Translations.load(babel_ext.translation_directories[0], use_translation)

17
searx/network/__init__.py
View file

@ -13,6 +13,7 @@ from contextlib import contextmanager
import httpx
import anyio
from searx.extended_types import SXNG_Response
from .network import get_network, initialize, check_network_configuration # pylint:disable=cyclic-import
from .client import get_loop
from .raise_for_httperror import raise_for_httperror
@ -85,7 +86,7 @@ def _get_timeout(start_time, kwargs):
return timeout
def request(method, url, **kwargs):
def request(method, url, **kwargs) -> SXNG_Response:
"""same as requests/requests/api.py request(...)"""
with _record_http_time() as start_time:
network = get_context_network()
@ -159,34 +160,34 @@ class Request(NamedTuple):
return Request('DELETE', url, kwargs)
def get(url, **kwargs):
def get(url, **kwargs) -> SXNG_Response:
kwargs.setdefault('allow_redirects', True)
return request('get', url, **kwargs)
def options(url, **kwargs):
def options(url, **kwargs) -> SXNG_Response:
kwargs.setdefault('allow_redirects', True)
return request('options', url, **kwargs)
def head(url, **kwargs):
def head(url, **kwargs) -> SXNG_Response:
kwargs.setdefault('allow_redirects', False)
return request('head', url, **kwargs)
def post(url, data=None, **kwargs):
def post(url, data=None, **kwargs) -> SXNG_Response:
return request('post', url, data=data, **kwargs)
def put(url, data=None, **kwargs):
def put(url, data=None, **kwargs) -> SXNG_Response:
return request('put', url, data=data, **kwargs)
def patch(url, data=None, **kwargs):
def patch(url, data=None, **kwargs) -> SXNG_Response:
return request('patch', url, data=data, **kwargs)
def delete(url, **kwargs):
def delete(url, **kwargs) -> SXNG_Response:
return request('delete', url, **kwargs)

8
searx/network/network.py
View file

@ -1,7 +1,9 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=global-statement
# pylint: disable=missing-module-docstring, missing-class-docstring
from __future__ import annotations
import typing
import atexit
import asyncio
import ipaddress
@ -11,6 +13,7 @@ from typing import Dict
import httpx
from searx import logger, searx_debug
from searx.extended_types import SXNG_Response
from .client import new_client, get_loop, AsyncHTTPTransportNoHttp
from .raise_for_httperror import raise_for_httperror
@ -233,8 +236,9 @@ class Network:
del kwargs['raise_for_httperror']
return do_raise_for_httperror
def patch_response(self, response, do_raise_for_httperror):
def patch_response(self, response, do_raise_for_httperror) -> SXNG_Response:
if isinstance(response, httpx.Response):
response = typing.cast(SXNG_Response, response)
# requests compatibility (response is not streamed)
# see also https://www.python-httpx.org/compatibility/#checking-for-4xx5xx-responses
response.ok = not response.is_error
@ -258,7 +262,7 @@ class Network:
return False
return True
async def call_client(self, stream, method, url, **kwargs):
async def call_client(self, stream, method, url, **kwargs) -> SXNG_Response:
retries = self.retries
was_disconnected = False
do_raise_for_httperror = Network.extract_do_raise_for_httperror(kwargs)

250
searx/plugins/__init__.py
View file

@ -1,232 +1,68 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring, missing-class-docstring
""".. sidebar:: Further reading ..
import sys
from hashlib import sha256
from importlib import import_module
from os import listdir, makedirs, remove, stat, utime
from os.path import abspath, basename, dirname, exists, join
from shutil import copyfile
from pkgutil import iter_modules
from logging import getLogger
from typing import List, Tuple
- :ref:`plugins admin`
- :ref:`SearXNG settings <settings plugins>`
- :ref:`builtin plugins`
from searx import logger, settings
Plugins can extend or replace functionality of various components of SearXNG.
Here is an example of a very simple plugin that adds a "Hello" into the answer
area:
.. code:: python
class Plugin: # pylint: disable=too-few-public-methods
"""This class is currently never initialized and only used for type hinting."""
from flask_babel import gettext as _
from searx.plugins import Plugin
from searx.result_types import Answer
id: str
name: str
description: str
default_on: bool
js_dependencies: Tuple[str]
css_dependencies: Tuple[str]
preference_section: str
class MyPlugin(Plugin):
id = "self_info"
default_on = True
logger = logger.getChild("plugins")
def __init__(self):
super().__init__()
info = PluginInfo(id=self.id, name=_("Hello"), description=_("demo plugin"))
required_attrs = (
# fmt: off
("name", str),
("description", str),
("default_on", bool)
# fmt: on
)
def post_search(self, request, search):
return [ Answer(answer="Hello") ]
optional_attrs = (
# fmt: off
("js_dependencies", tuple),
("css_dependencies", tuple),
("preference_section", str),
# fmt: on
)
Entry points (hooks) define when a plugin runs. Right now only three hooks are
implemented. So feel free to implement a hook if it fits the behaviour of your
plugin / a plugin doesn't need to implement all the hooks.
- pre search: :py:obj:`Plugin.pre_search`
- post search: :py:obj:`Plugin.post_search`
- on each result item: :py:obj:`Plugin.on_result`
def sha_sum(filename):
with open(filename, "rb") as f:
file_content_bytes = f.read()
return sha256(file_content_bytes).hexdigest()
For a coding example have a look at :ref:`self_info plugin`.
----
def sync_resource(base_path, resource_path, name, target_dir, plugin_dir):
dep_path = join(base_path, resource_path)
file_name = basename(dep_path)
resource_path = join(target_dir, file_name)
if not exists(resource_path) or sha_sum(dep_path) != sha_sum(resource_path):
try:
copyfile(dep_path, resource_path)
# copy atime_ns and mtime_ns, so the weak ETags (generated by
# the HTTP server) do not change
dep_stat = stat(dep_path)
utime(resource_path, ns=(dep_stat.st_atime_ns, dep_stat.st_mtime_ns))
except IOError:
logger.critical("failed to copy plugin resource {0} for plugin {1}".format(file_name, name))
sys.exit(3)
.. autoclass:: Plugin
:members:
# returning with the web path of the resource
return join("plugins/external_plugins", plugin_dir, file_name)
.. autoclass:: PluginInfo
:members:
.. autoclass:: PluginStorage
:members:
def prepare_package_resources(plugin, plugin_module_name):
plugin_base_path = dirname(abspath(plugin.__file__))
.. autoclass:: searx.plugins._core.ModulePlugin
:members:
:show-inheritance:
plugin_dir = plugin_module_name
target_dir = join(settings["ui"]["static_path"], "plugins/external_plugins", plugin_dir)
try:
makedirs(target_dir, exist_ok=True)
except IOError:
logger.critical("failed to create resource directory {0} for plugin {1}".format(target_dir, plugin_module_name))
sys.exit(3)
"""
resources = []
from __future__ import annotations
if hasattr(plugin, "js_dependencies"):
resources.extend(map(basename, plugin.js_dependencies))
plugin.js_dependencies = [
sync_resource(plugin_base_path, x, plugin_module_name, target_dir, plugin_dir)
for x in plugin.js_dependencies
]
__all__ = ["PluginInfo", "Plugin", "PluginStorage"]
if hasattr(plugin, "css_dependencies"):
resources.extend(map(basename, plugin.css_dependencies))
plugin.css_dependencies = [
sync_resource(plugin_base_path, x, plugin_module_name, target_dir, plugin_dir)
for x in plugin.css_dependencies
]
from ._core import PluginInfo, Plugin, PluginStorage
for f in listdir(target_dir):
if basename(f) not in resources:
resource_path = join(target_dir, basename(f))
try:
remove(resource_path)
except IOError:
logger.critical(
"failed to remove unused resource file {0} for plugin {1}".format(resource_path, plugin_module_name)
)
sys.exit(3)
def load_plugin(plugin_module_name, external):
# pylint: disable=too-many-branches
try:
plugin = import_module(plugin_module_name)
except (
SyntaxError,
KeyboardInterrupt,
SystemExit,
SystemError,
ImportError,
RuntimeError,
) as e:
logger.critical("%s: fatal exception", plugin_module_name, exc_info=e)
sys.exit(3)
except BaseException:
logger.exception("%s: exception while loading, the plugin is disabled", plugin_module_name)
return None
# difference with searx: use module name instead of the user name
plugin.id = plugin_module_name
#
plugin.logger = getLogger(plugin_module_name)
for plugin_attr, plugin_attr_type in required_attrs:
if not hasattr(plugin, plugin_attr):
logger.critical('%s: missing attribute "%s", cannot load plugin', plugin, plugin_attr)
sys.exit(3)
attr = getattr(plugin, plugin_attr)
if not isinstance(attr, plugin_attr_type):
type_attr = str(type(attr))
logger.critical(
'{1}: attribute "{0}" is of type {2}, must be of type {3}, cannot load plugin'.format(
plugin, plugin_attr, type_attr, plugin_attr_type
)
)
sys.exit(3)
for plugin_attr, plugin_attr_type in optional_attrs:
if not hasattr(plugin, plugin_attr) or not isinstance(getattr(plugin, plugin_attr), plugin_attr_type):
setattr(plugin, plugin_attr, plugin_attr_type())
if not hasattr(plugin, "preference_section"):
plugin.preference_section = "general"
# query plugin
if plugin.preference_section == "query":
for plugin_attr in ("query_keywords", "query_examples"):
if not hasattr(plugin, plugin_attr):
logger.critical('missing attribute "{0}", cannot load plugin: {1}'.format(plugin_attr, plugin))
sys.exit(3)
if settings.get("enabled_plugins"):
# searx compatibility: plugin.name in settings['enabled_plugins']
plugin.default_on = plugin.name in settings["enabled_plugins"] or plugin.id in settings["enabled_plugins"]
# copy resources if this is an external plugin
if external:
prepare_package_resources(plugin, plugin_module_name)
logger.debug("%s: loaded", plugin_module_name)
return plugin
def load_and_initialize_plugin(plugin_module_name, external, init_args):
plugin = load_plugin(plugin_module_name, external)
if plugin and hasattr(plugin, 'init'):
try:
return plugin if plugin.init(*init_args) else None
except Exception: # pylint: disable=broad-except
plugin.logger.exception("Exception while calling init, the plugin is disabled")
return None
return plugin
class PluginStore:
def __init__(self):
self.plugins: List[Plugin] = []
def __iter__(self):
yield from self.plugins
def register(self, plugin):
self.plugins.append(plugin)
def call(self, ordered_plugin_list, plugin_type, *args, **kwargs):
ret = True
for plugin in ordered_plugin_list:
if hasattr(plugin, plugin_type):
try:
ret = getattr(plugin, plugin_type)(*args, **kwargs)
if not ret:
break
except Exception: # pylint: disable=broad-except
plugin.logger.exception("Exception while calling %s", plugin_type)
return ret
plugins = PluginStore()
def plugin_module_names():
yield_plugins = set()
# embedded plugins
for module in iter_modules(path=[dirname(__file__)]):
yield (__name__ + "." + module.name, False)
yield_plugins.add(module.name)
# external plugins
for module_name in settings['plugins']:
if module_name not in yield_plugins:
yield (module_name, True)
yield_plugins.add(module_name)
STORAGE: PluginStorage = PluginStorage()
def initialize(app):
for module_name, external in plugin_module_names():
plugin = load_and_initialize_plugin(module_name, external, (app, settings))
if plugin:
plugins.register(plugin)
STORAGE.load_builtins()
STORAGE.init(app)

394
searx/plugins/_core.py Normal file
View file

@ -0,0 +1,394 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=too-few-public-methods,missing-module-docstring
from __future__ import annotations
__all__ = ["PluginInfo", "Plugin", "PluginStorage"]
import abc
import importlib
import logging
import pathlib
import types
import typing
import warnings
from dataclasses import dataclass, field
import flask
import searx
from searx.utils import load_module
from searx.extended_types import SXNG_Request
from searx.result_types import Result
if typing.TYPE_CHECKING:
from searx.search import SearchWithPlugins
_default = pathlib.Path(__file__).parent
log: logging.Logger = logging.getLogger("searx.plugins")
@dataclass
class PluginInfo:
"""Object that holds informations about a *plugin*, these infos are shown to
the user in the Preferences menu.
To be able to translate the information into other languages, the text must
be written in English and translated with :py:obj:`flask_babel.gettext`.
"""
id: str
"""The ID-selector in HTML/CSS `#<id>`."""
name: str
"""Name of the *plugin*."""
description: str
"""Short description of the *answerer*."""
preference_section: typing.Literal["general", "ui", "privacy", "query"] | None = "general"
"""Section (tab/group) in the preferences where this plugin is shown to the
user.
The value ``query`` is reserved for plugins that are activated via a
*keyword* as part of a search query, see:
- :py:obj:`PluginInfo.examples`
- :py:obj:`Plugin.keywords`
Those plugins are shown in the preferences in tab *Special Queries*.
"""
examples: list[str] = field(default_factory=list)
"""List of short examples of the usage / of query terms."""
keywords: list[str] = field(default_factory=list)
"""See :py:obj:`Plugin.keywords`"""
class Plugin(abc.ABC):
"""Abstract base class of all Plugins."""
id: typing.ClassVar[str]
"""The ID (suffix) in the HTML form."""
default_on: typing.ClassVar[bool]
"""Plugin is enabled/disabled by default."""
keywords: list[str] = []
"""Keywords in the search query that activate the plugin. The *keyword* is
the first word in a search query. If a plugin should be executed regardless
of the search query, the list of keywords should be empty (which is also the
default in the base class for Plugins)."""
log: logging.Logger
"""A logger object, is automatically initialized when calling the
constructor (if not already set in the subclass)."""
info: PluginInfo
"""Informations about the *plugin*, see :py:obj:`PluginInfo`."""
def __init__(self) -> None:
super().__init__()
for attr in ["id", "default_on"]:
if getattr(self, attr, None) is None:
raise NotImplementedError(f"plugin {self} is missing attribute {attr}")
if not self.id:
self.id = f"{self.__class__.__module__}.{self.__class__.__name__}"
if not getattr(self, "log", None):
self.log = log.getChild(self.id)
def __hash__(self) -> int:
"""The hash value is used in :py:obj:`set`, for example, when an object
is added to the set. The hash value is also used in other contexts,
e.g. when checking for equality to identify identical plugins from
different sources (name collisions)."""
return id(self)
def __eq__(self, other):
"""py:obj:`Plugin` objects are equal if the hash values of the two
objects are equal."""
return hash(self) == hash(other)
def init(self, app: flask.Flask) -> bool: # pylint: disable=unused-argument
"""Initialization of the plugin, the return value decides whether this
plugin is active or not. Initialization only takes place once, at the
time the WEB application is set up. The base methode always returns
``True``, the methode can be overwritten in the inheritances,
- ``True`` plugin is active
- ``False`` plugin is inactive
"""
return True
# pylint: disable=unused-argument
def pre_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> bool:
"""Runs BEFORE the search request and returns a boolean:
- ``True`` to continue the search
- ``False`` to stop the search
"""
return True
def on_result(self, request: SXNG_Request, search: "SearchWithPlugins", result: Result) -> bool:
"""Runs for each result of each engine and returns a boolean:
- ``True`` to keep the result
- ``False`` to remove the result from the result list
The ``result`` can be modified to the needs.
.. hint::
If :py:obj:`Result.url` is modified, :py:obj:`Result.parsed_url` must
be changed accordingly:
.. code:: python
result["parsed_url"] = urlparse(result["url"])
"""
return True
def post_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> None | typing.Sequence[Result]:
"""Runs AFTER the search request. Can return a list of :py:obj:`Result`
objects to be added to the final result list."""
return
class ModulePlugin(Plugin):
"""A wrapper class for legacy *plugins*.
.. note::
For internal use only!
In a module plugin, the follwing names are mapped:
- `module.query_keywords` --> :py:obj:`Plugin.keywords`
- `module.plugin_id` --> :py:obj:`Plugin.id`
- `module.logger` --> :py:obj:`Plugin.log`
"""
_required_attrs = (("name", str), ("description", str), ("default_on", bool))
def __init__(self, mod: types.ModuleType):
"""In case of missing attributes in the module or wrong types are given,
a :py:obj:`TypeError` exception is raised."""
self.module = mod
self.id = getattr(self.module, "plugin_id", self.module.__name__)
self.log = logging.getLogger(self.module.__name__)
self.keywords = getattr(self.module, "query_keywords", [])
for attr, attr_type in self._required_attrs:
if not hasattr(self.module, attr):
msg = f"missing attribute {attr}, cannot load plugin"
self.log.critical(msg)
raise TypeError(msg)
if not isinstance(getattr(self.module, attr), attr_type):
msg = f"attribute {attr} is not of type {attr_type}"
self.log.critical(msg)
raise TypeError(msg)
self.default_on = mod.default_on
self.info = PluginInfo(
id=self.id,
name=self.module.name,
description=self.module.description,
preference_section=getattr(self.module, "preference_section", None),
examples=getattr(self.module, "query_examples", []),
keywords=self.keywords,
)
# monkeypatch module
self.module.logger = self.log # type: ignore
super().__init__()
def init(self, app: flask.Flask) -> bool:
if not hasattr(self.module, "init"):
return True
return self.module.init(app)
def pre_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> bool:
if not hasattr(self.module, "pre_search"):
return True
return self.module.pre_search(request, search)
def on_result(self, request: SXNG_Request, search: "SearchWithPlugins", result: Result) -> bool:
if not hasattr(self.module, "on_result"):
return True
return self.module.on_result(request, search, result)
def post_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> None | list[Result]:
if not hasattr(self.module, "post_search"):
return None
return self.module.post_search(request, search)
class PluginStorage:
"""A storage for managing the *plugins* of SearXNG."""
plugin_list: set[Plugin]
"""The list of :py:obj:`Plugins` in this storage."""
legacy_plugins = [
"ahmia_filter",
"calculator",
"hostnames",
"oa_doi_rewrite",
"tor_check",
"tracker_url_remover",
"unit_converter",
]
"""Internal plugins implemented in the legacy style (as module / deprecated!)."""
def __init__(self):
self.plugin_list = set()
def __iter__(self):
yield from self.plugin_list
def __len__(self):
return len(self.plugin_list)
@property
def info(self) -> list[PluginInfo]:
return [p.info for p in self.plugin_list]
def load_builtins(self):
"""Load plugin modules from:
- the python packages in :origin:`searx/plugins` and
- the external plugins from :ref:`settings plugins`.
"""
for f in _default.iterdir():
if f.name.startswith("_"):
continue
if f.stem not in self.legacy_plugins:
self.register_by_fqn(f"searx.plugins.{f.stem}.SXNGPlugin")
continue
# for backward compatibility
mod = load_module(f.name, str(f.parent))
self.register(ModulePlugin(mod))
for fqn in searx.get_setting("plugins"): # type: ignore
self.register_by_fqn(fqn)
def register(self, plugin: Plugin):
"""Register a :py:obj:`Plugin`. In case of name collision (if two
plugins have same ID) a :py:obj:`KeyError` exception is raised.
"""
if plugin in self.plugin_list:
msg = f"name collision '{plugin.id}'"
plugin.log.critical(msg)
raise KeyError(msg)
self.plugin_list.add(plugin)
plugin.log.debug("plugin has been loaded")
def register_by_fqn(self, fqn: str):
"""Register a :py:obj:`Plugin` via its fully qualified class name (FQN).
The FQNs of external plugins could be read from a configuration, for
example, and registered using this method
"""
mod_name, _, obj_name = fqn.rpartition('.')
if not mod_name:
# for backward compatibility
code_obj = importlib.import_module(fqn)
else:
mod = importlib.import_module(mod_name)
code_obj = getattr(mod, obj_name, None)
if code_obj is None:
msg = f"plugin {fqn} is not implemented"
log.critical(msg)
raise ValueError(msg)
if isinstance(code_obj, types.ModuleType):
# for backward compatibility
warnings.warn(
f"plugin {fqn} is implemented in a legacy module / migrate to searx.plugins.Plugin", DeprecationWarning
)
self.register(ModulePlugin(code_obj))
return
self.register(code_obj())
def init(self, app: flask.Flask) -> None:
"""Calls the method :py:obj:`Plugin.init` of each plugin in this
storage. Depending on its return value, the plugin is removed from
*this* storage or not."""
for plg in self.plugin_list.copy():
if not plg.init(app):
self.plugin_list.remove(plg)
def pre_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> bool:
ret = True
for plugin in [p for p in self.plugin_list if p.id in search.user_plugins]:
try:
ret = bool(plugin.pre_search(request=request, search=search))
except Exception: # pylint: disable=broad-except
plugin.log.exception("Exception while calling pre_search")
continue
if not ret:
# skip this search on the first False from a plugin
break
return ret
def on_result(self, request: SXNG_Request, search: "SearchWithPlugins", result: Result) -> bool:
ret = True
for plugin in [p for p in self.plugin_list if p.id in search.user_plugins]:
try:
ret = bool(plugin.on_result(request=request, search=search, result=result))
except Exception: # pylint: disable=broad-except
plugin.log.exception("Exception while calling on_result")
continue
if not ret:
# ignore this result item on the first False from a plugin
break
return ret
def post_search(self, request: SXNG_Request, search: "SearchWithPlugins") -> None:
"""Extend :py:obj:`search.result_container
<searx.results.ResultContainer`> with result items from plugins listed
in :py:obj:`search.user_plugins <SearchWithPlugins.user_plugins>`.
"""
keyword = None
for keyword in search.search_query.query.split():
if keyword:
break
for plugin in [p for p in self.plugin_list if p.id in search.user_plugins]:
if plugin.keywords:
# plugin with keywords: skip plugin if no keyword match
if keyword and keyword not in plugin.keywords:
continue
try:
results = plugin.post_search(request=request, search=search) or []
except Exception: # pylint: disable=broad-except
plugin.log.exception("Exception while calling post_search")
continue
# In case of *plugins* prefix ``plugin:`` is set, see searx.result_types.Result
search.result_container.extend(f"plugin: {plugin.id}", results)

14
searx/plugins/ahmia_filter.py
View file

@ -1,27 +1,33 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
from hashlib import md5
import flask
from searx.data import ahmia_blacklist_loader
from searx import get_setting
name = "Ahmia blacklist"
description = "Filter out onion results that appear in Ahmia's blacklist. (See https://ahmia.fi/blacklist)"
default_on = True
preference_section = 'onions'
ahmia_blacklist = None
ahmia_blacklist: list = []
def on_result(_request, _search, result):
def on_result(_request, _search, result) -> bool:
if not result.get('is_onion') or not result.get('parsed_url'):
return True
result_hash = md5(result['parsed_url'].hostname.encode()).hexdigest()
return result_hash not in ahmia_blacklist
def init(_app, settings):
def init(app=flask.Flask) -> bool: # pylint: disable=unused-argument
global ahmia_blacklist # pylint: disable=global-statement
if not settings['outgoing']['using_tor_proxy']:
if not get_setting("outgoing.using_tor_proxy"):
# disable the plugin
return False
ahmia_blacklist = ahmia_blacklist_loader()

70
searx/plugins/calculator.py
View file

@ -1,28 +1,27 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""Calculate mathematical expressions using ack#eval
"""Calculate mathematical expressions using :py:obj`ast.parse` (mode="eval").
"""
from __future__ import annotations
from typing import Callable
import ast
import re
import operator
from multiprocessing import Process, Queue
from typing import Callable
import multiprocessing
import flask
import babel
import babel.numbers
from flask_babel import gettext
from searx.plugins import logger
from searx.result_types import Answer
name = "Basic Calculator"
description = gettext("Calculate mathematical expressions via the search bar")
default_on = True
preference_section = 'general'
plugin_id = 'calculator'
logger = logger.getChild(plugin_id)
operators: dict[type, Callable] = {
ast.Add: operator.add,
ast.Sub: operator.sub,
@ -33,11 +32,17 @@ operators: dict[type, Callable] = {
ast.USub: operator.neg,
}
# with multiprocessing.get_context("fork") we are ready for Py3.14 (by emulating
# the old behavior "fork") but it will not solve the core problem of fork, nor
# will it remove the deprecation warnings in py3.12 & py3.13. Issue is
# ddiscussed here: https://github.com/searxng/searxng/issues/4159
mp_fork = multiprocessing.get_context("fork")
def _eval_expr(expr):
"""
>>> _eval_expr('2^6')
4
64
>>> _eval_expr('2**6')
64
>>> _eval_expr('1 + 2*3**(4^5) / (6 + -7)')
@ -63,46 +68,49 @@ def _eval(node):
raise TypeError(node)
def handler(q: multiprocessing.Queue, func, args, **kwargs): # pylint:disable=invalid-name
try:
q.put(func(*args, **kwargs))
except:
q.put(None)
raise
def timeout_func(timeout, func, *args, **kwargs):
def handler(q: Queue, func, args, **kwargs): # pylint:disable=invalid-name
try:
q.put(func(*args, **kwargs))
except:
q.put(None)
raise
que = Queue()
p = Process(target=handler, args=(que, func, args), kwargs=kwargs)
que = mp_fork.Queue()
p = mp_fork.Process(target=handler, args=(que, func, args), kwargs=kwargs)
p.start()
p.join(timeout=timeout)
ret_val = None
# pylint: disable=used-before-assignment,undefined-variable
if not p.is_alive():
ret_val = que.get()
else:
logger.debug("terminate function after timeout is exceeded")
logger.debug("terminate function after timeout is exceeded") # type: ignore
p.terminate()
p.join()
p.close()
return ret_val
def post_search(_request, search):
def post_search(request, search) -> list[Answer]:
results = []
# only show the result of the expression on the first page
if search.search_query.pageno > 1:
return True
return results
query = search.search_query.query
# in order to avoid DoS attacks with long expressions, ignore long expressions
if len(query) > 100:
return True
return results
# replace commonly used math operators with their proper Python operator
query = query.replace("x", "*").replace(":", "/")
# use UI language
ui_locale = babel.Locale.parse(flask.request.preferences.get_value('locale'), sep='-')
ui_locale = babel.Locale.parse(request.preferences.get_value('locale'), sep='-')
# parse the number system in a localized way
def _decimal(match: re.Match) -> str:
@ -116,15 +124,17 @@ def post_search(_request, search):
# only numbers and math operators are accepted
if any(str.isalpha(c) for c in query):
return True
return results
# in python, powers are calculated via **
query_py_formatted = query.replace("^", "**")
# Prevent the runtime from being longer than 50 ms
result = timeout_func(0.05, _eval_expr, query_py_formatted)
if result is None or result == "":
return True
result = babel.numbers.format_decimal(result, locale=ui_locale)
search.result_container.answers['calculate'] = {'answer': f"{search.search_query.query} = {result}"}
return True
res = timeout_func(0.05, _eval_expr, query_py_formatted)
if res is None or res == "":
return results
res = babel.numbers.format_decimal(res, locale=ui_locale)
Answer(results=results, answer=f"{search.search_query.query} = {res}")
return results

83
searx/plugins/hash_plugin.py
View file

@ -1,43 +1,66 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
# pylint: disable=missing-module-docstring, missing-class-docstring
from __future__ import annotations
import typing
import hashlib
import re
import hashlib
from flask_babel import gettext
name = "Hash plugin"
description = gettext("Converts strings to different hash digests.")
default_on = True
preference_section = 'query'
query_keywords = ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']
query_examples = 'sha512 The quick brown fox jumps over the lazy dog'
from searx.plugins import Plugin, PluginInfo
from searx.result_types import Answer
parser_re = re.compile('(md5|sha1|sha224|sha256|sha384|sha512) (.*)', re.I)
if typing.TYPE_CHECKING:
from searx.search import SearchWithPlugins
from searx.extended_types import SXNG_Request
def post_search(_request, search):
# process only on first page
if search.search_query.pageno > 1:
return True
m = parser_re.match(search.search_query.query)
if not m:
# wrong query
return True
class SXNGPlugin(Plugin):
"""Plugin converts strings to different hash digests. The results are
displayed in area for the "answers".
"""
function, string = m.groups()
if not string.strip():
# end if the string is empty
return True
id = "hash_plugin"
default_on = True
keywords = ["md5", "sha1", "sha224", "sha256", "sha384", "sha512"]
# select hash function
f = hashlib.new(function.lower())
def __init__(self):
super().__init__()
# make digest from the given string
f.update(string.encode('utf-8').strip())
answer = function + " " + gettext('hash digest') + ": " + f.hexdigest()
self.parser_re = re.compile(f"({'|'.join(self.keywords)}) (.*)", re.I)
self.info = PluginInfo(
id=self.id,
name=gettext("Hash plugin"),
description=gettext("Converts strings to different hash digests."),
examples=["sha512 The quick brown fox jumps over the lazy dog"],
preference_section="query",
)
# print result
search.result_container.answers.clear()
search.result_container.answers['hash'] = {'answer': answer}
return True
def post_search(self, request: "SXNG_Request", search: "SearchWithPlugins") -> list[Answer]:
"""Returns a result list only for the first page."""
results = []
if search.search_query.pageno > 1:
return results
m = self.parser_re.match(search.search_query.query)
if not m:
# wrong query
return results
function, string = m.groups()
if not string.strip():
# end if the string is empty
return results
# select hash function
f = hashlib.new(function.lower())
# make digest from the given string
f.update(string.encode("utf-8").strip())
answer = function + " " + gettext("hash digest") + ": " + f.hexdigest()
Answer(results=results, answer=answer)
return results

21
searx/plugins/hostnames.py
View file

@ -91,15 +91,17 @@ something like this:
"""
from __future__ import annotations
import re
from urllib.parse import urlunparse, urlparse
from flask_babel import gettext
from searx import settings
from searx.plugins import logger
from searx.settings_loader import get_yaml_cfg
name = gettext('Hostnames plugin')
description = gettext('Rewrite hostnames, remove results or prioritize them based on the hostname')
default_on = False
@ -107,16 +109,15 @@ preference_section = 'general'
plugin_id = 'hostnames'
logger = logger.getChild(plugin_id)
parsed = 'parsed_url'
_url_fields = ['iframe_src', 'audio_src']
def _load_regular_expressions(settings_key):
def _load_regular_expressions(settings_key) -> dict | set | None:
setting_value = settings.get(plugin_id, {}).get(settings_key)
if not setting_value:
return {}
return None
# load external file with configuration
if isinstance(setting_value, str):
@ -128,20 +129,20 @@ def _load_regular_expressions(settings_key):
if isinstance(setting_value, dict):
return {re.compile(p): r for (p, r) in setting_value.items()}
return {}
return None
replacements = _load_regular_expressions('replace')
removables = _load_regular_expressions('remove')
high_priority = _load_regular_expressions('high_priority')
low_priority = _load_regular_expressions('low_priority')
replacements: dict = _load_regular_expressions('replace') or {} # type: ignore
removables: set = _load_regular_expressions('remove') or set() # type: ignore
high_priority: set = _load_regular_expressions('high_priority') or set() # type: ignore
low_priority: set = _load_regular_expressions('low_priority') or set() # type: ignore
def _matches_parsed_url(result, pattern):
return parsed in result and pattern.search(result[parsed].netloc)
def on_result(_request, _search, result):
def on_result(_request, _search, result) -> bool:
for pattern, replacement in replacements.items():
if _matches_parsed_url(result, pattern):
# logger.debug(result['url'])

10
searx/plugins/oa_doi_rewrite.py
View file

@ -1,18 +1,21 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
import re
from urllib.parse import urlparse, parse_qsl
from flask_babel import gettext
from searx import settings
regex = re.compile(r'10\.\d{4,9}/[^\s]+')
name = gettext('Open Access DOI rewrite')
description = gettext('Avoid paywalls by redirecting to open-access versions of publications when available')
default_on = False
preference_section = 'general'
preference_section = 'general/doi_resolver'
def extract_doi(url):
@ -34,8 +37,9 @@ def get_doi_resolver(preferences):
return doi_resolvers[selected_resolver]
def on_result(request, _search, result):
if 'parsed_url' not in result:
def on_result(request, _search, result) -> bool:
if not result.parsed_url:
return True
doi = extract_doi(result['parsed_url'])

69
searx/plugins/self_info.py
View file

@ -1,32 +1,57 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring,invalid-name
# pylint: disable=missing-module-docstring, missing-class-docstring
from __future__ import annotations
import typing
import re
from flask_babel import gettext
from searx.botdetection._helpers import get_real_ip
from searx.result_types import Answer
name = gettext('Self Information')
description = gettext('Displays your IP if the query is "ip" and your user agent if the query contains "user agent".')
default_on = True
preference_section = 'query'
query_keywords = ['user-agent']
query_examples = ''
from . import Plugin, PluginInfo
# "ip" or "my ip" regex
ip_regex = re.compile('^ip$|my ip', re.IGNORECASE)
# Self User Agent regex
ua_regex = re.compile('.*user[ -]agent.*', re.IGNORECASE)
if typing.TYPE_CHECKING:
from searx.search import SearchWithPlugins
from searx.extended_types import SXNG_Request
def post_search(request, search):
if search.search_query.pageno > 1:
return True
if ip_regex.search(search.search_query.query):
ip = get_real_ip(request)
search.result_container.answers['ip'] = {'answer': gettext('Your IP is: ') + ip}
elif ua_regex.match(search.search_query.query):
ua = request.user_agent
search.result_container.answers['user-agent'] = {'answer': gettext('Your user-agent is: ') + ua.string}
return True
class SXNGPlugin(Plugin):
"""Simple plugin that displays information about user's request, including
the IP or HTTP User-Agent. The information is displayed in area for the
"answers".
"""
id = "self_info"
default_on = True
keywords = ["ip", "user-agent"]
def __init__(self):
super().__init__()
self.ip_regex = re.compile(r"^ip", re.IGNORECASE)
self.ua_regex = re.compile(r"^user-agent", re.IGNORECASE)
self.info = PluginInfo(
id=self.id,
name=gettext("Self Information"),
description=gettext(
"""Displays your IP if the query is "ip" and your user agent if the query is "user-agent"."""
),
preference_section="query",
)
def post_search(self, request: "SXNG_Request", search: "SearchWithPlugins") -> list[Answer]:
"""Returns a result list only for the first page."""
results = []
if search.search_query.pageno > 1:
return results
if self.ip_regex.search(search.search_query.query):
Answer(results=results, answer=gettext("Your IP is: ") + get_real_ip(request))
if self.ua_regex.match(search.search_query.query):
Answer(results=results, answer=gettext("Your user-agent is: ") + str(request.user_agent))
return results

53
searx/plugins/tor_check.py
View file

@ -1,8 +1,8 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""A plugin to check if the ip address of the request is a Tor exit-node if the
user searches for ``tor-check``. It fetches the tor exit node list from
https://check.torproject.org/exit-addresses and parses all the IPs into a list,
then checks if the user's IP address is in it.
:py:obj:`url_exit_list` and parses all the IPs into a list, then checks if the
user's IP address is in it.
Enable in ``settings.yml``:
@ -14,10 +14,15 @@ Enable in ``settings.yml``:
"""
from __future__ import annotations
import re
from flask_babel import gettext
from httpx import HTTPError
from searx.network import get
from searx.result_types import Answer
default_on = False
@ -42,27 +47,28 @@ query_examples = ''
# Regex for exit node addresses in the list.
reg = re.compile(r"(?<=ExitAddress )\S+")
url_exit_list = "https://check.torproject.org/exit-addresses"
"""URL to load Tor exit list from."""
def post_search(request, search):
def post_search(request, search) -> list[Answer]:
results = []
if search.search_query.pageno > 1:
return True
return results
if search.search_query.query.lower() == "tor-check":
# Request the list of tor exit nodes.
try:
resp = get("https://check.torproject.org/exit-addresses")
node_list = re.findall(reg, resp.text)
resp = get(url_exit_list)
node_list = re.findall(reg, resp.text) # type: ignore
except HTTPError:
# No answer, return error
search.result_container.answers["tor"] = {
"answer": gettext(
"Could not download the list of Tor exit-nodes from: https://check.torproject.org/exit-addresses"
)
}
return True
msg = gettext("Could not download the list of Tor exit-nodes from")
Answer(results=results, answer=f"{msg} {url_exit_list}")
return results
x_forwarded_for = request.headers.getlist("X-Forwarded-For")
@ -72,20 +78,11 @@ def post_search(request, search):
ip_address = request.remote_addr
if ip_address in node_list:
search.result_container.answers["tor"] = {
"answer": gettext(
"You are using Tor and it looks like you have this external IP address: {ip_address}".format(
ip_address=ip_address
)
)
}
else:
search.result_container.answers["tor"] = {
"answer": gettext(
"You are not using Tor and you have this external IP address: {ip_address}".format(
ip_address=ip_address
)
)
}
msg = gettext("You are using Tor and it looks like you have the external IP address")
Answer(results=results, answer=f"{msg} {ip_address}")
return True
else:
msg = gettext("You are not using Tor and you have the external IP address")
Answer(results=results, answer=f"{msg} {ip_address}")
return results

18
searx/plugins/tracker_url_remover.py
View file

@ -1,6 +1,8 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
import re
from urllib.parse import urlunparse, parse_qsl, urlencode
@ -19,24 +21,24 @@ default_on = True
preference_section = 'privacy'
def on_result(_request, _search, result):
if 'parsed_url' not in result:
def on_result(_request, _search, result) -> bool:
parsed_url = getattr(result, "parsed_url", None)
if not parsed_url:
return True
query = result['parsed_url'].query
if query == "":
if parsed_url.query == "":
return True
parsed_query = parse_qsl(query)
parsed_query = parse_qsl(parsed_url.query)
changes = 0
for i, (param_name, _) in enumerate(list(parsed_query)):
for reg in regexes:
if reg.match(param_name):
parsed_query.pop(i - changes)
changes += 1
result['parsed_url'] = result['parsed_url']._replace(query=urlencode(parsed_query))
result['url'] = urlunparse(result['parsed_url'])
result.parsed_url = result.parsed_url._replace(query=urlencode(parsed_query))
result.url = urlunparse(result.parsed_url)
break
return True

30
searx/plugins/unit_converter.py
View file

@ -18,11 +18,14 @@ Enable in ``settings.yml``:
"""
from __future__ import annotations
import re
import babel.numbers
from flask_babel import gettext, get_locale
from searx import data
from searx.result_types import Answer
name = "Unit converter plugin"
@ -171,16 +174,16 @@ def symbol_to_si():
return SYMBOL_TO_SI
def _parse_text_and_convert(search, from_query, to_query):
def _parse_text_and_convert(from_query, to_query) -> str | None:
# pylint: disable=too-many-branches, too-many-locals
if not (from_query and to_query):
return
return None
measured = re.match(RE_MEASURE, from_query, re.VERBOSE)
if not (measured and measured.group('number'), measured.group('unit')):
return
return None
# Symbols are not unique, if there are several hits for the from-unit, then
# the correct one must be determined by comparing it with the to-unit
@ -198,7 +201,7 @@ def _parse_text_and_convert(search, from_query, to_query):
target_list.append((si_name, from_si, orig_symbol))
if not (source_list and target_list):
return
return None
source_to_si = target_from_si = target_symbol = None
@ -212,7 +215,7 @@ def _parse_text_and_convert(search, from_query, to_query):
target_symbol = target[2]
if not (source_to_si and target_from_si):
return
return None
_locale = get_locale() or 'en_US'
@ -239,25 +242,28 @@ def _parse_text_and_convert(search, from_query, to_query):
else:
result = babel.numbers.format_decimal(value, locale=_locale, format='#,##0.##########;-#')
search.result_container.answers['conversion'] = {'answer': f'{result} {target_symbol}'}
return f'{result} {target_symbol}'
def post_search(_request, search):
def post_search(_request, search) -> list[Answer]:
results = []
# only convert between units on the first page
if search.search_query.pageno > 1:
return True
return results
query = search.search_query.query
query_parts = query.split(" ")
if len(query_parts) < 3:
return True
return results
for query_part in query_parts:
for keyword in CONVERT_KEYWORDS:
if query_part == keyword:
from_query, to_query = query.split(keyword, 1)
_parse_text_and_convert(search, from_query.strip(), to_query.strip())
return True
target_val = _parse_text_and_convert(from_query.strip(), to_query.strip())
if target_val:
Answer(results=results, answer=target_val)
return True
return results

21
searx/preferences.py
View file

@ -1,6 +1,7 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""Searx preferences implementation.
"""
from __future__ import annotations
# pylint: disable=useless-object-inheritance
@ -13,12 +14,14 @@ from collections import OrderedDict
import flask
import babel
import searx.plugins
from searx import settings, autocomplete, favicons
from searx.enginelib import Engine
from searx.plugins import Plugin
from searx.engines import DEFAULT_CATEGORY
from searx.extended_types import SXNG_Request
from searx.locales import LOCALE_NAMES
from searx.webutils import VALID_LANGUAGE_CODE
from searx.engines import DEFAULT_CATEGORY
COOKIE_MAX_AGE = 60 * 60 * 24 * 365 * 5 # 5 years
@ -312,7 +315,7 @@ class EnginesSetting(BooleanChoices):
class PluginsSetting(BooleanChoices):
"""Plugin settings"""
def __init__(self, default_value, plugins: Iterable[Plugin]):
def __init__(self, default_value, plugins: Iterable[searx.plugins.Plugin]):
super().__init__(default_value, {plugin.id: plugin.default_on for plugin in plugins})
def transform_form_items(self, items):
@ -340,7 +343,7 @@ class ClientPref:
return tag
@classmethod
def from_http_request(cls, http_request: flask.Request):
def from_http_request(cls, http_request: SXNG_Request):
"""Build ClientPref object from HTTP request.
- `Accept-Language used for locale setting
@ -375,11 +378,11 @@ class Preferences:
def __init__(
self,
themes: List[str],
categories: List[str],
engines: Dict[str, Engine],
plugins: Iterable[Plugin],
client: Optional[ClientPref] = None,
themes: list[str],
categories: list[str],
engines: dict[str, Engine],
plugins: searx.plugins.PluginStorage,
client: ClientPref | None = None,
):
super().__init__()

18
searx/result_types/__init__.py Normal file
View file

@ -0,0 +1,18 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""Typification of the result items generated by the *engines*, *answerers* and
*plugins*.
.. note::
We are at the beginning of typing the results. Further typing will follow,
but this is a very large task that we will only be able to implement
gradually. For more, please read :ref:`result types`.
"""
from __future__ import annotations
__all__ = ["Result", "AnswerSet", "Answer", "Translations"]
from ._base import Result, LegacyResult
from .answer import AnswerSet, Answer, Translations

223
searx/result_types/_base.py Normal file
View file

@ -0,0 +1,223 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=too-few-public-methods, missing-module-docstring
"""Basic types for the typification of results.
- :py:obj:`Result` base class
- :py:obj:`LegacyResult` for internal use only
----
.. autoclass:: Result
:members:
.. autoclass:: LegacyResult
:members:
"""
from __future__ import annotations
__all__ = ["Result"]
import re
import urllib.parse
import warnings
import msgspec
class Result(msgspec.Struct, kw_only=True):
"""Base class of all result types :ref:`result types`."""
url: str | None = None
"""A link related to this *result*"""
template: str = "default.html"
"""Name of the template used to render the result.
By default :origin:`result_templates/default.html
<searx/templates/simple/result_templates/default.html>` is used.
"""
engine: str | None = ""
"""Name of the engine *this* result comes from. In case of *plugins* a
prefix ``plugin:`` is set, in case of *answerer* prefix ``answerer:`` is
set.
The field is optional and is initialized from the context if necessary.
"""
parsed_url: urllib.parse.ParseResult | None = None
""":py:obj:`urllib.parse.ParseResult` of :py:obj:`Result.url`.
The field is optional and is initialized from the context if necessary.
"""
results: list = [] # https://jcristharif.com/msgspec/structs.html#default-values
"""Result list of an :origin:`engine <searx/engines>` response or a
:origin:`answerer <searx/answerers>` to which the answer should be added.
This field is only present for the sake of simplicity. Typically, the
response function of an engine has a result list that is returned at the
end. By specifying the result list in the constructor of the result, this
result is then immediately added to the list (this parameter does not have
another function).
.. code:: python
def response(resp):
results = []
...
Answer(results=results, answer=answer, url=url)
...
return results
"""
def normalize_result_fields(self):
"""Normalize a result ..
- if field ``url`` is set and field ``parse_url`` is unset, init
``parse_url`` from field ``url``. This method can be extended in the
inheritance.
"""
if not self.parsed_url and self.url:
self.parsed_url = urllib.parse.urlparse(self.url)
# if the result has no scheme, use http as default
if not self.parsed_url.scheme:
self.parsed_url = self.parsed_url._replace(scheme="http")
self.url = self.parsed_url.geturl()
def __post_init__(self):
"""Add *this* result to the result list."""
self.results.append(self)
def __hash__(self) -> int:
"""Generates a hash value that uniquely identifies the content of *this*
result. The method can be adapted in the inheritance to compare results
from different sources.
If two result objects are not identical but have the same content, their
hash values should also be identical.
The hash value is used in contexts, e.g. when checking for equality to
identify identical results from different sources (engines).
"""
return id(self)
def __eq__(self, other):
"""py:obj:`Result` objects are equal if the hash values of the two
objects are equal. If needed, its recommended to overwrite
"py:obj:`Result.__hash__`."""
return hash(self) == hash(other)
# for legacy code where a result is treated as a Python dict
def __setitem__(self, field_name, value):
return setattr(self, field_name, value)
def __getitem__(self, field_name):
if field_name not in self.__struct_fields__:
raise KeyError(f"{field_name}")
return getattr(self, field_name)
def __iter__(self):
return iter(self.__struct_fields__)
class LegacyResult(dict):
"""A wrapper around a legacy result item. The SearXNG core uses this class
for untyped dictionaries / to be downward compatible.
This class is needed until we have implemented an :py:obj:`Result` class for
each result type and the old usages in the codebase have been fully
migrated.
There is only one place where this class is used, in the
:py:obj:`searx.results.ResultContainer`.
.. attention::
Do not use this class in your own implementations!
"""
UNSET = object()
WHITESPACE_REGEX = re.compile('( |\t|\n)+', re.M | re.U)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.__dict__ = self
# Init fields with defaults / compare with defaults of the fields in class Result
self.engine = self.get("engine", "")
self.template = self.get("template", "default.html")
self.url = self.get("url", None)
self.parsed_url = self.get("parsed_url", None)
self.content = self.get("content", "")
self.title = self.get("title", "")
# Legacy types that have already been ported to a type ..
if "answer" in self:
warnings.warn(
f"engine {self.engine} is using deprecated `dict` for answers"
f" / use a class from searx.result_types.answer",
DeprecationWarning,
)
self.template = "answer/legacy.html"
def __hash__(self) -> int: # type: ignore
if "answer" in self:
return hash(self["answer"])
if not any(cls in self for cls in ["suggestion", "correction", "infobox", "number_of_results", "engine_data"]):
# it is a commun url-result ..
return hash(self.url)
return id(self)
def __eq__(self, other):
return hash(self) == hash(other)
def __repr__(self) -> str:
return f"LegacyResult: {super().__repr__()}"
def __getattr__(self, name: str, default=UNSET):
if default == self.UNSET and name not in self:
raise AttributeError(f"LegacyResult object has no field named: {name}")
return self[name]
def __setattr__(self, name: str, val):
self[name] = val
def normalize_result_fields(self):
self.title = self.WHITESPACE_REGEX.sub(" ", self.title)
if not self.parsed_url and self.url:
self.parsed_url = urllib.parse.urlparse(self.url)
# if the result has no scheme, use http as default
if not self.parsed_url.scheme:
self.parsed_url = self.parsed_url._replace(scheme="http")
self.url = self.parsed_url.geturl()
if self.content:
self.content = self.WHITESPACE_REGEX.sub(" ", self.content)
if self.content == self.title:
# avoid duplicate content between the content and title fields
self.content = ""

141
searx/result_types/answer.py Normal file
View file

@ -0,0 +1,141 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
"""
Typification of the *answer* results. Results of this type are rendered in
the :origin:`answers.html <searx/templates/simple/elements/answers.html>`
template.
----
.. autoclass:: BaseAnswer
:members:
:show-inheritance:
.. autoclass:: Answer
:members:
:show-inheritance:
.. autoclass:: Translations
:members:
:show-inheritance:
.. autoclass:: AnswerSet
:members:
:show-inheritance:
"""
# pylint: disable=too-few-public-methods
from __future__ import annotations
__all__ = ["AnswerSet", "Answer", "Translations"]
import msgspec
from ._base import Result
class BaseAnswer(Result, kw_only=True):
"""Base class of all answer types. It is not intended to build instances of
this class (aka *abstract*)."""
class AnswerSet:
"""Aggregator for :py:obj:`BaseAnswer` items in a result container."""
def __init__(self):
self._answerlist = []
def __len__(self):
return len(self._answerlist)
def __bool__(self):
return bool(self._answerlist)
def add(self, answer: BaseAnswer) -> None:
a_hash = hash(answer)
for i in self._answerlist:
if hash(i) == a_hash:
return
self._answerlist.append(answer)
def __iter__(self):
"""Sort items in this set and iterate over the items."""
self._answerlist.sort(key=lambda answer: answer.template)
yield from self._answerlist
def __contains__(self, answer: BaseAnswer) -> bool:
a_hash = hash(answer)
for i in self._answerlist:
if hash(i) == a_hash:
return True
return False
class Answer(BaseAnswer, kw_only=True):
"""Simple answer type where the *answer* is a simple string with an optional
:py:obj:`url field <Result.url>` field to link a resource (article, map, ..)
related to the answer."""
template: str = "answer/legacy.html"
answer: str
"""Text of the answer."""
def __hash__(self):
"""The hash value of field *answer* is the hash value of the
:py:obj:`Answer` object. :py:obj:`Answer <Result.__eq__>` objects are
equal, when the hash values of both objects are equal."""
return hash(self.answer)
class Translations(BaseAnswer, kw_only=True):
"""Answer type with a list of translations.
The items in the list of :py:obj:`Translations.translations` are of type
:py:obj:`Translations.Item`:
.. code:: python
def response(resp):
results = []
...
foo_1 = Translations.Item(
text="foobar",
synonyms=["bar", "foo"],
examples=["foo and bar are placeholders"],
)
foo_url="https://www.deepl.com/de/translator#en/de/foo"
...
Translations(results=results, translations=[foo], url=foo_url)
"""
template: str = "answer/translations.html"
"""The template in :origin:`answer/translations.html
<searx/templates/simple/answer/translations.html>`"""
translations: list[Translations.Item]
"""List of translations."""
class Item(msgspec.Struct, kw_only=True):
"""A single element of the translations / a translation. A translation
consists of at least a mandatory ``text`` property (the translation) ,
optional properties such as *definitions*, *synonyms* and *examples* are
possible."""
text: str
"""Translated text."""
transliteration: str = ""
"""Transliteration_ of the requested translation.
.. _Transliteration: https://en.wikipedia.org/wiki/Transliteration
"""
examples: list[str] = []
"""List of examples for the requested translation."""
definitions: list[str] = []
"""List of definitions for the requested translation."""
synonyms: list[str] = []
"""List of synonyms for the requested translation."""

121
searx/results.py
View file

@ -1,6 +1,8 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring
from __future__ import annotations
import warnings
import re
from collections import defaultdict
from operator import itemgetter
@ -12,8 +14,10 @@ from searx import logger
from searx.engines import engines
from searx.metrics import histogram_observe, counter_add, count_error
from searx.result_types import Result, LegacyResult
from searx.result_types.answer import AnswerSet, BaseAnswer
CONTENT_LEN_IGNORED_CHARS_REGEX = re.compile(r'[,;:!?\./\\\\ ()-_]', re.M | re.U)
WHITESPACE_REGEX = re.compile('( |\t|\n)+', re.M | re.U)
# return the meaningful length of the content for a result
@ -183,56 +187,76 @@ class ResultContainer:
def __init__(self):
super().__init__()
self._merged_results = []
self.infoboxes = []
self.suggestions = set()
self.answers = {}
self._merged_results: list[LegacyResult] = []
self.infoboxes: list[dict] = []
self.suggestions: set[str] = set()
self.answers = AnswerSet()
self.corrections = set()
self._number_of_results = []
self.engine_data = defaultdict(dict)
self._closed = False
self.paging = False
self._number_of_results: list[int] = []
self.engine_data: dict[str, str | dict] = defaultdict(dict)
self._closed: bool = False
self.paging: bool = False
self.unresponsive_engines: Set[UnresponsiveEngine] = set()
self.timings: List[Timing] = []
self.redirect_url = None
self.on_result = lambda _: True
self._lock = RLock()
def extend(self, engine_name, results): # pylint: disable=too-many-branches
def extend(self, engine_name: str | None, results): # pylint: disable=too-many-branches
if self._closed:
return
standard_result_count = 0
error_msgs = set()
for result in list(results):
result['engine'] = engine_name
if 'suggestion' in result and self.on_result(result):
self.suggestions.add(result['suggestion'])
elif 'answer' in result and self.on_result(result):
self.answers[result['answer']] = result
elif 'correction' in result and self.on_result(result):
self.corrections.add(result['correction'])
elif 'infobox' in result and self.on_result(result):
self._merge_infobox(result)
elif 'number_of_results' in result and self.on_result(result):
self._number_of_results.append(result['number_of_results'])
elif 'engine_data' in result and self.on_result(result):
self.engine_data[engine_name][result['key']] = result['engine_data']
elif 'url' in result:
# standard result (url, title, content)
if not self._is_valid_url_result(result, error_msgs):
continue
# normalize the result
self._normalize_url_result(result)
# call on_result call searx.search.SearchWithPlugins._on_result
# which calls the plugins
if not self.on_result(result):
continue
self.__merge_url_result(result, standard_result_count + 1)
standard_result_count += 1
elif self.on_result(result):
self.__merge_result_no_url(result, standard_result_count + 1)
standard_result_count += 1
if isinstance(result, Result):
result.engine = result.engine or engine_name
result.normalize_result_fields()
if isinstance(result, BaseAnswer) and self.on_result(result):
self.answers.add(result)
else:
# more types need to be implemented in the future ..
raise NotImplementedError(f"no handler implemented to process the result of type {result}")
else:
result['engine'] = result.get('engine') or engine_name or ""
result = LegacyResult(result) # for backward compatibility, will be romeved one day
if 'suggestion' in result and self.on_result(result):
self.suggestions.add(result['suggestion'])
elif 'answer' in result and self.on_result(result):
warnings.warn(
f"answer results from engine {result.engine}"
" are without typification / migrate to Answer class.",
DeprecationWarning,
)
self.answers.add(result)
elif 'correction' in result and self.on_result(result):
self.corrections.add(result['correction'])
elif 'infobox' in result and self.on_result(result):
self._merge_infobox(result)
elif 'number_of_results' in result and self.on_result(result):
self._number_of_results.append(result['number_of_results'])
elif 'engine_data' in result and self.on_result(result):
self.engine_data[result.engine][result['key']] = result['engine_data']
elif result.url:
# standard result (url, title, content)
if not self._is_valid_url_result(result, error_msgs):
continue
# normalize the result
result.normalize_result_fields()
# call on_result call searx.search.SearchWithPlugins._on_result
# which calls the plugins
if not self.on_result(result):
continue
self.__merge_url_result(result, standard_result_count + 1)
standard_result_count += 1
elif self.on_result(result):
self.__merge_result_no_url(result, standard_result_count + 1)
standard_result_count += 1
if len(error_msgs) > 0:
for msg in error_msgs:
@ -279,27 +303,6 @@ class ResultContainer:
return True
def _normalize_url_result(self, result):
"""Return True if the result is valid"""
result['parsed_url'] = urlparse(result['url'])
# if the result has no scheme, use http as default
if not result['parsed_url'].scheme:
result['parsed_url'] = result['parsed_url']._replace(scheme="http")
result['url'] = result['parsed_url'].geturl()
# avoid duplicate content between the content and title fields
if result.get('content') == result.get('title'):
del result['content']
# make sure there is a template
if 'template' not in result:
result['template'] = 'default.html'
# strip multiple spaces and carriage returns from content
if result.get('content'):
result['content'] = WHITESPACE_REGEX.sub(' ', result['content'])
def __merge_url_result(self, result, position):
result['engines'] = set([result['engine']])
with self._lock:

51
searx/search/__init__.py
View file

@ -1,28 +1,30 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# pylint: disable=missing-module-docstring, too-few-public-methods
# the public namespace has not yet been finally defined ..
# __all__ = ["EngineRef", "SearchQuery"]
import threading
from copy import copy
from timeit import default_timer
from uuid import uuid4
import flask
from flask import copy_current_request_context
import babel
from searx import settings
from searx.answerers import ask
from searx.external_bang import get_bang_url
from searx.results import ResultContainer
from searx import logger
from searx.plugins import plugins
from searx.search.models import EngineRef, SearchQuery
from searx import settings
import searx.answerers
import searx.plugins
from searx.engines import load_engines
from searx.network import initialize as initialize_network, check_network_configuration
from searx.extended_types import SXNG_Request
from searx.external_bang import get_bang_url
from searx.metrics import initialize as initialize_metrics, counter_inc, histogram_observe_time
from searx.search.processors import PROCESSORS, initialize as initialize_processors
from searx.network import initialize as initialize_network, check_network_configuration
from searx.results import ResultContainer
from searx.search.checker import initialize as initialize_checker
from searx.search.models import SearchQuery
from searx.search.processors import PROCESSORS, initialize as initialize_processors
from .models import EngineRef, SearchQuery
logger = logger.getChild('search')
@ -68,17 +70,10 @@ class Search:
return False
def search_answerers(self):
"""
Check if an answer return a result.
If yes, update self.result_container and return True
"""
answerers_results = ask(self.search_query)
if answerers_results:
for results in answerers_results:
self.result_container.extend('answer', results)
return True
return False
results = searx.answerers.STORAGE.ask(self.search_query.query)
self.result_container.extend(None, results)
return bool(results)
# do search-request
def _get_requests(self):
@ -184,11 +179,11 @@ class Search:
class SearchWithPlugins(Search):
"""Inherit from the Search class, add calls to the plugins."""
__slots__ = 'ordered_plugin_list', 'request'
__slots__ = 'user_plugins', 'request'
def __init__(self, search_query: SearchQuery, ordered_plugin_list, request: flask.Request):
def __init__(self, search_query: SearchQuery, request: SXNG_Request, user_plugins: list[str]):
super().__init__(search_query)
self.ordered_plugin_list = ordered_plugin_list
self.user_plugins = user_plugins
self.result_container.on_result = self._on_result
# pylint: disable=line-too-long
# get the "real" request to use it outside the Flask context.
@ -200,14 +195,14 @@ class SearchWithPlugins(Search):
self.request = request._get_current_object()
def _on_result(self, result):
return plugins.call(self.ordered_plugin_list, 'on_result', self.request, self, result)
return searx.plugins.STORAGE.on_result(self.request, self, result)
def search(self) -> ResultContainer:
if plugins.call(self.ordered_plugin_list, 'pre_search', self.request, self):
if searx.plugins.STORAGE.pre_search(self.request, self):
super().search()
plugins.call(self.ordered_plugin_list, 'post_search', self.request, self)
searx.plugins.STORAGE.post_search(self.request, self)
self.result_container.close()
return self.result_container

1
searx/search/processors/abstract.py
View file

@ -158,6 +158,7 @@ class EngineProcessor(ABC):
return None
params = {}
params["query"] = search_query.query
params['category'] = engine_category
params['pageno'] = search_query.pageno
params['safesearch'] = search_query.safesearch

24
searx/settings.yml
View file

@ -148,14 +148,24 @@ ui:
# URL formatting: pretty, full or host
url_formatting: pretty
# Lock arbitrary settings on the preferences page. To find the ID of the user
# setting you want to lock, check the ID of the form on the page "preferences".
# Lock arbitrary settings on the preferences page.
#
# preferences:
# lock:
# - categories
# - language
# - autocomplete
# - favicon
# - safesearch
# - method
# - doi_resolver
# - locale
# - theme
# - results_on_new_tab
# - infinite_scroll
# - search_on_category_select
# - method
# - image_proxy
# - query_in_title
# searx supports result proxification using an external service:
@ -217,14 +227,15 @@ outgoing:
# - fe80::/126
# External plugin configuration, for more details see
# https://docs.searxng.org/dev/plugins.html
# https://docs.searxng.org/admin/settings/settings_plugins.html
#
# plugins:
# - plugin1
# - plugin2
# - mypackage.mymodule.MyPlugin
# - mypackage.mymodule.MyOtherPlugin
# - ...
# Comment or un-comment plugin to activate / deactivate by default.
# https://docs.searxng.org/admin/settings/settings_plugins.html
#
# enabled_plugins:
# # these plugins are enabled if nothing is configured ..
@ -1138,8 +1149,7 @@ engines:
engine: libretranslate
# https://github.com/LibreTranslate/LibreTranslate?tab=readme-ov-file#mirrors
base_url:
- https://translate.terraprint.co
- https://trans.zillyhuhn.com
- https://libretranslate.com/translate
# api_key: abc123
shortcut: lt
disabled: true

5
searx/settings_defaults.py
View file

@ -245,8 +245,3 @@ SCHEMA = {
'engines': SettingsValue(list, []),
'doi_resolvers': {},
}
def settings_set_defaults(settings):
apply_schema(settings, SCHEMA, [])
return settings

3
searx/static/plugins/external_plugins/.gitignore vendored
View file

@ -1,3 +0,0 @@
*
*/
!.gitignore

8
searx/static/themes/simple/src/less/preferences.less
View file

@ -48,6 +48,14 @@ table {
}
}
div.pref-group {
width: 100%;
font-weight: normal;
padding: 1rem 0.5rem;
.ltr-text-align-left();
background: var(--color-settings-table-group-background);
}
.value {
margin: 0;
padding: 0;

8
searx/templates/simple/answer/legacy.html Normal file
View file

@ -0,0 +1,8 @@
<span>{{ answer.answer }}</span>
{%- if answer.url -%}
<a href="{{ answer.url }}" class="answer-url"
{%- if results_on_new_tab %} target="_blank" rel="noopener noreferrer"
{%- else -%} rel="noreferrer"
{%- endif -%}
>{{ urlparse(answer.url).hostname }}</a>
{% endif -%}

52
searx/templates/simple/answer/translations.html Normal file
View file

@ -0,0 +1,52 @@
<details class="answer-translations">
<summary>{{ answer.translations[0].text }}</summary>
<dl>
{%- for item in answer.translations -%}
<dt>{{ item.text }}</dt>
<dd>
{%- if item.transliteration -%}
<div class="item-transliteration">{{ item.transliteration }}</div>
{%- endif -%}
{%- if item.examples -%}
<div>{{ _('Examples') }}</div>
<ul>
{%- for i in item.examples -%}
<li>{{ i }}</li>
{%- endfor -%}
</ul>
{%- endif -%}
{%- if item.definitions -%}
<div>{{ _('Definitions') }}</div>
<ul>
{%- for i in item.definitions -%}
<li>{{ i }}</li>
{%- endfor -%}
</ul>
{%- endif -%}
{%- if item.synonyms -%}
<div>{{ _('Synonyms') }}</div>
<ul>
{%- for i in item.synonyms -%}
<li>{{ i }}</li>
{%- endfor -%}
</ul>
{%- endif -%}
</dd>
{%- endfor -%}
</dl>
</details>
{%- if answer.url -%}
<a href="{{ answer.url }}" class="answer-url"
{%- if results_on_new_tab %}
target="_blank" rel="noopener noreferrer"
{%- else -%}
rel="noreferrer"
{%- endif -%}
>{{ answer.engine }}</a>
{%- else -%}
<span class="answer-url">{{ answer.engine }}</span>
{% endif -%}

38
searx/templates/simple/answerers/translate.html
View file

@ -1,38 +0,0 @@
<div class="answer-translations">
{% for translation in translations %}
{% if loop.index > 1 %}
<hr />
{% endif %}
<h3>{{ translation.text }}</h3>
{% if translation.transliteration %}
<b>translation.transliteration</b>
{% endif %} {% if translation.definitions %}
<dl>
<dt>{{ _('Definitions') }}</dt>
<ul>
{% for definition in translation.definitions %}
<li>{{ definition }}</li>
{% endfor %}
<ul>
</dl>
{% endif %} {% if translation.examples %}
<dl>
<dt>{{ _('Examples') }}</dt>
<ul>
{% for example in translation.examples %}
<li>{{ example }}</li>
{% endfor %}
</ul>
</dl>
{% endif %} {% if translation.synonyms %}
<dl>
<dt>{{ _('Synonyms') }}</dt>
<ul>
{% for synonym in translation.synonyms %}
<li>{{ synonym }}</li>
{% endfor %}
</ul>
</dl>
{% endif %}
{% endfor %}
</div>

1
searx/templates/simple/base.html
View file

@ -20,7 +20,6 @@
{% if get_setting('server.limiter') or get_setting('server.public_instance') %}
<link rel="stylesheet" href="{{ url_for('client_token', token=link_token) }}" type="text/css">
{% endif %}
{% block styles %}{% endblock %}
<!--[if gte IE 9]>-->
<script src="{{ url_for('static', filename='js/searxng.head.min.js') }}" client_settings="{{ client_settings }}"></script>
<!--<![endif]-->

8
searx/templates/simple/elements/answers.html Normal file
View file

@ -0,0 +1,8 @@
<div id="answers" role="complementary" aria-labelledby="answers-title">
<h4 class="title" id="answers-title">{{ _('Answers') }} : </h4>
{%- for answer in answers -%}
<div class="answer">
{%- include ("simple/" + (answer.template or "answer/legacy.html")) -%}
</div>
{%- endfor -%}
</div>

19
searx/templates/simple/elements/corrections.html Normal file
View file

@ -0,0 +1,19 @@
<div id="corrections" role="complementary" aria-labelledby="corrections-title">
<h4 id="corrections-title">{{ _('Try searching for:') }}</h4>
{% for correction in corrections %}
<div class="left">
<form method="{{ method or 'POST' }}" action="{{ url_for('search') }}" role="navigation">
{% for category in selected_categories %}
<input type="hidden" name="category_{{ category }}" value="1">
{% endfor %}
<input type="hidden" name="q" value="{{ correction.url }}">
<input type="hidden" name="language" value="{{ current_language }}">
<input type="hidden" name="time_range" value="{{ time_range }}">
<input type="hidden" name="safesearch" value="{{ safesearch }}">
<input type="hidden" name="theme" value="{{ theme }}">
{% if timeout_limit %}<input type="hidden" name="timeout_limit" value="{{ timeout_limit }}" >{% endif %}
<input type="submit" role="link" value="{{ correction.title }}">
</form>
</div>
{% endfor %}
</div>

44
searx/templates/simple/preferences.html
View file

@ -37,19 +37,19 @@
{%- endmacro -%}
{%- macro plugin_preferences(section) -%}
{%- for plugin in plugins -%}
{%- if plugin.preference_section == section -%}
<fieldset>{{- '' -}}
<legend>{{ _(plugin.name) }}</legend>{{- '' -}}
<div class="value">
{{- checkbox_onoff_reversed('plugin_' + plugin.id, plugin.id not in allowed_plugins, 'plugin_labelledby' + plugin.id) -}}
</div>{{- '' -}}
<div class="description" id="{{ 'plugin_labelledby' + plugin.id }}">
{{- _(plugin.description) -}}
</div>{{- '' -}}
</fieldset>
{%- endif -%}
{%- endfor -%}
{%- for plugin in plugins_storage -%}
{%- if plugin.preference_section == section -%}
<fieldset>{{- '' -}}
<legend>{{ _(plugin.name) }}</legend>{{- '' -}}
<div class="value">
{{- checkbox_onoff_reversed('plugin_' + plugin.id, plugin.id not in allowed_plugins, 'plugin_labelledby' + plugin.id) -}}
</div>{{- '' -}}
<div class="description" id="{{ 'plugin_labelledby' + plugin.id }}">
{{- _(plugin.description) -}}
</div>{{- '' -}}
</fieldset>
{%- endif -%}
{%- endfor -%}
{%- endmacro -%}
{%- macro engine_about(search_engine) -%}
@ -158,6 +158,7 @@
<form id="search_form" method="post" action="{{ url_for('preferences') }}" autocomplete="off">
{{- tabs_open() -}}
{# tab: general #}
{{- tab_header('maintab', 'general', _('General'), True) -}}
{%- if 'categories' not in locked_preferences -%}
@ -179,13 +180,16 @@
{% if 'safesearch' not in locked_preferences %}
{%- include 'simple/preferences/safesearch.html' -%}
{%- endif -%}
{%- include 'simple/preferences/tokens.html' -%}
{{- plugin_preferences('general') -}}
{%- if 'doi_resolver' not in locked_preferences %}
{%- include 'simple/preferences/doi_resolver.html' -%}
{%- endif -%}
{%- include 'simple/preferences/tokens.html' -%}
{{- tab_footer() -}}
{# tab: ui #}
{{- tab_header('maintab', 'ui', _('User interface')) -}}
{%- if 'locale' not in locked_preferences -%}
@ -208,6 +212,7 @@
{{- plugin_preferences('ui') -}}
{{- tab_footer() -}}
{# tab: privacy #}
{{- tab_header('maintab', 'privacy', _('Privacy')) -}}
{%- if 'method' not in locked_preferences -%}
@ -222,6 +227,8 @@
{{- plugin_preferences('privacy') -}}
{{- tab_footer() -}}
{# tab: enignes #}
{{- tab_header('maintab', 'engines', _('Engines')) -}}
<p>
{{- _('Currently used search engines') -}}
@ -231,18 +238,23 @@
{{- tabs_close() -}}
{{- tab_footer() -}}
{# tab: query #}
{{- tab_header('maintab', 'query', _('Special Queries')) -}}
{%- if answerers -%}
{%- if answer_storage -%}
{%- include 'simple/preferences/answerers.html' -%}
{%- endif -%}
{{- tab_footer() -}}
{# tab: cookies #}
{{- tab_header('maintab', 'cookies', _('Cookies')) -}}
{%- include 'simple/preferences/cookies.html' -%}
{{- tab_footer() -}}
{{- tabs_close() -}}
{# footer with save & reset buttons #}
{%- include 'simple/preferences/footer.html' -%}
</form>{{- '' -}}

18
searx/templates/simple/preferences/answerers.html
View file

@ -2,7 +2,7 @@
<table class="striped">{{- '' -}}
<tr>{{- '' -}}
<th>{{ _('Allow') }}</th>{{- '' -}}
<th>{{ _('Keywords') }}</th>{{- '' -}}
<th>{{ _('Keywords (first word in query)') }}</th>{{- '' -}}
<th>{{ _('Name') }}</th>{{- '' -}}
<th>{{ _('Description') }}</th>{{- '' -}}
<th>{{ _('Examples') }}</th>{{- '' -}}
@ -14,13 +14,13 @@
</th>{{- '' -}}
</tr>
{%- for answerer in answerers -%}
{%- for info in answer_storage -%}
<tr>{{- '' -}}
<td class="checkbox-col"></td>{{- '' -}}
<td>{{ answerer.keywords|join(', ') }}</td>{{- '' -}}
<td>{{ answerer.info.name }}</td>{{- '' -}}
<td>{{ answerer.info.description }}</td>{{- '' -}}
<td>{{ answerer.info.examples|join(', ') }}</td>{{- '' -}}
<td>{{ info.keywords|join(', ') }}</td>{{- '' -}}
<td>{{ info.name }}</td>{{- '' -}}
<td>{{ info.description }}</td>{{- '' -}}
<td>{{ info.examples|join(', ') }}</td>{{- '' -}}
</tr>
{%- endfor -%}
@ -30,14 +30,14 @@
</th>{{- '' -}}
</th>
{%- for plugin in plugins -%}
{%- for plugin in plugins_storage -%}
{%- if plugin.preference_section == 'query' -%}
<tr>{{- '' -}}
<td class="checkbox-col">{{- checkbox_onoff_reversed('plugin_' + plugin.id, plugin.id not in allowed_plugins, 'plugin_labelledby' + plugin.id) -}}</td>{{- '' -}}
<td>{{ plugin.query_keywords|join(', ') }}</td>{{- '' -}}
<td>{{ plugin.keywords|join(', ') }}</td>{{- '' -}}
<td>{{ _(plugin.name) }}</td>{{- '' -}}
<td id="{{ 'plugin_labelledby' + plugin.id }}">{{ _(plugin.description) }}</td>{{- '' -}}
<td>{{ plugin.query_examples }}</td>{{- '' -}}
<td>{{ plugin.examples|join(', ') }}</td>{{- '' -}}
</tr>
{%- endif -%}
{%- endfor -%}

6
searx/templates/simple/preferences/doi_resolver.html
View file

@ -1,3 +1,7 @@
<div class="pref-group">{{- _('Digital Object Identifier (DOI)') -}}</div>
{{- plugin_preferences('general/doi_resolver') -}}
<fieldset>{{- '' -}}
<legend id="pref_doi_resolver">{{- _('Open Access DOI resolver') -}}</legend>{{- '' -}}
<div class="value">{{- '' -}}
@ -5,7 +9,7 @@
{%- for doi_resolver_name,doi_resolver_url in doi_resolvers.items() -%}
<option value="{{ doi_resolver_name }}"
{%- if doi_resolver_url == current_doi_resolver %} selected="selected" {%- endif -%}>
{{- doi_resolver_name }} - {{ doi_resolver_url -}}
{{- doi_resolver_name }} - {{ doi_resolver_url -}}
</option>
{%- endfor -%}
</select>{{- '' -}}

50
searx/templates/simple/results.html
View file

@ -19,28 +19,10 @@
{% endif %}
<div id="results" class="{{ only_template }}">
{% if answers -%}
<div id="answers" role="complementary" aria-labelledby="answers-title"><h4 class="title" id="answers-title">{{ _('Answers') }} : </h4>
{%- for answer in answers.values() -%}
<div class="answer">
{%- if answer.answer_type == 'translations' -%}
{% with translations=answer.translations %}
{% include 'simple/answerers/translate.html' %}
{% endwith %}
{%- else -%}
<span>{{ answer.answer }}</span>
{%- if answer.url -%}
<a href="{{ answer.url }}" class="answer-url"
{%- if results_on_new_tab %} target="_blank" rel="noopener noreferrer"
{%- else -%} rel="noreferrer"
{%- endif -%}
>{{ urlparse(answer.url).hostname }}</a>
{% endif -%}
{%- endif -%}
</div>
{%- endfor -%}
</div>
{%- endif %}
{%- if answers -%}
{%- include 'simple/elements/answers.html' -%}
{%- endif %}
<div id="sidebar">
@ -76,27 +58,9 @@
<div id="sidebar-end-collapsible"></div>
</div>
{% if corrections %}
<div id="corrections" role="complementary" aria-labelledby="corrections-title">
<h4 id="corrections-title">{{ _('Try searching for:') }}</h4>
{% for correction in corrections %}
<div class="left">
<form method="{{ method or 'POST' }}" action="{{ url_for('search') }}" role="navigation">
{% for category in selected_categories %}
<input type="hidden" name="category_{{ category }}" value="1">
{% endfor %}
<input type="hidden" name="q" value="{{ correction.url }}">
<input type="hidden" name="language" value="{{ current_language }}">
<input type="hidden" name="time_range" value="{{ time_range }}">
<input type="hidden" name="safesearch" value="{{ safesearch }}">
<input type="hidden" name="theme" value="{{ theme }}">
{% if timeout_limit %}<input type="hidden" name="timeout_limit" value="{{ timeout_limit }}" >{% endif %}
<input type="submit" role="link" value="{{ correction.title }}">
</form>
</div>
{% endfor %}
</div>
{% endif %}
{%- if corrections -%}
{%- include 'simple/elements/corrections' -%}
{%- endif -%}
<div id="urls" role="main">
{% for result in results %}

290
searx/webapp.py
View file

@ -4,6 +4,7 @@
"""
# pylint: disable=use-dict-literal
from __future__ import annotations
import hashlib
import hmac
@ -16,12 +17,12 @@ from timeit import default_timer
from html import escape
from io import StringIO
import typing
from typing import List, Dict, Iterable
import urllib
import urllib.parse
from urllib.parse import urlencode, urlparse, unquote
import warnings
import httpx
from pygments import highlight
@ -47,6 +48,7 @@ from flask_babel import (
format_decimal,
)
from searx.extended_types import sxng_request
from searx import (
logger,
get_setting,
@ -59,7 +61,7 @@ from searx import limiter
from searx.botdetection import link_token
from searx.data import ENGINE_DESCRIPTIONS
from searx.results import Timing
from searx.result_types import Answer
from searx.settings_defaults import OUTPUT_FORMATS
from searx.settings_loader import DEFAULT_SETTINGS_FILE
from searx.exceptions import SearxParameterException
@ -90,17 +92,16 @@ from searx.webadapter import (
from searx.utils import gen_useragent, dict_subset
from searx.version import VERSION_STRING, GIT_URL, GIT_BRANCH
from searx.query import RawTextQuery
from searx.plugins import Plugin, plugins, initialize as plugin_initialize
from searx.plugins.oa_doi_rewrite import get_doi_resolver
from searx.preferences import (
Preferences,
ClientPref,
ValidationException,
)
from searx.answerers import (
answerers,
ask,
)
import searx.answerers
import searx.plugins
from searx.metrics import get_engines_stats, get_engine_errors, get_reliabilities, histogram, counter, openmetrics
from searx.flaskfix import patch_application
@ -118,12 +119,15 @@ from searx import favicons
from searx.redisdb import initialize as redis_initialize
from searx.sxng_locales import sxng_locales
from searx.search import SearchWithPlugins, initialize as search_initialize
import searx.search
from searx.network import stream as http_stream, set_context_network_name
from searx.search.checker import get_result as checker_get_result
logger = logger.getChild('webapp')
warnings.simplefilter("always")
# check secret_key
if not searx_debug and settings['server']['secret_key'] == 'ultrasecretkey':
logger.error('server.secret_key is not changed. Please use something else instead of ultrasecretkey.')
@ -158,24 +162,9 @@ app.jinja_env.filters['group_engines_in_tab'] = group_engines_in_tab # pylint:
app.secret_key = settings['server']['secret_key']
class ExtendedRequest(flask.Request):
"""This class is never initialized and only used for type checking."""
preferences: Preferences
errors: List[str]
user_plugins: List[Plugin]
form: Dict[str, str]
start_time: float
render_time: float
timings: List[Timing]
request = typing.cast(ExtendedRequest, flask.request)
def get_locale():
locale = localeselector()
logger.debug("%s uses locale `%s`", urllib.parse.quote(request.url), locale)
logger.debug("%s uses locale `%s`", urllib.parse.quote(sxng_request.url), locale)
return locale
@ -260,7 +249,7 @@ def custom_url_for(endpoint: str, **values):
file_hash = static_files.get(values['filename'])
if not file_hash:
# try file in the current theme
theme_name = request.preferences.get_value('theme')
theme_name = sxng_request.preferences.get_value('theme')
filename_with_theme = "themes/{}/{}".format(theme_name, values['filename'])
file_hash = static_files.get(filename_with_theme)
if file_hash:
@ -268,7 +257,7 @@ def custom_url_for(endpoint: str, **values):
if get_setting('ui.static_use_hash') and file_hash:
suffix = "?" + file_hash
if endpoint == 'info' and 'locale' not in values:
locale = request.preferences.get_value('locale')
locale = sxng_request.preferences.get_value('locale')
if infopage.INFO_PAGES.get_page(values['pagename'], locale) is None:
locale = infopage.INFO_PAGES.locale_default
values['locale'] = locale
@ -295,7 +284,7 @@ def image_proxify(url: str):
if url.startswith('//'):
url = 'https:' + url
if not request.preferences.get_value('image_proxy'):
if not sxng_request.preferences.get_value('image_proxy'):
return url
if url.startswith('data:image/'):
@ -328,11 +317,11 @@ def get_translations():
}
def get_enabled_categories(category_names: Iterable[str]):
def get_enabled_categories(category_names: typing.Iterable[str]):
"""The categories in ``category_names```for which there is no active engine
are filtered out and a reduced list is returned."""
enabled_engines = [item[0] for item in request.preferences.engines.get_enabled()]
enabled_engines = [item[0] for item in sxng_request.preferences.engines.get_enabled()]
enabled_categories = set()
for engine_name in enabled_engines:
enabled_categories.update(engines[engine_name].categories)
@ -340,7 +329,7 @@ def get_enabled_categories(category_names: Iterable[str]):
def get_pretty_url(parsed_url: urllib.parse.ParseResult):
url_formatting_pref = request.preferences.get_value('url_formatting')
url_formatting_pref = sxng_request.preferences.get_value('url_formatting')
if url_formatting_pref == 'full':
return [parsed_url.geturl()]
@ -355,7 +344,7 @@ def get_pretty_url(parsed_url: urllib.parse.ParseResult):
def get_client_settings():
req_pref = request.preferences
req_pref = sxng_request.preferences
return {
'autocomplete': req_pref.get_value('autocomplete'),
'autocomplete_min': get_setting('search.autocomplete_min'),
@ -389,13 +378,13 @@ def render(template_name: str, **kwargs):
),
encoding='utf-8',
)
kwargs['preferences'] = request.preferences
kwargs['preferences'] = sxng_request.preferences
kwargs.update(client_settings)
# values from the HTTP requests
kwargs['endpoint'] = 'results' if 'q' in kwargs else request.endpoint
kwargs['cookies'] = request.cookies
kwargs['errors'] = request.errors
kwargs['endpoint'] = 'results' if 'q' in kwargs else sxng_request.endpoint
kwargs['cookies'] = sxng_request.cookies
kwargs['errors'] = sxng_request.errors
kwargs['link_token'] = link_token.get_token()
kwargs['categories_as_tabs'] = list(settings['categories_as_tabs'].keys())
@ -405,14 +394,14 @@ def render(template_name: str, **kwargs):
# i18n
kwargs['sxng_locales'] = [l for l in sxng_locales if l[0] in settings['search']['languages']]
locale = request.preferences.get_value('locale')
locale = sxng_request.preferences.get_value('locale')
kwargs['locale_rfc5646'] = _get_locale_rfc5646(locale)
if locale in RTL_LOCALES and 'rtl' not in kwargs:
kwargs['rtl'] = True
if 'current_language' not in kwargs:
kwargs['current_language'] = parse_lang(request.preferences, {}, RawTextQuery('', []))
kwargs['current_language'] = parse_lang(sxng_request.preferences, {}, RawTextQuery('', []))
# values from settings
kwargs['search_formats'] = [x for x in settings['search']['formats'] if x != 'html']
@ -442,92 +431,80 @@ def render(template_name: str, **kwargs):
+ '?'
+ urlencode(
{
'method': request.preferences.get_value('method'),
'autocomplete': request.preferences.get_value('autocomplete'),
'method': sxng_request.preferences.get_value('method'),
'autocomplete': sxng_request.preferences.get_value('autocomplete'),
}
)
)
kwargs['urlparse'] = urlparse
# scripts from plugins
kwargs['scripts'] = set()
for plugin in request.user_plugins:
for script in plugin.js_dependencies:
kwargs['scripts'].add(script)
# styles from plugins
kwargs['styles'] = set()
for plugin in request.user_plugins:
for css in plugin.css_dependencies:
kwargs['styles'].add(css)
start_time = default_timer()
result = render_template('{}/{}'.format(kwargs['theme'], template_name), **kwargs)
request.render_time += default_timer() - start_time # pylint: disable=assigning-non-slot
sxng_request.render_time += default_timer() - start_time # pylint: disable=assigning-non-slot
return result
@app.before_request
def pre_request():
request.start_time = default_timer() # pylint: disable=assigning-non-slot
request.render_time = 0 # pylint: disable=assigning-non-slot
request.timings = [] # pylint: disable=assigning-non-slot
request.errors = [] # pylint: disable=assigning-non-slot
sxng_request.start_time = default_timer() # pylint: disable=assigning-non-slot
sxng_request.render_time = 0 # pylint: disable=assigning-non-slot
sxng_request.timings = [] # pylint: disable=assigning-non-slot
sxng_request.errors = [] # pylint: disable=assigning-non-slot
client_pref = ClientPref.from_http_request(request)
client_pref = ClientPref.from_http_request(sxng_request)
# pylint: disable=redefined-outer-name
preferences = Preferences(themes, list(categories.keys()), engines, plugins, client_pref)
preferences = Preferences(themes, list(categories.keys()), engines, searx.plugins.STORAGE, client_pref)
user_agent = request.headers.get('User-Agent', '').lower()
user_agent = sxng_request.headers.get('User-Agent', '').lower()
if 'webkit' in user_agent and 'android' in user_agent:
preferences.key_value_settings['method'].value = 'GET'
request.preferences = preferences # pylint: disable=assigning-non-slot
sxng_request.preferences = preferences # pylint: disable=assigning-non-slot
try:
preferences.parse_dict(request.cookies)
preferences.parse_dict(sxng_request.cookies)
except Exception as e: # pylint: disable=broad-except
logger.exception(e, exc_info=True)
request.errors.append(gettext('Invalid settings, please edit your preferences'))
sxng_request.errors.append(gettext('Invalid settings, please edit your preferences'))
# merge GET, POST vars
# request.form
request.form = dict(request.form.items()) # pylint: disable=assigning-non-slot
for k, v in request.args.items():
if k not in request.form:
request.form[k] = v
# HINT request.form is of type werkzeug.datastructures.ImmutableMultiDict
sxng_request.form = dict(sxng_request.form.items()) # type: ignore
for k, v in sxng_request.args.items():
if k not in sxng_request.form:
sxng_request.form[k] = v
if request.form.get('preferences'):
preferences.parse_encoded_data(request.form['preferences'])
if sxng_request.form.get('preferences'):
preferences.parse_encoded_data(sxng_request.form['preferences'])
else:
try:
preferences.parse_dict(request.form)
preferences.parse_dict(sxng_request.form)
except Exception as e: # pylint: disable=broad-except
logger.exception(e, exc_info=True)
request.errors.append(gettext('Invalid settings'))
sxng_request.errors.append(gettext('Invalid settings'))
# language is defined neither in settings nor in preferences
# use browser headers
if not preferences.get_value("language"):
language = _get_browser_language(request, settings['search']['languages'])
language = _get_browser_language(sxng_request, settings['search']['languages'])
preferences.parse_dict({"language": language})
logger.debug('set language %s (from browser)', preferences.get_value("language"))
# UI locale is defined neither in settings nor in preferences
# use browser headers
if not preferences.get_value("locale"):
locale = _get_browser_language(request, LOCALE_NAMES.keys())
locale = _get_browser_language(sxng_request, LOCALE_NAMES.keys())
preferences.parse_dict({"locale": locale})
logger.debug('set locale %s (from browser)', preferences.get_value("locale"))
# request.user_plugins
request.user_plugins = [] # pylint: disable=assigning-non-slot
sxng_request.user_plugins = [] # pylint: disable=assigning-non-slot
allowed_plugins = preferences.plugins.get_enabled()
disabled_plugins = preferences.plugins.get_disabled()
for plugin in plugins:
if (plugin.default_on and plugin.id not in disabled_plugins) or plugin.id in allowed_plugins:
request.user_plugins.append(plugin)
for plugin in searx.plugins.STORAGE:
if (plugin.id not in disabled_plugins) or plugin.id in allowed_plugins:
sxng_request.user_plugins.append(plugin.id)
@app.after_request
@ -542,13 +519,13 @@ def add_default_headers(response: flask.Response):
@app.after_request
def post_request(response: flask.Response):
total_time = default_timer() - request.start_time
total_time = default_timer() - sxng_request.start_time
timings_all = [
'total;dur=' + str(round(total_time * 1000, 3)),
'render;dur=' + str(round(request.render_time * 1000, 3)),
'render;dur=' + str(round(sxng_request.render_time * 1000, 3)),
]
if len(request.timings) > 0:
timings = sorted(request.timings, key=lambda t: t.total)
if len(sxng_request.timings) > 0:
timings = sorted(sxng_request.timings, key=lambda t: t.total)
timings_total = [
'total_' + str(i) + '_' + t.engine + ';dur=' + str(round(t.total * 1000, 3)) for i, t in enumerate(timings)
]
@ -575,18 +552,18 @@ def index_error(output_format: str, error_message: str):
response_rss = render(
'opensearch_response_rss.xml',
results=[],
q=request.form['q'] if 'q' in request.form else '',
q=sxng_request.form['q'] if 'q' in sxng_request.form else '',
number_of_results=0,
error_message=error_message,
)
return Response(response_rss, mimetype='text/xml')
# html
request.errors.append(gettext('search error'))
sxng_request.errors.append(gettext('search error'))
return render(
# fmt: off
'index.html',
selected_categories=get_selected_categories(request.preferences, request.form),
selected_categories=get_selected_categories(sxng_request.preferences, sxng_request.form),
# fmt: on
)
@ -596,15 +573,15 @@ def index():
"""Render index page."""
# redirect to search if there's a query in the request
if request.form.get('q'):
query = ('?' + request.query_string.decode()) if request.query_string else ''
if sxng_request.form.get('q'):
query = ('?' + sxng_request.query_string.decode()) if sxng_request.query_string else ''
return redirect(url_for('search') + query, 308)
return render(
# fmt: off
'index.html',
selected_categories=get_selected_categories(request.preferences, request.form),
current_locale = request.preferences.get_value("locale"),
selected_categories=get_selected_categories(sxng_request.preferences, sxng_request.form),
current_locale = sxng_request.preferences.get_value("locale"),
# fmt: on
)
@ -616,14 +593,14 @@ def health():
@app.route('/client<token>.css', methods=['GET', 'POST'])
def client_token(token=None):
link_token.ping(request, token)
link_token.ping(sxng_request, token)
return Response('', mimetype='text/css')
@app.route('/rss.xsl', methods=['GET', 'POST'])
def rss_xsl():
return render_template(
f"{request.preferences.get_value('theme')}/rss.xsl",
f"{sxng_request.preferences.get_value('theme')}/rss.xsl",
url_for=custom_url_for,
)
@ -638,7 +615,7 @@ def search():
# pylint: disable=too-many-statements
# output_format
output_format = request.form.get('format', 'html')
output_format = sxng_request.form.get('format', 'html')
if output_format not in OUTPUT_FORMATS:
output_format = 'html'
@ -646,12 +623,12 @@ def search():
flask.abort(403)
# check if there is query (not None and not an empty string)
if not request.form.get('q'):
if not sxng_request.form.get('q'):
if output_format == 'html':
return render(
# fmt: off
'index.html',
selected_categories=get_selected_categories(request.preferences, request.form),
selected_categories=get_selected_categories(sxng_request.preferences, sxng_request.form),
# fmt: on
)
return index_error(output_format, 'No query'), 400
@ -662,10 +639,10 @@ def search():
result_container = None
try:
search_query, raw_text_query, _, _, selected_locale = get_search_query_from_webapp(
request.preferences, request.form
sxng_request.preferences, sxng_request.form
)
search = SearchWithPlugins(search_query, request.user_plugins, request) # pylint: disable=redefined-outer-name
result_container = search.search()
search_obj = searx.search.SearchWithPlugins(search_query, sxng_request, sxng_request.user_plugins)
result_container = search_obj.search()
except SearxParameterException as e:
logger.exception('search error: SearxParameterException')
@ -680,7 +657,7 @@ def search():
# 2. add Server-Timing header for measuring performance characteristics of
# web applications
request.timings = result_container.get_timings() # pylint: disable=assigning-non-slot
sxng_request.timings = result_container.get_timings() # pylint: disable=assigning-non-slot
# 3. formats without a template
@ -745,7 +722,7 @@ def search():
response_rss = render(
'opensearch_response_rss.xml',
results=results,
q=request.form['q'],
q=sxng_request.form['q'],
number_of_results=result_container.number_of_results,
)
return Response(response_rss, mimetype='text/xml')
@ -779,7 +756,7 @@ def search():
# fmt: off
'results.html',
results = results,
q=request.form['q'],
q=sxng_request.form['q'],
selected_categories = search_query.categories,
pageno = search_query.pageno,
time_range = search_query.time_range or '',
@ -793,14 +770,14 @@ def search():
unresponsive_engines = webutils.get_translated_errors(
result_container.unresponsive_engines
),
current_locale = request.preferences.get_value("locale"),
current_locale = sxng_request.preferences.get_value("locale"),
current_language = selected_locale,
search_language = match_locale(
search.search_query.lang,
search_obj.search_query.lang,
settings['search']['languages'],
fallback=request.preferences.get_value("language")
fallback=sxng_request.preferences.get_value("language")
),
timeout_limit = request.form.get('timeout_limit', None),
timeout_limit = sxng_request.form.get('timeout_limit', None),
timings = engine_timings_pairs,
max_response_time = max_response_time
# fmt: on
@ -821,7 +798,7 @@ def info(pagename, locale):
if page is None:
flask.abort(404)
user_locale = request.preferences.get_value('locale')
user_locale = sxng_request.preferences.get_value('locale')
return render(
'info.html',
all_pages=infopage.INFO_PAGES.iter_pages(user_locale, fallback_to_default=True),
@ -838,19 +815,23 @@ def autocompleter():
results = []
# set blocked engines
disabled_engines = request.preferences.engines.get_disabled()
disabled_engines = sxng_request.preferences.engines.get_disabled()
# parse query
raw_text_query = RawTextQuery(request.form.get('q', ''), disabled_engines)
raw_text_query = RawTextQuery(sxng_request.form.get('q', ''), disabled_engines)
sug_prefix = raw_text_query.getQuery()
for obj in searx.answerers.STORAGE.ask(sug_prefix):
if isinstance(obj, Answer):
results.append(obj.answer)
# normal autocompletion results only appear if no inner results returned
# and there is a query part
if len(raw_text_query.autocomplete_list) == 0 and len(sug_prefix) > 0:
# get SearXNG's locale and autocomplete backend from cookie
sxng_locale = request.preferences.get_value('language')
backend_name = request.preferences.get_value('autocomplete')
sxng_locale = sxng_request.preferences.get_value('language')
backend_name = sxng_request.preferences.get_value('autocomplete')
for result in search_autocomplete(backend_name, sug_prefix, sxng_locale):
# attention: this loop will change raw_text_query object and this is
@ -862,11 +843,7 @@ def autocompleter():
for autocomplete_text in raw_text_query.autocomplete_list:
results.append(raw_text_query.get_autocomplete_full_query(autocomplete_text))
for answers in ask(raw_text_query):
for answer in answers:
results.append(str(answer['answer']))
if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
if sxng_request.headers.get('X-Requested-With') == 'XMLHttpRequest':
# the suggestion request comes from the searx search form
suggestions = json.dumps(results)
mimetype = 'application/json'
@ -887,27 +864,27 @@ def preferences():
# pylint: disable=too-many-statements
# save preferences using the link the /preferences?preferences=...
if request.args.get('preferences') or request.form.get('preferences'):
if sxng_request.args.get('preferences') or sxng_request.form.get('preferences'):
resp = make_response(redirect(url_for('index', _external=True)))
return request.preferences.save(resp)
return sxng_request.preferences.save(resp)
# save preferences
if request.method == 'POST':
if sxng_request.method == 'POST':
resp = make_response(redirect(url_for('index', _external=True)))
try:
request.preferences.parse_form(request.form)
sxng_request.preferences.parse_form(sxng_request.form)
except ValidationException:
request.errors.append(gettext('Invalid settings, please edit your preferences'))
sxng_request.errors.append(gettext('Invalid settings, please edit your preferences'))
return resp
return request.preferences.save(resp)
return sxng_request.preferences.save(resp)
# render preferences
image_proxy = request.preferences.get_value('image_proxy') # pylint: disable=redefined-outer-name
disabled_engines = request.preferences.engines.get_disabled()
allowed_plugins = request.preferences.plugins.get_enabled()
image_proxy = sxng_request.preferences.get_value('image_proxy') # pylint: disable=redefined-outer-name
disabled_engines = sxng_request.preferences.engines.get_disabled()
allowed_plugins = sxng_request.preferences.plugins.get_enabled()
# stats for preferences page
filtered_engines = dict(filter(lambda kv: request.preferences.validate_token(kv[1]), engines.items()))
filtered_engines = dict(filter(lambda kv: sxng_request.preferences.validate_token(kv[1]), engines.items()))
engines_by_category = {}
@ -938,7 +915,7 @@ def preferences():
'rate95': rate95,
'warn_timeout': e.timeout > settings['outgoing']['request_timeout'],
'supports_selected_language': e.traits.is_locale_supported(
str(request.preferences.get_value('language') or 'all')
str(sxng_request.preferences.get_value('language') or 'all')
),
'result_count': result_count,
}
@ -991,7 +968,7 @@ def preferences():
supports = {}
for _, e in filtered_engines.items():
supports_selected_language = e.traits.is_locale_supported(
str(request.preferences.get_value('language') or 'all')
str(sxng_request.preferences.get_value('language') or 'all')
)
safesearch = e.safesearch
time_range_support = e.time_range_support
@ -1011,31 +988,28 @@ def preferences():
return render(
# fmt: off
'preferences.html',
selected_categories = get_selected_categories(request.preferences, request.form),
preferences = True,
selected_categories = get_selected_categories(sxng_request.preferences, sxng_request.form),
locales = LOCALE_NAMES,
current_locale = request.preferences.get_value("locale"),
current_locale = sxng_request.preferences.get_value("locale"),
image_proxy = image_proxy,
engines_by_category = engines_by_category,
stats = stats,
max_rate95 = max_rate95,
reliabilities = reliabilities,
supports = supports,
answerers = [
{'info': a.self_info(), 'keywords': a.keywords}
for a in answerers
],
answer_storage = searx.answerers.STORAGE.info,
disabled_engines = disabled_engines,
autocomplete_backends = autocomplete_backends,
favicon_resolver_names = favicons.proxy.CFG.resolver_map.keys(),
shortcuts = {y: x for x, y in engine_shortcuts.items()},
themes = themes,
plugins = plugins,
doi_resolvers = settings['doi_resolvers'],
current_doi_resolver = get_doi_resolver(request.preferences),
plugins_storage = searx.plugins.STORAGE.info,
current_doi_resolver = get_doi_resolver(sxng_request.preferences),
allowed_plugins = allowed_plugins,
preferences_url_params = request.preferences.get_as_url_params(),
locked_preferences = settings['preferences']['lock'],
preferences = True
preferences_url_params = sxng_request.preferences.get_as_url_params(),
locked_preferences = get_setting("preferences.lock", []),
doi_resolvers = get_setting("doi_resolvers", {}),
# fmt: on
)
@ -1047,11 +1021,11 @@ app.add_url_rule('/favicon_proxy', methods=['GET'], endpoint="favicon_proxy", vi
def image_proxy():
# pylint: disable=too-many-return-statements, too-many-branches
url = request.args.get('url')
url = sxng_request.args.get('url')
if not url:
return '', 400
if not is_hmac_of(settings['server']['secret_key'], url.encode(), request.args.get('h', '')):
if not is_hmac_of(settings['server']['secret_key'], url.encode(), sxng_request.args.get('h', '')):
return '', 400
maximum_size = 5 * 1024 * 1024
@ -1143,10 +1117,10 @@ def engine_descriptions():
@app.route('/stats', methods=['GET'])
def stats():
"""Render engine statistics page."""
sort_order = request.args.get('sort', default='name', type=str)
selected_engine_name = request.args.get('engine', default=None, type=str)
sort_order = sxng_request.args.get('sort', default='name', type=str)
selected_engine_name = sxng_request.args.get('engine', default=None, type=str)
filtered_engines = dict(filter(lambda kv: request.preferences.validate_token(kv[1]), engines.items()))
filtered_engines = dict(filter(lambda kv: sxng_request.preferences.validate_token(kv[1]), engines.items()))
if selected_engine_name:
if selected_engine_name not in filtered_engines:
selected_engine_name = None
@ -1209,7 +1183,7 @@ def stats():
@app.route('/stats/errors', methods=['GET'])
def stats_errors():
filtered_engines = dict(filter(lambda kv: request.preferences.validate_token(kv[1]), engines.items()))
filtered_engines = dict(filter(lambda kv: sxng_request.preferences.validate_token(kv[1]), engines.items()))
result = get_engine_errors(filtered_engines)
return jsonify(result)
@ -1227,10 +1201,10 @@ def stats_open_metrics():
if not (settings['general'].get("enable_metrics") and password):
return Response('open metrics is disabled', status=404, mimetype='text/plain')
if not request.authorization or request.authorization.password != password:
if not sxng_request.authorization or sxng_request.authorization.password != password:
return Response('access forbidden', status=401, mimetype='text/plain')
filtered_engines = dict(filter(lambda kv: request.preferences.validate_token(kv[1]), engines.items()))
filtered_engines = dict(filter(lambda kv: sxng_request.preferences.validate_token(kv[1]), engines.items()))
checker_results = checker_get_result()
checker_results = (
@ -1260,11 +1234,11 @@ Disallow: /*?*q=*
@app.route('/opensearch.xml', methods=['GET'])
def opensearch():
method = request.preferences.get_value('method')
autocomplete = request.preferences.get_value('autocomplete')
method = sxng_request.preferences.get_value('method')
autocomplete = sxng_request.preferences.get_value('autocomplete')
# chrome/chromium only supports HTTP GET....
if request.headers.get('User-Agent', '').lower().find('webkit') >= 0:
if sxng_request.headers.get('User-Agent', '').lower().find('webkit') >= 0:
method = 'GET'
if method not in ('POST', 'GET'):
@ -1277,9 +1251,9 @@ def opensearch():
@app.route('/favicon.ico')
def favicon():
theme = request.preferences.get_value("theme")
theme = sxng_request.preferences.get_value("theme")
return send_from_directory(
os.path.join(app.root_path, settings['ui']['static_path'], 'themes', theme, 'img'), # pyright: ignore
os.path.join(app.root_path, settings['ui']['static_path'], 'themes', theme, 'img'), # type: ignore
'favicon.png',
mimetype='image/vnd.microsoft.icon',
)
@ -1288,7 +1262,7 @@ def favicon():
@app.route('/clear_cookies')
def clear_cookies():
resp = make_response(redirect(url_for('index', _external=True)))
for cookie_name in request.cookies:
for cookie_name in sxng_request.cookies:
resp.delete_cookie(cookie_name)
return resp
@ -1298,7 +1272,7 @@ def config():
"""Return configuration in JSON format."""
_engines = []
for name, engine in engines.items():
if not request.preferences.validate_token(engine):
if not sxng_request.preferences.validate_token(engine):
continue
_languages = engine.traits.languages.keys()
@ -1319,8 +1293,8 @@ def config():
)
_plugins = []
for _ in plugins:
_plugins.append({'name': _.name, 'enabled': _.default_on})
for _ in searx.plugins.STORAGE:
_plugins.append({'name': _.id, 'enabled': _.default_on})
_limiter_cfg = limiter.get_cfg()
@ -1375,8 +1349,12 @@ werkzeug_reloader = flask_run_development or (searx_debug and __name__ == "__mai
if not werkzeug_reloader or (werkzeug_reloader and os.environ.get("WERKZEUG_RUN_MAIN") == "true"):
locales_initialize()
redis_initialize()
plugin_initialize(app)
search_initialize(enable_checker=True, check_network=True, enable_metrics=settings['general']['enable_metrics'])
searx.plugins.initialize(app)
searx.search.initialize(
enable_checker=True,
check_network=True,
enable_metrics=get_setting("general.enable_metrics"),
)
limiter.initialize(app, settings)
favicons.init()