From e4e7d73e8eec2ee845e8e077c2b2e78c22accfd6 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Sun, 15 Dec 2024 09:59:50 +0100 Subject: [PATCH] [refactor] typification of SearXNG (initial) / result items (part 1) Typification of SearXNG ======================= This patch introduces the typing of the results. The why and how is described in the documentation, please generate the documentation .. $ make docs.clean docs.live and read the following articles in the "Developer documentation": - result types --> http://0.0.0.0:8000/dev/result_types/index.html The result types are available from the `searx.result_types` module. The following have been implemented so far: - base result type: `searx.result_type.Result` --> http://0.0.0.0:8000/dev/result_types/base_result.html - answer results --> http://0.0.0.0:8000/dev/result_types/answer.html including the type for translations (inspired by #3925). For all other types (which still need to be set up in subsequent PRs), template documentation has been created for the transition period. Doc of the fields used in Templates =================================== The template documentation is the basis for the typing and is the first complete documentation of the results (needed for engine development). It is the "working paper" (the plan) with which further typifications can be implemented in subsequent PRs. - https://github.com/searxng/searxng/issues/357 Answer Templates ================ With the new (sub) types for `Answer`, the templates for the answers have also been revised, `Translation` are now displayed with collapsible entries (inspired by #3925). !en-de dog Plugins & Answerer ================== The implementation for `Plugin` and `Answer` has been revised, see documentation: - Plugin: http://0.0.0.0:8000/dev/plugins/index.html - Answerer: http://0.0.0.0:8000/dev/answerers/index.html With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented) Autocomplete ============ The autocompletion had a bug where the results from `Answer` had not been shown in the past. To test activate autocompletion and try search terms for which we have answerers - statistics: type `min 1 2 3` .. in the completion list you should find an entry like `[de] min(1, 2, 3) = 1` - random: type `random uuid` .. in the completion list, the first item is a random UUID Extended Types ============== SearXNG extends e.g. the request and response types of flask and httpx, a module has been set up for type extensions: - Extended Types --> http://0.0.0.0:8000/dev/extended_types.html Unit-Tests ========== The unit tests have been completely revised. In the previous implementation, the runtime (the global variables such as `searx.settings`) was not initialized before each test, so the runtime environment with which a test ran was always determined by the tests that ran before it. This was also the reason why we sometimes had to observe non-deterministic errors in the tests in the past: - https://github.com/searxng/searxng/issues/2988 is one example for the Runtime issues, with non-deterministic behavior .. - https://github.com/searxng/searxng/pull/3650 - https://github.com/searxng/searxng/pull/3654 - https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469 - https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005 Why msgspec.Struct ================== We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past: - https://github.com/searxng/searxng/pull/1562/files - https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2 - https://github.com/searxng/searxng/pull/1412/files - https://github.com/searxng/searxng/pull/1356 In my opinion, TypeDict is unsuitable because the objects are still dictionaries and not instances of classes / the `dataclass` are classes but ... The `msgspec.Struct` combine the advantages of typing, runtime behaviour and also offer the option of (fast) serializing (incl. type check) the objects. Currently not possible but conceivable with `msgspec`: Outsourcing the engines into separate processes, what possibilities this opens up in the future is left to the imagination! Internally, we have already defined that it is desirable to decouple the development of the engines from the development of the SearXNG core / The serialization of the `Result` objects is a prerequisite for this. HINT: The threads listed above were the template for this PR, even though the implementation here is based on msgspec. They should also be an inspiration for the following PRs of typification, as the models and implementations can provide a good direction. Why just one commit? ==================== I tried to create several (thematically separated) commits, but gave up at some point ... there are too many things to tackle at once / The comprehensibility of the commits would not be improved by a thematic separation. On the contrary, we would have to make multiple changes at the same places and the goal of a change would be vaguely recognizable in the fog of the commits. Signed-off-by: Markus Heiser --- Makefile | 4 +- docs/admin/plugins.rst | 19 +- docs/admin/settings/index.rst | 2 +- docs/admin/settings/settings_plugins.rst | 67 ++ docs/conf.py | 5 +- docs/dev/answerers/builtins.rst | 11 + docs/dev/answerers/development.rst | 7 + docs/dev/answerers/index.rst | 9 + docs/dev/answerers/random.rst | 8 + docs/dev/answerers/statistics.rst | 8 + docs/dev/engines/engine_overview.rst | 335 +--------- docs/dev/extended_types.rst | 7 + docs/dev/index.rst | 6 +- docs/dev/plugins.rst | 106 ---- docs/dev/plugins/builtins.rst | 15 + docs/dev/plugins/calculator.rst | 8 + docs/dev/plugins/development.rst | 7 + docs/dev/plugins/hash_plugin.rst | 8 + .../plugins/hostnames.rst} | 9 +- docs/dev/plugins/index.rst | 9 + docs/dev/plugins/self_info.rst | 8 + .../plugins/tor_check.rst} | 9 +- .../plugins/unit_converter.rst} | 7 +- docs/dev/result_types/answer.rst | 7 + docs/dev/result_types/base_result.rst | 5 + docs/dev/result_types/correction.rst | 34 ++ docs/dev/result_types/index.rst | 95 +++ docs/dev/result_types/infobox.rst | 60 ++ docs/dev/result_types/main_result.rst | 17 + docs/dev/result_types/suggestion.rst | 38 ++ docs/dev/searxng_extra/update.rst | 1 + docs/dev/templates.rst | 577 ++++++++++++++++++ docs/src/index.rst | 6 +- searx/__init__.py | 77 ++- searx/answerers/__init__.py | 86 ++- searx/answerers/_core.py | 169 +++++ searx/answerers/random.py | 80 +++ searx/answerers/random/__init__.py | 2 - searx/answerers/random/answerer.py | 79 --- searx/answerers/statistics.py | 64 ++ searx/answerers/statistics/__init__.py | 2 - searx/answerers/statistics/answerer.py | 53 -- searx/autocomplete.py | 12 +- searx/botdetection/_helpers.py | 6 +- searx/botdetection/http_accept.py | 8 +- searx/botdetection/http_accept_encoding.py | 8 +- searx/botdetection/http_accept_language.py | 9 +- searx/botdetection/http_connection.py | 8 +- searx/botdetection/http_user_agent.py | 8 +- searx/botdetection/ip_limit.py | 3 +- searx/botdetection/link_token.py | 8 +- searx/engines/brave.py | 7 +- searx/engines/deepl.py | 7 +- searx/engines/dictzone.py | 95 ++- searx/engines/duckduckgo.py | 8 +- searx/engines/duckduckgo_definitions.py | 8 +- searx/engines/google.py | 8 +- searx/engines/libretranslate.py | 25 +- searx/engines/lingva.py | 41 +- searx/engines/mozhi.py | 29 +- searx/engines/openstreetmap.py | 35 +- searx/engines/tineye.py | 9 +- searx/engines/translated.py | 45 +- searx/engines/xpath.py | 2 +- searx/extended_types.py | 82 +++ searx/favicons/proxy.py | 13 +- searx/infopage/__init__.py | 3 +- searx/limiter.py | 7 +- searx/locales.py | 9 +- searx/network/__init__.py | 17 +- searx/network/network.py | 8 +- searx/plugins/__init__.py | 250 ++------ searx/plugins/_core.py | 394 ++++++++++++ searx/plugins/ahmia_filter.py | 14 +- searx/plugins/calculator.py | 70 ++- searx/plugins/hash_plugin.py | 83 ++- searx/plugins/hostnames.py | 21 +- searx/plugins/oa_doi_rewrite.py | 10 +- searx/plugins/self_info.py | 69 ++- searx/plugins/tor_check.py | 53 +- searx/plugins/tracker_url_remover.py | 18 +- searx/plugins/unit_converter.py | 30 +- searx/preferences.py | 21 +- searx/result_types/__init__.py | 18 + searx/result_types/_base.py | 223 +++++++ searx/result_types/answer.py | 141 +++++ searx/results.py | 121 ++-- searx/search/__init__.py | 51 +- searx/search/processors/abstract.py | 1 + searx/settings.yml | 24 +- searx/settings_defaults.py | 5 - .../plugins/external_plugins/.gitignore | 3 - .../themes/simple/src/less/preferences.less | 8 + searx/templates/simple/answer/legacy.html | 8 + .../templates/simple/answer/translations.html | 52 ++ .../templates/simple/answerers/translate.html | 38 -- searx/templates/simple/base.html | 1 - searx/templates/simple/elements/answers.html | 8 + .../simple/elements/corrections.html | 19 + searx/templates/simple/preferences.html | 44 +- .../simple/preferences/answerers.html | 18 +- .../simple/preferences/doi_resolver.html | 6 +- searx/templates/simple/results.html | 50 +- searx/webapp.py | 290 ++++----- tests/__init__.py | 60 +- tests/robot/__init__.py | 2 +- tests/robot/__main__.py | 3 +- tests/robot/test_webapp.py | 2 +- tests/unit/__init__.py | 10 +- tests/unit/engines/__init__.py | 2 +- tests/unit/engines/test_command.py | 21 +- tests/unit/engines/test_xpath.py | 5 +- tests/unit/network/__init__.py | 2 +- tests/unit/network/test_network.py | 22 +- tests/unit/processors/__init__.py | 2 +- tests/unit/processors/test_online.py | 21 +- tests/unit/settings/limiter.toml | 2 + tests/unit/settings/test_result_container.yml | 8 + tests/unit/settings/test_settings.yml | 24 +- tests/unit/settings/test_tineye.yml | 16 + tests/unit/test_answerers.py | 37 +- tests/unit/test_engine_mariadb_server.py | 5 +- tests/unit/test_engine_tineye.py | 21 +- tests/unit/test_engines_init.py | 13 +- tests/unit/test_exceptions.py | 5 +- tests/unit/test_external_bangs.py | 13 +- tests/unit/test_locales.py | 3 +- tests/unit/test_plugin_calculator.py | 105 ++-- tests/unit/test_plugin_hash.py | 96 +-- tests/unit/test_plugin_self_info.py | 96 +-- tests/unit/test_plugins.py | 116 +++- tests/unit/test_preferences.py | 47 +- tests/unit/test_query.py | 37 +- tests/unit/test_results.py | 93 ++- tests/unit/test_search.py | 34 +- tests/unit/test_settings_loader.py | 11 +- tests/unit/test_toml.py | 4 +- tests/unit/test_utils.py | 8 +- tests/unit/test_webadapter.py | 42 +- tests/unit/test_webapp.py | 87 ++- tests/unit/test_webutils.py | 11 +- utils/lib_sxng_test.sh | 40 +- utils/lib_sxng_themes.sh | 4 +- 143 files changed, 3846 insertions(+), 2100 deletions(-) create mode 100644 docs/admin/settings/settings_plugins.rst create mode 100644 docs/dev/answerers/builtins.rst create mode 100644 docs/dev/answerers/development.rst create mode 100644 docs/dev/answerers/index.rst create mode 100644 docs/dev/answerers/random.rst create mode 100644 docs/dev/answerers/statistics.rst create mode 100644 docs/dev/extended_types.rst delete mode 100644 docs/dev/plugins.rst create mode 100644 docs/dev/plugins/builtins.rst create mode 100644 docs/dev/plugins/calculator.rst create mode 100644 docs/dev/plugins/development.rst create mode 100644 docs/dev/plugins/hash_plugin.rst rename docs/{src/searx.plugins.hostnames.rst => dev/plugins/hostnames.rst} (50%) create mode 100644 docs/dev/plugins/index.rst create mode 100644 docs/dev/plugins/self_info.rst rename docs/{src/searx.plugins.tor_check.rst => dev/plugins/tor_check.rst} (50%) rename docs/{src/searx.plugins.unit_converter.rst => dev/plugins/unit_converter.rst} (56%) create mode 100644 docs/dev/result_types/answer.rst create mode 100644 docs/dev/result_types/base_result.rst create mode 100644 docs/dev/result_types/correction.rst create mode 100644 docs/dev/result_types/index.rst create mode 100644 docs/dev/result_types/infobox.rst create mode 100644 docs/dev/result_types/main_result.rst create mode 100644 docs/dev/result_types/suggestion.rst create mode 100644 docs/dev/templates.rst create mode 100644 searx/answerers/_core.py create mode 100644 searx/answerers/random.py delete mode 100644 searx/answerers/random/__init__.py delete mode 100644 searx/answerers/random/answerer.py create mode 100644 searx/answerers/statistics.py delete mode 100644 searx/answerers/statistics/__init__.py delete mode 100644 searx/answerers/statistics/answerer.py create mode 100644 searx/extended_types.py create mode 100644 searx/plugins/_core.py create mode 100644 searx/result_types/__init__.py create mode 100644 searx/result_types/_base.py create mode 100644 searx/result_types/answer.py delete mode 100644 searx/static/plugins/external_plugins/.gitignore create mode 100644 searx/templates/simple/answer/legacy.html create mode 100644 searx/templates/simple/answer/translations.html delete mode 100644 searx/templates/simple/answerers/translate.html create mode 100644 searx/templates/simple/elements/answers.html create mode 100644 searx/templates/simple/elements/corrections.html create mode 100644 tests/unit/settings/limiter.toml create mode 100644 tests/unit/settings/test_result_container.yml create mode 100644 tests/unit/settings/test_tineye.yml diff --git a/Makefile b/Makefile index 073b4de27..fea044702 100644 --- a/Makefile +++ b/Makefile @@ -50,7 +50,7 @@ search.checker.%: install $(Q)./manage pyenv.cmd searxng-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))" PHONY += test ci.test test.shell -ci.test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.pybabel +ci.test: test.yamllint test.black test.types test.pylint test.unit test.robot test.rst test.pybabel test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.shell test.shell: $(Q)shellcheck -x -s dash \ @@ -83,7 +83,7 @@ MANAGE += node.env node.env.dev node.clean MANAGE += py.build py.clean MANAGE += pyenv pyenv.install pyenv.uninstall MANAGE += format.python -MANAGE += test.yamllint test.pylint test.pyright test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean +MANAGE += test.yamllint test.pylint test.pyright test.types test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean MANAGE += themes.all themes.simple themes.simple.test pygments.less MANAGE += static.build.commit static.build.drop static.build.restore MANAGE += nvm.install nvm.clean nvm.status nvm.nodejs diff --git a/docs/admin/plugins.rst b/docs/admin/plugins.rst index d97b3dada..d3d137fcf 100644 --- a/docs/admin/plugins.rst +++ b/docs/admin/plugins.rst @@ -1,12 +1,14 @@ -.. _plugins generic: +.. _plugins admin: =============== -Plugins builtin +List of plugins =============== .. sidebar:: Further reading .. + - :ref:`SearXNG settings ` - :ref:`dev plugin` + - :ref:`builtin plugins` Configuration defaults (at built time): @@ -25,15 +27,10 @@ Configuration defaults (at built time): - DO - Description - JS & CSS dependencies + {% for plg in plugins %} - {% for plgin in plugins %} - - * - {{plgin.name}} - - {{(plgin.default_on and "y") or ""}} - - {{plgin.description}} - - {% for dep in (plgin.js_dependencies + plgin.css_dependencies) %} - | ``{{dep}}`` {% endfor %} + * - {{plg.info.name}} + - {{(plg.default_on and "y") or ""}} + - {{plg.info.description}} {% endfor %} diff --git a/docs/admin/settings/index.rst b/docs/admin/settings/index.rst index acc91dbdd..d1f808f3d 100644 --- a/docs/admin/settings/index.rst +++ b/docs/admin/settings/index.rst @@ -22,6 +22,6 @@ Settings settings_redis settings_outgoing settings_categories_as_tabs - + settings_plugins diff --git a/docs/admin/settings/settings_plugins.rst b/docs/admin/settings/settings_plugins.rst new file mode 100644 index 000000000..991e7ee53 --- /dev/null +++ b/docs/admin/settings/settings_plugins.rst @@ -0,0 +1,67 @@ +.. _settings plugins: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins admin` + - :ref:`dev plugin` + - :ref:`builtin plugins` + + +The built-in plugins can be activated or deactivated via the settings +(:ref:`settings enabled_plugins`) and external plugins can be integrated into +SearXNG (:ref:`settings external_plugins`). + + +.. _settings enabled_plugins: + +``enabled_plugins:`` (internal) +=============================== + +In :ref:`plugins admin` you find a complete list of all plugins, the default +configuration looks like: + +.. code:: yaml + + enabled_plugins: + - 'Basic Calculator' + - 'Hash plugin' + - 'Self Information' + - 'Tracker URL remover' + - 'Unit converter plugin' + - 'Ahmia blacklist' + + +.. _settings external_plugins: + +``plugins:`` (external) +======================= + +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +`__: + +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash -c + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + - only_show_green_results + # - mypackage.mymodule.MyPlugin + # - mypackage.mymodule.MyOtherPlugin + +.. hint:: + + ``only_show_green_results`` is an old plugin that was still implemented in + the old style. There is a legacy treatment for backward compatibility, but + new plugins should be implemented as a :py:obj:`searx.plugins.Plugin` class. diff --git a/docs/conf.py b/docs/conf.py index fec9eb64b..aeb62e337 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -54,7 +54,7 @@ searx.engines.load_engines(searx.settings['engines']) jinja_contexts = { 'searx': { 'engines': searx.engines.engines, - 'plugins': searx.plugins.plugins, + 'plugins': searx.plugins.STORAGE, 'version': { 'node': os.getenv('NODE_MINIMUM_VERSION') }, @@ -129,8 +129,9 @@ extensions = [ 'notfound.extension', # https://github.com/readthedocs/sphinx-notfound-page ] +# autodoc_typehints = "description" autodoc_default_options = { - 'member-order': 'groupwise', + 'member-order': 'bysource', } myst_enable_extensions = [ diff --git a/docs/dev/answerers/builtins.rst b/docs/dev/answerers/builtins.rst new file mode 100644 index 000000000..cca020a7a --- /dev/null +++ b/docs/dev/answerers/builtins.rst @@ -0,0 +1,11 @@ +.. _builtin answerers: + +================== +Built-in Answerers +================== + +.. toctree:: + :maxdepth: 1 + + random + statistics diff --git a/docs/dev/answerers/development.rst b/docs/dev/answerers/development.rst new file mode 100644 index 000000000..04a1da0bb --- /dev/null +++ b/docs/dev/answerers/development.rst @@ -0,0 +1,7 @@ +.. _dev answerers: + +==================== +Answerer Development +==================== + +.. automodule:: searx.answerers diff --git a/docs/dev/answerers/index.rst b/docs/dev/answerers/index.rst new file mode 100644 index 000000000..02af238bf --- /dev/null +++ b/docs/dev/answerers/index.rst @@ -0,0 +1,9 @@ +========= +Answerers +========= + +.. toctree:: + :maxdepth: 2 + + development + builtins diff --git a/docs/dev/answerers/random.rst b/docs/dev/answerers/random.rst new file mode 100644 index 000000000..4489d2a34 --- /dev/null +++ b/docs/dev/answerers/random.rst @@ -0,0 +1,8 @@ +.. _answerer.random: + +====== +Random +====== + +.. autoclass:: searx.answerers.random.SXNGAnswerer + :members: diff --git a/docs/dev/answerers/statistics.rst b/docs/dev/answerers/statistics.rst new file mode 100644 index 000000000..86ae6861e --- /dev/null +++ b/docs/dev/answerers/statistics.rst @@ -0,0 +1,8 @@ +.. _answerer.statistics: + +========== +Statistics +========== + +.. autoclass:: searx.answerers.statistics.SXNGAnswerer + :members: diff --git a/docs/dev/engines/engine_overview.rst b/docs/dev/engines/engine_overview.rst index 1a6270a3b..4aeb5b624 100644 --- a/docs/dev/engines/engine_overview.rst +++ b/docs/dev/engines/engine_overview.rst @@ -237,335 +237,18 @@ following parameters can be used to specify a search request: =================== =========== ========================================================================== -.. _engine results: -.. _engine media types: +Making a Response +================= -Result Types (``template``) -=========================== +In the ``response`` function of the engine, the HTTP response (``resp``) is +parsed and a list of results is returned. -Each result item of an engine can be of different media-types. Currently the -following media-types are supported. To set another media-type as -:ref:`template default`, the parameter ``template`` must be set to the desired -type. +A engine can append result-items of different media-types and different +result-types to the result list. The list of the result items is render to HTML +by templates. For more details read section: -.. _template default: +- :ref:`simple theme templates` +- :ref:`result types` -``default`` ------------ -.. table:: Parameter of the **default** media type: - :width: 100% - ========================= ===================================================== - result-parameter information - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - ========================= ===================================================== - - -.. _template images: - -``images`` ----------- - -.. list-table:: Parameter of the **images** media type - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``images.html`` - - * - url - - :py:class:`str` - - url to the result site - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - description of the image - - * - publishedDate - - :py:class:`datetime ` - - time of publish - - * - img_src - - :py:class:`str` - - url to the result image - - * - thumbnail_src - - :py:class:`str` - - url to a small-preview image - - * - resolution - - :py:class:`str` - - the resolution of the image (e.g. ``1920 x 1080`` pixel) - - * - img_format - - :py:class:`str` - - the format of the image (e.g. ``png``) - - * - filesize - - :py:class:`str` - - size of bytes in :py:obj:`human readable ` notation - (e.g. ``MB`` for 1024 \* 1024 Bytes filesize). - - -.. _template videos: - -``videos`` ----------- - -.. table:: Parameter of the **videos** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``videos.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content *(not implemented yet)* - publishedDate :py:class:`datetime.datetime`, time of publish - thumbnail string, url to a small-preview image - length :py:class:`datetime.timedelta`, duration of result - views string, view count in humanized number format - ========================= ===================================================== - - -.. _template torrent: - -``torrent`` ------------ - -.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme - -.. table:: Parameter of the **torrent** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``torrent.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, - time of publish *(not implemented yet)* - seed int, number of seeder - leech int, number of leecher - filesize int, size of file in bytes - files int, number of files - magnetlink string, magnetlink_ of the result - torrentfile string, torrentfile of the result - ========================= ===================================================== - - -.. _template map: - -``map`` -------- - -.. table:: Parameter of the **map** media type: - :width: 100% - - ========================= ===================================================== - result-parameter information - ------------------------- ----------------------------------------------------- - template is set to ``map.html`` - ========================= ===================================================== - url string, url of the result - title string, title of the result - content string, general result-text - publishedDate :py:class:`datetime.datetime`, time of publish - latitude latitude of result (in decimal format) - longitude longitude of result (in decimal format) - boundingbox boundingbox of result (array of 4. values - ``[lat-min, lat-max, lon-min, lon-max]``) - geojson geojson of result (https://geojson.org/) - osm.type type of osm-object (if OSM-Result) - osm.id id of osm-object (if OSM-Result) - address.name name of object - address.road street name of object - address.house_number house number of object - address.locality city, place of object - address.postcode postcode of object - address.country country of object - ========================= ===================================================== - - -.. _template paper: - -``paper`` ---------- - -.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ -.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types - -.. list-table:: Parameter of the **paper** media type / - see `BibTeX field types`_ and `BibTeX format`_ - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``paper.html`` - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - abstract - - * - comments - - :py:class:`str` - - free text display in italic below the content - - * - tags - - :py:class:`List `\ [\ :py:class:`str`\ ] - - free tag list - - * - publishedDate - - :py:class:`datetime ` - - last publication date - - * - type - - :py:class:`str` - - short description of medium type, e.g. *book*, *pdf* or *html* ... - - * - authors - - :py:class:`List `\ [\ :py:class:`str`\ ] - - list of authors of the work (authors with a "s") - - * - editor - - :py:class:`str` - - list of editors of a book - - * - publisher - - :py:class:`str` - - name of the publisher - - * - journal - - :py:class:`str` - - name of the journal or magazine the article was - published in - - * - volume - - :py:class:`str` - - volume number - - * - pages - - :py:class:`str` - - page range where the article is - - * - number - - :py:class:`str` - - number of the report or the issue number for a journal article - - * - doi - - :py:class:`str` - - DOI number (like ``10.1038/d41586-018-07848-2``) - - * - issn - - :py:class:`List `\ [\ :py:class:`str`\ ] - - ISSN number like ``1476-4687`` - - * - isbn - - :py:class:`List `\ [\ :py:class:`str`\ ] - - ISBN number like ``9780201896831`` - - * - pdf_url - - :py:class:`str` - - URL to the full article, the PDF version - - * - html_url - - :py:class:`str` - - URL to full article, HTML version - - -.. _template packages: - -``packages`` ------------- - -.. list-table:: Parameter of the **packages** media type - :header-rows: 2 - :width: 100% - - * - result-parameter - - Python type - - information - - * - template - - :py:class:`str` - - is set to ``packages.html`` - - * - title - - :py:class:`str` - - title of the result - - * - content - - :py:class:`str` - - abstract - - * - package_name - - :py:class:`str` - - the name of the package - - * - version - - :py:class:`str` - - the current version of the package - - * - maintainer - - :py:class:`str` - - the maintainer or author of the project - - * - publishedDate - - :py:class:`datetime ` - - date of latest update or release - - * - tags - - :py:class:`List `\ [\ :py:class:`str`\ ] - - free tag list - - * - popularity - - :py:class:`str` - - the popularity of the package, e.g. rating or download count - - * - license_name - - :py:class:`str` - - the name of the license - - * - license_url - - :py:class:`str` - - the web location of a license copy - - * - homepage - - :py:class:`str` - - the url of the project's homepage - - * - source_code_url - - :py:class:`str` - - the location of the project's source code - - * - links - - :py:class:`dict` - - additional links in the form of ``{'link_name': 'http://example.com'}`` diff --git a/docs/dev/extended_types.rst b/docs/dev/extended_types.rst new file mode 100644 index 000000000..59013eef2 --- /dev/null +++ b/docs/dev/extended_types.rst @@ -0,0 +1,7 @@ +.. _extended_types.: + +============== +Extended Types +============== + +.. automodule:: searx.extended_types diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 09be9de5e..01a16ba80 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -8,9 +8,13 @@ Developer documentation quickstart rtm_asdf contribution_guide + extended_types engines/index + result_types/index + templates search_api - plugins + plugins/index + answerers/index translation lxcdev makefile diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst deleted file mode 100644 index fb3201e66..000000000 --- a/docs/dev/plugins.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _dev plugin: - -======= -Plugins -======= - -.. sidebar:: Further reading .. - - - :ref:`plugins generic` - -Plugins can extend or replace functionality of various components of searx. - -Example plugin -============== - -.. code:: python - - name = 'Example plugin' - description = 'This plugin extends the suggestions with the word "example"' - default_on = False # disabled by default - - # attach callback to the post search hook - # request: flask request object - # ctx: the whole local context of the post search hook - def post_search(request, search): - search.result_container.suggestions.add('example') - return True - -External plugins -================ - -SearXNG supports *external plugins* / there is no need to install one, SearXNG -runs out of the box. But to demonstrate; in the example below we install the -SearXNG plugins from *The Green Web Foundation* `[ref] -`__: - -.. code:: bash - - $ sudo utils/searxng.sh instance cmd bash -c - (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins - -In the :ref:`settings.yml` activate the ``plugins:`` section and add module -``only_show_green_results`` from ``tgwf-searx-plugins``. - -.. code:: yaml - - plugins: - ... - - only_show_green_results - ... - - -Plugin entry points -=================== - -Entry points (hooks) define when a plugin runs. Right now only three hooks are -implemented. So feel free to implement a hook if it fits the behaviour of your -plugin. A plugin doesn't need to implement all the hooks. - - -.. py:function:: pre_search(request, search) -> bool - - Runs BEFORE the search request. - - `search.result_container` can be changed. - - Return a boolean: - - * True to continue the search - * False to stop the search - - :param flask.request request: - :param searx.search.SearchWithPlugins search: - :return: False to stop the search - :rtype: bool - - -.. py:function:: post_search(request, search) -> None - - Runs AFTER the search request. - - :param flask.request request: Flask request. - :param searx.search.SearchWithPlugins search: Context. - - -.. py:function:: on_result(request, search, result) -> bool - - Runs for each result of each engine. - - `result` can be changed. - - If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])` - - .. warning:: - `result["url"]` can be changed, but `result["parsed_url"]` must be updated too. - - Return a boolean: - - * True to keep the result - * False to remove the result - - :param flask.request request: - :param searx.search.SearchWithPlugins search: - :param typing.Dict result: Result, see - :ref:`engine results` - :return: True to keep the result - :rtype: bool diff --git a/docs/dev/plugins/builtins.rst b/docs/dev/plugins/builtins.rst new file mode 100644 index 000000000..902092e11 --- /dev/null +++ b/docs/dev/plugins/builtins.rst @@ -0,0 +1,15 @@ +.. _builtin plugins: + +================ +Built-in Plugins +================ + +.. toctree:: + :maxdepth: 1 + + calculator + hash_plugin + hostnames + self_info + tor_check + unit_converter diff --git a/docs/dev/plugins/calculator.rst b/docs/dev/plugins/calculator.rst new file mode 100644 index 000000000..7ca9e32ed --- /dev/null +++ b/docs/dev/plugins/calculator.rst @@ -0,0 +1,8 @@ +.. _plugins.calculator: + +========== +Calculator +========== + +.. automodule:: searx.plugins.calculator + :members: diff --git a/docs/dev/plugins/development.rst b/docs/dev/plugins/development.rst new file mode 100644 index 000000000..6bece72c0 --- /dev/null +++ b/docs/dev/plugins/development.rst @@ -0,0 +1,7 @@ +.. _dev plugin: + +================== +Plugin Development +================== + +.. automodule:: searx.plugins diff --git a/docs/dev/plugins/hash_plugin.rst b/docs/dev/plugins/hash_plugin.rst new file mode 100644 index 000000000..a5ac0728f --- /dev/null +++ b/docs/dev/plugins/hash_plugin.rst @@ -0,0 +1,8 @@ +.. _hash_plugin plugin: + +=========== +Hash Values +=========== + +.. autoclass:: searx.plugins.hash_plugin.SXNGPlugin + :members: diff --git a/docs/src/searx.plugins.hostnames.rst b/docs/dev/plugins/hostnames.rst similarity index 50% rename from docs/src/searx.plugins.hostnames.rst rename to docs/dev/plugins/hostnames.rst index 5d0365536..434f170ea 100644 --- a/docs/src/searx.plugins.hostnames.rst +++ b/docs/dev/plugins/hostnames.rst @@ -1,9 +1,8 @@ .. _hostnames plugin: -================ -Hostnames plugin -================ +========= +Hostnames +========= .. automodule:: searx.plugins.hostnames - :members: - + :members: diff --git a/docs/dev/plugins/index.rst b/docs/dev/plugins/index.rst new file mode 100644 index 000000000..bed8095e2 --- /dev/null +++ b/docs/dev/plugins/index.rst @@ -0,0 +1,9 @@ +======= +Plugins +======= + +.. toctree:: + :maxdepth: 2 + + development + builtins diff --git a/docs/dev/plugins/self_info.rst b/docs/dev/plugins/self_info.rst new file mode 100644 index 000000000..a1dd66b17 --- /dev/null +++ b/docs/dev/plugins/self_info.rst @@ -0,0 +1,8 @@ +.. _self_info plugin: + +========= +Self-Info +========= + +.. autoclass:: searx.plugins.self_info.SXNGPlugin + :members: diff --git a/docs/src/searx.plugins.tor_check.rst b/docs/dev/plugins/tor_check.rst similarity index 50% rename from docs/src/searx.plugins.tor_check.rst rename to docs/dev/plugins/tor_check.rst index 905328ebf..f1a02381e 100644 --- a/docs/src/searx.plugins.tor_check.rst +++ b/docs/dev/plugins/tor_check.rst @@ -1,9 +1,8 @@ .. _tor check plugin: -================ -Tor check plugin -================ +========= +Tor check +========= .. automodule:: searx.plugins.tor_check - :members: - + :members: diff --git a/docs/src/searx.plugins.unit_converter.rst b/docs/dev/plugins/unit_converter.rst similarity index 56% rename from docs/src/searx.plugins.unit_converter.rst rename to docs/dev/plugins/unit_converter.rst index 48d495d80..2e19bc38e 100644 --- a/docs/src/searx.plugins.unit_converter.rst +++ b/docs/dev/plugins/unit_converter.rst @@ -1,9 +1,8 @@ .. _unit converter plugin: -===================== -Unit converter plugin -===================== +============== +Unit Converter +============== .. automodule:: searx.plugins.unit_converter :members: - diff --git a/docs/dev/result_types/answer.rst b/docs/dev/result_types/answer.rst new file mode 100644 index 000000000..72368be39 --- /dev/null +++ b/docs/dev/result_types/answer.rst @@ -0,0 +1,7 @@ +.. _result_types.answer: + +============== +Answer Results +============== + +.. automodule:: searx.result_types.answer diff --git a/docs/dev/result_types/base_result.rst b/docs/dev/result_types/base_result.rst new file mode 100644 index 000000000..24fd96a9d --- /dev/null +++ b/docs/dev/result_types/base_result.rst @@ -0,0 +1,5 @@ +====== +Result +====== + +.. automodule:: searx.result_types._base diff --git a/docs/dev/result_types/correction.rst b/docs/dev/result_types/correction.rst new file mode 100644 index 000000000..51cdd45d2 --- /dev/null +++ b/docs/dev/result_types/correction.rst @@ -0,0 +1,34 @@ +.. _result_types.corrections: + +================== +Correction Results +================== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The corrections area shows the user alternative search terms. + +A result of this type is a very simple dictionary with only one key/value pair + +.. code:: python + + {"correction" : "lorem ipsum .."} + +From this simple dict another dict is build up: + +.. code:: python + + # use RawTextQuery to get the corrections URLs with the same bang + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum .." } + +and used in the template :origin:`corrections.html +`: + +title : :py:class:`str` + Corrected search term. + +url : :py:class:`str` + Not really an URL, its the value to insert in a HTML form for a SearXNG query. diff --git a/docs/dev/result_types/index.rst b/docs/dev/result_types/index.rst new file mode 100644 index 000000000..e90dba849 --- /dev/null +++ b/docs/dev/result_types/index.rst @@ -0,0 +1,95 @@ +.. _result types: + +============ +Result Types +============ + +To understand the typification of the results, let's take a brief look at the +structure of SearXNG .. At its core, SearXNG is nothing more than an aggregator +that aggregates the results from various sources, renders them via templates and +displays them to the user. + +The **sources** can be: + +1. :ref:`engines ` +2. :ref:`plugins ` +3. :ref:`answerers ` + +The sources provide the results, which are displayed in different **areas** +depending on the type of result. The areas are: + +main results: + It is the main area in which -- as is typical for search engines -- the + results that a search engine has found for the search term are displayed. + +answers: + This area displays short answers that could be found for the search term. + +info box: + An area in which additional information can be displayed, e.g. excerpts from + wikipedia or other sources such as maps. + +suggestions: + Suggestions for alternative search terms can be found in this area. These can + be clicked on and a search is carried out with these search terms. + +corrections: + Results in this area are like the suggestion of alternative search terms, + which usually result from spelling corrections + +At this point it is important to note that all **sources** can contribute +results to all of the areas mentioned above. + +In most cases, however, the :ref:`engines ` will fill +the *main results* and the :ref:`answerers ` will generally +provide the contributions for the *answer* area. Not necessary to mention here +but for a better understanding: the plugins can also filter out or change +results from the main results area (e.g. the URL of the link). + +The result items are organized in the :py:obj:`results.ResultContainer` and +after all sources have delivered their results, this container is passed to the +templating to build a HTML output. The output is usually HTML, but it is also +possible to output the result lists as JSON or RSS feed. Thats quite all we need +to know before we dive into typification of result items. + +.. hint:: + + Typification of result items: we are at the very first beginng! + +The first thing we have to realize is that there is no typification of the +result items so far, we have to build it up first .. and that is quite a big +task, which we will only be able to accomplish gradually. + +The foundation for the typeless results was laid back in 2013 in the very first +commit :commit:`ae9fb1d7d`, and the principle has not changed since then. At +the time, the approach was perfectly adequate, but we have since evolved and the +demands on SearXNG increase with every feature request. + +**Motivation:** in the meantime, it has become very difficult to develop new +features that require structural changes and it is especially hard for newcomers +to find their way in this typeless world. As long as the results are only +simple key/value dictionaries, it is not even possible for the IDEs to support +the application developer in his work. + +**Planning:** The procedure for subsequent typing will have to be based on the +circumstances .. + +.. attention:: + + As long as there is no type defined for a kind of result the HTML template + specify what the properties of a type are. + + In this sense, you will either find a type definition here in the + documentation or, if this does not yet exist, a description of the HTML + template. + + +.. toctree:: + :maxdepth: 2 + + base_result + main_result + answer + correction + suggestion + infobox diff --git a/docs/dev/result_types/infobox.rst b/docs/dev/result_types/infobox.rst new file mode 100644 index 000000000..428dc8db7 --- /dev/null +++ b/docs/dev/result_types/infobox.rst @@ -0,0 +1,60 @@ +.. _result_types.infobox: + +=============== +Infobox Results +=============== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The infobox is an area where addtional infos shown to the user. + +Fields used in the :origin:`infobox.html +`: + +img_src: :py:class:`str` + URL of a image or thumbnail that is displayed in the infobox. + +infobox: :py:class:`str` + Title of the info box. + +content: :py:class:`str` + Text of the info box. + +The infobox has additional subsections for *attributes*, *urls* and +*relatedTopics*: + +attributes: :py:class:`List `\ [\ :py:class:`dict`\ ] + A list of attributes. An *attribute* is a dictionary with keys: + + - label :py:class:`str`: (mandatory) + + - value :py:class:`str`: (mandatory) + + - image :py:class:`List `\ [\ :py:class:`dict`\ ] (optional) + + A list of images. An *image* is a dictionary with keys: + + - src :py:class:`str`: URL of an image/thumbnail (mandatory) + - alt :py:class:`str`: alternative text for the image (mandatory) + +urls: :py:class:`List `\ [\ :py:class:`dict`\ ] + A list of links. An *link* is a dictionary with keys: + + - url :py:class:`str`: URL of the link (mandatory) + - title :py:class:`str`: Title of the link (mandatory) + +relatedTopics: :py:class:`List `\ [\ :py:class:`dict`\ ] + A list of topics. An *topic* is a dictionary with keys: + + - name: :py:class:`str`: (mandatory) + + - suggestions: :py:class:`List `\ [\ :py:class:`dict`\ ] (optional) + + A list of suggestions. A *suggestion* is simple dictionary with just one + key/value pair: + + - suggestion: :py:class:`str`: suggested search term (mandatory) + diff --git a/docs/dev/result_types/main_result.rst b/docs/dev/result_types/main_result.rst new file mode 100644 index 000000000..1f178cbd1 --- /dev/null +++ b/docs/dev/result_types/main_result.rst @@ -0,0 +1,17 @@ +============ +Main Results +============ + +There is still no typing for the items in the :ref:`main result list`. The +templates can be used as orientation until the final typing is complete. + +- :ref:`template default` +- :ref:`template images` +- :ref:`template videos` +- :ref:`template torrent` +- :ref:`template map` +- :ref:`template paper` +- :ref:`template packages` +- :ref:`template code` +- :ref:`template files` +- :ref:`template products` diff --git a/docs/dev/result_types/suggestion.rst b/docs/dev/result_types/suggestion.rst new file mode 100644 index 000000000..52e8a05a8 --- /dev/null +++ b/docs/dev/result_types/suggestion.rst @@ -0,0 +1,38 @@ +.. _result_types.suggestion: + +================== +Suggestion Results +================== + +.. hint:: + + There is still no typing for these result items. The templates can be used as + orientation until the final typing is complete. + +The suggestions area shows the user alternative search terms. + +A result of this type is a very simple dictionary with only one key/value pair + +.. code:: python + + {"suggestion" : "lorem ipsum .."} + +From this simple dict another dict is build up: + +.. code:: python + + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" } + +and used in the template :origin:`suggestions.html +`: + +.. code:: python + + # use RawTextQuery to get the suggestion URLs with the same bang + {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" } + +title : :py:class:`str` + Suggested search term + +url : :py:class:`str` + Not really an URL, its the value to insert in a HTML form for a SearXNG query. diff --git a/docs/dev/searxng_extra/update.rst b/docs/dev/searxng_extra/update.rst index dc3b06744..000085970 100644 --- a/docs/dev/searxng_extra/update.rst +++ b/docs/dev/searxng_extra/update.rst @@ -60,6 +60,7 @@ Scripts to update static data in :origin:`searx/data/` .. automodule:: searxng_extra.update.update_engine_traits :members: +.. _update_osm_keys_tags.py: ``update_osm_keys_tags.py`` =========================== diff --git a/docs/dev/templates.rst b/docs/dev/templates.rst new file mode 100644 index 000000000..b62c6b732 --- /dev/null +++ b/docs/dev/templates.rst @@ -0,0 +1,577 @@ +.. _simple theme templates: + +====================== +Simple Theme Templates +====================== + +The simple template is complex, it consists of many different elements and also +uses macros and include statements. The following is a rough overview that we +would like to give the developerat hand, details must still be taken from the +:origin:`sources `. + +A :ref:`result item ` can be of different media types. The media +type of a result is defined by the :py:obj:`result_type.Result.template`. To +set another media-type as :ref:`template default`, the field ``template`` +in the result item must be set to the desired type. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +.. _result template macros: + +Result template macros +====================== + +.. _macro result_header: + +``result_header`` +----------------- + +Execpt ``image.html`` and some others this macro is used in nearly all result +types in the :ref:`main result list`. + +Fields used in the template :origin:`macro result_header +`: + +url : :py:class:`str` + Link URL of the result item. + +title : :py:class:`str` + Link title of the result item. + +img_src, thumbnail : :py:class:`str` + URL of a image or thumbnail that is displayed in the result item. + + +.. _macro result_sub_header: + +``result_sub_header`` +--------------------- + +Execpt ``image.html`` and some others this macro is used in nearly all result +types in the :ref:`main result list`. + +Fields used in the template :origin:`macro result_sub_header +`: + +publishedDate : :py:obj:`datetime.datetime` + The date on which the object was published. + +length: :py:obj:`time.struct_time` + Playing duration in seconds. + +views: :py:class:`str` + View count in humanized number format. + +author : :py:class:`str` + Author of the title. + +metadata : :py:class:`str` + Miscellaneous metadata. + + +.. _engine_data: + +``engine_data_form`` +-------------------- + +The ``engine_data_form`` macro is used in :origin:`results,html +` in a HTML ``
`` element. The +intention of this macro is to pass data of a engine from one :py:obj:`response +` to the :py:obj:`searx.search.SearchQuery` +of the next :py:obj:`request `. + +To pass data, engine's response handler can append result items of typ +``engine_data``. This is by example used to pass a token from the response to +the next request: + +.. code:: python + + def response(resp): + ... + results.append({ + 'engine_data': token, + 'key': 'next_page_token', + }) + ... + return results + + def request(query, params): + page_token = params['engine_data'].get('next_page_token') + + +.. _main result list: + +Main Result List +================ + +The **media types** of the **main result type** are the template files in +the :origin:`result_templates `. + +.. _template default: + +``default.html`` +---------------- + +Displays result fields from: + +- :ref:`macro result_header` and +- :ref:`macro result_sub_header` + +Additional fields used in the :origin:`default.html +`: + +content : :py:class:`str` + General text of the result item. + +iframe_src : :py:class:`str` + URL of an embedded ``