commit 5849f6a4b3e34aca3ed8fe0c98ec7d661340cd03 Author: return42 Date: Fri Aug 11 10:34:02 2023 +0000 build from commit 905ce2a6f60785d674c85eb09d5cd9067c1a45bd diff --git a/.buildinfo b/.buildinfo new file mode 100644 index 000000000..cddded564 --- /dev/null +++ b/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 762fcdd87899abab84bfae2e9c119a31 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 000000000..e69de29bb diff --git a/404.html b/404.html new file mode 100644 index 000000000..da6a85931 --- /dev/null +++ b/404.html @@ -0,0 +1,102 @@ + + + + + + + + Page not found — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Page not found

+ +Unfortunately we couldn't find the content you were looking for. + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/CNAME b/CNAME new file mode 100644 index 000000000..5e170067d --- /dev/null +++ b/CNAME @@ -0,0 +1 @@ +docs.searxng.org \ No newline at end of file diff --git a/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst b/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst new file mode 100644 index 000000000..47af0b130 --- /dev/null +++ b/_downloads/ad0ebe55d6b53b1559e0ca8dee6f30b9/reST.rst @@ -0,0 +1,1438 @@ +.. _reST primer: + +=========== +reST primer +=========== + +.. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +We at SearXNG are using reStructuredText (aka reST_) markup for all kind of +documentation. With the builders from the Sphinx_ project a HTML output is +generated and deployed at docs.searxng.org_. For build prerequisites read +:ref:`docs build`. + +.. _docs.searxng.org: https://docs.searxng.org/ + +The source files of SearXNG's documentation are located at :origin:`docs`. +Sphinx assumes source files to be encoded in UTF-8 by default. Run :ref:`make +docs.live ` to build HTML while editing. + +.. sidebar:: Further reading + + - Sphinx-Primer_ + - `Sphinx markup constructs`_ + - reST_, docutils_, `docutils FAQ`_ + - Sphinx_, `sphinx-doc FAQ`_ + - `sphinx config`_, doctree_ + - `sphinx cross references`_ + - linuxdoc_ + - intersphinx_ + - sphinx-jinja_ + - `Sphinx's autodoc`_ + - `Sphinx's Python domain`_, `Sphinx's C domain`_ + - SVG_, ImageMagick_ + - DOT_, `Graphviz's dot`_, Graphviz_ + + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +Sphinx_ and reST_ have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation `[kernel doc]`_. + +.. _[kernel doc]: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html + +.. sidebar:: Content matters + + The readability_ of the reST sources has its value, therefore we recommend to + make sparse usage of reST markup / .. content matters! + +**reST** is a plaintext markup language, its markup is *mostly* intuitive and +you will not need to learn much to produce well formed articles with. I use the +word *mostly*: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups). + +Soft skills +=========== + +Before going any deeper into the markup let's face on some **soft skills** a +trained author brings with, to reach a well feedback from readers: + +- Documentation is dedicated to an audience and answers questions from the + audience point of view. +- Don't detail things which are general knowledge from the audience point of + view. +- Limit the subject, use cross links for any further reading. + +To be more concrete what a *point of view* means. In the (:origin:`docs`) +folder we have three sections (and the *blog* folder), each dedicate to a +different group of audience. + +User's POV: :origin:`docs/user` + A typical user knows about search engines and might have heard about + meta crawlers and privacy. + +Admin's POV: :origin:`docs/admin` + A typical Admin knows about setting up services on a linux system, but he does + not know all the pros and cons of a SearXNG setup. + +Developer's POV: :origin:`docs/dev` + Depending on the readability_ of code, a typical developer is able to read and + understand source code. Describe what a item aims to do (e.g. a function). + If the chronological order matters, describe it. Name the *out-of-limits + conditions* and all the side effects a external developer will not know. + +.. _reST inline markup: + +Basic inline markup +=================== + +.. sidebar:: Inline markup + + - :ref:`reST roles` + - :ref:`reST smart ref` + +Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (``\*pointer``). + +.. table:: basic inline markup + :widths: 4 3 7 + + ================================================ ==================== ======================== + description rendered markup + ================================================ ==================== ======================== + one asterisk for emphasis *italics* ``*italics*`` + two asterisks for strong emphasis **boldface** ``**boldface**`` + backquotes for code samples and literals ``foo()`` ````foo()```` + quote asterisks or backquotes \*foo is a pointer ``\*foo is a pointer`` + ================================================ ==================== ======================== + +.. _reST basic structure: + +Basic article structure +======================= + +The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections. + +.. _reST template: + +reST template +------------- + +reST template for an simple article: + +.. code:: reST + + .. _doc refname: + + ============== + Document title + ============== + + Lorem ipsum dolor sit amet, consectetur adipisici elit .. Further read + :ref:`chapter refname`. + + .. _chapter refname: + + Chapter + ======= + + Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut + aliquid ex ea commodi consequat ... + + .. _section refname: + + Section + ------- + + lorem .. + + .. _subsection refname: + + Subsection + ~~~~~~~~~~ + + lorem .. + + +Headings +-------- + +#. title - with overline for document title: + + .. code:: reST + + ============== + Document title + ============== + + +#. chapter - with anchor named ``anchor name``: + + .. code:: reST + + .. _anchor name: + + Chapter + ======= + +#. section + + .. code:: reST + + Section + ------- + +#. subsection + + .. code:: reST + + Subsection + ~~~~~~~~~~ + + + +Anchors & Links +=============== + +.. _reST anchor: + +Anchors +------- + +.. _ref role: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref + +To refer a point in the documentation a anchor is needed. The :ref:`reST +template ` shows an example where a chapter titled *"Chapters"* +gets an anchor named ``chapter title``. Another example from *this* document, +where the anchor named ``reST anchor``: + +.. code:: reST + + .. _reST anchor: + + Anchors + ------- + + To refer a point in the documentation a anchor is needed ... + +To refer anchors use the `ref role`_ markup: + +.. code:: reST + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar `. + +.. admonition:: ``:ref:`` role + :class: rst-example + + Visist chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar `. + +.. _reST ordinary ref: + +Link ordinary URL +----------------- + +If you need to reference external URLs use *named* hyperlinks to maintain +readability of reST sources. Here is a example taken from *this* article: + +.. code:: reST + + .. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + `__. + +.. admonition:: Named hyperlink + :class: rst-example + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + `__. + + +.. _reST smart ref: + +Smart refs +---------- + +With the power of sphinx.ext.extlinks_ and intersphinx_ referencing external +content becomes smart. + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + refer ... rendered example markup + ========================== ================================== ==================================== + :rst:role:`rfc` :rfc:`822` ``:rfc:`822``` + :rst:role:`pep` :pep:`8` ``:pep:`8``` + sphinx.ext.extlinks_ + -------------------------------------------------------------------------------------------------- + project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines``` + to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` + files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` + pull request :pull:`4` ``:pull:`4``` + patch :patch:`af2cae6` ``:patch:`af2cae6``` + PyPi package :pypi:`searx` ``:pypi:`searx``` + manual page man :man:`bash` ``:man:`bash``` + intersphinx_ + -------------------------------------------------------------------------------------------------- + external anchor :ref:`python:and` ``:ref:`python:and``` + external doc anchor :doc:`jinja:templates` ``:doc:`jinja:templates``` + python code object :py:obj:`datetime.datetime` ``:py:obj:`datetime.datetime``` + flask code object :py:obj:`flask.Flask` ``:py:obj:`flask.Flask``` + ========================== ================================== ==================================== + + +Intersphinx is configured in :origin:`docs/conf.py`: + +.. code:: python + + intersphinx_mapping = { + "python": ("https://docs.python.org/3/", None), + "flask": ("https://flask.palletsprojects.com/", None), + "jinja": ("https://jinja.palletsprojects.com/", None), + "linuxdoc" : ("https://return42.github.io/linuxdoc/", None), + "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), + } + + +To list all anchors of the inventory (e.g. ``python``) use: + +.. code:: sh + + $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + ... + $ python -m sphinx.ext.intersphinx https://docs.searxng.org/objects.inv + ... + +Literal blocks +============== + +The simplest form of :duref:`literal-blocks` is a indented block introduced by +two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST +code` directive. To include literals from external files use +:rst:dir:`literalinclude` or :ref:`kernel-include ` +directive (latter one expands environment variables in the path name). + +.. _reST literal: + +``::`` +------ + +.. code:: reST + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + +.. admonition:: Literal block + :class: rst-example + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + + +.. _reST code: + +``code-block`` +-------------- + +.. _pygments: https://pygments.org/languages/ + +.. sidebar:: Syntax highlighting + + is handled by pygments_. + +The :rst:dir:`code-block` directive is a variant of the :dudir:`code` directive +with additional options. To learn more about code literals visit +:ref:`sphinx:code-examples`. + +.. code-block:: reST + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +.. code-block:: reST + +.. admonition:: Code block + :class: rst-example + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +Unicode substitution +==================== + +The :dudir:`unicode directive ` converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition. + +.. code-block:: reST + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + +.. admonition:: Unicode + :class: rst-example + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + + +.. _reST roles: + +Roles +===== + +.. sidebar:: Further reading + + - `Sphinx Roles`_ + - :doc:`sphinx:usage/restructuredtext/domains` + + +A *custom interpreted text role* (:duref:`ref `) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way. + +The general markup is one of: + +.. code:: reST + + :rolename:`ref-name` + :rolename:`ref text ` + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + role rendered example markup + ========================== ================================== ==================================== + :rst:role:`guilabel` :guilabel:`&Cancel` ``:guilabel:`&Cancel``` + :rst:role:`kbd` :kbd:`C-x C-f` ``:kbd:`C-x C-f``` + :rst:role:`menuselection` :menuselection:`Open --> File` ``:menuselection:`Open --> File``` + :rst:role:`download` :download:`this file ` ``:download:`this file ``` + math_ :math:`a^2 + b^2 = c^2` ``:math:`a^2 + b^2 = c^2``` + :rst:role:`ref` :ref:`svg image example` ``:ref:`svg image example``` + :rst:role:`command` :command:`ls -la` ``:command:`ls -la``` + :durole:`emphasis` :emphasis:`italic` ``:emphasis:`italic``` + :durole:`strong` :strong:`bold` ``:strong:`bold``` + :durole:`literal` :literal:`foo()` ``:literal:`foo()``` + :durole:`subscript` H\ :sub:`2`\ O ``H\ :sub:`2`\ O`` + :durole:`superscript` E = mc\ :sup:`2` ``E = mc\ :sup:`2``` + :durole:`title-reference` :title:`Time` ``:title:`Time``` + ========================== ================================== ==================================== + +Figures & Images +================ + +.. sidebar:: Image processing + + With the directives from :ref:`linuxdoc ` the build process + is flexible. To get best results in the generated output format, install + ImageMagick_ and Graphviz_. + +SearXNG's sphinx setup includes: :ref:`linuxdoc:kfigure`. Scalable here means; +scalable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With :ref:`linuxdoc:kfigure` the build process +continues and scales output quality in dependence of installed image processors. + +If you want to add an image, you should use the ``kernel-figure`` (inheritance +of :dudir:`figure`) and ``kernel-image`` (inheritance of :dudir:`image`) +directives. E.g. to insert a figure with a scalable image format use SVG +(:ref:`svg image example`): + +.. code:: reST + + .. _svg image example: + + .. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image + + To refer the figure, a caption block is needed: :ref:`svg image example`. + +.. _svg image example: + +.. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image. + +To refer the figure, a caption block is needed: :ref:`svg image example`. + +DOT files (aka Graphviz) +------------------------ + +With :ref:`linuxdoc:kernel-figure` reST support for **DOT** formatted files is +given. + +- `Graphviz's dot`_ +- DOT_ +- Graphviz_ + +A simple example is shown in :ref:`dot file example`: + +.. code:: reST + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. admonition:: hello.dot + :class: rst-example + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +``kernel-render`` DOT +--------------------- + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +:ref:`linuxdoc:kernel-render` directive. A simple example of embedded DOT_ is +shown in figure :ref:`dot render example`: + +.. code:: reST + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +Please note :ref:`build tools `. If Graphviz_ is +installed, you will see an vector image. If not, the raw markup is inserted as +*literal-block*. + +.. admonition:: kernel-render DOT + :class: rst-example + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +``kernel-render`` SVG +--------------------- + +A simple example of embedded SVG_ is shown in figure :ref:`svg render example`: + +.. code:: reST + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow +.. + + .. code:: xml + + + + + + + +.. admonition:: kernel-render SVG + :class: rst-example + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow + + + + + + + + + + +.. _reST lists: + +List markups +============ + +Bullet list +----------- + +List markup (:duref:`ref `) is simple: + +.. code:: reST + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + +.. admonition:: bullet list + :class: rst-example + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + + +Horizontal list +--------------- + +The :rst:dir:`.. hlist:: ` transforms a bullet list into a more compact +list. + +.. code:: reST + + .. hlist:: + + - first list item + - second list item + - third list item + ... + +.. admonition:: hlist + :class: rst-example + + .. hlist:: + + - first list item + - second list item + - third list item + - next list item + - next list item xxxx + - next list item yyyy + - next list item zzzz + + +Definition list +--------------- + +.. sidebar:: Note .. + + - the term cannot have more than one line of text + + - there is **no blank line between term and definition block** // this + distinguishes definition lists (:duref:`ref `) from block + quotes (:duref:`ref `). + +Each definition list (:duref:`ref `) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ' : ' (**space, colon, space**). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(*this distinguishes definition lists from block quotes*). Blank lines are +required before the first and after the last definition list item, but are +optional in-between. + +Definition lists are created as follows: + +.. code:: reST + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref `) and not as term's definition! + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + Definition 4. + +.. admonition:: definition list + :class: rst-example + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref `) and not as term's definition! + + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + + +Quoted paragraphs +----------------- + +Quoted paragraphs (:duref:`ref `) are created by just indenting +them more than the surrounding paragraphs. Line blocks (:duref:`ref +`) are a way of preserving line breaks: + +.. code:: reST + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. admonition:: Quoted paragraph and line block + :class: rst-example + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. _reST field list: + +Field Lists +----------- + +.. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + +.. sidebar:: bibliographic fields + + First lines fields are bibliographic fields, see `Sphinx Field Lists`_. + +Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this: + +.. code:: reST + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + +.. admonition:: Field List + :class: rst-example + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + + +They are commonly used in Python documentation: + +.. code:: python + + def my_function(my_arg, my_other_arg): + """A function just for me. + + :param my_arg: The first of my arguments. + :param my_other_arg: The second of my arguments. + + :returns: A message (just for me, of course). + """ + +Further list blocks +------------------- + +- field lists (:duref:`ref `, with caveats noted in + :ref:`reST field list`) +- option lists (:duref:`ref `) +- quoted literal blocks (:duref:`ref `) +- doctest blocks (:duref:`ref `) + + +Admonitions +=========== + +Sidebar +------- + +Sidebar is an eye catcher, often used for admonitions pointing further stuff or +site effects. Here is the source of the sidebar :ref:`on top of this page `. + +.. code:: reST + + .. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +Generic admonition +------------------ + +The generic :dudir:`admonition ` needs a title: + +.. code:: reST + + .. admonition:: generic admonition title + + lorem ipsum .. + + +.. admonition:: generic admonition title + + lorem ipsum .. + + +Specific admonitions +-------------------- + +Specific admonitions: :dudir:`hint`, :dudir:`note`, :dudir:`tip` :dudir:`attention`, +:dudir:`caution`, :dudir:`danger`, :dudir:`error`, , :dudir:`important`, and +:dudir:`warning` . + +.. code:: reST + + .. hint:: + + lorem ipsum .. + + .. note:: + + lorem ipsum .. + + .. warning:: + + lorem ipsum .. + + +.. hint:: + + lorem ipsum .. + +.. note:: + + lorem ipsum .. + +.. tip:: + + lorem ipsum .. + +.. attention:: + + lorem ipsum .. + +.. caution:: + + lorem ipsum .. + +.. danger:: + + lorem ipsum .. + +.. important:: + + lorem ipsum .. + +.. error:: + + lorem ipsum .. + +.. warning:: + + lorem ipsum .. + + +Tables +====== + +.. sidebar:: Nested tables + + Nested tables are ugly! Not all builder support nested tables, don't use + them! + +ASCII-art tables like :ref:`reST simple table` and :ref:`reST grid table` might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables. + + +.. sidebar:: List tables + + For meaningful patch and diff use :ref:`reST flat table`. + +Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers: + +* `Emacs Table Mode`_ +* `Online Tables Generator`_ + +.. _reST simple table: + +Simple tables +------------- + +:duref:`Simple tables ` allow *colspan* but not *rowspan*. If +your table need some metadata (e.g. a title) you need to add the ``.. table:: +directive`` :dudir:`(ref) ` in front and place the table in its body: + +.. code:: reST + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + +.. admonition:: Simple ASCII table + :class: rst-example + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + + + +.. _reST grid table: + +Grid tables +----------- + +:duref:`Grid tables ` allow colspan *colspan* and *rowspan*: + +.. code:: reST + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + +.. admonition:: ASCII grid table + :class: rst-example + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + + +.. _reST flat table: + +flat-table +---------- + +The ``flat-table`` is a further developed variant of the :ref:`list tables +`. It is a double-stage list similar to the +:dudir:`list-table` with some additional features: + +column-span: ``cspan`` + with the role ``cspan`` a cell can be extended through additional columns + +row-span: ``rspan`` + with the role ``rspan`` a cell can be extended through additional rows + +auto-span: + spans rightmost cell of a table row over the missing cells on the right side + of that table-row. With Option ``:fill-cells:`` this behavior can changed + from *auto span* to *auto fill*, which automatically inserts (empty) cells + instead of spanning the last cell. + +options: + :header-rows: [int] count of header rows + :stub-columns: [int] count of stub columns + :widths: [[int] [int] ... ] widths of columns + :fill-cells: instead of auto-span missing cells, insert missing cells + +roles: + :cspan: [int] additional columns (*morecols*) + :rspan: [int] additional rows (*morerows*) + +The example below shows how to use this markup. The first level of the staged +list is the *table-row*. In the *table-row* there is only one markup allowed, +the list of the cells in this *table-row*. Exception are *comments* ( ``..`` ) +and *targets* (e.g. a ref to :ref:`row 2 of table's body `). + +.. code:: reST + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +.. admonition:: List table + :class: rst-example + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +CSV table +--------- + +CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation. + +.. code:: reST + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 2 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Content of file ``csv_table.txt``: + +.. literalinclude:: csv_table.txt + +.. admonition:: CSV table + :class: rst-example + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 3 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Templating +========== + +.. sidebar:: Build environment + + All *generic-doc* tasks are running in the :ref:`make install`. + +Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with SearXNG modules installed). We use this e.g. to build chapter: +:ref:`configured engines`. Below the jinja directive from the +:origin:`docs/admin/engines.rst` is shown: + +.. literalinclude:: ../user/configured_engines.rst + :language: reST + :start-after: .. _configured engines: + +The context for the template is selected in the line ``.. jinja:: searx``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``searx`` context +contains the ``engines`` and ``plugins``. + +.. code:: py + + import searx.search + import searx.engines + import searx.plugins + searx.search.initialize() + jinja_contexts = { + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, + } + + +Tabbed views +============ + +.. _sphinx-tabs: https://github.com/djungelorm/sphinx-tabs +.. _basic-tabs: https://github.com/djungelorm/sphinx-tabs#basic-tabs +.. _group-tabs: https://github.com/djungelorm/sphinx-tabs#group-tabs +.. _code-tabs: https://github.com/djungelorm/sphinx-tabs#code-tabs + +With `sphinx-tabs`_ extension we have *tabbed views*. To provide installation +instructions with one tab per distribution we use the `group-tabs`_ directive, +others are basic-tabs_ and code-tabs_. Below a *group-tab* example from +:ref:`docs build` is shown: + +.. literalinclude:: ../admin/buildhosts.rst + :language: reST + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements + +.. _math: + +Math equations +============== + +.. _Mathematics: https://en.wikibooks.org/wiki/LaTeX/Mathematics +.. _amsmath user guide: + http://vesta.informatik.rwth-aachen.de/ftp/pub/mirror/ctan/macros/latex/required/amsmath/amsldoc.pdf + +.. sidebar:: About LaTeX + + - `amsmath user guide`_ + - Mathematics_ + - :ref:`docs build` + +The input language for mathematics is LaTeX markup using the :ctan:`amsmath` +package. + +To embed LaTeX markup in reST documents, use role :rst:role:`:math: ` for +inline and directive :rst:dir:`.. math:: ` for block markup. + +.. code:: reST + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + +.. admonition:: LaTeX math equation + :class: rst-example + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + + +The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` +(*displaystyle*) used in a inline markup or another fraction. + +.. code:: reST + + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + ``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + +.. admonition:: Line spacing + :class: rst-example + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + +.. _KISS: https://en.wikipedia.org/wiki/KISS_principle + +.. _readability: https://docs.python-guide.org/writing/style/ +.. _Sphinx-Primer: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _reST: https://docutils.sourceforge.io/rst.html +.. _Sphinx Roles: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html +.. _Sphinx: https://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx markup constructs: + https://www.sphinx-doc.org/en/stable/markup/index.html +.. _`sphinx cross references`: + https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations +.. _sphinx.ext.extlinks: + https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html +.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _Sphinx's Python domain: + https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain +.. _Sphinx's C domain: + https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs +.. _doctree: + https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases +.. _docutils: http://docutils.sourceforge.net/docs/index.html +.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html +.. _linuxdoc: https://return42.github.io/linuxdoc +.. _jinja: https://jinja.palletsprojects.com/ +.. _sphinx-jinja: https://github.com/tardyp/sphinx-jinja +.. _SVG: https://www.w3.org/TR/SVG11/expanded-toc.html +.. _DOT: https://graphviz.gitlab.io/_pages/doc/info/lang.html +.. _`Graphviz's dot`: https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org + +.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode +.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables +.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.dot b/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.dot new file mode 100644 index 000000000..eaec9f36b --- /dev/null +++ b/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.dot @@ -0,0 +1,3 @@ +digraph foo { + "bar" -> "baz"; +} \ No newline at end of file diff --git a/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.svg b/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.svg new file mode 100644 index 000000000..a5b367334 --- /dev/null +++ b/_images/DOT-57a4a7f78690d0b6b884bc59f36e84cfb0b61f76.svg @@ -0,0 +1,31 @@ + + + + + + +foo + + + +bar + +bar + + + +baz + +baz + + + +bar->baz + + + + + diff --git a/_images/SVG-1fb7029fa2cc454a267bae271cccb2c591387416.svg b/_images/SVG-1fb7029fa2cc454a267bae271cccb2c591387416.svg new file mode 100644 index 000000000..783577ecd --- /dev/null +++ b/_images/SVG-1fb7029fa2cc454a267bae271cccb2c591387416.svg @@ -0,0 +1,13 @@ + + + + + \ No newline at end of file diff --git a/_images/arch_public.dot b/_images/arch_public.dot new file mode 100644 index 000000000..526fb53da --- /dev/null +++ b/_images/arch_public.dot @@ -0,0 +1,30 @@ +digraph G { + + node [style=filled, shape=box, fillcolor="#ffffcc", fontname=Sans]; + edge [fontname="Sans"]; + + browser [label="browser", shape=tab, fillcolor=aliceblue]; + rp [label="reverse proxy"]; + static [label="static files", shape=folder, href="url to configure static files", fillcolor=lightgray]; + uwsgi [label="uwsgi", shape=parallelogram href="https://docs.searxng.org/utils/searx.sh.html"] + redis [label="redis DB", shape=cylinder]; + searxng1 [label="SearXNG #1", fontcolor=blue3]; + searxng2 [label="SearXNG #2", fontcolor=blue3]; + searxng3 [label="SearXNG #3", fontcolor=blue3]; + searxng4 [label="SearXNG #4", fontcolor=blue3]; + + browser -> rp [label="HTTPS"] + + subgraph cluster_searxng { + label = "SearXNG instance" fontname=Sans; + bgcolor="#fafafa"; + { rank=same; static rp }; + rp -> static [label="optional: reverse proxy serves static files", fillcolor=slategray, fontcolor=slategray]; + rp -> uwsgi [label="http:// (tcp) or unix:// (socket)"]; + uwsgi -> searxng1 -> redis; + uwsgi -> searxng2 -> redis; + uwsgi -> searxng3 -> redis; + uwsgi -> searxng4 -> redis; + } + +} diff --git a/_images/arch_public.svg b/_images/arch_public.svg new file mode 100644 index 000000000..0a0a79ae2 --- /dev/null +++ b/_images/arch_public.svg @@ -0,0 +1,149 @@ + + + + + + +G + + +cluster_searxng + +SearXNG instance + + + +browser + + +browser + + + +rp + +reverse proxy + + + +browser->rp + + +HTTPS + + + +static + + +static files + + + + + +rp->static + + +optional: reverse proxy serves static files + + + +uwsgi + + +uwsgi + + + + + +rp->uwsgi + + +http:// (tcp) or unix:// (socket) + + + +searxng1 + +SearXNG #1 + + + +uwsgi->searxng1 + + + + + +searxng2 + +SearXNG #2 + + + +uwsgi->searxng2 + + + + + +searxng3 + +SearXNG #3 + + + +uwsgi->searxng3 + + + + + +searxng4 + +SearXNG #4 + + + +uwsgi->searxng4 + + + + + +redis + + +redis DB + + + +searxng1->redis + + + + + +searxng2->redis + + + + + +searxng3->redis + + + + + +searxng4->redis + + + + + diff --git a/_images/ffox-setting-proxy-socks.png b/_images/ffox-setting-proxy-socks.png new file mode 100644 index 000000000..3da85d2c0 Binary files /dev/null and b/_images/ffox-setting-proxy-socks.png differ diff --git a/_images/hello.dot b/_images/hello.dot new file mode 100644 index 000000000..504621dfc --- /dev/null +++ b/_images/hello.dot @@ -0,0 +1,3 @@ +graph G { + Hello -- World +} diff --git a/_images/hello.svg b/_images/hello.svg new file mode 100644 index 000000000..7d7af8e24 --- /dev/null +++ b/_images/hello.svg @@ -0,0 +1,30 @@ + + + + + + +G + + + +Hello + +Hello + + + +World + +World + + + +Hello--World + + + + diff --git a/_images/math/07c9ff4251510b06013159f4e45ec9ab97044096.svg b/_images/math/07c9ff4251510b06013159f4e45ec9ab97044096.svg new file mode 100644 index 000000000..80edca1d9 --- /dev/null +++ b/_images/math/07c9ff4251510b06013159f4e45ec9ab97044096.svg @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/math/3b8127a8eed95247f9249ea6c85e8e86df1baa82.svg b/_images/math/3b8127a8eed95247f9249ea6c85e8e86df1baa82.svg new file mode 100644 index 000000000..7c11c8b42 --- /dev/null +++ b/_images/math/3b8127a8eed95247f9249ea6c85e8e86df1baa82.svg @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/math/6673b43f9fe29455c1fcd1164e5844698cc64d38.svg b/_images/math/6673b43f9fe29455c1fcd1164e5844698cc64d38.svg new file mode 100644 index 000000000..8be77010f --- /dev/null +++ b/_images/math/6673b43f9fe29455c1fcd1164e5844698cc64d38.svg @@ -0,0 +1,22 @@ + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/math/a6a994cb6e7278ec30eaebe7e636046d3deccb5b.svg b/_images/math/a6a994cb6e7278ec30eaebe7e636046d3deccb5b.svg new file mode 100644 index 000000000..27ae39410 --- /dev/null +++ b/_images/math/a6a994cb6e7278ec30eaebe7e636046d3deccb5b.svg @@ -0,0 +1,43 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/_images/svg_image.svg b/_images/svg_image.svg new file mode 100644 index 000000000..5405f85b8 --- /dev/null +++ b/_images/svg_image.svg @@ -0,0 +1,10 @@ + + + + + + + + diff --git a/_images/translation.svg b/_images/translation.svg new file mode 100644 index 000000000..71f4172b5 --- /dev/null +++ b/_images/translation.svg @@ -0,0 +1,47 @@ +master branchmaster branchtranslations_update branchtranslations_update branchtranslations branchtranslations branchweblate clone oftranslations branchweblate clone oftranslations branchweblatepending changesweblatepending changesfor each commit on master.github/workflows/integration.ymlmake weblate.push.translationspybabel extractextract messages, store messages.pot on translations branchalt[if there are some changes in messages.pot]wlc lockwlc pullwlc commitgit merge weblate/translationspybabel update (messages.po)git add searx/translationsgit commitgit pushwlc unlockevery Friday.github/workflows/translations-update.ymlmake weblate.translations.commitwlc lockwlc pullwlc commitgit merge weblate/translationspybabel compilecp searx/translationsgit addgit commitwlc unlockcreate or update pull request"Update translations"developper's reviewmerge pull request \ No newline at end of file diff --git a/_modules/index.html b/_modules/index.html new file mode 100644 index 000000000..036626842 --- /dev/null +++ b/_modules/index.html @@ -0,0 +1,155 @@ + + + + + + + + Overview: module code — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

All modules for which code is available

+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/autocomplete.html b/_modules/searx/autocomplete.html new file mode 100644 index 000000000..9c42994cc --- /dev/null +++ b/_modules/searx/autocomplete.html @@ -0,0 +1,336 @@ + + + + + + + + searx.autocomplete — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.autocomplete

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This module implements functions needed for the autocompleter.
+
+"""
+# pylint: disable=use-dict-literal
+
+import json
+from urllib.parse import urlencode
+
+import lxml
+from httpx import HTTPError
+
+from searx import settings
+from searx.engines import (
+    engines,
+    google,
+)
+from searx.network import get as http_get
+from searx.exceptions import SearxEngineResponseException
+
+
+def get(*args, **kwargs):
+    if 'timeout' not in kwargs:
+        kwargs['timeout'] = settings['outgoing']['request_timeout']
+    kwargs['raise_for_httperror'] = True
+    return http_get(*args, **kwargs)
+
+
+def brave(query, _lang):
+    # brave search autocompleter
+    url = 'https://search.brave.com/api/suggest?'
+    url += urlencode({'q': query})
+    country = 'all'
+    # if lang in _brave:
+    #    country = lang
+    kwargs = {'cookies': {'country': country}}
+    resp = get(url, **kwargs)
+
+    results = []
+
+    if resp.ok:
+        data = resp.json()
+        for item in data[1]:
+            results.append(item)
+    return results
+
+
+def dbpedia(query, _lang):
+    # dbpedia autocompleter, no HTTPS
+    autocomplete_url = 'https://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'
+
+    response = get(autocomplete_url + urlencode(dict(QueryString=query)))
+
+    results = []
+
+    if response.ok:
+        dom = lxml.etree.fromstring(response.content)
+        results = dom.xpath('//Result/Label//text()')
+
+    return results
+
+
+def duckduckgo(query, sxng_locale):
+    """Autocomplete from DuckDuckGo. Supports DuckDuckGo's languages"""
+
+    traits = engines['duckduckgo'].traits
+    args = {
+        'q': query,
+        'kl': traits.get_region(sxng_locale, traits.all_locale),
+    }
+
+    url = 'https://duckduckgo.com/ac/?type=list&' + urlencode(args)
+    resp = get(url)
+
+    ret_val = []
+    if resp.ok:
+        j = resp.json()
+        if len(j) > 1:
+            ret_val = j[1]
+    return ret_val
+
+
+
[docs]def google_complete(query, sxng_locale):
+    """Autocomplete from Google.  Supports Google's languages and subdomains
+    (:py:obj:`searx.engines.google.get_google_info`) by using the async REST
+    API::
+
+        https://{subdomain}/complete/search?{args}
+
+    """
+
+    google_info = google.get_google_info({'searxng_locale': sxng_locale}, engines['google'].traits)
+
+    url = 'https://{subdomain}/complete/search?{args}'
+    args = urlencode(
+        {
+            'q': query,
+            'client': 'gws-wiz',
+            'hl': google_info['params']['hl'],
+        }
+    )
+    results = []
+    resp = get(url.format(subdomain=google_info['subdomain'], args=args))
+    if resp.ok:
+        json_txt = resp.text[resp.text.find('[') : resp.text.find(']', -3) + 1]
+        data = json.loads(json_txt)
+        for item in data[0]:
+            results.append(lxml.html.fromstring(item[0]).text_content())
+    return results

+
+
+def seznam(query, _lang):
+    # seznam search autocompleter
+    url = 'https://suggest.seznam.cz/fulltext/cs?{query}'
+
+    resp = get(
+        url.format(
+            query=urlencode(
+                {'phrase': query, 'cursorPosition': len(query), 'format': 'json-2', 'highlight': '1', 'count': '6'}
+            )
+        )
+    )
+
+    if not resp.ok:
+        return []
+
+    data = resp.json()
+    return [
+        ''.join([part.get('text', '') for part in item.get('text', [])])
+        for item in data.get('result', [])
+        if item.get('itemType', None) == 'ItemType.TEXT'
+    ]
+
+
+def startpage(query, sxng_locale):
+    """Autocomplete from Startpage. Supports Startpage's languages"""
+    lui = engines['startpage'].traits.get_language(sxng_locale, 'english')
+    url = 'https://startpage.com/suggestions?{query}'
+    resp = get(url.format(query=urlencode({'q': query, 'segment': 'startpage.udog', 'lui': lui})))
+    data = resp.json()
+    return [e['text'] for e in data.get('suggestions', []) if 'text' in e]
+
+
+def swisscows(query, _lang):
+    # swisscows autocompleter
+    url = 'https://swisscows.ch/api/suggest?{query}&itemsCount=5'
+
+    resp = json.loads(get(url.format(query=urlencode({'query': query}))).text)
+    return resp
+
+
+def qwant(query, sxng_locale):
+    """Autocomplete from Qwant. Supports Qwant's regions."""
+    results = []
+
+    locale = engines['qwant'].traits.get_region(sxng_locale, 'en_US')
+    url = 'https://api.qwant.com/v3/suggest?{query}'
+    resp = get(url.format(query=urlencode({'q': query, 'locale': locale, 'version': '2'})))
+
+    if resp.ok:
+        data = resp.json()
+        if data['status'] == 'success':
+            for item in data['data']['items']:
+                results.append(item['value'])
+
+    return results
+
+
+def wikipedia(query, sxng_locale):
+    """Autocomplete from Wikipedia. Supports Wikipedia's languages (aka netloc)."""
+    results = []
+    eng_traits = engines['wikipedia'].traits
+    wiki_lang = eng_traits.get_language(sxng_locale, 'en')
+    wiki_netloc = eng_traits.custom['wiki_netloc'].get(wiki_lang, 'en.wikipedia.org')
+
+    url = 'https://{wiki_netloc}/w/api.php?{args}'
+    args = urlencode(
+        {
+            'action': 'opensearch',
+            'format': 'json',
+            'formatversion': '2',
+            'search': query,
+            'namespace': '0',
+            'limit': '10',
+        }
+    )
+    resp = get(url.format(args=args, wiki_netloc=wiki_netloc))
+    if resp.ok:
+        data = resp.json()
+        if len(data) > 1:
+            results = data[1]
+
+    return results
+
+
+def yandex(query, _lang):
+    # yandex autocompleter
+    url = "https://suggest.yandex.com/suggest-ff.cgi?{0}"
+
+    resp = json.loads(get(url.format(urlencode(dict(part=query)))).text)
+    if len(resp) > 1:
+        return resp[1]
+    return []
+
+
+backends = {
+    'dbpedia': dbpedia,
+    'duckduckgo': duckduckgo,
+    'google': google_complete,
+    'seznam': seznam,
+    'startpage': startpage,
+    'swisscows': swisscows,
+    'qwant': qwant,
+    'wikipedia': wikipedia,
+    'brave': brave,
+    'yandex': yandex,
+}
+
+
+def search_autocomplete(backend_name, query, sxng_locale):
+    backend = backends.get(backend_name)
+    if backend is None:
+        return []
+    try:
+        return backend(query, sxng_locale)
+    except (HTTPError, SearxEngineResponseException):
+        return []
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/babel_extract.html b/_modules/searx/babel_extract.html new file mode 100644 index 000000000..9d882c0a7 --- /dev/null +++ b/_modules/searx/babel_extract.html @@ -0,0 +1,159 @@ + + + + + + + + searx.babel_extract — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.babel_extract

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This module implements the :origin:`searxng_msg <babel.cfg>` extractor to
+extract messages from:
+
+- :origin:`searx/searxng.msg`
+
+The ``searxng.msg`` files are selected by Babel_, see Babel's configuration in
+:origin:`babel.cfg`::
+
+    searxng_msg = searx.babel_extract.extract
+    ...
+    [searxng_msg: **/searxng.msg]
+
+A ``searxng.msg`` file is a python file that is *executed* by the
+:py:obj:`extract` function.  Additional ``searxng.msg`` files can be added by:
+
+1. Adding a ``searxng.msg`` file in one of the SearXNG python packages and
+2. implement a method in :py:obj:`extract` that yields messages from this file.
+
+.. _Babel: https://babel.pocoo.org/en/latest/index.html
+
+"""
+
+from os import path
+
+SEARXNG_MSG_FILE = "searxng.msg"
+_MSG_FILES = [path.join(path.dirname(__file__), SEARXNG_MSG_FILE)]
+
+
+
[docs]def extract(
+    # pylint: disable=unused-argument
+    fileobj,
+    keywords,
+    comment_tags,
+    options,
+):
+    """Extract messages from ``searxng.msg`` files by a custom extractor_.
+
+    .. _extractor:
+       https://babel.pocoo.org/en/latest/messages.html#writing-extraction-methods
+    """
+    if fileobj.name not in _MSG_FILES:
+        raise RuntimeError("don't know how to extract messages from %s" % fileobj.name)
+
+    namespace = {}
+    exec(fileobj.read(), {}, namespace)  # pylint: disable=exec-used
+
+    for name in namespace['__all__']:
+        for k, v in namespace[name].items():
+            yield 0, '_', v, ["%s['%s']" % (name, k)]

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/botdetection/_helpers.html b/_modules/searx/botdetection/_helpers.html new file mode 100644 index 000000000..068f34220 --- /dev/null +++ b/_modules/searx/botdetection/_helpers.html @@ -0,0 +1,228 @@ + + + + + + + + searx.botdetection._helpers — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.botdetection._helpers

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+# pylint: disable=missing-module-docstring, invalid-name
+from __future__ import annotations
+
+from ipaddress import (
+    IPv4Network,
+    IPv6Network,
+    IPv4Address,
+    IPv6Address,
+    ip_network,
+)
+import flask
+import werkzeug
+
+from searx.tools import config
+from searx import logger
+
+logger = logger.getChild('botdetection')
+
+
+def dump_request(request: flask.Request):
+    return (
+        request.path
+        + " || X-Forwarded-For: %s" % request.headers.get('X-Forwarded-For')
+        + " || X-Real-IP: %s" % request.headers.get('X-Real-IP')
+        + " || form: %s" % request.form
+        + " || Accept: %s" % request.headers.get('Accept')
+        + " || Accept-Language: %s" % request.headers.get('Accept-Language')
+        + " || Accept-Encoding: %s" % request.headers.get('Accept-Encoding')
+        + " || Content-Type: %s" % request.headers.get('Content-Type')
+        + " || Content-Length: %s" % request.headers.get('Content-Length')
+        + " || Connection: %s" % request.headers.get('Connection')
+        + " || User-Agent: %s" % request.headers.get('User-Agent')
+    )
+
+
+def too_many_requests(network: IPv4Network | IPv6Network, log_msg: str) -> werkzeug.Response | None:
+    """Returns a HTTP 429 response object and writes a ERROR message to the
+    'botdetection' logger.  This function is used in part by the filter methods
+    to return the default ``Too Many Requests`` response.
+
+    """
+
+    logger.debug("BLOCK %s: %s", network.compressed, log_msg)
+    return flask.make_response(('Too Many Requests', 429))
+
+
+def get_network(real_ip: IPv4Address | IPv6Address, cfg: config.Config) -> IPv4Network | IPv6Network:
+    """Returns the (client) network of whether the real_ip is part of."""
+
+    if real_ip.version == 6:
+        prefix = cfg['real_ip.ipv6_prefix']
+    else:
+        prefix = cfg['real_ip.ipv4_prefix']
+    network = ip_network(f"{real_ip}/{prefix}", strict=False)
+    # logger.debug("get_network(): %s", network.compressed)
+    return network
+
+
+
[docs]def get_real_ip(request: flask.Request) -> str:
+    """Returns real IP of the request.  Since not all proxies set all the HTTP
+    headers and incoming headers can be faked it may happen that the IP cannot
+    be determined correctly.
+
+    .. sidebar:: :py:obj:`flask.Request.remote_addr`
+
+       SearXNG uses Werkzeug's ProxyFix_ (with it default ``x_for=1``).
+
+    This function tries to get the remote IP in the order listed below,
+    additional some tests are done and if inconsistencies or errors are
+    detected, they are logged.
+
+    The remote IP of the request is taken from (first match):
+
+    - X-Forwarded-For_ header
+    - `X-real-IP header <https://github.com/searxng/searxng/issues/1237#issuecomment-1147564516>`__
+    - :py:obj:`flask.Request.remote_addr`
+
+    .. _ProxyFix:
+       https://werkzeug.palletsprojects.com/middleware/proxy_fix/
+
+    .. _X-Forwarded-For:
+      https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
+
+    """
+
+    forwarded_for = request.headers.get("X-Forwarded-For")
+    real_ip = request.headers.get('X-Real-IP')
+    remote_addr = request.remote_addr
+    # logger.debug(
+    #     "X-Forwarded-For: %s || X-Real-IP: %s || request.remote_addr: %s", forwarded_for, real_ip, remote_addr
+    # )
+
+    if not forwarded_for:
+        logger.error("X-Forwarded-For header is not set!")
+    else:
+        from .limiter import get_cfg  # pylint: disable=import-outside-toplevel, cyclic-import
+
+        forwarded_for = [x.strip() for x in forwarded_for.split(',')]
+        x_for: int = get_cfg()['real_ip.x_for']  # type: ignore
+        forwarded_for = forwarded_for[-min(len(forwarded_for), x_for)]
+
+    if not real_ip:
+        logger.error("X-Real-IP header is not set!")
+
+    if forwarded_for and real_ip and forwarded_for != real_ip:
+        logger.warning("IP from X-Real-IP (%s) is not equal to IP from X-Forwarded-For (%s)", real_ip, forwarded_for)
+
+    if forwarded_for and remote_addr and forwarded_for != remote_addr:
+        logger.warning(
+            "IP from WSGI environment (%s) is not equal to IP from X-Forwarded-For (%s)", remote_addr, forwarded_for
+        )
+
+    if real_ip and remote_addr and real_ip != remote_addr:
+        logger.warning("IP from WSGI environment (%s) is not equal to IP from X-Real-IP (%s)", remote_addr, real_ip)
+
+    request_ip = forwarded_for or real_ip or remote_addr or '0.0.0.0'
+    # logger.debug("get_real_ip() -> %s", request_ip)
+    return request_ip

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/botdetection/ip_lists.html b/_modules/searx/botdetection/ip_lists.html new file mode 100644 index 000000000..99831b677 --- /dev/null +++ b/_modules/searx/botdetection/ip_lists.html @@ -0,0 +1,193 @@ + + + + + + + + searx.botdetection.ip_lists — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.botdetection.ip_lists

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+""".. _botdetection.ip_lists:
+
+Method ``ip_lists``
+-------------------
+
+The ``ip_lists`` method implements IP :py:obj:`block- <block_ip>` and
+:py:obj:`pass-lists <pass_ip>`.
+
+.. code:: toml
+
+   [botdetection.ip_lists]
+
+   pass_ip = [
+    '140.238.172.132', # IPv4 of check.searx.space
+    '192.168.0.0/16',  # IPv4 private network
+    'fe80::/10'        # IPv6 linklocal
+   ]
+   block_ip = [
+      '93.184.216.34', # IPv4 of example.org
+      '257.1.1.1',     # invalid IP --> will be ignored, logged in ERROR class
+   ]
+
+"""
+# pylint: disable=unused-argument
+
+from __future__ import annotations
+from typing import Tuple
+from ipaddress import (
+    ip_network,
+    IPv4Address,
+    IPv6Address,
+)
+
+from searx.tools import config
+from ._helpers import logger
+
+logger = logger.getChild('ip_limit')
+
+SEARXNG_ORG = [
+    # https://github.com/searxng/searxng/pull/2484#issuecomment-1576639195
+    '140.238.172.132',  # IPv4 check.searx.space
+    '2603:c022:0:4900::/56',  # IPv6 check.searx.space
+]
+"""Passlist of IPs from the SearXNG organization, e.g. `check.searx.space`."""
+
+
+
[docs]def pass_ip(real_ip: IPv4Address | IPv6Address, cfg: config.Config) -> Tuple[bool, str]:
+    """Checks if the IP on the subnet is in one of the members of the
+    ``botdetection.ip_lists.pass_ip`` list.
+    """
+
+    if cfg.get('botdetection.ip_lists.pass_searxng_org', default=True):
+        for net in SEARXNG_ORG:
+            net = ip_network(net, strict=False)
+            if real_ip.version == net.version and real_ip in net:
+                return True, f"IP matches {net.compressed} in SEARXNG_ORG list."
+    return ip_is_subnet_of_member_in_list(real_ip, 'botdetection.ip_lists.pass_ip', cfg)

+
+
+
[docs]def block_ip(real_ip: IPv4Address | IPv6Address, cfg: config.Config) -> Tuple[bool, str]:
+    """Checks if the IP on the subnet is in one of the members of the
+    ``botdetection.ip_lists.block_ip`` list.
+    """
+
+    block, msg = ip_is_subnet_of_member_in_list(real_ip, 'botdetection.ip_lists.block_ip', cfg)
+    if block:
+        msg += " To remove IP from list, please contact the maintainer of the service."
+    return block, msg

+
+
+def ip_is_subnet_of_member_in_list(
+    real_ip: IPv4Address | IPv6Address, list_name: str, cfg: config.Config
+) -> Tuple[bool, str]:
+
+    for net in cfg.get(list_name, default=[]):
+        try:
+            net = ip_network(net, strict=False)
+        except ValueError:
+            logger.error("invalid IP %s in %s", net, list_name)
+            continue
+        if real_ip.version == net.version and real_ip in net:
+            return True, f"IP matches {net.compressed} in {list_name}."
+    return False, f"IP is not a member of an item in the f{list_name} list"
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/botdetection/link_token.html b/_modules/searx/botdetection/link_token.html new file mode 100644 index 000000000..1440ba191 --- /dev/null +++ b/_modules/searx/botdetection/link_token.html @@ -0,0 +1,265 @@ + + + + + + + + searx.botdetection.link_token — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.botdetection.link_token

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""
+Method ``link_token``
+---------------------
+
+The ``link_token`` method evaluates a request as :py:obj:`suspicious
+<is_suspicious>` if the URL ``/client<token>.css`` is not requested by the
+client.  By adding a random component (the token) in the URL, a bot can not send
+a ping by request a static URL.
+
+.. note::
+
+   This method requires a redis DB and needs a HTTP X-Forwarded-For_ header.
+
+To get in use of this method a flask URL route needs to be added:
+
+.. code:: python
+
+   @app.route('/client<token>.css', methods=['GET', 'POST'])
+   def client_token(token=None):
+       link_token.ping(request, token)
+       return Response('', mimetype='text/css')
+
+And in the HTML template from flask a stylesheet link is needed (the value of
+``link_token`` comes from :py:obj:`get_token`):
+
+.. code:: html
+
+   <link rel="stylesheet"
+         href="{{ url_for('client_token', token=link_token) }}"
+         type="text/css" />
+
+.. _X-Forwarded-For:
+   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
+
+"""
+from __future__ import annotations
+from ipaddress import (
+    IPv4Network,
+    IPv6Network,
+    ip_address,
+)
+
+import string
+import random
+import flask
+
+from searx import logger
+from searx import redisdb
+from searx.redislib import secret_hash
+
+from ._helpers import (
+    get_network,
+    get_real_ip,
+)
+
+TOKEN_LIVE_TIME = 600
+"""Livetime (sec) of limiter's CSS token."""
+
+PING_LIVE_TIME = 3600
+"""Livetime (sec) of the ping-key from a client (request)"""
+
+PING_KEY = 'SearXNG_limiter.ping'
+"""Prefix of all ping-keys generated by :py:obj:`get_ping_key`"""
+
+TOKEN_KEY = 'SearXNG_limiter.token'
+"""Key for which the current token is stored in the DB"""
+
+logger = logger.getChild('botdetection.link_token')
+
+
+
[docs]def is_suspicious(network: IPv4Network | IPv6Network, request: flask.Request, renew: bool = False):
+    """Checks whether a valid ping is exists for this (client) network, if not
+    this request is rated as *suspicious*.  If a valid ping exists and argument
+    ``renew`` is ``True`` the expire time of this ping is reset to
+    :py:obj:`PING_LIVE_TIME`.
+
+    """
+    redis_client = redisdb.client()
+    if not redis_client:
+        return False
+
+    ping_key = get_ping_key(network, request)
+    if not redis_client.get(ping_key):
+        logger.warning("missing ping (IP: %s) / request: %s", network.compressed, ping_key)
+        return True
+
+    if renew:
+        redis_client.set(ping_key, 1, ex=PING_LIVE_TIME)
+
+    logger.debug("found ping for (client) network %s -> %s", network.compressed, ping_key)
+    return False

+
+
+
[docs]def ping(request: flask.Request, token: str):
+    """This function is called by a request to URL ``/client<token>.css``.  If
+    ``token`` is valid a :py:obj:`PING_KEY` for the client is stored in the DB.
+    The expire time of this ping-key is :py:obj:`PING_LIVE_TIME`.
+
+    """
+    from . import limiter  # pylint: disable=import-outside-toplevel, cyclic-import
+
+    redis_client = redisdb.client()
+    if not redis_client:
+        return
+    if not token_is_valid(token):
+        return
+
+    cfg = limiter.get_cfg()
+    real_ip = ip_address(get_real_ip(request))
+    network = get_network(real_ip, cfg)
+
+    ping_key = get_ping_key(network, request)
+    logger.debug("store ping_key for (client) network %s (IP %s) -> %s", network.compressed, real_ip, ping_key)
+    redis_client.set(ping_key, 1, ex=PING_LIVE_TIME)

+
+
+
[docs]def get_ping_key(network: IPv4Network | IPv6Network, request: flask.Request) -> str:
+    """Generates a hashed key that fits (more or less) to a *WEB-browser
+    session* in a network."""
+    return (
+        PING_KEY
+        + "["
+        + secret_hash(
+            network.compressed + request.headers.get('Accept-Language', '') + request.headers.get('User-Agent', '')
+        )
+        + "]"
+    )

+
+
+def token_is_valid(token) -> bool:
+    valid = token == get_token()
+    logger.debug("token is valid --> %s", valid)
+    return valid
+
+
+
[docs]def get_token() -> str:
+    """Returns current token.  If there is no currently active token a new token
+    is generated randomly and stored in the redis DB.
+
+    - :py:obj:`TOKEN_LIVE_TIME`
+    - :py:obj:`TOKEN_KEY`
+
+    """
+    redis_client = redisdb.client()
+    if not redis_client:
+        # This function is also called when limiter is inactive / no redis DB
+        # (see render function in webapp.py)
+        return '12345678'
+    token = redis_client.get(TOKEN_KEY)
+    if token:
+        token = token.decode('UTF-8')
+    else:
+        token = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(16))
+        redis_client.set(TOKEN_KEY, token, ex=TOKEN_LIVE_TIME)
+    return token

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/enginelib.html b/_modules/searx/enginelib.html new file mode 100644 index 000000000..89b0e3794 --- /dev/null +++ b/_modules/searx/enginelib.html @@ -0,0 +1,253 @@ + + + + + + + + searx.enginelib — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.enginelib

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Implementations of the framework for the SearXNG engines.
+
+.. hint::
+
+   The long term goal is to modularize all implementations of the engine
+   framework here in this Python package.  ToDo:
+
+   - move implementations of the :ref:`searx.engines loader` to a new module in
+     the :py:obj:`searx.enginelib` namespace.
+
+"""
+
+
+from __future__ import annotations
+from typing import List, Callable, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from searx.enginelib import traits
+
+
+
[docs]class Engine:  # pylint: disable=too-few-public-methods
+    """Class of engine instances build from YAML settings.
+
+    Further documentation see :ref:`general engine configuration`.
+
+    .. hint::
+
+       This class is currently never initialized and only used for type hinting.
+    """
+
+    # Common options in the engine module
+
+    engine_type: str
+    """Type of the engine (:ref:`searx.search.processors`)"""
+
+    paging: bool
+    """Engine supports multiple pages."""
+
+    time_range_support: bool
+    """Engine supports search time range."""
+
+    safesearch: bool
+    """Engine supports SafeSearch"""
+
+    language_support: bool
+    """Engine supports languages (locales) search."""
+
+    language: str
+    """For an engine, when there is ``language: ...`` in the YAML settings the engine
+    does support only this one language:
+
+    .. code:: yaml
+
+      - name: google french
+        engine: google
+        language: fr
+    """
+
+    region: str
+    """For an engine, when there is ``region: ...`` in the YAML settings the engine
+    does support only this one region::
+
+    .. code:: yaml
+
+      - name: google belgium
+        engine: google
+        region: fr-BE
+    """
+
+    fetch_traits: Callable
+    """Function to to fetch engine's traits from origin."""
+
+    traits: traits.EngineTraits
+    """Traits of the engine."""
+
+    # settings.yml
+
+    categories: List[str]
+    """Specifies to which :ref:`engine categories` the engine should be added."""
+
+    name: str
+    """Name that will be used across SearXNG to define this engine.  In settings, on
+    the result page .."""
+
+    engine: str
+    """Name of the python file used to handle requests and responses to and from
+    this search engine (file name from :origin:`searx/engines` without
+    ``.py``)."""
+
+    enable_http: bool
+    """Enable HTTP (by default only HTTPS is enabled)."""
+
+    shortcut: str
+    """Code used to execute bang requests (``!foo``)"""
+
+    timeout: float
+    """Specific timeout for search-engine."""
+
+    display_error_messages: bool
+    """Display error messages on the web UI."""
+
+    proxies: dict
+    """Set proxies for a specific engine (YAML):
+
+    .. code:: yaml
+
+       proxies :
+         http:  socks5://proxy:port
+         https: socks5://proxy:port
+    """
+
+    disabled: bool
+    """To disable by default the engine, but not deleting it.  It will allow the
+    user to manually activate it in the settings."""
+
+    inactive: bool
+    """Remove the engine from the settings (*disabled & removed*)."""
+
+    about: dict
+    """Additional fileds describing the engine.
+
+    .. code:: yaml
+
+       about:
+          website: https://example.com
+          wikidata_id: Q306656
+          official_api_documentation: https://example.com/api-doc
+          use_official_api: true
+          require_api_key: true
+          results: HTML
+    """
+
+    using_tor_proxy: bool
+    """Using tor proxy (``true``) or not (``false``) for this engine."""
+
+    send_accept_language_header: bool
+    """When this option is activated, the language (locale) that is selected by
+    the user is used to build and send a ``Accept-Language`` header in the
+    request to the origin search engine."""
+
+    tokens: List[str]
+    """A list of secret tokens to make this engine *private*, more details see
+    :ref:`private engines`."""

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/enginelib/traits.html b/_modules/searx/enginelib/traits.html new file mode 100644 index 000000000..1ac22081d --- /dev/null +++ b/_modules/searx/enginelib/traits.html @@ -0,0 +1,364 @@ + + + + + + + + searx.enginelib.traits — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.enginelib.traits

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Engine's traits are fetched from the origin engines and stored in a JSON file
+in the *data folder*.  Most often traits are languages and region codes and
+their mapping from SearXNG's representation to the representation in the origin
+search engine.  For new traits new properties can be added to the class
+:py:class:`EngineTraits`.
+
+To load traits from the persistence :py:obj:`EngineTraitsMap.from_data` can be
+used.
+"""
+
+from __future__ import annotations
+import json
+import dataclasses
+import types
+from typing import Dict, Iterable, Union, Callable, Optional, TYPE_CHECKING
+from typing_extensions import Literal, Self
+
+from searx import locales
+from searx.data import data_dir, ENGINE_TRAITS
+
+if TYPE_CHECKING:
+    from . import Engine
+
+
+
[docs]class EngineTraitsEncoder(json.JSONEncoder):
+    """Encodes :class:`EngineTraits` to a serializable object, see
+    :class:`json.JSONEncoder`."""
+
+
[docs]    def default(self, o):
+        """Return dictionary of a :class:`EngineTraits` object."""
+        if isinstance(o, EngineTraits):
+            return o.__dict__
+        return super().default(o)

+
+
+
[docs]@dataclasses.dataclass
+class EngineTraits:
+    """The class is intended to be instantiated for each engine."""
+
+    regions: Dict[str, str] = dataclasses.field(default_factory=dict)
+    """Maps SearXNG's internal representation of a region to the one of the engine.
+
+    SearXNG's internal representation can be parsed by babel and the value is
+    send to the engine:
+
+    .. code:: python
+
+       regions ={
+           'fr-BE' : <engine's region name>,
+       }
+
+       for key, egnine_region regions.items():
+          searxng_region = babel.Locale.parse(key, sep='-')
+          ...
+    """
+
+    languages: Dict[str, str] = dataclasses.field(default_factory=dict)
+    """Maps SearXNG's internal representation of a language to the one of the engine.
+
+    SearXNG's internal representation can be parsed by babel and the value is
+    send to the engine:
+
+    .. code:: python
+
+       languages = {
+           'ca' : <engine's language name>,
+       }
+
+       for key, egnine_lang in languages.items():
+          searxng_lang = babel.Locale.parse(key)
+          ...
+    """
+
+    all_locale: Optional[str] = None
+    """To which locale value SearXNG's ``all`` language is mapped (shown a "Default
+    language").
+    """
+
+    data_type: Literal['traits_v1'] = 'traits_v1'
+    """Data type, default is 'traits_v1'.
+    """
+
+    custom: Dict[str, Union[Dict[str, Dict], Iterable[str]]] = dataclasses.field(default_factory=dict)
+    """A place to store engine's custom traits, not related to the SearXNG core.
+    """
+
+
[docs]    def get_language(self, searxng_locale: str, default=None):
+        """Return engine's language string that *best fits* to SearXNG's locale.
+
+        :param searxng_locale: SearXNG's internal representation of locale
+          selected by the user.
+
+        :param default: engine's default language
+
+        The *best fits* rules are implemented in
+        :py:obj:`searx.locales.get_engine_locale`.  Except for the special value ``all``
+        which is determined from :py:obj:`EngineTraits.all_locale`.
+        """
+        if searxng_locale == 'all' and self.all_locale is not None:
+            return self.all_locale
+        return locales.get_engine_locale(searxng_locale, self.languages, default=default)

+
+
[docs]    def get_region(self, searxng_locale: str, default=None):
+        """Return engine's region string that best fits to SearXNG's locale.
+
+        :param searxng_locale: SearXNG's internal representation of locale
+          selected by the user.
+
+        :param default: engine's default region
+
+        The *best fits* rules are implemented in
+        :py:obj:`searx.locales.get_engine_locale`.  Except for the special value ``all``
+        which is determined from :py:obj:`EngineTraits.all_locale`.
+        """
+        if searxng_locale == 'all' and self.all_locale is not None:
+            return self.all_locale
+        return locales.get_engine_locale(searxng_locale, self.regions, default=default)

+
+
[docs]    def is_locale_supported(self, searxng_locale: str) -> bool:
+        """A *locale* (SearXNG's internal representation) is considered to be
+        supported by the engine if the *region* or the *language* is supported
+        by the engine.
+
+        For verification the functions :py:func:`EngineTraits.get_region` and
+        :py:func:`EngineTraits.get_language` are used.
+        """
+        if self.data_type == 'traits_v1':
+            return bool(self.get_region(searxng_locale) or self.get_language(searxng_locale))
+
+        raise TypeError('engine traits of type %s is unknown' % self.data_type)

+
+
[docs]    def copy(self):
+        """Create a copy of the dataclass object."""
+        return EngineTraits(**dataclasses.asdict(self))

+
+
[docs]    @classmethod
+    def fetch_traits(cls, engine: Engine) -> Union[Self, None]:
+        """Call a function ``fetch_traits(engine_traits)`` from engines namespace to fetch
+        and set properties from the origin engine in the object ``engine_traits``.  If
+        function does not exists, ``None`` is returned.
+        """
+
+        fetch_traits = getattr(engine, 'fetch_traits', None)
+        engine_traits = None
+
+        if fetch_traits:
+            engine_traits = cls()
+            fetch_traits(engine_traits)
+        return engine_traits

+
+
[docs]    def set_traits(self, engine: Engine):
+        """Set traits from self object in a :py:obj:`.Engine` namespace.
+
+        :param engine: engine instance build by :py:func:`searx.engines.load_engine`
+        """
+
+        if self.data_type == 'traits_v1':
+            self._set_traits_v1(engine)
+        else:
+            raise TypeError('engine traits of type %s is unknown' % self.data_type)

+
+    def _set_traits_v1(self, engine: Engine):
+        # For an engine, when there is `language: ...` in the YAML settings the engine
+        # does support only this one language (region)::
+        #
+        #   - name: google italian
+        #     engine: google
+        #     language: it
+        #     region: it-IT
+
+        traits = self.copy()
+
+        _msg = "settings.yml - engine: '%s' / %s: '%s' not supported"
+
+        languages = traits.languages
+        if hasattr(engine, 'language'):
+            if engine.language not in languages:
+                raise ValueError(_msg % (engine.name, 'language', engine.language))
+            traits.languages = {engine.language: languages[engine.language]}
+
+        regions = traits.regions
+        if hasattr(engine, 'region'):
+            if engine.region not in regions:
+                raise ValueError(_msg % (engine.name, 'region', engine.region))
+            traits.regions = {engine.region: regions[engine.region]}
+
+        engine.language_support = bool(traits.languages or traits.regions)
+
+        # set the copied & modified traits in engine's namespace
+        engine.traits = traits

+
+
+
[docs]class EngineTraitsMap(Dict[str, EngineTraits]):
+    """A python dictionary to map :class:`EngineTraits` by engine name."""
+
+    ENGINE_TRAITS_FILE = (data_dir / 'engine_traits.json').resolve()
+    """File with persistence of the :py:obj:`EngineTraitsMap`."""
+
+
[docs]    def save_data(self):
+        """Store EngineTraitsMap in in file :py:obj:`self.ENGINE_TRAITS_FILE`"""
+        with open(self.ENGINE_TRAITS_FILE, 'w', encoding='utf-8') as f:
+            json.dump(self, f, indent=2, sort_keys=True, cls=EngineTraitsEncoder)

+
+
[docs]    @classmethod
+    def from_data(cls) -> Self:
+        """Instantiate :class:`EngineTraitsMap` object from :py:obj:`ENGINE_TRAITS`"""
+        obj = cls()
+        for k, v in ENGINE_TRAITS.items():
+            obj[k] = EngineTraits(**v)
+        return obj

+
+    @classmethod
+    def fetch_traits(cls, log: Callable) -> Self:
+        from searx import engines  # pylint: disable=cyclic-import, import-outside-toplevel
+
+        names = list(engines.engines)
+        names.sort()
+        obj = cls()
+
+        for engine_name in names:
+            engine = engines.engines[engine_name]
+
+            traits = EngineTraits.fetch_traits(engine)
+            if traits is not None:
+                log("%-20s: SearXNG languages --> %s " % (engine_name, len(traits.languages)))
+                log("%-20s: SearXNG regions   --> %s" % (engine_name, len(traits.regions)))
+                obj[engine_name] = traits
+
+        return obj
+
+
[docs]    def set_traits(self, engine: Engine | types.ModuleType):
+        """Set traits in a :py:obj:`Engine` namespace.
+
+        :param engine: engine instance build by :py:func:`searx.engines.load_engine`
+        """
+
+        engine_traits = EngineTraits(data_type='traits_v1')
+        if engine.name in self.keys():
+            engine_traits = self[engine.name]
+
+        elif engine.engine in self.keys():
+            # The key of the dictionary traits_map is the *engine name*
+            # configured in settings.xml.  When multiple engines are configured
+            # in settings.yml to use the same origin engine (python module)
+            # these additional engines can use the languages from the origin
+            # engine.  For this use the configured ``engine: ...`` from
+            # settings.yml
+            engine_traits = self[engine.engine]
+
+        engine_traits.set_traits(engine)

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines.html b/_modules/searx/engines.html new file mode 100644 index 000000000..ff8893732 --- /dev/null +++ b/_modules/searx/engines.html @@ -0,0 +1,361 @@ + + + + + + + + searx.engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Load and initialize the ``engines``, see :py:func:`load_engines` and register
+:py:obj:`engine_shortcuts`.
+
+usage::
+
+    load_engines( settings['engines'] )
+
+"""
+
+from __future__ import annotations
+
+import sys
+import copy
+from os.path import realpath, dirname
+
+from typing import TYPE_CHECKING, Dict
+import types
+import inspect
+
+from searx import logger, settings
+from searx.utils import load_module
+
+if TYPE_CHECKING:
+    from searx.enginelib import Engine
+
+logger = logger.getChild('engines')
+ENGINE_DIR = dirname(realpath(__file__))
+ENGINE_DEFAULT_ARGS = {
+    # Common options in the engine module
+    "engine_type": "online",
+    "paging": False,
+    "time_range_support": False,
+    "safesearch": False,
+    # settings.yml
+    "categories": ["general"],
+    "enable_http": False,
+    "shortcut": "-",
+    "timeout": settings["outgoing"]["request_timeout"],
+    "display_error_messages": True,
+    "disabled": False,
+    "inactive": False,
+    "about": {},
+    "using_tor_proxy": False,
+    "send_accept_language_header": False,
+    "tokens": [],
+}
+# set automatically when an engine does not have any tab category
+DEFAULT_CATEGORY = 'other'
+
+
+# Defaults for the namespace of an engine module, see :py:func:`load_engine`
+
+categories = {'general': []}
+engines: Dict[str, Engine | types.ModuleType] = {}
+engine_shortcuts = {}
+"""Simple map of registered *shortcuts* to name of the engine (or ``None``).
+
+::
+
+    engine_shortcuts[engine.shortcut] = engine.name
+
+:meta hide-value:
+"""
+
+
+def check_engine_module(module: types.ModuleType):
+    # probe unintentional name collisions / for example name collisions caused
+    # by import statements in the engine module ..
+
+    # network: https://github.com/searxng/searxng/issues/762#issuecomment-1605323861
+    obj = getattr(module, 'network', None)
+    if obj and inspect.ismodule(obj):
+        msg = f'type of {module.__name__}.network is a module ({obj.__name__}), expected a string'
+        # logger.error(msg)
+        raise TypeError(msg)
+
+
+
[docs]def load_engine(engine_data: dict) -> Engine | types.ModuleType | None:
+    """Load engine from ``engine_data``.
+
+    :param dict engine_data:  Attributes from YAML ``settings:engines/<engine>``
+    :return: initialized namespace of the ``<engine>``.
+
+    1. create a namespace and load module of the ``<engine>``
+    2. update namespace with the defaults from :py:obj:`ENGINE_DEFAULT_ARGS`
+    3. update namespace with values from ``engine_data``
+
+    If engine *is active*, return namespace of the engine, otherwise return
+    ``None``.
+
+    This function also returns ``None`` if initialization of the namespace fails
+    for one of the following reasons:
+
+    - engine name contains underscore
+    - engine name is not lowercase
+    - required attribute is not set :py:func:`is_missing_required_attributes`
+
+    """
+    # pylint: disable=too-many-return-statements
+
+    engine_name = engine_data.get('name')
+    if engine_name is None:
+        logger.error('An engine does not have a "name" field')
+        return None
+    if '_' in engine_name:
+        logger.error('Engine name contains underscore: "{}"'.format(engine_name))
+        return None
+
+    if engine_name.lower() != engine_name:
+        logger.warning('Engine name is not lowercase: "{}", converting to lowercase'.format(engine_name))
+        engine_name = engine_name.lower()
+        engine_data['name'] = engine_name
+
+    # load_module
+    module_name = engine_data.get('engine')
+    if module_name is None:
+        logger.error('The "engine" field is missing for the engine named "{}"'.format(engine_name))
+        return None
+    try:
+        engine = load_module(module_name + '.py', ENGINE_DIR)
+    except (SyntaxError, KeyboardInterrupt, SystemExit, SystemError, ImportError, RuntimeError):
+        logger.exception('Fatal exception in engine "{}"'.format(module_name))
+        sys.exit(1)
+    except BaseException:
+        logger.exception('Cannot load engine "{}"'.format(module_name))
+        return None
+
+    check_engine_module(engine)
+    update_engine_attributes(engine, engine_data)
+    update_attributes_for_tor(engine)
+
+    # avoid cyclic imports
+    # pylint: disable=import-outside-toplevel
+    from searx.enginelib.traits import EngineTraitsMap
+
+    trait_map = EngineTraitsMap.from_data()
+    trait_map.set_traits(engine)
+
+    if not is_engine_active(engine):
+        return None
+
+    if is_missing_required_attributes(engine):
+        return None
+
+    set_loggers(engine, engine_name)
+
+    if not any(cat in settings['categories_as_tabs'] for cat in engine.categories):
+        engine.categories.append(DEFAULT_CATEGORY)
+
+    return engine

+
+
+def set_loggers(engine, engine_name):
+    # set the logger for engine
+    engine.logger = logger.getChild(engine_name)
+    # the engine may have load some other engines
+    # may sure the logger is initialized
+    # use sys.modules.copy() to avoid "RuntimeError: dictionary changed size during iteration"
+    # see https://github.com/python/cpython/issues/89516
+    # and https://docs.python.org/3.10/library/sys.html#sys.modules
+    modules = sys.modules.copy()
+    for module_name, module in modules.items():
+        if (
+            module_name.startswith("searx.engines")
+            and module_name != "searx.engines.__init__"
+            and not hasattr(module, "logger")
+        ):
+            module_engine_name = module_name.split(".")[-1]
+            module.logger = logger.getChild(module_engine_name)  # type: ignore
+
+
+def update_engine_attributes(engine: Engine | types.ModuleType, engine_data):
+    # set engine attributes from engine_data
+    for param_name, param_value in engine_data.items():
+        if param_name == 'categories':
+            if isinstance(param_value, str):
+                param_value = list(map(str.strip, param_value.split(',')))
+            engine.categories = param_value  # type: ignore
+        elif hasattr(engine, 'about') and param_name == 'about':
+            engine.about = {**engine.about, **engine_data['about']}  # type: ignore
+        else:
+            setattr(engine, param_name, param_value)
+
+    # set default attributes
+    for arg_name, arg_value in ENGINE_DEFAULT_ARGS.items():
+        if not hasattr(engine, arg_name):
+            setattr(engine, arg_name, copy.deepcopy(arg_value))
+
+
+def update_attributes_for_tor(engine: Engine | types.ModuleType):
+    if using_tor_proxy(engine) and hasattr(engine, 'onion_url'):
+        engine.search_url = engine.onion_url + getattr(engine, 'search_path', '')  # type: ignore
+        engine.timeout += settings['outgoing'].get('extra_proxy_timeout', 0)  # type: ignore
+
+
+
[docs]def is_missing_required_attributes(engine):
+    """An attribute is required when its name doesn't start with ``_`` (underline).
+    Required attributes must not be ``None``.
+
+    """
+    missing = False
+    for engine_attr in dir(engine):
+        if not engine_attr.startswith('_') and getattr(engine, engine_attr) is None:
+            logger.error('Missing engine config attribute: "{0}.{1}"'.format(engine.name, engine_attr))
+            missing = True
+    return missing

+
+
+
[docs]def using_tor_proxy(engine: Engine | types.ModuleType):
+    """Return True if the engine configuration declares to use Tor."""
+    return settings['outgoing'].get('using_tor_proxy') or getattr(engine, 'using_tor_proxy', False)

+
+
+def is_engine_active(engine: Engine | types.ModuleType):
+    # check if engine is inactive
+    if engine.inactive is True:
+        return False
+
+    # exclude onion engines if not using tor
+    if 'onions' in engine.categories and not using_tor_proxy(engine):
+        return False
+
+    return True
+
+
+def register_engine(engine: Engine | types.ModuleType):
+    if engine.name in engines:
+        logger.error('Engine config error: ambiguous name: {0}'.format(engine.name))
+        sys.exit(1)
+    engines[engine.name] = engine
+
+    if engine.shortcut in engine_shortcuts:
+        logger.error('Engine config error: ambiguous shortcut: {0}'.format(engine.shortcut))
+        sys.exit(1)
+    engine_shortcuts[engine.shortcut] = engine.name
+
+    for category_name in engine.categories:
+        categories.setdefault(category_name, []).append(engine)
+
+
+
[docs]def load_engines(engine_list):
+    """usage: ``engine_list = settings['engines']``"""
+    engines.clear()
+    engine_shortcuts.clear()
+    categories.clear()
+    categories['general'] = []
+    for engine_data in engine_list:
+        engine = load_engine(engine_data)
+        if engine:
+            register_engine(engine)
+    return engines

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/annas_archive.html b/_modules/searx/engines/annas_archive.html new file mode 100644 index 000000000..3ff16d9c6 --- /dev/null +++ b/_modules/searx/engines/annas_archive.html @@ -0,0 +1,299 @@ + + + + + + + + searx.engines.annas_archive — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.annas_archive

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""`Anna's Archive`_ is a free non-profit online shadow library metasearch
+engine providing access to a variety of book resources (also via IPFS), created
+by a team of anonymous archivists (AnnaArchivist_).
+
+.. _Anna's Archive: https://annas-archive.org/
+.. _AnnaArchivist: https://annas-software.org/AnnaArchivist/annas-archive
+
+Configuration
+=============
+
+The engine has the following additional settings:
+
+- :py:obj:`aa_content`
+- :py:obj:`aa_ext`
+- :py:obj:`aa_sort`
+
+With this options a SearXNG maintainer is able to configure **additional**
+engines for specific searches in Anna's Archive.  For example a engine to search
+for *newest* articles and journals (PDF) / by shortcut ``!aaa <search-term>``.
+
+.. code:: yaml
+
+  - name: annas articles
+    engine: annas_archive
+    shortcut: aaa
+    aa_content: 'journal_article'
+    aa_ext: 'pdf'
+    aa_sort: 'newest'
+
+Implementations
+===============
+
+"""
+
+from typing import List, Dict, Any, Optional
+from urllib.parse import quote
+from lxml import html
+
+from searx.utils import extract_text, eval_xpath, eval_xpath_list
+from searx.enginelib.traits import EngineTraits
+from searx.data import ENGINE_TRAITS
+
+# about
+about: Dict[str, Any] = {
+    "website": "https://annas-archive.org/",
+    "wikidata_id": "Q115288326",
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": "HTML",
+}
+
+# engine dependent config
+categories: List[str] = ["files"]
+paging: bool = False
+
+# search-url
+base_url: str = "https://annas-archive.org"
+aa_content: str = ""
+"""Anan's search form field **Content** / possible values::
+
+    journal_article, book_any, book_fiction, book_unknown, book_nonfiction,
+    book_comic, magazine, standards_document
+
+To not filter use an empty string (default).
+"""
+aa_sort: str = ''
+"""Sort Anna's results, possible values::
+
+    newest, oldest, largest, smallest
+
+To sort by *most relevant* use an empty string (default)."""
+
+aa_ext: str = ''
+"""Filter Anna's results by a file ending.  Common filters for example are
+``pdf`` and ``epub``.
+
+.. note::
+
+   Anna's Archive is a beta release: Filter results by file extension does not
+   really work on Anna's Archive.
+
+"""
+
+
+
[docs]def init(engine_settings=None):  # pylint: disable=unused-argument
+    """Check of engine's settings."""
+    traits = EngineTraits(**ENGINE_TRAITS['annas archive'])
+
+    if aa_content and aa_content not in traits.custom['content']:
+        raise ValueError(f'invalid setting content: {aa_content}')
+
+    if aa_sort and aa_sort not in traits.custom['sort']:
+        raise ValueError(f'invalid setting sort: {aa_sort}')
+
+    if aa_ext and aa_ext not in traits.custom['ext']:
+        raise ValueError(f'invalid setting ext: {aa_ext}')

+
+
+def request(query, params: Dict[str, Any]) -> Dict[str, Any]:
+    q = quote(query)
+    lang = traits.get_language(params["language"], traits.all_locale)  # type: ignore
+    params["url"] = base_url + f"/search?lang={lang or ''}&content={aa_content}&ext={aa_ext}&sort={aa_sort}&q={q}"
+    return params
+
+
+def response(resp) -> List[Dict[str, Optional[str]]]:
+    results: List[Dict[str, Optional[str]]] = []
+    dom = html.fromstring(resp.text)
+
+    for item in eval_xpath_list(dom, '//main//div[contains(@class, "h-[125]")]/a'):
+        results.append(_get_result(item))
+
+    # The rendering of the WEB page is very strange; except the first position
+    # all other positions of Anna's result page are enclosed in SGML comments.
+    # These comments are *uncommented* by some JS code, see query of class
+    # '.js-scroll-hidden' in Anna's HTML template:
+    #   https://annas-software.org/AnnaArchivist/annas-archive/-/blob/main/allthethings/templates/macros/md5_list.html
+
+    for item in eval_xpath_list(dom, '//main//div[contains(@class, "js-scroll-hidden")]'):
+        item = html.fromstring(item.xpath('./comment()')[0].text)
+        results.append(_get_result(item))
+
+    return results
+
+
+def _get_result(item):
+    return {
+        'template': 'paper.html',
+        'url': base_url + item.xpath('./@href')[0],
+        'title': extract_text(eval_xpath(item, './/h3/text()[1]')),
+        'publisher': extract_text(eval_xpath(item, './/div[contains(@class, "text-sm")]')),
+        'authors': [extract_text(eval_xpath(item, './/div[contains(@class, "italic")]'))],
+        'content': extract_text(eval_xpath(item, './/div[contains(@class, "text-xs")]')),
+        'img_src': item.xpath('.//img/@src')[0],
+    }
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and other search arguments from Anna's search form."""
+    # pylint: disable=import-outside-toplevel
+
+    import babel
+    from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+    from searx.locales import language_tag
+
+    engine_traits.all_locale = ''
+    engine_traits.custom['content'] = []
+    engine_traits.custom['ext'] = []
+    engine_traits.custom['sort'] = []
+
+    resp = get(base_url + '/search')
+    if not resp.ok:  # type: ignore
+        raise RuntimeError("Response from Anna's search page is not OK.")
+    dom = html.fromstring(resp.text)  # type: ignore
+
+    # supported language codes
+
+    lang_map = {}
+    for x in eval_xpath_list(dom, "//form//select[@name='lang']//option"):
+        eng_lang = x.get("value")
+        if eng_lang in ('', '_empty', 'nl-BE', 'und'):
+            continue
+        try:
+            locale = babel.Locale.parse(lang_map.get(eng_lang, eng_lang), sep='-')
+        except babel.UnknownLocaleError:
+            # silently ignore unknown languages
+            # print("ERROR: %s -> %s is unknown by babel" % (x.get("data-name"), eng_lang))
+            continue
+        sxng_lang = language_tag(locale)
+        conflict = engine_traits.languages.get(sxng_lang)
+        if conflict:
+            if conflict != eng_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_lang, conflict, eng_lang))
+            continue
+        engine_traits.languages[sxng_lang] = eng_lang
+
+    for x in eval_xpath_list(dom, "//form//select[@name='content']//option"):
+        engine_traits.custom['content'].append(x.get("value"))
+
+    for x in eval_xpath_list(dom, "//form//select[@name='ext']//option"):
+        engine_traits.custom['ext'].append(x.get("value"))
+
+    for x in eval_xpath_list(dom, "//form//select[@name='sort']//option"):
+        engine_traits.custom['sort'].append(x.get("value"))

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/archlinux.html b/_modules/searx/engines/archlinux.html new file mode 100644 index 000000000..389e4bafb --- /dev/null +++ b/_modules/searx/engines/archlinux.html @@ -0,0 +1,264 @@ + + + + + + + + searx.engines.archlinux — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.archlinux

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""
+Arch Linux Wiki
+~~~~~~~~~~~~~~~
+
+This implementation does not use a official API: Mediawiki provides API, but
+Arch Wiki blocks access to it.
+
+"""
+
+from typing import TYPE_CHECKING
+from urllib.parse import urlencode, urljoin, urlparse
+import lxml
+import babel
+
+from searx.utils import extract_text, eval_xpath_list, eval_xpath_getindex
+from searx.enginelib.traits import EngineTraits
+from searx.locales import language_tag
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+
+about = {
+    "website": 'https://wiki.archlinux.org/',
+    "wikidata_id": 'Q101445877',
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['it', 'software wikis']
+paging = True
+main_wiki = 'wiki.archlinux.org'
+
+
+def request(query, params):
+
+    sxng_lang = params['searxng_locale'].split('-')[0]
+    netloc: str = traits.custom['wiki_netloc'].get(sxng_lang, main_wiki)  # type: ignore
+    title: str = traits.custom['title'].get(sxng_lang, 'Special:Search')  # type: ignore
+    base_url = 'https://' + netloc + '/index.php?'
+    offset = (params['pageno'] - 1) * 20
+
+    if netloc == main_wiki:
+        eng_lang: str = traits.get_language(sxng_lang, 'English')  # type: ignore
+        query += ' (' + eng_lang + ')'
+    elif netloc == 'wiki.archlinuxcn.org':
+        base_url = 'https://' + netloc + '/wzh/index.php?'
+
+    args = {
+        'search': query,
+        'title': title,
+        'limit': 20,
+        'offset': offset,
+        'profile': 'default',
+    }
+
+    params['url'] = base_url + urlencode(args)
+    return params
+
+
+def response(resp):
+
+    results = []
+    dom = lxml.html.fromstring(resp.text)  # type: ignore
+
+    # get the base URL for the language in which request was made
+    sxng_lang = resp.search_params['searxng_locale'].split('-')[0]
+    netloc: str = traits.custom['wiki_netloc'].get(sxng_lang, main_wiki)  # type: ignore
+    base_url = 'https://' + netloc + '/index.php?'
+
+    for result in eval_xpath_list(dom, '//ul[@class="mw-search-results"]/li'):
+        link = eval_xpath_getindex(result, './/div[@class="mw-search-result-heading"]/a', 0)
+        content = extract_text(result.xpath('.//div[@class="searchresult"]'))
+        results.append(
+            {
+                'url': urljoin(base_url, link.get('href')),  # type: ignore
+                'title': extract_text(link),
+                'content': content,
+            }
+        )
+
+    return results
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages from Archlinix-Wiki.  The location of the Wiki address of a
+    language is mapped in a :py:obj:`custom field
+    <searx.enginelib.traits.EngineTraits.custom>` (``wiki_netloc``).  Depending
+    on the location, the ``title`` argument in the request is translated.
+
+    .. code:: python
+
+       "custom": {
+         "wiki_netloc": {
+           "de": "wiki.archlinux.de",
+            # ...
+           "zh": "wiki.archlinuxcn.org"
+         }
+         "title": {
+           "de": "Spezial:Suche",
+            # ...
+           "zh": "Special:\u641c\u7d22"
+         },
+       },
+
+    """
+    # pylint: disable=import-outside-toplevel
+    from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+
+    engine_traits.custom['wiki_netloc'] = {}
+    engine_traits.custom['title'] = {}
+
+    title_map = {
+        'de': 'Spezial:Suche',
+        'fa': 'ویژه:جستجو',
+        'ja': '特別:検索',
+        'zh': 'Special:搜索',
+    }
+
+    resp = get('https://wiki.archlinux.org/')
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from wiki.archlinix.org is not OK.")
+
+    dom = lxml.html.fromstring(resp.text)  # type: ignore
+    for a in eval_xpath_list(dom, "//a[@class='interlanguage-link-target']"):
+
+        sxng_tag = language_tag(babel.Locale.parse(a.get('lang'), sep='-'))
+        # zh_Hans --> zh
+        sxng_tag = sxng_tag.split('_')[0]
+
+        netloc = urlparse(a.get('href')).netloc
+        if netloc != 'wiki.archlinux.org':
+            title = title_map.get(sxng_tag)
+            if not title:
+                print("ERROR: title tag from %s (%s) is unknown" % (netloc, sxng_tag))
+                continue
+            engine_traits.custom['wiki_netloc'][sxng_tag] = netloc
+            engine_traits.custom['title'][sxng_tag] = title  # type: ignore
+
+        eng_tag = extract_text(eval_xpath_list(a, ".//span"))
+        engine_traits.languages[sxng_tag] = eng_tag  # type: ignore
+
+    engine_traits.languages['en'] = 'English'

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/bing.html b/_modules/searx/engines/bing.html new file mode 100644 index 000000000..b13c6f113 --- /dev/null +++ b/_modules/searx/engines/bing.html @@ -0,0 +1,449 @@ + + + + + + + + searx.engines.bing — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.bing

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Bing-WEB engine. Some of this
+implementations are shared by other engines:
+
+- :ref:`bing images engine`
+- :ref:`bing news engine`
+- :ref:`bing videos engine`
+
+On the `preference page`_ Bing offers a lot of languages an regions (see section
+'Search results languages' and 'Country/region').  However, the abundant choice
+does not correspond to reality, where Bing has a full-text indexer only for a
+limited number of languages.  By example: you can select a language like Māori
+but you never get a result in this language.
+
+What comes a bit closer to the truth are the `search-APIs`_ but they don`t seem
+to be completely correct either (if you take a closer look you will find some
+inaccuracies there too):
+
+- :py:obj:`searx.engines.bing.bing_traits_url`
+- :py:obj:`searx.engines.bing_videos.bing_traits_url`
+- :py:obj:`searx.engines.bing_images.bing_traits_url`
+- :py:obj:`searx.engines.bing_news.bing_traits_url`
+
+.. _preference page: https://www.bing.com/account/general
+.. _search-APIs: https://learn.microsoft.com/en-us/bing/search-apis/
+
+"""
+# pylint: disable=too-many-branches, invalid-name
+
+from typing import TYPE_CHECKING
+import datetime
+import re
+import uuid
+from urllib.parse import urlencode
+from lxml import html
+import babel
+import babel.languages
+
+from searx.utils import eval_xpath, extract_text, eval_xpath_list, eval_xpath_getindex
+from searx.locales import language_tag, region_tag
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+about = {
+    "website": 'https://www.bing.com',
+    "wikidata_id": 'Q182496',
+    "official_api_documentation": 'https://www.microsoft.com/en-us/bing/apis/bing-web-search-api',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+send_accept_language_header = True
+"""Bing tries to guess user's language and territory from the HTTP
+Accept-Language.  Optional the user can select a search-language (can be
+different to the UI language) and a region (market code)."""
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+time_range_support = True
+safesearch = True
+safesearch_types = {2: 'STRICT', 1: 'DEMOTE', 0: 'OFF'}  # cookie: ADLT=STRICT
+
+base_url = 'https://www.bing.com/search'
+"""Bing (Web) search URL"""
+
+bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/reference/market-codes'
+"""Bing (Web) search API description"""
+
+
+def _get_offset_from_pageno(pageno):
+    return (pageno - 1) * 10 + 1
+
+
+def set_bing_cookies(params, engine_language, engine_region, SID):
+
+    # set cookies
+    # -----------
+
+    params['cookies']['_EDGE_V'] = '1'
+
+    # _EDGE_S: F=1&SID=3A5253BD6BCA609509B741876AF961CA&mkt=zh-tw
+    _EDGE_S = [
+        'F=1',
+        'SID=%s' % SID,
+        'mkt=%s' % engine_region.lower(),
+        'ui=%s' % engine_language.lower(),
+    ]
+    params['cookies']['_EDGE_S'] = '&'.join(_EDGE_S)
+    logger.debug("cookie _EDGE_S=%s", params['cookies']['_EDGE_S'])
+
+    # "_EDGE_CD": "m=zh-tw",
+
+    _EDGE_CD = [  # pylint: disable=invalid-name
+        'm=%s' % engine_region.lower(),  # search region: zh-cn
+        'u=%s' % engine_language.lower(),  # UI: en-us
+    ]
+
+    params['cookies']['_EDGE_CD'] = '&'.join(_EDGE_CD) + ';'
+    logger.debug("cookie _EDGE_CD=%s", params['cookies']['_EDGE_CD'])
+
+    SRCHHPGUSR = [  # pylint: disable=invalid-name
+        'SRCHLANG=%s' % engine_language,
+        # Trying to set ADLT cookie here seems not to have any effect, I assume
+        # there is some age verification by a cookie (and/or session ID) needed,
+        # to disable the SafeSearch.
+        'ADLT=%s' % safesearch_types.get(params['safesearch'], 'DEMOTE'),
+    ]
+    params['cookies']['SRCHHPGUSR'] = '&'.join(SRCHHPGUSR)
+    logger.debug("cookie SRCHHPGUSR=%s", params['cookies']['SRCHHPGUSR'])
+
+
+
[docs]def request(query, params):
+    """Assemble a Bing-Web request."""
+
+    engine_region = traits.get_region(params['searxng_locale'], 'en-US')
+    engine_language = traits.get_language(params['searxng_locale'], 'en')
+
+    SID = uuid.uuid1().hex.upper()
+    CVID = uuid.uuid1().hex.upper()
+
+    set_bing_cookies(params, engine_language, engine_region, SID)
+
+    # build URL query
+    # ---------------
+
+    # query term
+    page = int(params.get('pageno', 1))
+    query_params = {
+        # fmt: off
+        'q': query,
+        'pq': query,
+        'cvid': CVID,
+        'qs': 'n',
+        'sp': '-1'
+        # fmt: on
+    }
+
+    # page
+    if page > 1:
+        referer = base_url + '?' + urlencode(query_params)
+        params['headers']['Referer'] = referer
+        logger.debug("headers.Referer --> %s", referer)
+
+    query_params['first'] = _get_offset_from_pageno(page)
+
+    if page == 2:
+        query_params['FORM'] = 'PERE'
+    elif page > 2:
+        query_params['FORM'] = 'PERE%s' % (page - 2)
+
+    filters = ''
+    if params['time_range']:
+        query_params['filt'] = 'custom'
+
+        if params['time_range'] == 'day':
+            filters = 'ex1:"ez1"'
+        elif params['time_range'] == 'week':
+            filters = 'ex1:"ez2"'
+        elif params['time_range'] == 'month':
+            filters = 'ex1:"ez3"'
+        elif params['time_range'] == 'year':
+            epoch_1970 = datetime.date(1970, 1, 1)
+            today_no = (datetime.date.today() - epoch_1970).days
+            filters = 'ex1:"ez5_%s_%s"' % (today_no - 365, today_no)
+
+    params['url'] = base_url + '?' + urlencode(query_params)
+    if filters:
+        params['url'] = params['url'] + '&filters=' + filters
+    return params

+
+
+def response(resp):
+    # pylint: disable=too-many-locals,import-outside-toplevel
+
+    from searx.network import Request, multi_requests  # see https://github.com/searxng/searxng/issues/762
+
+    results = []
+    result_len = 0
+
+    dom = html.fromstring(resp.text)
+
+    # parse results again if nothing is found yet
+
+    url_to_resolve = []
+    url_to_resolve_index = []
+    i = 0
+    for result in eval_xpath_list(dom, '//ol[@id="b_results"]/li[contains(@class, "b_algo")]'):
+
+        link = eval_xpath_getindex(result, './/h2/a', 0, None)
+        if link is None:
+            continue
+        url = link.attrib.get('href')
+        title = extract_text(link)
+
+        content = eval_xpath(result, '(.//p)[1]')
+        for p in content:
+            # Make sure that the element is free of <a href> links
+            for e in p.xpath('.//a'):
+                e.getparent().remove(e)
+        content = extract_text(content)
+
+        # get the real URL either using the URL shown to user or following the Bing URL
+        if url.startswith('https://www.bing.com/ck/a?'):
+            url_cite = extract_text(eval_xpath(result, './/div[@class="b_attribution"]/cite'))
+            # Bing can shorten the URL either at the end or in the middle of the string
+            if (
+                url_cite
+                and url_cite.startswith('https://')
+                and '…' not in url_cite
+                and '...' not in url_cite
+                and '›' not in url_cite
+            ):
+                # no need for an additional HTTP request
+                url = url_cite
+            else:
+                # resolve the URL with an additional HTTP request
+                url_to_resolve.append(url.replace('&ntb=1', '&ntb=F'))
+                url_to_resolve_index.append(i)
+                url = None  # remove the result if the HTTP Bing redirect raise an exception
+
+        # append result
+        results.append({'url': url, 'title': title, 'content': content})
+        # increment result pointer for the next iteration in this loop
+        i += 1
+
+    # resolve all Bing redirections in parallel
+    request_list = [
+        Request.get(u, allow_redirects=False, headers=resp.search_params['headers']) for u in url_to_resolve
+    ]
+    response_list = multi_requests(request_list)
+    for i, redirect_response in enumerate(response_list):
+        if not isinstance(redirect_response, Exception):
+            results[url_to_resolve_index[i]]['url'] = redirect_response.headers['location']
+
+    # get number_of_results
+    try:
+        result_len_container = "".join(eval_xpath(dom, '//span[@class="sb_count"]//text()'))
+        if "-" in result_len_container:
+
+            # Remove the part "from-to" for paginated request ...
+            result_len_container = result_len_container[result_len_container.find("-") * 2 + 2 :]
+
+        result_len_container = re.sub('[^0-9]', '', result_len_container)
+
+        if len(result_len_container) > 0:
+            result_len = int(result_len_container)
+
+    except Exception as e:  # pylint: disable=broad-except
+        logger.debug('result error :\n%s', e)
+
+    if result_len and _get_offset_from_pageno(resp.search_params.get("pageno", 0)) > result_len:
+        return []
+
+    results.append({'number_of_results': result_len})
+    return results
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and regions from Bing-Web."""
+
+    xpath_market_codes = '//table[1]/tbody/tr/td[3]'
+    # xpath_country_codes = '//table[2]/tbody/tr/td[2]'
+    xpath_language_codes = '//table[3]/tbody/tr/td[2]'
+
+    _fetch_traits(engine_traits, bing_traits_url, xpath_language_codes, xpath_market_codes)

+
+
+def _fetch_traits(engine_traits: EngineTraits, url: str, xpath_language_codes: str, xpath_market_codes: str):
+    # pylint: disable=too-many-locals,import-outside-toplevel
+
+    from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+
+    # insert alias to map from a language (zh) to a language + script (zh_Hans)
+    engine_traits.languages['zh'] = 'zh-hans'
+
+    resp = get(url)
+
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from peertube is not OK.")
+
+    dom = html.fromstring(resp.text)  # type: ignore
+
+    map_lang = {'jp': 'ja'}
+    for td in eval_xpath(dom, xpath_language_codes):
+        eng_lang = td.text
+
+        if eng_lang in ('en-gb', 'pt-br'):
+            # language 'en' is already in the list and a language 'en-gb' can't
+            # be handled in SearXNG, same with pt-br which is covered by pt-pt.
+            continue
+
+        babel_lang = map_lang.get(eng_lang, eng_lang).replace('-', '_')
+        try:
+            sxng_tag = language_tag(babel.Locale.parse(babel_lang))
+        except babel.UnknownLocaleError:
+            print("ERROR: language (%s) is unknown by babel" % (eng_lang))
+            continue
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_lang))
+            continue
+        engine_traits.languages[sxng_tag] = eng_lang
+
+    map_region = {
+        'en-ID': 'id_ID',
+        'no-NO': 'nb_NO',
+    }
+
+    for td in eval_xpath(dom, xpath_market_codes):
+        eng_region = td.text
+        babel_region = map_region.get(eng_region, eng_region).replace('-', '_')
+
+        if eng_region == 'en-WW':
+            engine_traits.all_locale = eng_region
+            continue
+
+        try:
+            sxng_tag = region_tag(babel.Locale.parse(babel_region))
+        except babel.UnknownLocaleError:
+            print("ERROR: region (%s) is unknown by babel" % (eng_region))
+            continue
+        conflict = engine_traits.regions.get(sxng_tag)
+        if conflict:
+            if conflict != eng_region:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_region))
+            continue
+        engine_traits.regions[sxng_tag] = eng_region
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/bing_images.html b/_modules/searx/engines/bing_images.html new file mode 100644 index 000000000..af2ee5623 --- /dev/null +++ b/_modules/searx/engines/bing_images.html @@ -0,0 +1,244 @@ + + + + + + + + searx.engines.bing_images — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.bing_images

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Bing-Images: description see :py:obj:`searx.engines.bing`.
+"""
+# pylint: disable=invalid-name
+
+
+from typing import TYPE_CHECKING
+import uuid
+import json
+from urllib.parse import urlencode
+
+from lxml import html
+
+from searx.enginelib.traits import EngineTraits
+from searx.engines.bing import (
+    set_bing_cookies,
+    _fetch_traits,
+)
+from searx.engines.bing import send_accept_language_header  # pylint: disable=unused-import
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://www.bing.com/images',
+    "wikidata_id": 'Q182496',
+    "official_api_documentation": 'https://www.microsoft.com/en-us/bing/apis/bing-image-search-api',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['images', 'web']
+paging = True
+safesearch = True
+time_range_support = True
+
+base_url = 'https://www.bing.com/images/async'
+"""Bing (Images) search URL"""
+
+bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-image-search/reference/market-codes'
+"""Bing (Images) search API description"""
+
+time_map = {
+    # fmt: off
+    'day': 60 * 24,
+    'week': 60 * 24 * 7,
+    'month': 60 * 24 * 31,
+    'year': 60 * 24 * 365,
+    # fmt: on
+}
+
+
+
[docs]def request(query, params):
+    """Assemble a Bing-Image request."""
+
+    engine_region = traits.get_region(params['searxng_locale'], 'en-US')
+    engine_language = traits.get_language(params['searxng_locale'], 'en')
+
+    SID = uuid.uuid1().hex.upper()
+    set_bing_cookies(params, engine_language, engine_region, SID)
+
+    # build URL query
+    # - example: https://www.bing.com/images/async?q=foo&first=155&count=35
+
+    query_params = {
+        # fmt: off
+        'q': query,
+        'async' : 'content',
+        # to simplify the page count lets use the default of 35 images per page
+        'first' : (int(params.get('pageno', 1)) - 1) * 35 + 1,
+        'count' : 35,
+        # fmt: on
+    }
+
+    # time range
+    # - example: one year (525600 minutes) 'qft=+filterui:age-lt525600'
+
+    if params['time_range']:
+        query_params['qft'] = 'filterui:age-lt%s' % time_map[params['time_range']]
+
+    params['url'] = base_url + '?' + urlencode(query_params)
+
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from Bing-Images"""
+
+    results = []
+    dom = html.fromstring(resp.text)
+
+    for result in dom.xpath('//ul[contains(@class, "dgControl_list")]/li'):
+
+        metadata = result.xpath('.//a[@class="iusc"]/@m')
+        if not metadata:
+            continue
+
+        metadata = json.loads(result.xpath('.//a[@class="iusc"]/@m')[0])
+        title = ' '.join(result.xpath('.//div[@class="infnmpt"]//a/text()')).strip()
+        img_format = ' '.join(result.xpath('.//div[@class="imgpt"]/div/span/text()')).strip()
+        source = ' '.join(result.xpath('.//div[@class="imgpt"]//div[@class="lnkw"]//a/text()')).strip()
+        results.append(
+            {
+                'template': 'images.html',
+                'url': metadata['purl'],
+                'thumbnail_src': metadata['turl'],
+                'img_src': metadata['murl'],
+                'content': metadata['desc'],
+                'title': title,
+                'source': source,
+                'img_format': img_format,
+            }
+        )
+    return results

+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and regions from Bing-News."""
+
+    xpath_market_codes = '//table[1]/tbody/tr/td[3]'
+    # xpath_country_codes = '//table[2]/tbody/tr/td[2]'
+    xpath_language_codes = '//table[3]/tbody/tr/td[2]'
+
+    _fetch_traits(engine_traits, bing_traits_url, xpath_language_codes, xpath_market_codes)

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/bing_news.html b/_modules/searx/engines/bing_news.html new file mode 100644 index 000000000..09b21544a --- /dev/null +++ b/_modules/searx/engines/bing_news.html @@ -0,0 +1,262 @@ + + + + + + + + searx.engines.bing_news — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.bing_news

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Bing-News: description see :py:obj:`searx.engines.bing`.
+"""
+
+# pylint: disable=invalid-name
+
+from typing import TYPE_CHECKING
+import uuid
+from urllib.parse import urlencode
+
+from lxml import html
+
+from searx.enginelib.traits import EngineTraits
+from searx.engines.bing import (
+    set_bing_cookies,
+    _fetch_traits,
+)
+from searx.engines.bing import send_accept_language_header  # pylint: disable=unused-import
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+
+# about
+about = {
+    "website": 'https://www.bing.com/news',
+    "wikidata_id": 'Q2878637',
+    "official_api_documentation": 'https://www.microsoft.com/en-us/bing/apis/bing-news-search-api',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'RSS',
+}
+
+# engine dependent config
+categories = ['news']
+paging = True
+time_range_support = True
+time_map = {
+    'day': '4',
+    'week': '8',
+    'month': '9',
+}
+"""A string '4' means *last hour*. We use *last hour* for ``day`` here since the
+difference of *last day* and *last week* in the result list is just marginally.
+"""
+
+base_url = 'https://www.bing.com/news/infinitescrollajax'
+"""Bing (News) search URL"""
+
+bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-news-search/reference/market-codes'
+"""Bing (News) search API description"""
+
+mkt_alias = {
+    'zh': 'en-WW',
+    'zh-CN': 'en-WW',
+}
+"""Bing News has an official market code 'zh-CN' but we won't get a result with
+this market code.  For 'zh' and 'zh-CN' we better use the *Worldwide aggregate*
+market code (en-WW).
+"""
+
+
+
[docs]def request(query, params):
+    """Assemble a Bing-News request."""
+
+    sxng_locale = params['searxng_locale']
+    engine_region = traits.get_region(mkt_alias.get(sxng_locale, sxng_locale), traits.all_locale)
+    engine_language = traits.get_language(sxng_locale, 'en')
+
+    SID = uuid.uuid1().hex.upper()
+    set_bing_cookies(params, engine_language, engine_region, SID)
+
+    # build URL query
+    #
+    # example: https://www.bing.com/news/infinitescrollajax?q=london&first=1
+
+    query_params = {
+        # fmt: off
+        'q': query,
+        'InfiniteScroll': 1,
+        # to simplify the page count lets use the default of 10 images per page
+        'first' : (int(params.get('pageno', 1)) - 1) * 10 + 1,
+        # fmt: on
+    }
+
+    if params['time_range']:
+        # qft=interval:"7"
+        query_params['qft'] = 'qft=interval="%s"' % time_map.get(params['time_range'], '9')
+
+    params['url'] = base_url + '?' + urlencode(query_params)
+
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from Bing-Video"""
+    results = []
+
+    if not resp.ok or not resp.text:
+        return results
+
+    dom = html.fromstring(resp.text)
+
+    for newsitem in dom.xpath('//div[contains(@class, "newsitem")]'):
+
+        url = newsitem.xpath('./@url')[0]
+        title = ' '.join(newsitem.xpath('.//div[@class="caption"]//a[@class="title"]/text()')).strip()
+        content = ' '.join(newsitem.xpath('.//div[@class="snippet"]/text()')).strip()
+        thumbnail = None
+        author = newsitem.xpath('./@data-author')[0]
+        metadata = ' '.join(newsitem.xpath('.//div[@class="source"]/span/text()')).strip()
+
+        img_src = newsitem.xpath('.//a[@class="imagelink"]//img/@src')
+        if img_src:
+            thumbnail = 'https://www.bing.com/' + img_src[0]
+
+        results.append(
+            {
+                'url': url,
+                'title': title,
+                'content': content,
+                'img_src': thumbnail,
+                'author': author,
+                'metadata': metadata,
+            }
+        )
+
+    return results

+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and regions from Bing-News.
+
+    The :py:obj:`description <searx.engines.bing_news.bing_traits_url>` of the
+    first table says *"query parameter when calling the Video Search API."*
+    .. thats why I use the 4. table "News Category API markets" for the
+    ``xpath_market_codes``.
+
+    """
+
+    xpath_market_codes = '//table[4]/tbody/tr/td[3]'
+    # xpath_country_codes = '//table[2]/tbody/tr/td[2]'
+    xpath_language_codes = '//table[3]/tbody/tr/td[2]'
+
+    _fetch_traits(engine_traits, bing_traits_url, xpath_language_codes, xpath_market_codes)

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/bing_videos.html b/_modules/searx/engines/bing_videos.html new file mode 100644 index 000000000..4e3920054 --- /dev/null +++ b/_modules/searx/engines/bing_videos.html @@ -0,0 +1,240 @@ + + + + + + + + searx.engines.bing_videos — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.bing_videos

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Bing-Videos: description see :py:obj:`searx.engines.bing`.
+"""
+# pylint: disable=invalid-name
+
+from typing import TYPE_CHECKING
+import uuid
+import json
+from urllib.parse import urlencode
+
+from lxml import html
+
+from searx.enginelib.traits import EngineTraits
+from searx.engines.bing import (
+    set_bing_cookies,
+    _fetch_traits,
+)
+from searx.engines.bing import send_accept_language_header  # pylint: disable=unused-import
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+
+about = {
+    "website": 'https://www.bing.com/videos',
+    "wikidata_id": 'Q4914152',
+    "official_api_documentation": 'https://www.microsoft.com/en-us/bing/apis/bing-video-search-api',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['videos', 'web']
+paging = True
+safesearch = True
+time_range_support = True
+
+base_url = 'https://www.bing.com/videos/asyncv2'
+"""Bing (Videos) async search URL."""
+
+bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-video-search/reference/market-codes'
+"""Bing (Video) search API description"""
+
+time_map = {
+    # fmt: off
+    'day': 60 * 24,
+    'week': 60 * 24 * 7,
+    'month': 60 * 24 * 31,
+    'year': 60 * 24 * 365,
+    # fmt: on
+}
+
+
+
[docs]def request(query, params):
+    """Assemble a Bing-Video request."""
+
+    engine_region = traits.get_region(params['searxng_locale'], 'en-US')
+    engine_language = traits.get_language(params['searxng_locale'], 'en')
+
+    SID = uuid.uuid1().hex.upper()
+    set_bing_cookies(params, engine_language, engine_region, SID)
+
+    # build URL query
+    #
+    # example: https://www.bing.com/videos/asyncv2?q=foo&async=content&first=1&count=35
+
+    query_params = {
+        # fmt: off
+        'q': query,
+        'async' : 'content',
+        # to simplify the page count lets use the default of 35 images per page
+        'first' : (int(params.get('pageno', 1)) - 1) * 35 + 1,
+        'count' : 35,
+        # fmt: on
+    }
+
+    # time range
+    #
+    # example: one week (10080 minutes) '&qft= filterui:videoage-lt10080'  '&form=VRFLTR'
+
+    if params['time_range']:
+        query_params['form'] = 'VRFLTR'
+        query_params['qft'] = ' filterui:videoage-lt%s' % time_map[params['time_range']]
+
+    params['url'] = base_url + '?' + urlencode(query_params)
+
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from Bing-Video"""
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for result in dom.xpath('//div[@class="dg_u"]//div[contains(@id, "mc_vtvc_video")]'):
+        metadata = json.loads(result.xpath('.//div[@class="vrhdata"]/@vrhm')[0])
+        info = ' - '.join(result.xpath('.//div[@class="mc_vtvc_meta_block"]//span/text()')).strip()
+        content = '{0} - {1}'.format(metadata['du'], info)
+        thumbnail = result.xpath('.//div[contains(@class, "mc_vtvc_th")]//img/@src')[0]
+
+        results.append(
+            {
+                'url': metadata['murl'],
+                'thumbnail': thumbnail,
+                'title': metadata.get('vt', ''),
+                'content': content,
+                'template': 'videos.html',
+            }
+        )
+
+    return results

+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and regions from Bing-Videos."""
+
+    xpath_market_codes = '//table[1]/tbody/tr/td[3]'
+    # xpath_country_codes = '//table[2]/tbody/tr/td[2]'
+    xpath_language_codes = '//table[3]/tbody/tr/td[2]'
+
+    _fetch_traits(engine_traits, bing_traits_url, xpath_language_codes, xpath_market_codes)

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/brave.html b/_modules/searx/engines/brave.html new file mode 100644 index 000000000..1aa6ee3b6 --- /dev/null +++ b/_modules/searx/engines/brave.html @@ -0,0 +1,531 @@ + + + + + + + + searx.engines.brave — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.brave

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Brave supports the categories listed in :py:obj:`brave_category` (General,
+news, videos, images).  The support of :py:obj:`paging` and :py:obj:`time range
+<time_range_support>` is limited (see remarks).
+
+Configured ``brave`` engines:
+
+.. code:: yaml
+
+  - name: brave
+    engine: brave
+    ...
+    brave_category: search
+    time_range_support: true
+    paging: true
+
+  - name: brave.images
+    engine: brave
+    ...
+    brave_category: images
+
+  - name: brave.videos
+    engine: brave
+    ...
+    brave_category: videos
+
+  - name: brave.news
+    engine: brave
+    ...
+    brave_category: news
+
+
+.. _brave regions:
+
+Brave regions
+=============
+
+Brave uses two-digit tags for the regions like ``ca`` while SearXNG deals with
+locales.  To get a mapping, all *officatl de-facto* languages of the Brave
+region are mapped to regions in SearXNG (see :py:obj:`babel
+<babel.languages.get_official_languages>`):
+
+.. code:: python
+
+    "regions": {
+      ..
+      "en-CA": "ca",
+      "fr-CA": "ca",
+      ..
+     }
+
+
+.. note::
+
+   The language (aka region) support of Brave's index is limited to very basic
+   languages.  The search results for languages like Chinese or Arabic are of
+   low quality.
+
+
+.. _brave languages:
+
+Brave languages
+===============
+
+Brave's language support is limited to the UI (menues, area local notations,
+etc).  Brave's index only seems to support a locale, but it does not seem to
+support any languages in its index.  The choice of available languages is very
+small (and its not clear to me where the differencee in UI is when switching
+from en-us to en-ca or en-gb).
+
+In the :py:obj:`EngineTraits object <searx.enginelib.traits.EngineTraits>` the
+UI languages are stored in a custom field named ``ui_lang``:
+
+.. code:: python
+
+    "custom": {
+      "ui_lang": {
+        "ca": "ca",
+        "de-DE": "de-de",
+        "en-CA": "en-ca",
+        "en-GB": "en-gb",
+        "en-US": "en-us",
+        "es": "es",
+        "fr-CA": "fr-ca",
+        "fr-FR": "fr-fr",
+        "ja-JP": "ja-jp",
+        "pt-BR": "pt-br",
+        "sq-AL": "sq-al"
+      }
+    },
+
+Implementations
+===============
+
+"""
+
+from typing import TYPE_CHECKING
+
+import re
+from urllib.parse import (
+    urlencode,
+    urlparse,
+    parse_qs,
+)
+
+import chompjs
+from lxml import html
+
+from searx import locales
+from searx.utils import (
+    extract_text,
+    eval_xpath_list,
+    eval_xpath_getindex,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+about = {
+    "website": 'https://search.brave.com/',
+    "wikidata_id": 'Q22906900',
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+base_url = "https://search.brave.com/"
+categories = []
+brave_category = 'search'
+"""Brave supports common web-search, video search, image and video search.
+
+- ``search``: Common WEB search
+- ``videos``: search for videos
+- ``images``: search for images
+- ``news``: search for news
+"""
+
+brave_spellcheck = False
+"""Brave supports some kind of spell checking.  When activated, Brave tries to
+fix typos, e.g. it searches for ``food`` when the user queries for ``fooh``.  In
+the UI of Brave the user gets warned about this, since we can not warn the user
+in SearXNG, the spellchecking is disabled by default.
+"""
+
+send_accept_language_header = True
+paging = False
+"""Brave only supports paging in :py:obj:`brave_category` ``search`` (UI
+category All)."""
+
+safesearch = True
+safesearch_map = {2: 'strict', 1: 'moderate', 0: 'off'}  # cookie: safesearch=off
+
+time_range_support = False
+"""Brave only supports time-range in :py:obj:`brave_category` ``search`` (UI
+category All)."""
+
+time_range_map = {
+    'day': 'pd',
+    'week': 'pw',
+    'month': 'pm',
+    'year': 'py',
+}
+
+
+def request(query, params):
+
+    # Don't accept br encoding / see https://github.com/searxng/searxng/pull/1787
+    params['headers']['Accept-Encoding'] = 'gzip, deflate'
+
+    args = {
+        'q': query,
+    }
+    if brave_spellcheck:
+        args['spellcheck'] = '1'
+
+    if brave_category == 'search':
+        if params.get('pageno', 1) - 1:
+            args['offset'] = params.get('pageno', 1) - 1
+        if time_range_map.get(params['time_range']):
+            args['tf'] = time_range_map.get(params['time_range'])
+
+    params["url"] = f"{base_url}{brave_category}?{urlencode(args)}"
+
+    # set properties in the cookies
+
+    params['cookies']['safesearch'] = safesearch_map.get(params['safesearch'], 'off')
+    # the useLocation is IP based, we use cookie 'country' for the region
+    params['cookies']['useLocation'] = '0'
+    params['cookies']['summarizer'] = '0'
+
+    engine_region = traits.get_region(params['searxng_locale'], 'all')
+    params['cookies']['country'] = engine_region.split('-')[-1].lower()  # type: ignore
+
+    ui_lang = locales.get_engine_locale(params['searxng_locale'], traits.custom["ui_lang"], 'en-us')
+    params['cookies']['ui_lang'] = ui_lang
+
+    logger.debug("cookies %s", params['cookies'])
+
+
+def response(resp):
+
+    if brave_category == 'search':
+        return _parse_search(resp)
+
+    datastr = ""
+    for line in resp.text.split("\n"):
+        if "const data = " in line:
+            datastr = line.replace("const data = ", "").strip()[:-1]
+            break
+
+    json_data = chompjs.parse_js_object(datastr)
+    json_resp = json_data[1]['data']['body']['response']
+
+    if brave_category == 'news':
+        json_resp = json_resp['news']
+        return _parse_news(json_resp)
+
+    if brave_category == 'images':
+        return _parse_images(json_resp)
+    if brave_category == 'videos':
+        return _parse_videos(json_resp)
+
+    raise ValueError(f"Unsupported brave category: {brave_category}")
+
+
+def _parse_search(resp):
+
+    result_list = []
+    dom = html.fromstring(resp.text)
+
+    answer_tag = eval_xpath_getindex(dom, '//div[@class="answer"]', 0, default=None)
+    if answer_tag:
+        result_list.append({'answer': extract_text(answer_tag)})
+
+    # xpath_results = '//div[contains(@class, "snippet fdb") and @data-type="web"]'
+    xpath_results = '//div[contains(@class, "snippet")]'
+
+    for result in eval_xpath_list(dom, xpath_results):
+
+        url = eval_xpath_getindex(result, './/a[@class="result-header"]/@href', 0, default=None)
+        title_tag = eval_xpath_getindex(result, './/span[@class="snippet-title"]', 0, default=None)
+        if not (url and title_tag):
+            continue
+
+        content_tag = eval_xpath_getindex(result, './/p[@class="snippet-description"]', 0, default='')
+        img_src = eval_xpath_getindex(result, './/img[@class="thumb"]/@src', 0, default='')
+
+        item = {
+            'url': url,
+            'title': extract_text(title_tag),
+            'content': extract_text(content_tag),
+            'img_src': img_src,
+        }
+
+        video_tag = eval_xpath_getindex(
+            result, './/div[contains(@class, "video-snippet") and @data-macro="video"]', 0, default=None
+        )
+        if video_tag is not None:
+
+            # In my tests a video tag in the WEB search was mostoften not a
+            # video, except the ones from youtube ..
+
+            iframe_src = _get_iframe_src(url)
+            if iframe_src:
+                item['iframe_src'] = iframe_src
+                item['template'] = 'videos.html'
+                item['thumbnail'] = eval_xpath_getindex(video_tag, './/img/@src', 0, default='')
+            else:
+                item['img_src'] = eval_xpath_getindex(video_tag, './/img/@src', 0, default='')
+
+        result_list.append(item)
+
+    return result_list
+
+
+def _get_iframe_src(url):
+    parsed_url = urlparse(url)
+    if parsed_url.path == '/watch' and parsed_url.query:
+        video_id = parse_qs(parsed_url.query).get('v', [])  # type: ignore
+        if video_id:
+            return 'https://www.youtube-nocookie.com/embed/' + video_id[0]  # type: ignore
+    return None
+
+
+def _parse_news(json_resp):
+    result_list = []
+
+    for result in json_resp["results"]:
+        item = {
+            'url': result['url'],
+            'title': result['title'],
+            'content': result['description'],
+        }
+        if result['thumbnail'] != "null":
+            item['img_src'] = result['thumbnail']['src']
+        result_list.append(item)
+
+    return result_list
+
+
+def _parse_images(json_resp):
+    result_list = []
+
+    for result in json_resp["results"]:
+        item = {
+            'url': result['url'],
+            'title': result['title'],
+            'content': result['description'],
+            'template': 'images.html',
+            'img_format': result['properties']['format'],
+            'source': result['source'],
+            'img_src': result['properties']['url'],
+        }
+        result_list.append(item)
+
+    return result_list
+
+
+def _parse_videos(json_resp):
+    result_list = []
+
+    for result in json_resp["results"]:
+
+        url = result['url']
+        item = {
+            'url': url,
+            'title': result['title'],
+            'content': result['description'],
+            'template': 'videos.html',
+            'length': result['video']['duration'],
+            'duration': result['video']['duration'],
+        }
+
+        if result['thumbnail'] != "null":
+            item['thumbnail'] = result['thumbnail']['src']
+
+        iframe_src = _get_iframe_src(url)
+        if iframe_src:
+            item['iframe_src'] = iframe_src
+
+        result_list.append(item)
+
+    return result_list
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch :ref:`languages <brave languages>` and :ref:`regions <brave
+    regions>` from Brave."""
+
+    # pylint: disable=import-outside-toplevel
+
+    import babel.languages
+    from searx.locales import region_tag, language_tag
+    from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+
+    engine_traits.custom["ui_lang"] = {}
+
+    headers = {
+        'Accept-Encoding': 'gzip, deflate',
+    }
+    lang_map = {'no': 'nb'}  # norway
+
+    # languages (UI)
+
+    resp = get('https://search.brave.com/settings', headers=headers)
+
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from Brave is not OK.")
+    dom = html.fromstring(resp.text)  # type: ignore
+
+    for option in dom.xpath('//div[@id="language-select"]//option'):
+
+        ui_lang = option.get('value')
+        try:
+            if '-' in ui_lang:
+                sxng_tag = region_tag(babel.Locale.parse(ui_lang, sep='-'))
+            else:
+                sxng_tag = language_tag(babel.Locale.parse(ui_lang))
+
+        except babel.UnknownLocaleError:
+            print("ERROR: can't determine babel locale of Brave's (UI) language %s" % ui_lang)
+            continue
+
+        conflict = engine_traits.custom["ui_lang"].get(sxng_tag)
+        if conflict:
+            if conflict != ui_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, ui_lang))
+            continue
+        engine_traits.custom["ui_lang"][sxng_tag] = ui_lang
+
+    # search regions of brave
+
+    engine_traits.all_locale = 'all'
+
+    for country in dom.xpath('//div[@id="sidebar"]//ul/li/div[contains(@class, "country")]'):
+
+        flag = country.xpath('./span[contains(@class, "flag")]')[0]
+        # country_name = extract_text(flag.xpath('./following-sibling::*')[0])
+        country_tag = re.search(r'flag-([^\s]*)\s', flag.xpath('./@class')[0]).group(1)  # type: ignore
+
+        # add offical languages of the country ..
+        for lang_tag in babel.languages.get_official_languages(country_tag, de_facto=True):
+            lang_tag = lang_map.get(lang_tag, lang_tag)
+            sxng_tag = region_tag(babel.Locale.parse('%s_%s' % (lang_tag, country_tag.upper())))
+            # print("%-20s: %s <-- %s" % (country_name, country_tag, sxng_tag))
+
+            conflict = engine_traits.regions.get(sxng_tag)
+            if conflict:
+                if conflict != country_tag:
+                    print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, country_tag))
+                    continue
+            engine_traits.regions[sxng_tag] = country_tag

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/command.html b/_modules/searx/engines/command.html new file mode 100644 index 000000000..020bcf7a9 --- /dev/null +++ b/_modules/searx/engines/command.html @@ -0,0 +1,355 @@ + + + + + + + + searx.engines.command — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.command

+# SPDX-License-Identifier: AGPL-3.0-or-later
+"""With *command engines* administrators can run engines to integrate arbitrary
+shell commands.
+
+.. attention::
+
+   When creating and enabling a ``command`` engine on a public instance, you
+   must be careful to avoid leaking private data.
+
+The easiest solution is to limit the access by setting ``tokens`` as described
+in section :ref:`private engines`.  The engine base is flexible.  Only your
+imagination can limit the power of this engine (and maybe security concerns).
+
+Configuration
+=============
+
+The following options are available:
+
+``command``:
+  A comma separated list of the elements of the command.  A special token
+  ``{{QUERY}}`` tells where to put the search terms of the user. Example:
+
+  .. code:: yaml
+
+     ['ls', '-l', '-h', '{{QUERY}}']
+
+``delimiter``:
+  A mapping containing a delimiter ``char`` and the *titles* of each element in
+  ``keys``.
+
+``parse_regex``:
+  A dict containing the regular expressions for each result key.
+
+``query_type``:
+
+  The expected type of user search terms.  Possible values: ``path`` and
+  ``enum``.
+
+  ``path``:
+    Checks if the user provided path is inside the working directory.  If not,
+    the query is not executed.
+
+  ``enum``:
+    Is a list of allowed search terms.  If the user submits something which is
+    not included in the list, the query returns an error.
+
+``query_enum``:
+  A list containing allowed search terms if ``query_type`` is set to ``enum``.
+
+``working_dir``:
+  The directory where the command has to be executed.  Default: ``./``.
+
+``result_separator``:
+  The character that separates results. Default: ``\\n``.
+
+Example
+=======
+
+The example engine below can be used to find files with a specific name in the
+configured working directory:
+
+.. code:: yaml
+
+  - name: find
+    engine: command
+    command: ['find', '.', '-name', '{{QUERY}}']
+    query_type: path
+    shortcut: fnd
+    delimiter:
+        chars: ' '
+        keys: ['line']
+
+Implementations
+===============
+"""
+
+import re
+from os.path import expanduser, isabs, realpath, commonprefix
+from shlex import split as shlex_split
+from subprocess import Popen, PIPE
+from threading import Thread
+
+from searx import logger
+
+
+engine_type = 'offline'
+paging = True
+command = []
+delimiter = {}
+parse_regex = {}
+query_type = ''
+query_enum = []
+environment_variables = {}
+working_dir = realpath('.')
+result_separator = '\n'
+result_template = 'key-value.html'
+timeout = 4.0
+
+_command_logger = logger.getChild('command')
+_compiled_parse_regex = {}
+
+
+def init(engine_settings):
+    check_parsing_options(engine_settings)
+
+    if 'command' not in engine_settings:
+        raise ValueError('engine command : missing configuration key: command')
+
+    global command, working_dir, delimiter, parse_regex, environment_variables
+
+    command = engine_settings['command']
+
+    if 'working_dir' in engine_settings:
+        working_dir = engine_settings['working_dir']
+        if not isabs(engine_settings['working_dir']):
+            working_dir = realpath(working_dir)
+
+    if 'parse_regex' in engine_settings:
+        parse_regex = engine_settings['parse_regex']
+        for result_key, regex in parse_regex.items():
+            _compiled_parse_regex[result_key] = re.compile(regex, flags=re.MULTILINE)
+    if 'delimiter' in engine_settings:
+        delimiter = engine_settings['delimiter']
+
+    if 'environment_variables' in engine_settings:
+        environment_variables = engine_settings['environment_variables']
+
+
+def search(query, params):
+    cmd = _get_command_to_run(query)
+    if not cmd:
+        return []
+
+    results = []
+    reader_thread = Thread(target=_get_results_from_process, args=(results, cmd, params['pageno']))
+    reader_thread.start()
+    reader_thread.join(timeout=timeout)
+
+    return results
+
+
+def _get_command_to_run(query):
+    params = shlex_split(query)
+    __check_query_params(params)
+
+    cmd = []
+    for c in command:
+        if c == '{{QUERY}}':
+            cmd.extend(params)
+        else:
+            cmd.append(c)
+
+    return cmd
+
+
+def _get_results_from_process(results, cmd, pageno):
+    leftover = ''
+    count = 0
+    start, end = __get_results_limits(pageno)
+    with Popen(cmd, stdout=PIPE, stderr=PIPE, env=environment_variables) as process:
+        line = process.stdout.readline()
+        while line:
+            buf = leftover + line.decode('utf-8')
+            raw_results = buf.split(result_separator)
+            if raw_results[-1]:
+                leftover = raw_results[-1]
+            raw_results = raw_results[:-1]
+
+            for raw_result in raw_results:
+                result = __parse_single_result(raw_result)
+                if result is None:
+                    _command_logger.debug('skipped result:', raw_result)
+                    continue
+
+                if start <= count and count <= end:
+                    result['template'] = result_template
+                    results.append(result)
+
+                count += 1
+                if end < count:
+                    return results
+
+            line = process.stdout.readline()
+
+        return_code = process.wait(timeout=timeout)
+        if return_code != 0:
+            raise RuntimeError('non-zero return code when running command', cmd, return_code)
+
+
+def __get_results_limits(pageno):
+    start = (pageno - 1) * 10
+    end = start + 9
+    return start, end
+
+
+def __check_query_params(params):
+    if not query_type:
+        return
+
+    if query_type == 'path':
+        query_path = params[-1]
+        query_path = expanduser(query_path)
+        if commonprefix([realpath(query_path), working_dir]) != working_dir:
+            raise ValueError('requested path is outside of configured working directory')
+    elif query_type == 'enum' and len(query_enum) > 0:
+        for param in params:
+            if param not in query_enum:
+                raise ValueError('submitted query params is not allowed', param, 'allowed params:', query_enum)
+
+
+
[docs]def check_parsing_options(engine_settings):
+    """Checks if delimiter based parsing or regex parsing is configured correctly"""
+
+    if 'delimiter' not in engine_settings and 'parse_regex' not in engine_settings:
+        raise ValueError('failed to init settings for parsing lines: missing delimiter or parse_regex')
+    if 'delimiter' in engine_settings and 'parse_regex' in engine_settings:
+        raise ValueError('failed to init settings for parsing lines: too many settings')
+
+    if 'delimiter' in engine_settings:
+        if 'chars' not in engine_settings['delimiter'] or 'keys' not in engine_settings['delimiter']:
+            raise ValueError

+
+
+def __parse_single_result(raw_result):
+    """Parses command line output based on configuration"""
+
+    result = {}
+
+    if delimiter:
+        elements = raw_result.split(delimiter['chars'], maxsplit=len(delimiter['keys']) - 1)
+        if len(elements) != len(delimiter['keys']):
+            return {}
+        for i in range(len(elements)):
+            result[delimiter['keys'][i]] = elements[i]
+
+    if parse_regex:
+        for result_key, regex in _compiled_parse_regex.items():
+            found = regex.search(raw_result)
+            if not found:
+                return {}
+            result[result_key] = raw_result[found.start() : found.end()]
+
+    return result
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/dailymotion.html b/_modules/searx/engines/dailymotion.html new file mode 100644 index 000000000..23949583d --- /dev/null +++ b/_modules/searx/engines/dailymotion.html @@ -0,0 +1,364 @@ + + + + + + + + searx.engines.dailymotion — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.dailymotion

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""
+Dailymotion (Videos)
+~~~~~~~~~~~~~~~~~~~~
+
+.. _REST GET: https://developers.dailymotion.com/tools/
+.. _Global API Parameters: https://developers.dailymotion.com/api/#global-parameters
+.. _Video filters API: https://developers.dailymotion.com/api/#video-filters
+.. _Fields selection: https://developers.dailymotion.com/api/#fields-selection
+
+"""
+
+from typing import TYPE_CHECKING
+
+from datetime import datetime, timedelta
+from urllib.parse import urlencode
+import time
+import babel
+
+from searx.network import get, raise_for_httperror  # see https://github.com/searxng/searxng/issues/762
+from searx.utils import html_to_text
+from searx.exceptions import SearxEngineAPIException
+from searx.locales import region_tag, language_tag
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://www.dailymotion.com',
+    "wikidata_id": 'Q769222',
+    "official_api_documentation": 'https://www.dailymotion.com/developer',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# engine dependent config
+categories = ['videos']
+paging = True
+number_of_results = 10
+
+time_range_support = True
+time_delta_dict = {
+    "day": timedelta(days=1),
+    "week": timedelta(days=7),
+    "month": timedelta(days=31),
+    "year": timedelta(days=365),
+}
+
+safesearch = True
+safesearch_params = {
+    2: {'is_created_for_kids': 'true'},
+    1: {'is_created_for_kids': 'true'},
+    0: {},
+}
+"""True if this video is "Created for Kids" / intends to target an audience
+under the age of 16 (``is_created_for_kids`` in `Video filters API`_ )
+"""
+
+family_filter_map = {
+    2: 'true',
+    1: 'true',
+    0: 'false',
+}
+"""By default, the family filter is turned on. Setting this parameter to
+``false`` will stop filtering-out explicit content from searches and global
+contexts (``family_filter`` in `Global API Parameters`_ ).
+"""
+
+result_fields = [
+    'allow_embed',
+    'description',
+    'title',
+    'created_time',
+    'duration',
+    'url',
+    'thumbnail_360_url',
+    'id',
+]
+"""`Fields selection`_, by default, a few fields are returned. To request more
+specific fields, the ``fields`` parameter is used with the list of fields
+SearXNG needs in the response to build a video result list.
+"""
+
+search_url = 'https://api.dailymotion.com/videos?'
+"""URL to retrieve a list of videos.
+
+- `REST GET`_
+- `Global API Parameters`_
+- `Video filters API`_
+"""
+
+iframe_src = "https://www.dailymotion.com/embed/video/{video_id}"
+"""URL template to embed video in SearXNG's result list."""
+
+
+def request(query, params):
+
+    if not query:
+        return False
+
+    eng_region: str = traits.get_region(params['searxng_locale'], 'en_US')  # type: ignore
+    eng_lang = traits.get_language(params['searxng_locale'], 'en')
+
+    args = {
+        'search': query,
+        'family_filter': family_filter_map.get(params['safesearch'], 'false'),
+        'thumbnail_ratio': 'original',  # original|widescreen|square
+        # https://developers.dailymotion.com/api/#video-filters
+        'languages': eng_lang,
+        'page': params['pageno'],
+        'password_protected': 'false',
+        'private': 'false',
+        'sort': 'relevance',
+        'limit': number_of_results,
+        'fields': ','.join(result_fields),
+    }
+
+    args.update(safesearch_params.get(params['safesearch'], {}))
+
+    # Don't add localization and country arguments if the user does select a
+    # language (:de, :en, ..)
+
+    if len(params['searxng_locale'].split('-')) > 1:
+        # https://developers.dailymotion.com/api/#global-parameters
+        args['localization'] = eng_region
+        args['country'] = eng_region.split('_')[1]
+        # Insufficient rights for the `ams_country' parameter of route `GET /videos'
+        # 'ams_country': eng_region.split('_')[1],
+
+    time_delta = time_delta_dict.get(params["time_range"])
+    if time_delta:
+        created_after = datetime.now() - time_delta
+        args['created_after'] = datetime.timestamp(created_after)
+
+    query_str = urlencode(args)
+    params['url'] = search_url + query_str
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    search_res = resp.json()
+
+    # check for an API error
+    if 'error' in search_res:
+        raise SearxEngineAPIException(search_res['error'].get('message'))
+
+    raise_for_httperror(resp)
+
+    # parse results
+    for res in search_res.get('list', []):
+
+        title = res['title']
+        url = res['url']
+
+        content = html_to_text(res['description'])
+        if len(content) > 300:
+            content = content[:300] + '...'
+
+        publishedDate = datetime.fromtimestamp(res['created_time'], None)
+
+        length = time.gmtime(res.get('duration'))
+        if length.tm_hour:
+            length = time.strftime("%H:%M:%S", length)
+        else:
+            length = time.strftime("%M:%S", length)
+
+        thumbnail = res['thumbnail_360_url']
+        thumbnail = thumbnail.replace("http://", "https://")
+
+        item = {
+            'template': 'videos.html',
+            'url': url,
+            'title': title,
+            'content': content,
+            'publishedDate': publishedDate,
+            'length': length,
+            'thumbnail': thumbnail,
+        }
+
+        # HINT: no mater what the value is, without API token videos can't shown
+        # embedded
+        if res['allow_embed']:
+            item['iframe_src'] = iframe_src.format(video_id=res['id'])
+
+        results.append(item)
+
+    # return results
+    return results
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch locales & languages from dailymotion.
+
+    Locales fetched from `api/locales <https://api.dailymotion.com/locales>`_.
+    There are duplications in the locale codes returned from Dailymotion which
+    can be ignored::
+
+      en_EN --> en_GB, en_US
+      ar_AA --> ar_EG, ar_AE, ar_SA
+
+    The language list `api/languages <https://api.dailymotion.com/languages>`_
+    contains over 7000 *languages* codes (see PR1071_).  We use only those
+    language codes that are used in the locales.
+
+    .. _PR1071: https://github.com/searxng/searxng/pull/1071
+
+    """
+
+    resp = get('https://api.dailymotion.com/locales')
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from dailymotion/locales is not OK.")
+
+    for item in resp.json()['list']:  # type: ignore
+        eng_tag = item['locale']
+        if eng_tag in ('en_EN', 'ar_AA'):
+            continue
+        try:
+            sxng_tag = region_tag(babel.Locale.parse(eng_tag))
+        except babel.UnknownLocaleError:
+            print("ERROR: item unknown --> %s" % item)
+            continue
+
+        conflict = engine_traits.regions.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.regions[sxng_tag] = eng_tag
+
+    locale_lang_list = [x.split('_')[0] for x in engine_traits.regions.values()]
+
+    resp = get('https://api.dailymotion.com/languages')
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from dailymotion/languages is not OK.")
+
+    for item in resp.json()['list']:  # type: ignore
+        eng_tag = item['code']
+        if eng_tag in locale_lang_list:
+            sxng_tag = language_tag(babel.Locale.parse(eng_tag))
+            engine_traits.languages[sxng_tag] = eng_tag

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/demo_offline.html b/_modules/searx/engines/demo_offline.html new file mode 100644 index 000000000..2c6670cf0 --- /dev/null +++ b/_modules/searx/engines/demo_offline.html @@ -0,0 +1,185 @@ + + + + + + + + searx.engines.demo_offline — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.demo_offline

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Within this module we implement a *demo offline engine*.  Do not look to
+close to the implementation, its just a simple example.  To get in use of this
+*demo* engine add the following entry to your engines list in ``settings.yml``:
+
+.. code:: yaml
+
+  - name: my offline engine
+    engine: demo_offline
+    shortcut: demo
+    disabled: false
+
+"""
+
+import json
+
+engine_type = 'offline'
+categories = ['general']
+disabled = True
+timeout = 2.0
+
+about = {
+    "wikidata_id": None,
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# if there is a need for globals, use a leading underline
+_my_offline_engine = None
+
+
+
[docs]def init(engine_settings=None):
+    """Initialization of the (offline) engine.  The origin of this demo engine is a
+    simple json string which is loaded in this example while the engine is
+    initialized.
+
+    """
+    global _my_offline_engine  # pylint: disable=global-statement
+
+    _my_offline_engine = (
+        '[ {"value": "%s"}'
+        ', {"value":"first item"}'
+        ', {"value":"second item"}'
+        ', {"value":"third item"}'
+        ']' % engine_settings.get('name')
+    )

+
+
+
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/demo_online.html b/_modules/searx/engines/demo_online.html new file mode 100644 index 000000000..560ad4eb3 --- /dev/null +++ b/_modules/searx/engines/demo_online.html @@ -0,0 +1,212 @@ + + + + + + + + searx.engines.demo_online — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.demo_online

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Within this module we implement a *demo online engine*.  Do not look to
+close to the implementation, its just a simple example which queries `The Art
+Institute of Chicago <https://www.artic.edu>`_
+
+To get in use of this *demo* engine add the following entry to your engines
+list in ``settings.yml``:
+
+.. code:: yaml
+
+  - name: my online engine
+    engine: demo_online
+    shortcut: demo
+    disabled: false
+
+"""
+
+from json import loads
+from urllib.parse import urlencode
+
+engine_type = 'online'
+send_accept_language_header = True
+categories = ['general']
+disabled = True
+timeout = 2.0
+categories = ['images']
+paging = True
+page_size = 20
+
+search_api = 'https://api.artic.edu/api/v1/artworks/search?'
+image_api = 'https://www.artic.edu/iiif/2/'
+
+about = {
+    "website": 'https://www.artic.edu',
+    "wikidata_id": 'Q239303',
+    "official_api_documentation": 'http://api.artic.edu/docs/',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+
+# if there is a need for globals, use a leading underline
+_my_online_engine = None
+
+
+
[docs]def init(engine_settings):
+    """Initialization of the (online) engine.  If no initialization is needed, drop
+    this init function.
+
+    """
+    global _my_online_engine  # pylint: disable=global-statement
+    _my_online_engine = engine_settings.get('name')

+
+
+
[docs]def request(query, params):
+    """Build up the ``params`` for the online request.  In this example we build a
+    URL to fetch images from `artic.edu <https://artic.edu>`__
+
+    """
+    args = urlencode(
+        {
+            'q': query,
+            'page': params['pageno'],
+            'fields': 'id,title,artist_display,medium_display,image_id,date_display,dimensions,artist_titles',
+            'limit': page_size,
+        }
+    )
+    params['url'] = search_api + args
+    return params

+
+
+
[docs]def response(resp):
+    """Parse out the result items from the response.  In this example we parse the
+    response from `api.artic.edu <https://artic.edu>`__ and filter out all
+    images.
+
+    """
+    results = []
+    json_data = loads(resp.text)
+
+    for result in json_data['data']:
+
+        if not result['image_id']:
+            continue
+
+        results.append(
+            {
+                'url': 'https://artic.edu/artworks/%(id)s' % result,
+                'title': result['title'] + " (%(date_display)s) //  %(artist_display)s" % result,
+                'content': result['medium_display'],
+                'author': ', '.join(result['artist_titles']),
+                'img_src': image_api + '/%(image_id)s/full/843,/0/default.jpg' % result,
+                'img_format': result['dimensions'],
+                'template': 'images.html',
+            }
+        )
+
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/duckduckgo.html b/_modules/searx/engines/duckduckgo.html new file mode 100644 index 000000000..e36f66b5c --- /dev/null +++ b/_modules/searx/engines/duckduckgo.html @@ -0,0 +1,549 @@ + + + + + + + + searx.engines.duckduckgo — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.duckduckgo

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""
+DuckDuckGo Lite
+~~~~~~~~~~~~~~~
+"""
+
+from typing import TYPE_CHECKING
+import re
+from urllib.parse import urlencode
+import json
+import babel
+import lxml.html
+
+from searx import (
+    locales,
+    redislib,
+    external_bang,
+)
+from searx.utils import (
+    eval_xpath,
+    eval_xpath_getindex,
+    extract_text,
+)
+from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+from searx import redisdb
+from searx.enginelib.traits import EngineTraits
+from searx.exceptions import SearxEngineAPIException
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+about = {
+    "website": 'https://lite.duckduckgo.com/lite/',
+    "wikidata_id": 'Q12805',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+send_accept_language_header = True
+"""DuckDuckGo-Lite tries to guess user's prefered language from the HTTP
+``Accept-Language``.  Optional the user can select a region filter (but not a
+language).
+"""
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+time_range_support = True
+safesearch = True  # user can't select but the results are filtered
+
+url = 'https://lite.duckduckgo.com/lite/'
+# url_ping = 'https://duckduckgo.com/t/sl_l'
+
+time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
+form_data = {'v': 'l', 'api': 'd.js', 'o': 'json'}
+
+
+
[docs]def cache_vqd(query, value):
+    """Caches a ``vqd`` value from a query.
+
+    The vqd value depends on the query string and is needed for the follow up
+    pages or the images loaded by a XMLHttpRequest:
+
+    - DuckDuckGo Web: `https://links.duckduckgo.com/d.js?q=...&vqd=...`
+    - DuckDuckGo Images: `https://duckduckgo.com/i.js??q=...&vqd=...`
+
+    """
+    c = redisdb.client()
+    if c:
+        logger.debug("cache vqd value: %s", value)
+        key = 'SearXNG_ddg_vqd' + redislib.secret_hash(query)
+        c.set(key, value, ex=600)

+
+
+
[docs]def get_vqd(query, headers):
+    """Returns the ``vqd`` that fits to the *query*.  If there is no ``vqd`` cached
+    (:py:obj:`cache_vqd`) the query is sent to DDG to get a vqd value from the
+    response.
+
+    """
+    value = None
+    c = redisdb.client()
+    if c:
+        key = 'SearXNG_ddg_vqd' + redislib.secret_hash(query)
+        value = c.get(key)
+        if value:
+            value = value.decode('utf-8')
+            logger.debug("re-use cached vqd value: %s", value)
+            return value
+
+    query_url = 'https://duckduckgo.com/?q={query}&atb=v290-5'.format(query=urlencode({'q': query}))
+    res = get(query_url, headers=headers)
+    content = res.text  # type: ignore
+    if content.find('vqd=\"') == -1:
+        raise SearxEngineAPIException('Request failed')
+    value = content[content.find('vqd=\"') + 5 :]
+    value = value[: value.find('\'')]
+    logger.debug("new vqd value: %s", value)
+    cache_vqd(query, value)
+    return value

+
+
+
[docs]def get_ddg_lang(eng_traits: EngineTraits, sxng_locale, default='en_US'):
+    """Get DuckDuckGo's language identifier from SearXNG's locale.
+
+    DuckDuckGo defines its lanaguages by region codes (see
+    :py:obj:`fetch_traits`).
+
+    To get region and language of a DDG service use:
+
+    .. code: python
+
+       eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
+       eng_lang = get_ddg_lang(traits, params['searxng_locale'])
+
+    It might confuse, but the ``l`` value of the cookie is what SearXNG calls
+    the *region*:
+
+    .. code:: python
+
+        # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
+        params['cookies']['ad'] = eng_lang
+        params['cookies']['ah'] = eng_region
+        params['cookies']['l'] = eng_region
+
+    .. hint::
+
+       `DDG-lite <https://lite.duckduckgo.com/lite>`__ does not offer a language
+       selection to the user, only a region can be selected by the user
+       (``eng_region`` from the example above).  DDG-lite stores the selected
+       region in a cookie::
+
+         params['cookies']['kl'] = eng_region  # 'ar-es'
+
+    """
+    return eng_traits.custom['lang_region'].get(  # type: ignore
+        sxng_locale, eng_traits.get_language(sxng_locale, default)
+    )

+
+
+ddg_reg_map = {
+    'tw-tzh': 'zh_TW',
+    'hk-tzh': 'zh_HK',
+    'ct-ca': 'skip',  # ct-ca and es-ca both map to ca_ES
+    'es-ca': 'ca_ES',
+    'id-en': 'id_ID',
+    'no-no': 'nb_NO',
+    'jp-jp': 'ja_JP',
+    'kr-kr': 'ko_KR',
+    'xa-ar': 'ar_SA',
+    'sl-sl': 'sl_SI',
+    'th-en': 'th_TH',
+    'vn-en': 'vi_VN',
+}
+
+ddg_lang_map = {
+    # use ar --> ar_EG (Egypt's arabic)
+    "ar_DZ": 'lang_region',
+    "ar_JO": 'lang_region',
+    "ar_SA": 'lang_region',
+    # use bn --> bn_BD
+    'bn_IN': 'lang_region',
+    # use de --> de_DE
+    'de_CH': 'lang_region',
+    # use en --> en_US,
+    'en_AU': 'lang_region',
+    'en_CA': 'lang_region',
+    'en_GB': 'lang_region',
+    # Esperanto
+    'eo_XX': 'eo',
+    # use es --> es_ES,
+    'es_AR': 'lang_region',
+    'es_CL': 'lang_region',
+    'es_CO': 'lang_region',
+    'es_CR': 'lang_region',
+    'es_EC': 'lang_region',
+    'es_MX': 'lang_region',
+    'es_PE': 'lang_region',
+    'es_UY': 'lang_region',
+    'es_VE': 'lang_region',
+    # use fr --> rf_FR
+    'fr_CA': 'lang_region',
+    'fr_CH': 'lang_region',
+    'fr_BE': 'lang_region',
+    # use nl --> nl_NL
+    'nl_BE': 'lang_region',
+    # use pt --> pt_PT
+    'pt_BR': 'lang_region',
+    # skip these languages
+    'od_IN': 'skip',
+    'io_XX': 'skip',
+    'tokipona_XX': 'skip',
+}
+
+
+def request(query, params):
+
+    # quote ddg bangs
+    query_parts = []
+    # for val in re.split(r'(\s+)', query):
+    for val in re.split(r'(\s+)', query):
+        if not val.strip():
+            continue
+        if val.startswith('!') and external_bang.get_node(external_bang.EXTERNAL_BANGS, val[1:]):
+            val = f"'{val}'"
+        query_parts.append(val)
+    query = ' '.join(query_parts)
+
+    eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
+    # eng_lang = get_ddg_lang(traits, params['searxng_locale'])
+
+    params['url'] = url
+    params['method'] = 'POST'
+    params['data']['q'] = query
+
+    # The API is not documented, so we do some reverse engineering and emulate
+    # what https://lite.duckduckgo.com/lite/ does when you press "next Page"
+    # link again and again ..
+
+    params['headers']['Content-Type'] = 'application/x-www-form-urlencoded'
+    params['headers']['Referer'] = 'https://google.com/'
+
+    # initial page does not have an offset
+    if params['pageno'] == 2:
+        # second page does have an offset of 30
+        offset = (params['pageno'] - 1) * 30
+        params['data']['s'] = offset
+        params['data']['dc'] = offset + 1
+
+    elif params['pageno'] > 2:
+        # third and following pages do have an offset of 30 + n*50
+        offset = 30 + (params['pageno'] - 2) * 50
+        params['data']['s'] = offset
+        params['data']['dc'] = offset + 1
+
+    # request needs a vqd argument
+    params['data']['vqd'] = get_vqd(query, params["headers"])
+
+    # initial page does not have additional data in the input form
+    if params['pageno'] > 1:
+
+        params['data']['o'] = form_data.get('o', 'json')
+        params['data']['api'] = form_data.get('api', 'd.js')
+        params['data']['nextParams'] = form_data.get('nextParams', '')
+        params['data']['v'] = form_data.get('v', 'l')
+
+    params['data']['kl'] = eng_region
+    params['cookies']['kl'] = eng_region
+
+    params['data']['df'] = ''
+    if params['time_range'] in time_range_dict:
+        params['data']['df'] = time_range_dict[params['time_range']]
+        params['cookies']['df'] = time_range_dict[params['time_range']]
+
+    logger.debug("param data: %s", params['data'])
+    logger.debug("param cookies: %s", params['cookies'])
+    return params
+
+
+def response(resp):
+
+    if resp.status_code == 303:
+        return []
+
+    results = []
+    doc = lxml.html.fromstring(resp.text)
+
+    result_table = eval_xpath(doc, '//html/body/form/div[@class="filters"]/table')
+
+    if len(result_table) == 2:
+        # some locales (at least China) does not have a "next page" button and
+        # the layout of the HTML tables is different.
+        result_table = result_table[1]
+    elif not len(result_table) >= 3:
+        # no more results
+        return []
+    else:
+        result_table = result_table[2]
+        # update form data from response
+        form = eval_xpath(doc, '//html/body/form/div[@class="filters"]/table//input/..')
+        if len(form):
+
+            form = form[0]
+            form_data['v'] = eval_xpath(form, '//input[@name="v"]/@value')[0]
+            form_data['api'] = eval_xpath(form, '//input[@name="api"]/@value')[0]
+            form_data['o'] = eval_xpath(form, '//input[@name="o"]/@value')[0]
+            logger.debug('form_data: %s', form_data)
+
+            value = eval_xpath(form, '//input[@name="vqd"]/@value')[0]
+            query = resp.search_params['data']['q']
+            cache_vqd(query, value)
+
+    tr_rows = eval_xpath(result_table, './/tr')
+    # In the last <tr> is the form of the 'previous/next page' links
+    tr_rows = tr_rows[:-1]
+
+    len_tr_rows = len(tr_rows)
+    offset = 0
+
+    while len_tr_rows >= offset + 4:
+
+        # assemble table rows we need to scrap
+        tr_title = tr_rows[offset]
+        tr_content = tr_rows[offset + 1]
+        offset += 4
+
+        # ignore sponsored Adds <tr class="result-sponsored">
+        if tr_content.get('class') == 'result-sponsored':
+            continue
+
+        a_tag = eval_xpath_getindex(tr_title, './/td//a[@class="result-link"]', 0, None)
+        if a_tag is None:
+            continue
+
+        td_content = eval_xpath_getindex(tr_content, './/td[@class="result-snippet"]', 0, None)
+        if td_content is None:
+            continue
+
+        results.append(
+            {
+                'title': a_tag.text_content(),
+                'content': extract_text(td_content),
+                'url': a_tag.get('href'),
+            }
+        )
+
+    return results
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages & regions from DuckDuckGo.
+
+    SearXNG's ``all`` locale maps DuckDuckGo's "Alle regions" (``wt-wt``).
+    DuckDuckGo's language "Browsers prefered language" (``wt_WT``) makes no
+    sense in a SearXNG request since SearXNG's ``all`` will not add a
+    ``Accept-Language`` HTTP header.  The value in ``engine_traits.all_locale``
+    is ``wt-wt`` (the region).
+
+    Beside regions DuckDuckGo also defines its lanaguages by region codes.  By
+    example these are the english languages in DuckDuckGo:
+
+    - en_US
+    - en_AU
+    - en_CA
+    - en_GB
+
+    The function :py:obj:`get_ddg_lang` evaluates DuckDuckGo's language from
+    SearXNG's locale.
+
+    """
+    # pylint: disable=too-many-branches, too-many-statements
+    # fetch regions
+
+    engine_traits.all_locale = 'wt-wt'
+
+    # updated from u588 to u661 / should be updated automatically?
+    resp = get('https://duckduckgo.com/util/u661.js')
+
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from DuckDuckGo is not OK.")
+
+    pos = resp.text.find('regions:{') + 8  # type: ignore
+    js_code = resp.text[pos:]  # type: ignore
+    pos = js_code.find('}') + 1
+    regions = json.loads(js_code[:pos])
+
+    for eng_tag, name in regions.items():
+
+        if eng_tag == 'wt-wt':
+            engine_traits.all_locale = 'wt-wt'
+            continue
+
+        region = ddg_reg_map.get(eng_tag)
+        if region == 'skip':
+            continue
+
+        if not region:
+            eng_territory, eng_lang = eng_tag.split('-')
+            region = eng_lang + '_' + eng_territory.upper()
+
+        try:
+            sxng_tag = locales.region_tag(babel.Locale.parse(region))
+        except babel.UnknownLocaleError:
+            print("ERROR: %s (%s) -> %s is unknown by babel" % (name, eng_tag, region))
+            continue
+
+        conflict = engine_traits.regions.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.regions[sxng_tag] = eng_tag
+
+    # fetch languages
+
+    engine_traits.custom['lang_region'] = {}
+
+    pos = resp.text.find('languages:{') + 10  # type: ignore
+    js_code = resp.text[pos:]  # type: ignore
+    pos = js_code.find('}') + 1
+    js_code = '{"' + js_code[1:pos].replace(':', '":').replace(',', ',"')
+    languages = json.loads(js_code)
+
+    for eng_lang, name in languages.items():
+
+        if eng_lang == 'wt_WT':
+            continue
+
+        babel_tag = ddg_lang_map.get(eng_lang, eng_lang)
+        if babel_tag == 'skip':
+            continue
+
+        try:
+
+            if babel_tag == 'lang_region':
+                sxng_tag = locales.region_tag(babel.Locale.parse(eng_lang))
+                engine_traits.custom['lang_region'][sxng_tag] = eng_lang
+                continue
+
+            sxng_tag = locales.language_tag(babel.Locale.parse(babel_tag))
+
+        except babel.UnknownLocaleError:
+            print("ERROR: language %s (%s) is unknown by babel" % (name, eng_lang))
+            continue
+
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_lang))
+            continue
+        engine_traits.languages[sxng_tag] = eng_lang

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/duckduckgo_definitions.html b/_modules/searx/engines/duckduckgo_definitions.html new file mode 100644 index 000000000..107dbc749 --- /dev/null +++ b/_modules/searx/engines/duckduckgo_definitions.html @@ -0,0 +1,367 @@ + + + + + + + + searx.engines.duckduckgo_definitions — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.duckduckgo_definitions

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""
+DuckDuckGo Instant Answer API
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `DDG-API <https://duckduckgo.com/api>`__ is no longer documented but from
+reverse engineering we can see that some services (e.g. instant answers) still
+in use from the DDG search engine.
+
+As far we can say the *instant answers* API does not support languages, or at
+least we could not find out how language support should work.  It seems that
+most of the features are based on English terms.
+
+"""
+
+from typing import TYPE_CHECKING
+
+from urllib.parse import urlencode, urlparse, urljoin
+from lxml import html
+
+from searx.data import WIKIDATA_UNITS
+from searx.utils import extract_text, html_to_text, get_string_replaces_function
+from searx.external_urls import get_external_url, get_earth_coordinates_url, area_to_osm_zoom
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+# about
+about = {
+    "website": 'https://duckduckgo.com/',
+    "wikidata_id": 'Q12805',
+    "official_api_documentation": 'https://duckduckgo.com/api',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+send_accept_language_header = True
+
+URL = 'https://api.duckduckgo.com/' + '?{query}&format=json&pretty=0&no_redirect=1&d=1'
+
+WIKIDATA_PREFIX = ['http://www.wikidata.org/entity/', 'https://www.wikidata.org/entity/']
+
+replace_http_by_https = get_string_replaces_function({'http:': 'https:'})
+
+
+
[docs]def is_broken_text(text):
+    """duckduckgo may return something like ``<a href="xxxx">http://somewhere Related website<a/>``
+
+    The href URL is broken, the "Related website" may contains some HTML.
+
+    The best solution seems to ignore these results.
+    """
+    return text.startswith('http') and ' ' in text

+
+
+def result_to_text(text, htmlResult):
+    # TODO : remove result ending with "Meaning" or "Category"  # pylint: disable=fixme
+    result = None
+    dom = html.fromstring(htmlResult)
+    a = dom.xpath('//a')
+    if len(a) >= 1:
+        result = extract_text(a[0])
+    else:
+        result = text
+    if not is_broken_text(result):
+        return result
+    return None
+
+
+def request(query, params):
+    params['url'] = URL.format(query=urlencode({'q': query}))
+    return params
+
+
+def response(resp):
+    # pylint: disable=too-many-locals, too-many-branches, too-many-statements
+    results = []
+
+    search_res = resp.json()
+
+    # search_res.get('Entity') possible values (not exhaustive) :
+    # * continent / country / department / location / waterfall
+    # * actor / musician / artist
+    # * book / performing art / film / television  / media franchise / concert tour / playwright
+    # * prepared food
+    # * website / software / os / programming language / file format / software engineer
+    # * company
+
+    content = ''
+    heading = search_res.get('Heading', '')
+    attributes = []
+    urls = []
+    infobox_id = None
+    relatedTopics = []
+
+    # add answer if there is one
+    answer = search_res.get('Answer', '')
+    if answer:
+        logger.debug('AnswerType="%s" Answer="%s"', search_res.get('AnswerType'), answer)
+        if search_res.get('AnswerType') not in ['calc', 'ip']:
+            results.append({'answer': html_to_text(answer)})
+
+    # add infobox
+    if 'Definition' in search_res:
+        content = content + search_res.get('Definition', '')
+
+    if 'Abstract' in search_res:
+        content = content + search_res.get('Abstract', '')
+
+    # image
+    image = search_res.get('Image')
+    image = None if image == '' else image
+    if image is not None and urlparse(image).netloc == '':
+        image = urljoin('https://duckduckgo.com', image)
+
+    # urls
+    # Official website, Wikipedia page
+    for ddg_result in search_res.get('Results', []):
+        firstURL = ddg_result.get('FirstURL')
+        text = ddg_result.get('Text')
+        if firstURL is not None and text is not None:
+            urls.append({'title': text, 'url': firstURL})
+            results.append({'title': heading, 'url': firstURL})
+
+    # related topics
+    for ddg_result in search_res.get('RelatedTopics', []):
+        if 'FirstURL' in ddg_result:
+            firstURL = ddg_result.get('FirstURL')
+            text = ddg_result.get('Text')
+            if not is_broken_text(text):
+                suggestion = result_to_text(text, ddg_result.get('Result'))
+                if suggestion != heading and suggestion is not None:
+                    results.append({'suggestion': suggestion})
+        elif 'Topics' in ddg_result:
+            suggestions = []
+            relatedTopics.append({'name': ddg_result.get('Name', ''), 'suggestions': suggestions})
+            for topic_result in ddg_result.get('Topics', []):
+                suggestion = result_to_text(topic_result.get('Text'), topic_result.get('Result'))
+                if suggestion != heading and suggestion is not None:
+                    suggestions.append(suggestion)
+
+    # abstract
+    abstractURL = search_res.get('AbstractURL', '')
+    if abstractURL != '':
+        # add as result ? problem always in english
+        infobox_id = abstractURL
+        urls.append({'title': search_res.get('AbstractSource'), 'url': abstractURL, 'official': True})
+        results.append({'url': abstractURL, 'title': heading})
+
+    # definition
+    definitionURL = search_res.get('DefinitionURL', '')
+    if definitionURL != '':
+        # add as result ? as answer ? problem always in english
+        infobox_id = definitionURL
+        urls.append({'title': search_res.get('DefinitionSource'), 'url': definitionURL})
+
+    # to merge with wikidata's infobox
+    if infobox_id:
+        infobox_id = replace_http_by_https(infobox_id)
+
+    # attributes
+    # some will be converted to urls
+    if 'Infobox' in search_res:
+        infobox = search_res.get('Infobox')
+        if 'content' in infobox:
+            osm_zoom = 17
+            coordinates = None
+            for info in infobox.get('content'):
+                data_type = info.get('data_type')
+                data_label = info.get('label')
+                data_value = info.get('value')
+
+                # Workaround: ddg may return a double quote
+                if data_value == '""':
+                    continue
+
+                # Is it an external URL ?
+                # * imdb_id / facebook_profile / youtube_channel / youtube_video / twitter_profile
+                # * instagram_profile / rotten_tomatoes / spotify_artist_id / itunes_artist_id / soundcloud_id
+                # * netflix_id
+                external_url = get_external_url(data_type, data_value)
+                if external_url is not None:
+                    urls.append({'title': data_label, 'url': external_url})
+                elif data_type in ['instance', 'wiki_maps_trigger', 'google_play_artist_id']:
+                    # ignore instance: Wikidata value from "Instance Of" (Qxxxx)
+                    # ignore wiki_maps_trigger: reference to a javascript
+                    # ignore google_play_artist_id: service shutdown
+                    pass
+                elif data_type == 'string' and data_label == 'Website':
+                    # There is already an URL for the website
+                    pass
+                elif data_type == 'area':
+                    attributes.append({'label': data_label, 'value': area_to_str(data_value), 'entity': 'P2046'})
+                    osm_zoom = area_to_osm_zoom(data_value.get('amount'))
+                elif data_type == 'coordinates':
+                    if data_value.get('globe') == 'http://www.wikidata.org/entity/Q2':
+                        # coordinate on Earth
+                        # get the zoom information from the area
+                        coordinates = info
+                    else:
+                        # coordinate NOT on Earth
+                        attributes.append({'label': data_label, 'value': data_value, 'entity': 'P625'})
+                elif data_type == 'string':
+                    attributes.append({'label': data_label, 'value': data_value})
+
+            if coordinates:
+                data_label = coordinates.get('label')
+                data_value = coordinates.get('value')
+                latitude = data_value.get('latitude')
+                longitude = data_value.get('longitude')
+                url = get_earth_coordinates_url(latitude, longitude, osm_zoom)
+                urls.append({'title': 'OpenStreetMap', 'url': url, 'entity': 'P625'})
+
+    if len(heading) > 0:
+        # TODO get infobox.meta.value where .label='article_title'    # pylint: disable=fixme
+        if image is None and len(attributes) == 0 and len(urls) == 1 and len(relatedTopics) == 0 and len(content) == 0:
+            results.append({'url': urls[0]['url'], 'title': heading, 'content': content})
+        else:
+            results.append(
+                {
+                    'infobox': heading,
+                    'id': infobox_id,
+                    'content': content,
+                    'img_src': image,
+                    'attributes': attributes,
+                    'urls': urls,
+                    'relatedTopics': relatedTopics,
+                }
+            )
+
+    return results
+
+
+def unit_to_str(unit):
+    for prefix in WIKIDATA_PREFIX:
+        if unit.startswith(prefix):
+            wikidata_entity = unit[len(prefix) :]
+            return WIKIDATA_UNITS.get(wikidata_entity, unit)
+    return unit
+
+
+
[docs]def area_to_str(area):
+    """parse ``{'unit': 'https://www.wikidata.org/entity/Q712226', 'amount': '+20.99'}``"""
+    unit = unit_to_str(area.get('unit'))
+    if unit is not None:
+        try:
+            amount = float(area.get('amount'))
+            return '{} {}'.format(amount, unit)
+        except ValueError:
+            pass
+    return '{} {}'.format(area.get('amount', ''), area.get('unit', ''))

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/google.html b/_modules/searx/engines/google.html new file mode 100644 index 000000000..671c1de09 --- /dev/null +++ b/_modules/searx/engines/google.html @@ -0,0 +1,605 @@ + + + + + + + + searx.engines.google — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.google

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Google WEB engine.  Some of this
+implementations (manly the :py:obj:`get_google_info`) are shared by other
+engines:
+
+- :ref:`google images engine`
+- :ref:`google news engine`
+- :ref:`google videos engine`
+- :ref:`google scholar engine`
+- :ref:`google autocomplete`
+
+"""
+
+from typing import TYPE_CHECKING
+
+import re
+from urllib.parse import urlencode
+from lxml import html
+import babel
+import babel.core
+import babel.languages
+
+from searx.utils import extract_text, eval_xpath, eval_xpath_list, eval_xpath_getindex
+from searx.locales import language_tag, region_tag, get_offical_locales
+from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+from searx.exceptions import SearxEngineCaptchaException
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+
+# about
+about = {
+    "website": 'https://www.google.com',
+    "wikidata_id": 'Q9366',
+    "official_api_documentation": 'https://developers.google.com/custom-search/',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+time_range_support = True
+safesearch = True
+
+time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
+
+# Filter results. 0: None, 1: Moderate, 2: Strict
+filter_mapping = {0: 'off', 1: 'medium', 2: 'high'}
+
+# specific xpath variables
+# ------------------------
+
+results_xpath = './/div[contains(@jscontroller, "SC7lYd")]'
+title_xpath = './/a/h3[1]'
+href_xpath = './/a[h3]/@href'
+content_xpath = './/div[@data-sncf]'
+
+# Suggestions are links placed in a *card-section*, we extract only the text
+# from the links not the links itself.
+suggestion_xpath = '//div[contains(@class, "EIaa9b")]//a'
+
+# UI_ASYNC = 'use_ac:true,_fmt:html' # returns a HTTP 500 when user search for
+#                                    # celebrities like '!google natasha allegri'
+#                                    # or '!google chris evans'
+UI_ASYNC = 'use_ac:true,_fmt:prog'
+"""Format of the response from UI's async request."""
+
+
+
[docs]def get_google_info(params, eng_traits):
+    """Composing various (language) properties for the google engines (:ref:`google
+    API`).
+
+    This function is called by the various google engines (:ref:`google web
+    engine`, :ref:`google images engine`, :ref:`google news engine` and
+    :ref:`google videos engine`).
+
+    :param dict param: Request parameters of the engine.  At least
+        a ``searxng_locale`` key should be in the dictionary.
+
+    :param eng_traits: Engine's traits fetched from google preferences
+        (:py:obj:`searx.enginelib.traits.EngineTraits`)
+
+    :rtype: dict
+    :returns:
+        Py-Dictionary with the key/value pairs:
+
+        language:
+            The language code that is used by google (e.g. ``lang_en`` or
+            ``lang_zh-TW``)
+
+        country:
+            The country code that is used by google (e.g. ``US`` or ``TW``)
+
+        locale:
+            A instance of :py:obj:`babel.core.Locale` build from the
+            ``searxng_locale`` value.
+
+        subdomain:
+            Google subdomain :py:obj:`google_domains` that fits to the country
+            code.
+
+        params:
+            Py-Dictionary with additional request arguments (can be passed to
+            :py:func:`urllib.parse.urlencode`).
+
+            - ``hl`` parameter: specifies the interface language of user interface.
+            - ``lr`` parameter: restricts search results to documents written in
+              a particular language.
+            - ``cr`` parameter: restricts search results to documents
+              originating in a particular country.
+            - ``ie`` parameter: sets the character encoding scheme that should
+              be used to interpret the query string ('utf8').
+            - ``oe`` parameter: sets the character encoding scheme that should
+              be used to decode the XML result ('utf8').
+
+        headers:
+            Py-Dictionary with additional HTTP headers (can be passed to
+            request's headers)
+
+            - ``Accept: '*/*``
+
+    """
+
+    ret_val = {
+        'language': None,
+        'country': None,
+        'subdomain': None,
+        'params': {},
+        'headers': {},
+        'cookies': {},
+        'locale': None,
+    }
+
+    sxng_locale = params.get('searxng_locale', 'all')
+    try:
+        locale = babel.Locale.parse(sxng_locale, sep='-')
+    except babel.core.UnknownLocaleError:
+        locale = None
+
+    eng_lang = eng_traits.get_language(sxng_locale, 'lang_en')
+    lang_code = eng_lang.split('_')[-1]  # lang_zh-TW --> zh-TW / lang_en --> en
+    country = eng_traits.get_region(sxng_locale, eng_traits.all_locale)
+
+    # Test zh_hans & zh_hant --> in the topmost links in the result list of list
+    # TW and HK you should a find wiktionary.org zh_hant link.  In the result
+    # list of zh-CN should not be no hant link instead you should find
+    # zh.m.wikipedia.org/zh somewhere in the top.
+
+    # '!go 日 :zh-TW' --> https://zh.m.wiktionary.org/zh-hant/%E6%97%A5
+    # '!go 日 :zh-CN' --> https://zh.m.wikipedia.org/zh/%E6%97%A5
+
+    ret_val['language'] = eng_lang
+    ret_val['country'] = country
+    ret_val['locale'] = locale
+    ret_val['subdomain'] = eng_traits.custom['supported_domains'].get(country.upper(), 'www.google.com')
+
+    # hl parameter:
+    #   The hl parameter specifies the interface language (host language) of
+    #   your user interface. To improve the performance and the quality of your
+    #   search results, you are strongly encouraged to set this parameter
+    #   explicitly.
+    #   https://developers.google.com/custom-search/docs/xml_results#hlsp
+    # The Interface Language:
+    #   https://developers.google.com/custom-search/docs/xml_results_appendices#interfaceLanguages
+
+    # https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817
+    ret_val['params']['hl'] = f'{lang_code}-{country}'
+
+    # lr parameter:
+    #   The lr (language restrict) parameter restricts search results to
+    #   documents written in a particular language.
+    #   https://developers.google.com/custom-search/docs/xml_results#lrsp
+    #   Language Collection Values:
+    #   https://developers.google.com/custom-search/docs/xml_results_appendices#languageCollections
+    #
+    # To select 'all' languages an empty 'lr' value is used.
+    #
+    # Different to other google services, Google Schloar supports to select more
+    # than one language. The languages are seperated by a pipe '|' (logical OR).
+    # By example: &lr=lang_zh-TW%7Clang_de selects articles written in
+    # traditional chinese OR german language.
+
+    ret_val['params']['lr'] = eng_lang
+    if sxng_locale == 'all':
+        ret_val['params']['lr'] = ''
+
+    # cr parameter:
+    #   The cr parameter restricts search results to documents originating in a
+    #   particular country.
+    #   https://developers.google.com/custom-search/docs/xml_results#crsp
+
+    ret_val['params']['cr'] = 'country' + country
+    if sxng_locale == 'all':
+        ret_val['params']['cr'] = ''
+
+    # gl parameter: (mandatory by Geeogle News)
+    #   The gl parameter value is a two-letter country code. For WebSearch
+    #   results, the gl parameter boosts search results whose country of origin
+    #   matches the parameter value. See the Country Codes section for a list of
+    #   valid values.
+    #   Specifying a gl parameter value in WebSearch requests should improve the
+    #   relevance of results. This is particularly true for international
+    #   customers and, even more specifically, for customers in English-speaking
+    #   countries other than the United States.
+    #   https://developers.google.com/custom-search/docs/xml_results#glsp
+
+    # https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
+    # ret_val['params']['gl'] = country
+
+    # ie parameter:
+    #   The ie parameter sets the character encoding scheme that should be used
+    #   to interpret the query string. The default ie value is latin1.
+    #   https://developers.google.com/custom-search/docs/xml_results#iesp
+
+    ret_val['params']['ie'] = 'utf8'
+
+    # oe parameter:
+    #   The oe parameter sets the character encoding scheme that should be used
+    #   to decode the XML result. The default oe value is latin1.
+    #   https://developers.google.com/custom-search/docs/xml_results#oesp
+
+    ret_val['params']['oe'] = 'utf8'
+
+    # num parameter:
+    #   The num parameter identifies the number of search results to return.
+    #   The default num value is 10, and the maximum value is 20. If you request
+    #   more than 20 results, only 20 results will be returned.
+    #   https://developers.google.com/custom-search/docs/xml_results#numsp
+
+    # HINT: seems to have no effect (tested in google WEB & Images)
+    # ret_val['params']['num'] = 20
+
+    # HTTP headers
+
+    ret_val['headers']['Accept'] = '*/*'
+
+    # Cookies
+
+    # - https://github.com/searxng/searxng/pull/1679#issuecomment-1235432746
+    # - https://github.com/searxng/searxng/issues/1555
+    ret_val['cookies']['CONSENT'] = "YES+"
+
+    return ret_val

+
+
+def detect_google_sorry(resp):
+    if resp.url.host == 'sorry.google.com' or resp.url.path.startswith('/sorry'):
+        raise SearxEngineCaptchaException()
+
+
+
[docs]def request(query, params):
+    """Google search request"""
+    # pylint: disable=line-too-long
+    offset = (params['pageno'] - 1) * 10
+    google_info = get_google_info(params, traits)
+
+    # https://www.google.de/search?q=corona&hl=de&lr=lang_de&start=0&tbs=qdr%3Ad&safe=medium
+    query_url = (
+        'https://'
+        + google_info['subdomain']
+        + '/search'
+        + "?"
+        + urlencode(
+            {
+                'q': query,
+                **google_info['params'],
+                'filter': '0',
+                'start': offset,
+                # 'vet': '12ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0QxK8CegQIARAC..i',
+                # 'ved': '2ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0Q_skCegQIARAG',
+                # 'cs' : 1,
+                # 'sa': 'N',
+                # 'yv': 3,
+                # 'prmd': 'vin',
+                # 'ei': 'GASaY6TxOcy_xc8PtYeY6AE',
+                # 'sa': 'N',
+                # 'sstk': 'AcOHfVkD7sWCSAheZi-0tx_09XDO55gTWY0JNq3_V26cNN-c8lfD45aZYPI8s_Bqp8s57AHz5pxchDtAGCA_cikAWSjy9kw3kgg'
+                # formally known as use_mobile_ui
+                'asearch': 'arc',
+                'async': UI_ASYNC,
+            }
+        )
+    )
+
+    if params['time_range'] in time_range_dict:
+        query_url += '&' + urlencode({'tbs': 'qdr:' + time_range_dict[params['time_range']]})
+    if params['safesearch']:
+        query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
+    params['url'] = query_url
+
+    params['cookies'] = google_info['cookies']
+    params['headers'].update(google_info['headers'])
+    return params

+
+
+# =26;[3,"dimg_ZNMiZPCqE4apxc8P3a2tuAQ_137"]a87;data:image/jpeg;base64,/9j/4AAQSkZJRgABA
+# ...6T+9Nl4cnD+gr9OK8I56/tX3l86nWYw//2Q==26;
+RE_DATA_IMAGE = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*);')
+
+
+def _parse_data_images(dom):
+    data_image_map = {}
+    for img_id, data_image in RE_DATA_IMAGE.findall(dom.text_content()):
+        end_pos = data_image.rfind('=')
+        if end_pos > 0:
+            data_image = data_image[: end_pos + 1]
+        data_image_map[img_id] = data_image
+    logger.debug('data:image objects --> %s', list(data_image_map.keys()))
+    return data_image_map
+
+
+
[docs]def response(resp):
+    """Get response from google's search request"""
+    # pylint: disable=too-many-branches, too-many-statements
+    detect_google_sorry(resp)
+
+    results = []
+
+    # convert the text to dom
+    dom = html.fromstring(resp.text)
+    data_image_map = _parse_data_images(dom)
+
+    # results --> answer
+    answer_list = eval_xpath(dom, '//div[contains(@class, "LGOjhe")]')
+    if answer_list:
+        answer_list = [_.xpath("normalize-space()") for _ in answer_list]
+        results.append({'answer': ' '.join(answer_list)})
+    else:
+        logger.debug("did not find 'answer'")
+
+    # parse results
+
+    for result in eval_xpath_list(dom, results_xpath):  # pylint: disable=too-many-nested-blocks
+
+        try:
+            title_tag = eval_xpath_getindex(result, title_xpath, 0, default=None)
+            if title_tag is None:
+                # this not one of the common google results *section*
+                logger.debug('ignoring item from the result_xpath list: missing title')
+                continue
+            title = extract_text(title_tag)
+
+            url = eval_xpath_getindex(result, href_xpath, 0, None)
+            if url is None:
+                logger.debug('ignoring item from the result_xpath list: missing url of title "%s"', title)
+                continue
+
+            content_nodes = eval_xpath(result, content_xpath)
+            content = extract_text(content_nodes)
+
+            if not content:
+                logger.debug('ignoring item from the result_xpath list: missing content of title "%s"', title)
+                continue
+
+            img_src = content_nodes[0].xpath('.//img/@src')
+            if img_src:
+                img_src = img_src[0]
+                if img_src.startswith('data:image'):
+                    img_id = content_nodes[0].xpath('.//img/@id')
+                    if img_id:
+                        img_src = data_image_map.get(img_id[0])
+            else:
+                img_src = None
+
+            results.append({'url': url, 'title': title, 'content': content, 'img_src': img_src})
+
+        except Exception as e:  # pylint: disable=broad-except
+            logger.error(e, exc_info=True)
+            continue
+
+    # parse suggestion
+    for suggestion in eval_xpath_list(dom, suggestion_xpath):
+        # append suggestion
+        results.append({'suggestion': extract_text(suggestion)})
+
+    # return results
+    return results

+
+
+# get supported languages from their site
+
+
+skip_countries = [
+    # official language of google-country not in google-languages
+    'AL',  # Albanien (sq)
+    'AZ',  # Aserbaidschan  (az)
+    'BD',  # Bangladesch (bn)
+    'BN',  # Brunei Darussalam (ms)
+    'BT',  # Bhutan (dz)
+    'ET',  # Äthiopien (am)
+    'GE',  # Georgien (ka, os)
+    'GL',  # Grönland (kl)
+    'KH',  # Kambodscha (km)
+    'LA',  # Laos (lo)
+    'LK',  # Sri Lanka (si, ta)
+    'ME',  # Montenegro (sr)
+    'MK',  # Nordmazedonien (mk, sq)
+    'MM',  # Myanmar (my)
+    'MN',  # Mongolei (mn)
+    'MV',  # Malediven (dv) // dv_MV is unknown by babel
+    'MY',  # Malaysia (ms)
+    'NP',  # Nepal (ne)
+    'TJ',  # Tadschikistan (tg)
+    'TM',  # Turkmenistan (tk)
+    'UZ',  # Usbekistan (uz)
+]
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits, add_domains: bool = True):
+    """Fetch languages from Google."""
+    # pylint: disable=import-outside-toplevel, too-many-branches
+
+    engine_traits.custom['supported_domains'] = {}
+
+    resp = get('https://www.google.com/preferences')
+    if not resp.ok:  # type: ignore
+        raise RuntimeError("Response from Google's preferences is not OK.")
+
+    dom = html.fromstring(resp.text)  # type: ignore
+
+    # supported language codes
+
+    lang_map = {'no': 'nb'}
+    for x in eval_xpath_list(dom, '//*[@id="langSec"]//input[@name="lr"]'):
+
+        eng_lang = x.get("value").split('_')[-1]
+        try:
+            locale = babel.Locale.parse(lang_map.get(eng_lang, eng_lang), sep='-')
+        except babel.UnknownLocaleError:
+            print("ERROR: %s -> %s is unknown by babel" % (x.get("data-name"), eng_lang))
+            continue
+        sxng_lang = language_tag(locale)
+
+        conflict = engine_traits.languages.get(sxng_lang)
+        if conflict:
+            if conflict != eng_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_lang, conflict, eng_lang))
+            continue
+        engine_traits.languages[sxng_lang] = 'lang_' + eng_lang
+
+    # alias languages
+    engine_traits.languages['zh'] = 'lang_zh-CN'
+
+    # supported region codes
+
+    for x in eval_xpath_list(dom, '//*[@name="region"]/..//input[@name="region"]'):
+        eng_country = x.get("value")
+
+        if eng_country in skip_countries:
+            continue
+        if eng_country == 'ZZ':
+            engine_traits.all_locale = 'ZZ'
+            continue
+
+        sxng_locales = get_offical_locales(eng_country, engine_traits.languages.keys(), regional=True)
+
+        if not sxng_locales:
+            print("ERROR: can't map from google country %s (%s) to a babel region." % (x.get('data-name'), eng_country))
+            continue
+
+        for sxng_locale in sxng_locales:
+            engine_traits.regions[region_tag(sxng_locale)] = eng_country
+
+    # alias regions
+    engine_traits.regions['zh-CN'] = 'HK'
+
+    # supported domains
+
+    if add_domains:
+        resp = get('https://www.google.com/supported_domains')
+        if not resp.ok:  # type: ignore
+            raise RuntimeError("Response from https://www.google.com/supported_domains is not OK.")
+
+        for domain in resp.text.split():  # type: ignore
+            domain = domain.strip()
+            if not domain or domain in [
+                '.google.com',
+            ]:
+                continue
+            region = domain.split('.')[-1].upper()
+            engine_traits.custom['supported_domains'][region] = 'www' + domain  # type: ignore
+            if region == 'HK':
+                # There is no google.cn, we use .com.hk for zh-CN
+                engine_traits.custom['supported_domains']['CN'] = 'www' + domain  # type: ignore

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/google_images.html b/_modules/searx/engines/google_images.html new file mode 100644 index 000000000..c55333c2c --- /dev/null +++ b/_modules/searx/engines/google_images.html @@ -0,0 +1,241 @@ + + + + + + + + searx.engines.google_images — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.google_images

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Google Images engine using the internal
+Google API used by the Google Go Android app.
+
+This internal API offer results in
+
+- JSON (``_fmt:json``)
+- Protobuf_ (``_fmt:pb``)
+- Protobuf_ compressed? (``_fmt:pc``)
+- HTML (``_fmt:html``)
+- Protobuf_ encoded in JSON (``_fmt:jspb``).
+
+.. _Protobuf: https://en.wikipedia.org/wiki/Protocol_Buffers
+"""
+
+from typing import TYPE_CHECKING
+
+from urllib.parse import urlencode
+from json import loads
+
+from searx.engines.google import fetch_traits  # pylint: disable=unused-import
+from searx.engines.google import (
+    get_google_info,
+    time_range_dict,
+    detect_google_sorry,
+)
+
+if TYPE_CHECKING:
+    import logging
+    from searx.enginelib.traits import EngineTraits
+
+    logger: logging.Logger
+    traits: EngineTraits
+
+
+# about
+about = {
+    "website": 'https://images.google.com',
+    "wikidata_id": 'Q521550',
+    "official_api_documentation": 'https://developers.google.com/custom-search',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# engine dependent config
+categories = ['images', 'web']
+paging = True
+time_range_support = True
+safesearch = True
+send_accept_language_header = True
+
+filter_mapping = {0: 'images', 1: 'active', 2: 'active'}
+
+
+
[docs]def request(query, params):
+    """Google-Image search request"""
+
+    google_info = get_google_info(params, traits)
+
+    query_url = (
+        'https://'
+        + google_info['subdomain']
+        + '/search'
+        + "?"
+        + urlencode(
+            {
+                'q': query,
+                'tbm': "isch",
+                **google_info['params'],
+                'asearch': 'isch',
+                'async': '_fmt:json,p:1,ijn:' + str(params['pageno']),
+            }
+        )
+    )
+
+    if params['time_range'] in time_range_dict:
+        query_url += '&' + urlencode({'tbs': 'qdr:' + time_range_dict[params['time_range']]})
+    if params['safesearch']:
+        query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
+    params['url'] = query_url
+
+    params['cookies'] = google_info['cookies']
+    params['headers'].update(google_info['headers'])
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from google's search request"""
+    results = []
+
+    detect_google_sorry(resp)
+
+    json_start = resp.text.find('{"ischj":')
+    json_data = loads(resp.text[json_start:])
+
+    for item in json_data["ischj"]["metadata"]:
+
+        result_item = {
+            'url': item["result"]["referrer_url"],
+            'title': item["result"]["page_title"],
+            'content': item["text_in_grid"]["snippet"],
+            'source': item["result"]["site_title"],
+            'img_format': f'{item["original_image"]["width"]} x {item["original_image"]["height"]}',
+            'img_src': item["original_image"]["url"],
+            'thumbnail_src': item["thumbnail"]["url"],
+            'template': 'images.html',
+        }
+
+        author = item["result"].get('iptc', {}).get('creator')
+        if author:
+            result_item['author'] = ', '.join(author)
+
+        copyright_notice = item["result"].get('iptc', {}).get('copyright_notice')
+        if copyright_notice:
+            result_item['source'] += ' | ' + copyright_notice
+
+        freshness_date = item["result"].get("freshness_date")
+        if freshness_date:
+            result_item['source'] += ' | ' + freshness_date
+
+        file_size = item.get('gsa', {}).get('file_size')
+        if file_size:
+            result_item['source'] += ' (%s)' % file_size
+
+        results.append(result_item)
+
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/google_news.html b/_modules/searx/engines/google_news.html new file mode 100644 index 000000000..8627e5c1a --- /dev/null +++ b/_modules/searx/engines/google_news.html @@ -0,0 +1,417 @@ + + + + + + + + searx.engines.google_news — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.google_news

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Google News engine.
+
+Google News has a different region handling compared to Google WEB.
+
+- the ``ceid`` argument has to be set (:py:obj:`ceid_list`)
+- the hl_ argument has to be set correctly (and different to Google WEB)
+- the gl_ argument is mandatory
+
+If one of this argument is not set correctly, the request is redirected to
+CONSENT dialog::
+
+  https://consent.google.com/m?continue=
+
+The google news API ignores some parameters from the common :ref:`google API`:
+
+- num_ : the number of search results is ignored / there is no paging all
+  results for a query term are in the first response.
+- save_ : is ignored / Google-News results are always *SafeSearch*
+
+.. _hl: https://developers.google.com/custom-search/docs/xml_results#hlsp
+.. _gl: https://developers.google.com/custom-search/docs/xml_results#glsp
+.. _num: https://developers.google.com/custom-search/docs/xml_results#numsp
+.. _save: https://developers.google.com/custom-search/docs/xml_results#safesp
+"""
+
+from typing import TYPE_CHECKING
+
+from urllib.parse import urlencode
+import base64
+from lxml import html
+import babel
+
+from searx import locales
+from searx.utils import (
+    eval_xpath,
+    eval_xpath_list,
+    eval_xpath_getindex,
+    extract_text,
+)
+
+from searx.engines.google import fetch_traits as _fetch_traits  # pylint: disable=unused-import
+from searx.engines.google import (
+    get_google_info,
+    detect_google_sorry,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://news.google.com',
+    "wikidata_id": 'Q12020',
+    "official_api_documentation": 'https://developers.google.com/custom-search',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['news']
+paging = False
+time_range_support = False
+
+# Google-News results are always *SafeSearch*. Option 'safesearch' is set to
+# False here, otherwise checker will report safesearch-errors::
+#
+#  safesearch : results are identitical for safesearch=0 and safesearch=2
+safesearch = True
+# send_accept_language_header = True
+
+
+
[docs]def request(query, params):
+    """Google-News search request"""
+
+    sxng_locale = params.get('searxng_locale', 'en-US')
+    ceid = locales.get_engine_locale(sxng_locale, traits.custom['ceid'], default='US:en')
+    google_info = get_google_info(params, traits)
+    google_info['subdomain'] = 'news.google.com'  # google news has only one domain
+
+    ceid_region, ceid_lang = ceid.split(':')
+    ceid_lang, ceid_suffix = (
+        ceid_lang.split('-')
+        + [
+            None,
+        ]
+    )[:2]
+
+    google_info['params']['hl'] = ceid_lang
+
+    if ceid_suffix and ceid_suffix not in ['Hans', 'Hant']:
+
+        if ceid_region.lower() == ceid_lang:
+            google_info['params']['hl'] = ceid_lang + '-' + ceid_region
+        else:
+            google_info['params']['hl'] = ceid_lang + '-' + ceid_suffix
+
+    elif ceid_region.lower() != ceid_lang:
+
+        if ceid_region in ['AT', 'BE', 'CH', 'IL', 'SA', 'IN', 'BD', 'PT']:
+            google_info['params']['hl'] = ceid_lang
+        else:
+            google_info['params']['hl'] = ceid_lang + '-' + ceid_region
+
+    google_info['params']['lr'] = 'lang_' + ceid_lang.split('-')[0]
+    google_info['params']['gl'] = ceid_region
+
+    query_url = (
+        'https://'
+        + google_info['subdomain']
+        + "/search?"
+        + urlencode(
+            {
+                'q': query,
+                **google_info['params'],
+            }
+        )
+        # ceid includes a ':' character which must not be urlencoded
+        + ('&ceid=%s' % ceid)
+    )
+
+    params['url'] = query_url
+    params['cookies'] = google_info['cookies']
+    params['headers'].update(google_info['headers'])
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from google's search request"""
+    results = []
+    detect_google_sorry(resp)
+
+    # convert the text to dom
+    dom = html.fromstring(resp.text)
+
+    for result in eval_xpath_list(dom, '//div[@class="xrnccd"]'):
+
+        # The first <a> tag in the <article> contains the link to the article
+        # The href attribute of the <a> tag is a google internal link, we have
+        # to decode
+
+        href = eval_xpath_getindex(result, './article/a/@href', 0)
+        href = href.split('?')[0]
+        href = href.split('/')[-1]
+        href = base64.urlsafe_b64decode(href + '====')
+        href = href[href.index(b'http') :].split(b'\xd2')[0]
+        href = href.decode()
+
+        title = extract_text(eval_xpath(result, './article/h3[1]'))
+
+        # The pub_date is mostly a string like 'yesertday', not a real
+        # timezone date or time.  Therefore we can't use publishedDate.
+        pub_date = extract_text(eval_xpath(result, './article//time'))
+        pub_origin = extract_text(eval_xpath(result, './article//a[@data-n-tid]'))
+
+        content = ' / '.join([x for x in [pub_origin, pub_date] if x])
+
+        # The image URL is located in a preceding sibling <img> tag, e.g.:
+        # "https://lh3.googleusercontent.com/DjhQh7DMszk.....z=-p-h100-w100"
+        # These URL are long but not personalized (double checked via tor).
+
+        img_src = extract_text(result.xpath('preceding-sibling::a/figure/img/@src'))
+
+        results.append(
+            {
+                'url': href,
+                'title': title,
+                'content': content,
+                'img_src': img_src,
+            }
+        )
+
+    # return results
+    return results

+
+
+ceid_list = [
+    'AE:ar',
+    'AR:es-419',
+    'AT:de',
+    'AU:en',
+    'BD:bn',
+    'BE:fr',
+    'BE:nl',
+    'BG:bg',
+    'BR:pt-419',
+    'BW:en',
+    'CA:en',
+    'CA:fr',
+    'CH:de',
+    'CH:fr',
+    'CL:es-419',
+    'CN:zh-Hans',
+    'CO:es-419',
+    'CU:es-419',
+    'CZ:cs',
+    'DE:de',
+    'EG:ar',
+    'ES:es',
+    'ET:en',
+    'FR:fr',
+    'GB:en',
+    'GH:en',
+    'GR:el',
+    'HK:zh-Hant',
+    'HU:hu',
+    'ID:en',
+    'ID:id',
+    'IE:en',
+    'IL:en',
+    'IL:he',
+    'IN:bn',
+    'IN:en',
+    'IN:hi',
+    'IN:ml',
+    'IN:mr',
+    'IN:ta',
+    'IN:te',
+    'IT:it',
+    'JP:ja',
+    'KE:en',
+    'KR:ko',
+    'LB:ar',
+    'LT:lt',
+    'LV:en',
+    'LV:lv',
+    'MA:fr',
+    'MX:es-419',
+    'MY:en',
+    'NA:en',
+    'NG:en',
+    'NL:nl',
+    'NO:no',
+    'NZ:en',
+    'PE:es-419',
+    'PH:en',
+    'PK:en',
+    'PL:pl',
+    'PT:pt-150',
+    'RO:ro',
+    'RS:sr',
+    'RU:ru',
+    'SA:ar',
+    'SE:sv',
+    'SG:en',
+    'SI:sl',
+    'SK:sk',
+    'SN:fr',
+    'TH:th',
+    'TR:tr',
+    'TW:zh-Hant',
+    'TZ:en',
+    'UA:ru',
+    'UA:uk',
+    'UG:en',
+    'US:en',
+    'US:es-419',
+    'VE:es-419',
+    'VN:vi',
+    'ZA:en',
+    'ZW:en',
+]
+"""List of region/language combinations supported by Google News.  Values of the
+``ceid`` argument of the Google News REST API."""
+
+
+_skip_values = [
+    'ET:en',  # english (ethiopia)
+    'ID:en',  # english (indonesia)
+    'LV:en',  # english (latvia)
+]
+
+_ceid_locale_map = {'NO:no': 'nb-NO'}
+
+
+def fetch_traits(engine_traits: EngineTraits):
+    _fetch_traits(engine_traits, add_domains=False)
+
+    engine_traits.custom['ceid'] = {}
+
+    for ceid in ceid_list:
+        if ceid in _skip_values:
+            continue
+
+        region, lang = ceid.split(':')
+        x = lang.split('-')
+        if len(x) > 1:
+            if x[1] not in ['Hant', 'Hans']:
+                lang = x[0]
+
+        sxng_locale = _ceid_locale_map.get(ceid, lang + '-' + region)
+        try:
+            locale = babel.Locale.parse(sxng_locale, sep='-')
+        except babel.UnknownLocaleError:
+            print("ERROR: %s -> %s is unknown by babel" % (ceid, sxng_locale))
+            continue
+
+        engine_traits.custom['ceid'][locales.region_tag(locale)] = ceid
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/google_scholar.html b/_modules/searx/engines/google_scholar.html new file mode 100644 index 000000000..190d19b71 --- /dev/null +++ b/_modules/searx/engines/google_scholar.html @@ -0,0 +1,329 @@ + + + + + + + + searx.engines.google_scholar — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.google_scholar

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Google Scholar engine.
+
+Compared to other Google services the Scholar engine has a simple GET REST-API
+and there does not exists `async` API.  Even though the API slightly vintage we
+can make use of the :ref:`google API` to assemble the arguments of the GET
+request.
+"""
+
+from typing import TYPE_CHECKING
+from typing import Optional
+
+from urllib.parse import urlencode
+from datetime import datetime
+from lxml import html
+
+from searx.utils import (
+    eval_xpath,
+    eval_xpath_getindex,
+    eval_xpath_list,
+    extract_text,
+)
+
+from searx.exceptions import SearxEngineCaptchaException
+
+from searx.engines.google import fetch_traits  # pylint: disable=unused-import
+from searx.engines.google import (
+    get_google_info,
+    time_range_dict,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://scholar.google.com',
+    "wikidata_id": 'Q494817',
+    "official_api_documentation": 'https://developers.google.com/custom-search',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['science', 'scientific publications']
+paging = True
+language_support = True
+time_range_support = True
+safesearch = False
+send_accept_language_header = True
+
+
+
[docs]def time_range_args(params):
+    """Returns a dictionary with a time range arguments based on
+    ``params['time_range']``.
+
+    Google Scholar supports a detailed search by year.  Searching by *last
+    month* or *last week* (as offered by SearXNG) is uncommon for scientific
+    publications and is not supported by Google Scholar.
+
+    To limit the result list when the users selects a range, all the SearXNG
+    ranges (*day*, *week*, *month*, *year*) are mapped to *year*.  If no range
+    is set an empty dictionary of arguments is returned.  Example;  when
+    user selects a time range (current year minus one in 2022):
+
+    .. code:: python
+
+        { 'as_ylo' : 2021 }
+
+    """
+    ret_val = {}
+    if params['time_range'] in time_range_dict:
+        ret_val['as_ylo'] = datetime.now().year - 1
+    return ret_val

+
+
+
[docs]def detect_google_captcha(dom):
+    """In case of CAPTCHA Google Scholar open its own *not a Robot* dialog and is
+    not redirected to ``sorry.google.com``.
+    """
+    if eval_xpath(dom, "//form[@id='gs_captcha_f']"):
+        raise SearxEngineCaptchaException()

+
+
+
[docs]def request(query, params):
+    """Google-Scholar search request"""
+
+    google_info = get_google_info(params, traits)
+    # subdomain is: scholar.google.xy
+    google_info['subdomain'] = google_info['subdomain'].replace("www.", "scholar.")
+
+    args = {
+        'q': query,
+        **google_info['params'],
+        'start': (params['pageno'] - 1) * 10,
+        'as_sdt': '2007',  # include patents / to disable set '0,5'
+        'as_vis': '0',  # include citations / to disable set '1'
+    }
+    args.update(time_range_args(params))
+
+    params['url'] = 'https://' + google_info['subdomain'] + '/scholar?' + urlencode(args)
+    params['cookies'] = google_info['cookies']
+    params['headers'].update(google_info['headers'])
+    return params

+
+
+
[docs]def parse_gs_a(text: Optional[str]):
+    """Parse the text written in green.
+
+    Possible formats:
+    * "{authors} - {journal}, {year} - {publisher}"
+    * "{authors} - {year} - {publisher}"
+    * "{authors} - {publisher}"
+    """
+    if text is None or text == "":
+        return None, None, None, None
+
+    s_text = text.split(' - ')
+    authors = s_text[0].split(', ')
+    publisher = s_text[-1]
+    if len(s_text) != 3:
+        return authors, None, publisher, None
+
+    # the format is "{authors} - {journal}, {year} - {publisher}" or "{authors} - {year} - {publisher}"
+    # get journal and year
+    journal_year = s_text[1].split(', ')
+    # journal is optional and may contains some coma
+    if len(journal_year) > 1:
+        journal = ', '.join(journal_year[0:-1])
+        if journal == '…':
+            journal = None
+    else:
+        journal = None
+    # year
+    year = journal_year[-1]
+    try:
+        publishedDate = datetime.strptime(year.strip(), '%Y')
+    except ValueError:
+        publishedDate = None
+    return authors, journal, publisher, publishedDate

+
+
+
[docs]def response(resp):  # pylint: disable=too-many-locals
+    """Parse response from Google Scholar"""
+    results = []
+
+    # convert the text to dom
+    dom = html.fromstring(resp.text)
+    detect_google_captcha(dom)
+
+    # parse results
+    for result in eval_xpath_list(dom, '//div[@data-rp]'):
+
+        title = extract_text(eval_xpath(result, './/h3[1]//a'))
+
+        if not title:
+            # this is a [ZITATION] block
+            continue
+
+        pub_type = extract_text(eval_xpath(result, './/span[@class="gs_ctg2"]'))
+        if pub_type:
+            pub_type = pub_type[1:-1].lower()
+
+        url = eval_xpath_getindex(result, './/h3[1]//a/@href', 0)
+        content = extract_text(eval_xpath(result, './/div[@class="gs_rs"]'))
+        authors, journal, publisher, publishedDate = parse_gs_a(
+            extract_text(eval_xpath(result, './/div[@class="gs_a"]'))
+        )
+        if publisher in url:
+            publisher = None
+
+        # cited by
+        comments = extract_text(eval_xpath(result, './/div[@class="gs_fl"]/a[starts-with(@href,"/scholar?cites=")]'))
+
+        # link to the html or pdf document
+        html_url = None
+        pdf_url = None
+        doc_url = eval_xpath_getindex(result, './/div[@class="gs_or_ggsm"]/a/@href', 0, default=None)
+        doc_type = extract_text(eval_xpath(result, './/span[@class="gs_ctg2"]'))
+        if doc_type == "[PDF]":
+            pdf_url = doc_url
+        else:
+            html_url = doc_url
+
+        results.append(
+            {
+                'template': 'paper.html',
+                'type': pub_type,
+                'url': url,
+                'title': title,
+                'authors': authors,
+                'publisher': publisher,
+                'journal': journal,
+                'publishedDate': publishedDate,
+                'content': content,
+                'comments': comments,
+                'html_url': html_url,
+                'pdf_url': pdf_url,
+            }
+        )
+
+    # parse suggestion
+    for suggestion in eval_xpath(dom, '//div[contains(@class, "gs_qsuggest_wrap")]//li//a'):
+        # append suggestion
+        results.append({'suggestion': extract_text(suggestion)})
+
+    for correction in eval_xpath(dom, '//div[@class="gs_r gs_pda"]/a'):
+        results.append({'correction': extract_text(correction)})
+
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/google_videos.html b/_modules/searx/engines/google_videos.html new file mode 100644 index 000000000..09906b715 --- /dev/null +++ b/_modules/searx/engines/google_videos.html @@ -0,0 +1,251 @@ + + + + + + + + searx.engines.google_videos — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.google_videos

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This is the implementation of the Google Videos engine.
+
+.. admonition:: Content-Security-Policy (CSP)
+
+   This engine needs to allow images from the `data URLs`_ (prefixed with the
+   ``data:`` scheme)::
+
+     Header set Content-Security-Policy "img-src 'self' data: ;"
+
+.. _data URLs:
+   https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
+
+"""
+
+from typing import TYPE_CHECKING
+
+from urllib.parse import urlencode
+from lxml import html
+
+from searx.utils import (
+    eval_xpath,
+    eval_xpath_list,
+    eval_xpath_getindex,
+    extract_text,
+)
+
+from searx.engines.google import fetch_traits  # pylint: disable=unused-import
+from searx.engines.google import (
+    get_google_info,
+    time_range_dict,
+    filter_mapping,
+    suggestion_xpath,
+    detect_google_sorry,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://www.google.com',
+    "wikidata_id": 'Q219885',
+    "official_api_documentation": 'https://developers.google.com/custom-search',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+
+categories = ['videos', 'web']
+paging = True
+language_support = True
+time_range_support = True
+safesearch = True
+
+
+
[docs]def request(query, params):
+    """Google-Video search request"""
+
+    google_info = get_google_info(params, traits)
+
+    query_url = (
+        'https://'
+        + google_info['subdomain']
+        + '/search'
+        + "?"
+        + urlencode(
+            {
+                'q': query,
+                'tbm': "vid",
+                'start': 10 * params['pageno'],
+                **google_info['params'],
+                'asearch': 'arc',
+                'async': 'use_ac:true,_fmt:html',
+            }
+        )
+    )
+
+    if params['time_range'] in time_range_dict:
+        query_url += '&' + urlencode({'tbs': 'qdr:' + time_range_dict[params['time_range']]})
+    if params['safesearch']:
+        query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
+    params['url'] = query_url
+
+    params['cookies'] = google_info['cookies']
+    params['headers'].update(google_info['headers'])
+    return params

+
+
+
[docs]def response(resp):
+    """Get response from google's search request"""
+    results = []
+
+    detect_google_sorry(resp)
+
+    # convert the text to dom
+    dom = html.fromstring(resp.text)
+
+    # parse results
+    for result in eval_xpath_list(dom, '//div[contains(@class, "g ")]'):
+
+        img_src = eval_xpath_getindex(result, './/img/@src', 0, None)
+        if img_src is None:
+            continue
+
+        title = extract_text(eval_xpath_getindex(result, './/a/h3[1]', 0))
+        url = eval_xpath_getindex(result, './/a/h3[1]/../@href', 0)
+
+        c_node = eval_xpath_getindex(result, './/div[@class="Uroaid"]', 0)
+        content = extract_text(c_node)
+        pub_info = extract_text(eval_xpath(result, './/div[@class="P7xzyf"]'))
+        length = extract_text(eval_xpath(result, './/div[@class="J1mWY"]'))
+
+        results.append(
+            {
+                'url': url,
+                'title': title,
+                'content': content,
+                'author': pub_info,
+                'thumbnail': img_src,
+                'length': length,
+                'template': 'videos.html',
+            }
+        )
+
+    # parse suggestion
+    for suggestion in eval_xpath_list(dom, suggestion_xpath):
+        # append suggestion
+        results.append({'suggestion': extract_text(suggestion)})
+
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/peertube.html b/_modules/searx/engines/peertube.html new file mode 100644 index 000000000..f09ff1296 --- /dev/null +++ b/_modules/searx/engines/peertube.html @@ -0,0 +1,298 @@ + + + + + + + + searx.engines.peertube — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.peertube

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Peertube and :py:obj:`SepiaSearch <searx.engines.sepiasearch>` do share
+(more or less) the same REST API and the schema of the JSON result is identical.
+
+"""
+
+import re
+from urllib.parse import urlencode
+from datetime import datetime
+from dateutil.parser import parse
+from dateutil.relativedelta import relativedelta
+
+import babel
+
+from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+from searx.locales import language_tag
+from searx.utils import html_to_text
+from searx.enginelib.traits import EngineTraits
+
+traits: EngineTraits
+
+about = {
+    # pylint: disable=line-too-long
+    "website": 'https://joinpeertube.org',
+    "wikidata_id": 'Q50938515',
+    "official_api_documentation": 'https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# engine dependent config
+categories = ["videos"]
+paging = True
+base_url = "https://peer.tube"
+"""Base URL of the Peertube instance.  A list of instances is available at:
+
+- https://instances.joinpeertube.org/instances
+"""
+
+time_range_support = True
+time_range_table = {
+    'day': relativedelta(),
+    'week': relativedelta(weeks=-1),
+    'month': relativedelta(months=-1),
+    'year': relativedelta(years=-1),
+}
+
+safesearch = True
+safesearch_table = {0: 'both', 1: 'false', 2: 'false'}
+
+
+def minute_to_hm(minute):
+    if isinstance(minute, int):
+        return "%d:%02d" % (divmod(minute, 60))
+    return None
+
+
+
[docs]def request(query, params):
+    """Assemble request for the Peertube API"""
+
+    if not query:
+        return False
+
+    # eng_region = traits.get_region(params['searxng_locale'], 'en_US')
+    eng_lang = traits.get_language(params['searxng_locale'], None)
+
+    params['url'] = (
+        base_url.rstrip("/")
+        + "/api/v1/search/videos?"
+        + urlencode(
+            {
+                'search': query,
+                'searchTarget': 'search-index',  # Vidiversum
+                'resultType': 'videos',
+                'start': (params['pageno'] - 1) * 10,
+                'count': 10,
+                # -createdAt: sort by date ascending / createdAt: date descending
+                'sort': '-match',  # sort by *match descending*
+                'nsfw': safesearch_table[params['safesearch']],
+            }
+        )
+    )
+
+    if eng_lang is not None:
+        params['url'] += '&languageOneOf[]=' + eng_lang
+        params['url'] += '&boostLanguages[]=' + eng_lang
+
+    if params['time_range'] in time_range_table:
+        time = datetime.now().date() + time_range_table[params['time_range']]
+        params['url'] += '&startDate=' + time.isoformat()
+
+    return params

+
+
+def response(resp):
+    return video_response(resp)
+
+
+
[docs]def video_response(resp):
+    """Parse video response from SepiaSearch and Peertube instances."""
+    results = []
+
+    json_data = resp.json()
+
+    if 'data' not in json_data:
+        return []
+
+    for result in json_data['data']:
+        metadata = [
+            x
+            for x in [
+                result.get('channel', {}).get('displayName'),
+                result.get('channel', {}).get('name') + '@' + result.get('channel', {}).get('host'),
+                ', '.join(result.get('tags', [])),
+            ]
+            if x
+        ]
+
+        results.append(
+            {
+                'url': result['url'],
+                'title': result['name'],
+                'content': html_to_text(result.get('description') or ''),
+                'author': result.get('account', {}).get('displayName'),
+                'length': minute_to_hm(result.get('duration')),
+                'template': 'videos.html',
+                'publishedDate': parse(result['publishedAt']),
+                'iframe_src': result.get('embedUrl'),
+                'thumbnail': result.get('thumbnailUrl') or result.get('previewUrl'),
+                'metadata': ' | '.join(metadata),
+            }
+        )
+
+    return results

+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages from peertube's search-index source code.
+
+    See videoLanguages_ in commit `8ed5c729 - Refactor and redesign client`_
+
+    .. _8ed5c729 - Refactor and redesign client:
+       https://framagit.org/framasoft/peertube/search-index/-/commit/8ed5c729
+    .. _videoLanguages:
+       https://framagit.org/framasoft/peertube/search-index/-/commit/8ed5c729#3d8747f9a60695c367c70bb64efba8f403721fad_0_291
+    """
+
+    resp = get(
+        'https://framagit.org/framasoft/peertube/search-index/-/raw/master/client/src/components/Filters.vue',
+        # the response from search-index repository is very slow
+        timeout=60,
+    )
+
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from peertube is not OK.")
+        return
+
+    js_lang = re.search(r"videoLanguages \(\)[^\n]+(.*?)\]", resp.text, re.DOTALL)  # type: ignore
+    if not js_lang:
+        print("ERROR: can't determine languages from peertube")
+        return
+
+    for lang in re.finditer(r"\{ id: '([a-z]+)', label:", js_lang.group(1)):
+        eng_tag = lang.group(1)
+        if eng_tag == 'oc':
+            # Occitanis not known by babel, its closest relative is Catalan
+            # but 'ca' is already in the list of engine_traits.languages -->
+            # 'oc' will be ignored.
+            continue
+        try:
+            sxng_tag = language_tag(babel.Locale.parse(eng_tag))
+        except babel.UnknownLocaleError:
+            print("ERROR: %s is unknown by babel" % eng_tag)
+            continue
+
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.languages[sxng_tag] = eng_tag
+
+    engine_traits.languages['zh_Hans'] = 'zh'
+    engine_traits.languages['zh_Hant'] = 'zh'

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/sepiasearch.html b/_modules/searx/engines/sepiasearch.html new file mode 100644 index 000000000..9cf4092d3 --- /dev/null +++ b/_modules/searx/engines/sepiasearch.html @@ -0,0 +1,198 @@ + + + + + + + + searx.engines.sepiasearch — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.sepiasearch

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""SepiaSearch uses the same languages as :py:obj:`Peertube
+<searx.engines.peertube>` and the response is identical to the response from the
+peertube engines.
+
+"""
+
+from typing import TYPE_CHECKING
+
+from urllib.parse import urlencode
+from datetime import datetime
+
+from searx.engines.peertube import fetch_traits  # pylint: disable=unused-import
+from searx.engines.peertube import (
+    # pylint: disable=unused-import
+    video_response,
+    safesearch_table,
+    time_range_table,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+about = {
+    # pylint: disable=line-too-long
+    "website": 'https://sepiasearch.org',
+    "wikidata_id": None,
+    "official_api_documentation": 'https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# engine dependent config
+categories = ['videos']
+paging = True
+
+base_url = 'https://sepiasearch.org'
+
+time_range_support = True
+safesearch = True
+
+
+
[docs]def request(query, params):
+    """Assemble request for the SepiaSearch API"""
+
+    if not query:
+        return False
+
+    # eng_region = traits.get_region(params['searxng_locale'], 'en_US')
+    eng_lang = traits.get_language(params['searxng_locale'], None)
+
+    params['url'] = (
+        base_url.rstrip("/")
+        + "/api/v1/search/videos?"
+        + urlencode(
+            {
+                'search': query,
+                'start': (params['pageno'] - 1) * 10,
+                'count': 10,
+                # -createdAt: sort by date ascending / createdAt: date descending
+                'sort': '-match',  # sort by *match descending*
+                'nsfw': safesearch_table[params['safesearch']],
+            }
+        )
+    )
+
+    if eng_lang is not None:
+        params['url'] += '&languageOneOf[]=' + eng_lang
+        params['url'] += '&boostLanguages[]=' + eng_lang
+
+    if params['time_range'] in time_range_table:
+        time = datetime.now().date() + time_range_table[params['time_range']]
+        params['url'] += '&startDate=' + time.isoformat()
+
+    return params

+
+
+def response(resp):
+    return video_response(resp)
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/sqlite.html b/_modules/searx/engines/sqlite.html new file mode 100644 index 000000000..7c7a4a786 --- /dev/null +++ b/_modules/searx/engines/sqlite.html @@ -0,0 +1,213 @@ + + + + + + + + searx.engines.sqlite — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.sqlite

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""SQLite is a small, fast and reliable SQL database engine.  It does not require
+any extra dependency.
+
+Example
+=======
+
+.. _MediathekView: https://mediathekview.de/
+
+To demonstrate the power of database engines, here is a more complex example
+which reads from a MediathekView_ (DE) movie database.  For this example of the
+SQlite engine download the database:
+
+- https://liste.mediathekview.de/filmliste-v2.db.bz2
+
+and unpack into ``searx/data/filmliste-v2.db``.  To search the database use e.g
+Query to test: ``!mediathekview concert``
+
+.. code:: yaml
+
+   - name: mediathekview
+     engine: sqlite
+     disabled: False
+     categories: general
+     result_template: default.html
+     database: searx/data/filmliste-v2.db
+     query_str:  >-
+       SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
+              COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
+              description AS content
+         FROM film
+        WHERE title LIKE :wildcard OR description LIKE :wildcard
+        ORDER BY duration DESC
+
+Implementations
+===============
+
+"""
+
+import sqlite3
+import contextlib
+
+engine_type = 'offline'
+database = ""
+query_str = ""
+limit = 10
+paging = True
+result_template = 'key-value.html'
+
+
+def init(engine_settings):
+    if 'query_str' not in engine_settings:
+        raise ValueError('query_str cannot be empty')
+
+    if not engine_settings['query_str'].lower().startswith('select '):
+        raise ValueError('only SELECT query is supported')
+
+
+
[docs]@contextlib.contextmanager
+def sqlite_cursor():
+    """Implements a :py:obj:`Context Manager <contextlib.contextmanager>` for a
+    :py:obj:`sqlite3.Cursor`.
+
+    Open database in read only mode: if the database doesn't exist.  The default
+    mode creates an empty file on the file system.  See:
+
+    * https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
+    * https://www.sqlite.org/uri.html
+
+    """
+    uri = 'file:' + database + '?mode=ro'
+    with contextlib.closing(sqlite3.connect(uri, uri=True)) as connect:
+        connect.row_factory = sqlite3.Row
+        with contextlib.closing(connect.cursor()) as cursor:
+            yield cursor

+
+
+def search(query, params):
+    results = []
+
+    query_params = {
+        'query': query,
+        'wildcard': r'%' + query.replace(' ', r'%') + r'%',
+        'limit': limit,
+        'offset': (params['pageno'] - 1) * limit,
+    }
+    query_to_run = query_str + ' LIMIT :limit OFFSET :offset'
+
+    with sqlite_cursor() as cur:
+
+        cur.execute(query_to_run, query_params)
+        col_names = [cn[0] for cn in cur.description]
+
+        for row in cur.fetchall():
+            item = dict(zip(col_names, map(str, row)))
+            item['template'] = result_template
+            logger.debug("append result --> %s", item)
+            results.append(item)
+
+    return results
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/startpage.html b/_modules/searx/engines/startpage.html new file mode 100644 index 000000000..daae67d8b --- /dev/null +++ b/_modules/searx/engines/startpage.html @@ -0,0 +1,606 @@ + + + + + + + + searx.engines.startpage — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.startpage

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Startpage's language & region selectors are a mess ..
+
+.. _startpage regions:
+
+Startpage regions
+=================
+
+In the list of regions there are tags we need to map to common region tags::
+
+  pt-BR_BR --> pt_BR
+  zh-CN_CN --> zh_Hans_CN
+  zh-TW_TW --> zh_Hant_TW
+  zh-TW_HK --> zh_Hant_HK
+  en-GB_GB --> en_GB
+
+and there is at least one tag with a three letter language tag (ISO 639-2)::
+
+  fil_PH --> fil_PH
+
+The locale code ``no_NO`` from Startpage does not exists and is mapped to
+``nb-NO``::
+
+    babel.core.UnknownLocaleError: unknown locale 'no_NO'
+
+For reference see languages-subtag at iana; ``no`` is the macrolanguage [1]_ and
+W3C recommends subtag over macrolanguage [2]_.
+
+.. [1] `iana: language-subtag-registry
+   <https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry>`_ ::
+
+      type: language
+      Subtag: nb
+      Description: Norwegian Bokmål
+      Added: 2005-10-16
+      Suppress-Script: Latn
+      Macrolanguage: no
+
+.. [2]
+   Use macrolanguages with care.  Some language subtags have a Scope field set to
+   macrolanguage, i.e. this primary language subtag encompasses a number of more
+   specific primary language subtags in the registry.  ...  As we recommended for
+   the collection subtags mentioned above, in most cases you should try to use
+   the more specific subtags ... `W3: The primary language subtag
+   <https://www.w3.org/International/questions/qa-choosing-language-tags#langsubtag>`_
+
+.. _startpage languages:
+
+Startpage languages
+===================
+
+:py:obj:`send_accept_language_header`:
+  The displayed name in Startpage's settings page depend on the location of the
+  IP when ``Accept-Language`` HTTP header is unset.  In :py:obj:`fetch_traits`
+  we use::
+
+    'Accept-Language': "en-US,en;q=0.5",
+    ..
+
+  to get uniform names independent from the IP).
+
+.. _startpage categories:
+
+Startpage categories
+====================
+
+Startpage's category (for Web-search, News, Videos, ..) is set by
+:py:obj:`startpage_categ` in  settings.yml::
+
+  - name: startpage
+    engine: startpage
+    startpage_categ: web
+    ...
+
+.. hint::
+
+   The default category is ``web`` .. and other categories than ``web`` are not
+   yet implemented.
+
+"""
+
+from typing import TYPE_CHECKING
+from collections import OrderedDict
+import re
+from unicodedata import normalize, combining
+from time import time
+from datetime import datetime, timedelta
+
+import dateutil.parser
+import lxml.html
+import babel
+
+from searx.utils import extract_text, eval_xpath, gen_useragent
+from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+from searx.exceptions import SearxEngineCaptchaException
+from searx.locales import region_tag
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://startpage.com',
+    "wikidata_id": 'Q2333295',
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+startpage_categ = 'web'
+"""Startpage's category, visit :ref:`startpage categories`.
+"""
+
+send_accept_language_header = True
+"""Startpage tries to guess user's language and territory from the HTTP
+``Accept-Language``.  Optional the user can select a search-language (can be
+different to the UI language) and a region filter.
+"""
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+time_range_support = True
+safesearch = True
+
+time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
+safesearch_dict = {0: '0', 1: '1', 2: '1'}
+
+# search-url
+base_url = 'https://www.startpage.com'
+search_url = base_url + '/sp/search'
+
+# specific xpath variables
+# ads xpath //div[@id="results"]/div[@id="sponsored"]//div[@class="result"]
+# not ads: div[@class="result"] are the direct childs of div[@id="results"]
+results_xpath = '//div[@class="w-gl__result__main"]'
+link_xpath = './/a[@class="w-gl__result-title result-link"]'
+content_xpath = './/p[@class="w-gl__description"]'
+search_form_xpath = '//form[@id="search"]'
+"""XPath of Startpage's origin search form
+
+.. code: html
+
+    <form action="/sp/search" method="post">
+      <input type="text" name="query"  value="" ..>
+      <input type="hidden" name="t" value="device">
+      <input type="hidden" name="lui" value="english">
+      <input type="hidden" name="sc" value="Q7Mt5TRqowKB00">
+      <input type="hidden" name="cat" value="web">
+      <input type="hidden" class="abp" id="abp-input" name="abp" value="1">
+    </form>
+"""
+
+# timestamp of the last fetch of 'sc' code
+sc_code_ts = 0
+sc_code = ''
+sc_code_cache_sec = 30
+"""Time in seconds the sc-code is cached in memory :py:obj:`get_sc_code`."""
+
+
+
[docs]def get_sc_code(searxng_locale, params):
+    """Get an actual ``sc`` argument from Startpage's search form (HTML page).
+
+    Startpage puts a ``sc`` argument on every HTML :py:obj:`search form
+    <search_form_xpath>`.  Without this argument Startpage considers the request
+    is from a bot.  We do not know what is encoded in the value of the ``sc``
+    argument, but it seems to be a kind of a *time-stamp*.
+
+    Startpage's search form generates a new sc-code on each request.  This
+    function scrap a new sc-code from Startpage's home page every
+    :py:obj:`sc_code_cache_sec` seconds.
+
+    """
+
+    global sc_code_ts, sc_code  # pylint: disable=global-statement
+
+    if sc_code and (time() < (sc_code_ts + sc_code_cache_sec)):
+        logger.debug("get_sc_code: reuse '%s'", sc_code)
+        return sc_code
+
+    headers = {**params['headers']}
+    headers['Origin'] = base_url
+    headers['Referer'] = base_url + '/'
+    # headers['Connection'] = 'keep-alive'
+    # headers['Accept-Encoding'] = 'gzip, deflate, br'
+    # headers['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'
+    # headers['User-Agent'] = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0'
+
+    # add Accept-Language header
+    if searxng_locale == 'all':
+        searxng_locale = 'en-US'
+    locale = babel.Locale.parse(searxng_locale, sep='-')
+
+    if send_accept_language_header:
+        ac_lang = locale.language
+        if locale.territory:
+            ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
+                locale.language,
+                locale.territory,
+                locale.language,
+            )
+        headers['Accept-Language'] = ac_lang
+
+    get_sc_url = base_url + '/?sc=%s' % (sc_code)
+    logger.debug("query new sc time-stamp ... %s", get_sc_url)
+    logger.debug("headers: %s", headers)
+    resp = get(get_sc_url, headers=headers)
+
+    # ?? x = network.get('https://www.startpage.com/sp/cdn/images/filter-chevron.svg', headers=headers)
+    # ?? https://www.startpage.com/sp/cdn/images/filter-chevron.svg
+    # ?? ping-back URL: https://www.startpage.com/sp/pb?sc=TLsB0oITjZ8F21
+
+    if str(resp.url).startswith('https://www.startpage.com/sp/captcha'):  # type: ignore
+        raise SearxEngineCaptchaException(
+            message="get_sc_code: got redirected to https://www.startpage.com/sp/captcha",
+        )
+
+    dom = lxml.html.fromstring(resp.text)  # type: ignore
+
+    try:
+        sc_code = eval_xpath(dom, search_form_xpath + '//input[@name="sc"]/@value')[0]
+    except IndexError as exc:
+        logger.debug("suspend startpage API --> https://github.com/searxng/searxng/pull/695")
+        raise SearxEngineCaptchaException(
+            message="get_sc_code: [PR-695] query new sc time-stamp failed! (%s)" % resp.url,  # type: ignore
+        ) from exc
+
+    sc_code_ts = time()
+    logger.debug("get_sc_code: new value is: %s", sc_code)
+    return sc_code

+
+
+
[docs]def request(query, params):
+    """Assemble a Startpage request.
+
+    To avoid CAPTCHA we need to send a well formed HTTP POST request with a
+    cookie.  We need to form a request that is identical to the request build by
+    Startpage's search form:
+
+    - in the cookie the **region** is selected
+    - in the HTTP POST data the **language** is selected
+
+    Additionally the arguments form Startpage's search form needs to be set in
+    HTML POST data / compare ``<input>`` elements: :py:obj:`search_form_xpath`.
+    """
+    if startpage_categ == 'web':
+        return _request_cat_web(query, params)
+
+    logger.error("Startpages's category '%' is not yet implemented.", startpage_categ)
+    return params

+
+
+def _request_cat_web(query, params):
+
+    engine_region = traits.get_region(params['searxng_locale'], 'en-US')
+    engine_language = traits.get_language(params['searxng_locale'], 'en')
+
+    # build arguments
+    args = {
+        'query': query,
+        'cat': 'web',
+        't': 'device',
+        'sc': get_sc_code(params['searxng_locale'], params),  # hint: this func needs HTTP headers,
+        'with_date': time_range_dict.get(params['time_range'], ''),
+    }
+
+    if engine_language:
+        args['language'] = engine_language
+        args['lui'] = engine_language
+
+    args['abp'] = '1'
+    if params['pageno'] > 1:
+        args['page'] = params['pageno']
+
+    # build cookie
+    lang_homepage = 'en'
+    cookie = OrderedDict()
+    cookie['date_time'] = 'world'
+    cookie['disable_family_filter'] = safesearch_dict[params['safesearch']]
+    cookie['disable_open_in_new_window'] = '0'
+    cookie['enable_post_method'] = '1'  # hint: POST
+    cookie['enable_proxy_safety_suggest'] = '1'
+    cookie['enable_stay_control'] = '1'
+    cookie['instant_answers'] = '1'
+    cookie['lang_homepage'] = 's/device/%s/' % lang_homepage
+    cookie['num_of_results'] = '10'
+    cookie['suggestions'] = '1'
+    cookie['wt_unit'] = 'celsius'
+
+    if engine_language:
+        cookie['language'] = engine_language
+        cookie['language_ui'] = engine_language
+
+    if engine_region:
+        cookie['search_results_region'] = engine_region
+
+    params['cookies']['preferences'] = 'N1N'.join(["%sEEE%s" % x for x in cookie.items()])
+    logger.debug('cookie preferences: %s', params['cookies']['preferences'])
+
+    # POST request
+    logger.debug("data: %s", args)
+    params['data'] = args
+    params['method'] = 'POST'
+    params['url'] = search_url
+    params['headers']['Origin'] = base_url
+    params['headers']['Referer'] = base_url + '/'
+    # is the Accept header needed?
+    # params['headers']['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    dom = lxml.html.fromstring(resp.text)
+
+    if startpage_categ == 'web':
+        return _response_cat_web(dom)
+
+    logger.error("Startpages's category '%' is not yet implemented.", startpage_categ)
+    return []
+
+
+def _response_cat_web(dom):
+    results = []
+
+    # parse results
+    for result in eval_xpath(dom, results_xpath):
+        links = eval_xpath(result, link_xpath)
+        if not links:
+            continue
+        link = links[0]
+        url = link.attrib.get('href')
+
+        # block google-ad url's
+        if re.match(r"^http(s|)://(www\.)?google\.[a-z]+/aclk.*$", url):
+            continue
+
+        # block startpage search url's
+        if re.match(r"^http(s|)://(www\.)?startpage\.com/do/search\?.*$", url):
+            continue
+
+        title = extract_text(link)
+
+        if eval_xpath(result, content_xpath):
+            content: str = extract_text(eval_xpath(result, content_xpath))  # type: ignore
+        else:
+            content = ''
+
+        published_date = None
+
+        # check if search result starts with something like: "2 Sep 2014 ... "
+        if re.match(r"^([1-9]|[1-2][0-9]|3[0-1]) [A-Z][a-z]{2} [0-9]{4} \.\.\. ", content):
+            date_pos = content.find('...') + 4
+            date_string = content[0 : date_pos - 5]
+            # fix content string
+            content = content[date_pos:]
+
+            try:
+                published_date = dateutil.parser.parse(date_string, dayfirst=True)
+            except ValueError:
+                pass
+
+        # check if search result starts with something like: "5 days ago ... "
+        elif re.match(r"^[0-9]+ days? ago \.\.\. ", content):
+            date_pos = content.find('...') + 4
+            date_string = content[0 : date_pos - 5]
+
+            # calculate datetime
+            published_date = datetime.now() - timedelta(days=int(re.match(r'\d+', date_string).group()))  # type: ignore
+
+            # fix content string
+            content = content[date_pos:]
+
+        if published_date:
+            # append result
+            results.append({'url': url, 'title': title, 'content': content, 'publishedDate': published_date})
+        else:
+            # append result
+            results.append({'url': url, 'title': title, 'content': content})
+
+    # return results
+    return results
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch :ref:`languages <startpage languages>` and :ref:`regions <startpage
+    regions>` from Startpage."""
+    # pylint: disable=too-many-branches
+
+    headers = {
+        'User-Agent': gen_useragent(),
+        'Accept-Language': "en-US,en;q=0.5",  # bing needs to set the English language
+    }
+    resp = get('https://www.startpage.com/do/settings', headers=headers)
+
+    if not resp.ok:  # type: ignore
+        print("ERROR: response from Startpage is not OK.")
+
+    dom = lxml.html.fromstring(resp.text)  # type: ignore
+
+    # regions
+
+    sp_region_names = []
+    for option in dom.xpath('//form[@name="settings"]//select[@name="search_results_region"]/option'):
+        sp_region_names.append(option.get('value'))
+
+    for eng_tag in sp_region_names:
+        if eng_tag == 'all':
+            continue
+        babel_region_tag = {'no_NO': 'nb_NO'}.get(eng_tag, eng_tag)  # norway
+
+        if '-' in babel_region_tag:
+            l, r = babel_region_tag.split('-')
+            r = r.split('_')[-1]
+            sxng_tag = region_tag(babel.Locale.parse(l + '_' + r, sep='_'))
+
+        else:
+            try:
+                sxng_tag = region_tag(babel.Locale.parse(babel_region_tag, sep='_'))
+
+            except babel.UnknownLocaleError:
+                print("ERROR: can't determine babel locale of startpage's locale %s" % eng_tag)
+                continue
+
+        conflict = engine_traits.regions.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.regions[sxng_tag] = eng_tag
+
+    # languages
+
+    catalog_engine2code = {name.lower(): lang_code for lang_code, name in babel.Locale('en').languages.items()}
+
+    # get the native name of every language known by babel
+
+    for lang_code in filter(
+        lambda lang_code: lang_code.find('_') == -1, babel.localedata.locale_identifiers()  # type: ignore
+    ):
+        native_name = babel.Locale(lang_code).get_language_name().lower()  # type: ignore
+        # add native name exactly as it is
+        catalog_engine2code[native_name] = lang_code
+
+        # add "normalized" language name (i.e. français becomes francais and español becomes espanol)
+        unaccented_name = ''.join(filter(lambda c: not combining(c), normalize('NFKD', native_name)))
+        if len(unaccented_name) == len(unaccented_name.encode()):
+            # add only if result is ascii (otherwise "normalization" didn't work)
+            catalog_engine2code[unaccented_name] = lang_code
+
+    # values that can't be determined by babel's languages names
+
+    catalog_engine2code.update(
+        {
+            # traditional chinese used in ..
+            'fantizhengwen': 'zh_Hant',
+            # Korean alphabet
+            'hangul': 'ko',
+            # Malayalam is one of 22 scheduled languages of India.
+            'malayam': 'ml',
+            'norsk': 'nb',
+            'sinhalese': 'si',
+        }
+    )
+
+    skip_eng_tags = {
+        'english_uk',  # SearXNG lang 'en' already maps to 'english'
+    }
+
+    for option in dom.xpath('//form[@name="settings"]//select[@name="language"]/option'):
+
+        eng_tag = option.get('value')
+        if eng_tag in skip_eng_tags:
+            continue
+        name = extract_text(option).lower()  # type: ignore
+
+        sxng_tag = catalog_engine2code.get(eng_tag)
+        if sxng_tag is None:
+            sxng_tag = catalog_engine2code[name]
+
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.languages[sxng_tag] = eng_tag

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/tineye.html b/_modules/searx/engines/tineye.html new file mode 100644 index 000000000..1cbbc11fd --- /dev/null +++ b/_modules/searx/engines/tineye.html @@ -0,0 +1,337 @@ + + + + + + + + searx.engines.tineye — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.tineye

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This engine implements *Tineye - reverse image search*
+
+Using TinEye, you can search by image or perform what we call a reverse image
+search.  You can do that by uploading an image or searching by URL. You can also
+simply drag and drop your images to start your search.  TinEye constantly crawls
+the web and adds images to its index.  Today, the TinEye index is over 50.2
+billion images `[tineye.com] <https://tineye.com/how>`_.
+
+.. hint::
+
+   This SearXNG engine only supports *'searching by URL'* and it does not use
+   the official API `[api.tineye.com] <https://api.tineye.com/python/docs/>`_.
+
+"""
+
+from urllib.parse import urlencode
+from datetime import datetime
+from flask_babel import gettext
+
+about = {
+    "website": 'https://tineye.com',
+    "wikidata_id": 'Q2382535',
+    "official_api_documentation": 'https://api.tineye.com/python/docs/',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+engine_type = 'online_url_search'
+""":py:obj:`searx.search.processors.online_url_search`"""
+
+categories = ['general']
+paging = True
+safesearch = False
+base_url = 'https://tineye.com'
+search_string = '/result_json/?page={page}&{query}'
+
+FORMAT_NOT_SUPPORTED = gettext(
+    "Could not read that image url. This may be due to an unsupported file"
+    " format. TinEye only supports images that are JPEG, PNG, GIF, BMP, TIFF or WebP."
+)
+"""TinEye error message"""
+
+NO_SIGNATURE_ERROR = gettext(
+    "The image is too simple to find matches. TinEye requires a basic level of"
+    " visual detail to successfully identify matches."
+)
+"""TinEye error message"""
+
+DOWNLOAD_ERROR = gettext("The image could not be downloaded.")
+"""TinEye error message"""
+
+
+
[docs]def request(query, params):
+    """Build TinEye HTTP request using ``search_urls`` of a :py:obj:`engine_type`."""
+
+    params['raise_for_httperror'] = False
+
+    if params['search_urls']['data:image']:
+        query = params['search_urls']['data:image']
+    elif params['search_urls']['http']:
+        query = params['search_urls']['http']
+
+    logger.debug("query URL: %s", query)
+    query = urlencode({'url': query})
+
+    # see https://github.com/TinEye/pytineye/blob/main/pytineye/api.py
+    params['url'] = base_url + search_string.format(query=query, page=params['pageno'])
+
+    params['headers'].update(
+        {
+            'Connection': 'keep-alive',
+            'Accept-Encoding': 'gzip, defalte, br',
+            'Host': 'tineye.com',
+            'DNT': '1',
+            'TE': 'trailers',
+        }
+    )
+    return params

+
+
+
[docs]def parse_tineye_match(match_json):
+    """Takes parsed JSON from the API server and turns it into a :py:obj:`dict`
+    object.
+
+    Attributes `(class Match) <https://github.com/TinEye/pytineye/blob/main/pytineye/api.py>`__
+
+    - `image_url`, link to the result image.
+    - `domain`, domain this result was found on.
+    - `score`, a number (0 to 100) that indicates how closely the images match.
+    - `width`, image width in pixels.
+    - `height`, image height in pixels.
+    - `size`, image area in pixels.
+    - `format`, image format.
+    - `filesize`, image size in bytes.
+    - `overlay`, overlay URL.
+    - `tags`, whether this match belongs to a collection or stock domain.
+
+    - `backlinks`, a list of Backlink objects pointing to the original websites
+      and image URLs. List items are instances of :py:obj:`dict`, (`Backlink
+      <https://github.com/TinEye/pytineye/blob/main/pytineye/api.py>`__):
+
+      - `url`, the image URL to the image.
+      - `backlink`, the original website URL.
+      - `crawl_date`, the date the image was crawled.
+
+    """
+
+    # HINT: there exists an alternative backlink dict in the domains list / e.g.::
+    #
+    #     match_json['domains'][0]['backlinks']
+
+    backlinks = []
+    if "backlinks" in match_json:
+
+        for backlink_json in match_json["backlinks"]:
+            if not isinstance(backlink_json, dict):
+                continue
+
+            crawl_date = backlink_json.get("crawl_date")
+            if crawl_date:
+                crawl_date = datetime.fromisoformat(crawl_date[:-3])
+            else:
+                crawl_date = datetime.min
+
+            backlinks.append(
+                {
+                    'url': backlink_json.get("url"),
+                    'backlink': backlink_json.get("backlink"),
+                    'crawl_date': crawl_date,
+                    'image_name': backlink_json.get("image_name"),
+                }
+            )
+
+    return {
+        'image_url': match_json.get("image_url"),
+        'domain': match_json.get("domain"),
+        'score': match_json.get("score"),
+        'width': match_json.get("width"),
+        'height': match_json.get("height"),
+        'size': match_json.get("size"),
+        'image_format': match_json.get("format"),
+        'filesize': match_json.get("filesize"),
+        'overlay': match_json.get("overlay"),
+        'tags': match_json.get("tags"),
+        'backlinks': backlinks,
+    }

+
+
+
[docs]def response(resp):
+    """Parse HTTP response from TinEye."""
+    results = []
+
+    try:
+        json_data = resp.json()
+    except Exception as exc:  # pylint: disable=broad-except
+        msg = "can't parse JSON response // %s" % exc
+        logger.error(msg)
+        json_data = {'error': msg}
+
+    # handle error codes from Tineye
+
+    if resp.is_error:
+        if resp.status_code in (400, 422):
+
+            message = 'HTTP status: %s' % resp.status_code
+            error = json_data.get('error')
+            s_key = json_data.get('suggestions', {}).get('key', '')
+
+            if error and s_key:
+                message = "%s (%s)" % (error, s_key)
+            elif error:
+                message = error
+
+            if s_key == "Invalid image URL":
+                # test https://docs.searxng.org/_static/searxng-wordmark.svg
+                message = FORMAT_NOT_SUPPORTED
+            elif s_key == 'NO_SIGNATURE_ERROR':
+                # test https://pngimg.com/uploads/dot/dot_PNG4.png
+                message = NO_SIGNATURE_ERROR
+            elif s_key == 'Download Error':
+                # test https://notexists
+                message = DOWNLOAD_ERROR
+
+            # see https://github.com/searxng/searxng/pull/1456#issuecomment-1193105023
+            # results.append({'answer': message})
+            logger.error(message)
+
+            return results
+
+        resp.raise_for_status()
+
+    # append results from matches
+
+    for match_json in json_data['matches']:
+
+        tineye_match = parse_tineye_match(match_json)
+        if not tineye_match['backlinks']:
+            continue
+
+        backlink = tineye_match['backlinks'][0]
+        results.append(
+            {
+                'template': 'images.html',
+                'url': backlink['backlink'],
+                'thumbnail_src': tineye_match['image_url'],
+                'source': backlink['url'],
+                'title': backlink['image_name'],
+                'img_src': backlink['url'],
+                'format': tineye_match['image_format'],
+                'widht': tineye_match['width'],
+                'height': tineye_match['height'],
+                'publishedDate': backlink['crawl_date'],
+            }
+        )
+
+    # append number of results
+
+    number_of_results = json_data.get('num_matches')
+    if number_of_results:
+        results.append({'number_of_results': number_of_results})
+
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/torznab.html b/_modules/searx/engines/torznab.html new file mode 100644 index 000000000..840a94ba5 --- /dev/null +++ b/_modules/searx/engines/torznab.html @@ -0,0 +1,355 @@ + + + + + + + + searx.engines.torznab — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.torznab

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Torznab_ is an API specification that provides a standardized way to query
+torrent site for content. It is used by a number of torrent applications,
+including Prowlarr_ and Jackett_.
+
+Using this engine together with Prowlarr_ or Jackett_ allows you to search
+a huge number of torrent sites which are not directly supported.
+
+Configuration
+=============
+
+The engine has the following settings:
+
+``base_url``:
+  Torznab endpoint URL.
+
+``api_key``:
+  The API key to use for authentication.
+
+``torznab_categories``:
+  The categories to use for searching. This is a list of category IDs.  See
+  Prowlarr-categories_ or Jackett-categories_ for more information.
+
+``show_torrent_files``:
+  Whether to show the torrent file in the search results.  Be carful as using
+  this with Prowlarr_ or Jackett_ leaks the API key.  This should be used only
+  if you are querying a Torznab endpoint without authentication or if the
+  instance is private.  Be aware that private trackers may ban you if you share
+  the torrent file.  Defaults to ``false``.
+
+``show_magnet_links``:
+  Whether to show the magnet link in the search results.  Be aware that private
+  trackers may ban you if you share the magnet link.  Defaults to ``true``.
+
+.. _Torznab:
+   https://torznab.github.io/spec-1.3-draft/index.html
+.. _Prowlarr:
+   https://github.com/Prowlarr/Prowlarr
+.. _Jackett:
+   https://github.com/Jackett/Jackett
+.. _Prowlarr-categories:
+   https://wiki.servarr.com/en/prowlarr/cardigann-yml-definition#categories
+.. _Jackett-categories:
+   https://github.com/Jackett/Jackett/wiki/Jackett-Categories
+
+Implementations
+===============
+
+"""
+from __future__ import annotations
+from typing import TYPE_CHECKING
+
+from typing import List, Dict, Any
+from datetime import datetime
+from urllib.parse import quote
+from lxml import etree  # type: ignore
+
+from searx.exceptions import SearxEngineAPIException
+
+if TYPE_CHECKING:
+    import httpx
+    import logging
+
+    logger: logging.Logger
+
+# engine settings
+about: Dict[str, Any] = {
+    "website": None,
+    "wikidata_id": None,
+    "official_api_documentation": "https://torznab.github.io/spec-1.3-draft",
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'XML',
+}
+categories: List[str] = ['files']
+paging: bool = False
+time_range_support: bool = False
+
+# defined in settings.yml
+# example (Jackett): "http://localhost:9117/api/v2.0/indexers/all/results/torznab"
+base_url: str = ''
+api_key: str = ''
+# https://newznab.readthedocs.io/en/latest/misc/api/#predefined-categories
+torznab_categories: List[str] = []
+show_torrent_files: bool = False
+show_magnet_links: bool = True
+
+
+
[docs]def init(engine_settings=None):  # pylint: disable=unused-argument
+    """Initialize the engine."""
+    if len(base_url) < 1:
+        raise ValueError('missing torznab base_url')

+
+
+
[docs]def request(query: str, params: Dict[str, Any]) -> Dict[str, Any]:
+    """Build the request params."""
+    search_url: str = base_url + '?t=search&q={search_query}'
+
+    if len(api_key) > 0:
+        search_url += '&apikey={api_key}'
+    if len(torznab_categories) > 0:
+        search_url += '&cat={torznab_categories}'
+
+    params['url'] = search_url.format(
+        search_query=quote(query), api_key=api_key, torznab_categories=",".join([str(x) for x in torznab_categories])
+    )
+
+    return params

+
+
+
[docs]def response(resp: httpx.Response) -> List[Dict[str, Any]]:
+    """Parse the XML response and return a list of results."""
+    results = []
+    search_results = etree.XML(resp.content)
+
+    # handle errors:  https://newznab.readthedocs.io/en/latest/misc/api/#newznab-error-codes
+    if search_results.tag == "error":
+        raise SearxEngineAPIException(search_results.get("description"))
+
+    channel: etree.Element = search_results[0]
+
+    item: etree.Element
+    for item in channel.iterfind('item'):
+        result: Dict[str, Any] = build_result(item)
+        results.append(result)
+
+    return results

+
+
+
[docs]def build_result(item: etree.Element) -> Dict[str, Any]:
+    """Build a result from a XML item."""
+
+    # extract attributes from XML
+    # see https://torznab.github.io/spec-1.3-draft/torznab/Specification-v1.3.html#predefined-attributes
+    enclosure: etree.Element | None = item.find('enclosure')
+    enclosure_url: str | None = None
+    if enclosure is not None:
+        enclosure_url = enclosure.get('url')
+
+    size = get_attribute(item, 'size')
+    if not size and enclosure:
+        size = enclosure.get('length')
+    if size:
+        size = int(size)
+
+    guid = get_attribute(item, 'guid')
+    comments = get_attribute(item, 'comments')
+    pubDate = get_attribute(item, 'pubDate')
+    seeders = get_torznab_attribute(item, 'seeders')
+    leechers = get_torznab_attribute(item, 'leechers')
+    peers = get_torznab_attribute(item, 'peers')
+
+    # map attributes to searx result
+    result: Dict[str, Any] = {
+        'template': 'torrent.html',
+        'title': get_attribute(item, 'title'),
+        'filesize': size,
+        'files': get_attribute(item, 'files'),
+        'seed': seeders,
+        'leech': _map_leechers(leechers, seeders, peers),
+        'url': _map_result_url(guid, comments),
+        'publishedDate': _map_published_date(pubDate),
+        'torrentfile': None,
+        'magnetlink': None,
+    }
+
+    link = get_attribute(item, 'link')
+    if show_torrent_files:
+        result['torrentfile'] = _map_torrent_file(link, enclosure_url)
+    if show_magnet_links:
+        magneturl = get_torznab_attribute(item, 'magneturl')
+        result['magnetlink'] = _map_magnet_link(magneturl, guid, enclosure_url, link)
+    return result

+
+
+def _map_result_url(guid: str | None, comments: str | None) -> str | None:
+    if guid and guid.startswith('http'):
+        return guid
+    if comments and comments.startswith('http'):
+        return comments
+    return None
+
+
+def _map_leechers(leechers: str | None, seeders: str | None, peers: str | None) -> str | None:
+    if leechers:
+        return leechers
+    if seeders and peers:
+        return str(int(peers) - int(seeders))
+    return None
+
+
+def _map_published_date(pubDate: str | None) -> datetime | None:
+    if pubDate is not None:
+        try:
+            return datetime.strptime(pubDate, '%a, %d %b %Y %H:%M:%S %z')
+        except (ValueError, TypeError) as e:
+            logger.debug("ignore exception (publishedDate): %s", e)
+    return None
+
+
+def _map_torrent_file(link: str | None, enclosure_url: str | None) -> str | None:
+    if link and link.startswith('http'):
+        return link
+    if enclosure_url and enclosure_url.startswith('http'):
+        return enclosure_url
+    return None
+
+
+def _map_magnet_link(
+    magneturl: str | None,
+    guid: str | None,
+    enclosure_url: str | None,
+    link: str | None,
+) -> str | None:
+    if magneturl and magneturl.startswith('magnet'):
+        return magneturl
+    if guid and guid.startswith('magnet'):
+        return guid
+    if enclosure_url and enclosure_url.startswith('magnet'):
+        return enclosure_url
+    if link and link.startswith('magnet'):
+        return link
+    return None
+
+
+
[docs]def get_attribute(item: etree.Element, property_name: str) -> str | None:
+    """Get attribute from item."""
+    property_element: etree.Element | None = item.find(property_name)
+    if property_element is not None:
+        return property_element.text
+    return None

+
+
+
[docs]def get_torznab_attribute(item: etree.Element, attribute_name: str) -> str | None:
+    """Get torznab special attribute from item."""
+    element: etree.Element | None = item.find(
+        './/torznab:attr[@name="{attribute_name}"]'.format(attribute_name=attribute_name),
+        {'torznab': 'http://torznab.com/schemas/2015/feed'},
+    )
+    if element is not None:
+        return element.get("value")
+    return None

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/wikidata.html b/_modules/searx/engines/wikidata.html new file mode 100644 index 000000000..0a6a02327 --- /dev/null +++ b/_modules/searx/engines/wikidata.html @@ -0,0 +1,895 @@ + + + + + + + + searx.engines.wikidata — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.wikidata

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This module implements the Wikidata engine.  Some implementations are shared
+from :ref:`wikipedia engine`.
+
+"""
+# pylint: disable=missing-class-docstring
+
+from typing import TYPE_CHECKING
+from hashlib import md5
+from urllib.parse import urlencode, unquote
+from json import loads
+
+from dateutil.parser import isoparse
+from babel.dates import format_datetime, format_date, format_time, get_datetime_format
+
+from searx.data import WIKIDATA_UNITS
+from searx.network import post, get
+from searx.utils import searx_useragent, get_string_replaces_function
+from searx.external_urls import get_external_url, get_earth_coordinates_url, area_to_osm_zoom
+from searx.engines.wikipedia import (
+    fetch_wikimedia_traits,
+    get_wiki_params,
+)
+from searx.enginelib.traits import EngineTraits
+
+if TYPE_CHECKING:
+    import logging
+
+    logger: logging.Logger
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://wikidata.org/',
+    "wikidata_id": 'Q2013',
+    "official_api_documentation": 'https://query.wikidata.org/',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+# SPARQL
+SPARQL_ENDPOINT_URL = 'https://query.wikidata.org/sparql'
+SPARQL_EXPLAIN_URL = 'https://query.wikidata.org/bigdata/namespace/wdq/sparql?explain'
+WIKIDATA_PROPERTIES = {
+    'P434': 'MusicBrainz',
+    'P435': 'MusicBrainz',
+    'P436': 'MusicBrainz',
+    'P966': 'MusicBrainz',
+    'P345': 'IMDb',
+    'P2397': 'YouTube',
+    'P1651': 'YouTube',
+    'P2002': 'Twitter',
+    'P2013': 'Facebook',
+    'P2003': 'Instagram',
+}
+
+# SERVICE wikibase:mwapi : https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI
+# SERVICE wikibase:label: https://en.wikibooks.org/wiki/SPARQL/SERVICE_-_Label#Manual_Label_SERVICE
+# https://en.wikibooks.org/wiki/SPARQL/WIKIDATA_Precision,_Units_and_Coordinates
+# https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Data_model
+# optimization:
+# * https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/query_optimization
+# * https://github.com/blazegraph/database/wiki/QueryHints
+QUERY_TEMPLATE = """
+SELECT ?item ?itemLabel ?itemDescription ?lat ?long %SELECT%
+WHERE
+{
+  SERVICE wikibase:mwapi {
+        bd:serviceParam wikibase:endpoint "www.wikidata.org";
+        wikibase:api "EntitySearch";
+        wikibase:limit 1;
+        mwapi:search "%QUERY%";
+        mwapi:language "%LANGUAGE%".
+        ?item wikibase:apiOutputItem mwapi:item.
+  }
+  hint:Prior hint:runFirst "true".
+
+  %WHERE%
+
+  SERVICE wikibase:label {
+      bd:serviceParam wikibase:language "%LANGUAGE%,en".
+      ?item rdfs:label ?itemLabel .
+      ?item schema:description ?itemDescription .
+      %WIKIBASE_LABELS%
+  }
+
+}
+GROUP BY ?item ?itemLabel ?itemDescription ?lat ?long %GROUP_BY%
+"""
+
+# Get the calendar names and the property names
+QUERY_PROPERTY_NAMES = """
+SELECT ?item ?name
+WHERE {
+    {
+      SELECT ?item
+      WHERE { ?item wdt:P279* wd:Q12132 }
+    } UNION {
+      VALUES ?item { %ATTRIBUTES% }
+    }
+    OPTIONAL { ?item rdfs:label ?name. }
+}
+"""
+
+# see the property "dummy value" of https://www.wikidata.org/wiki/Q2013 (Wikidata)
+# hard coded here to avoid to an additional SPARQL request when the server starts
+DUMMY_ENTITY_URLS = set(
+    "http://www.wikidata.org/entity/" + wid for wid in ("Q4115189", "Q13406268", "Q15397819", "Q17339402")
+)
+
+
+# https://www.w3.org/TR/sparql11-query/#rSTRING_LITERAL1
+# https://lists.w3.org/Archives/Public/public-rdf-dawg/2011OctDec/0175.html
+sparql_string_escape = get_string_replaces_function(
+    # fmt: off
+    {
+        '\t': '\\\t',
+        '\n': '\\\n',
+        '\r': '\\\r',
+        '\b': '\\\b',
+        '\f': '\\\f',
+        '\"': '\\\"',
+        '\'': '\\\'',
+        '\\': '\\\\'
+    }
+    # fmt: on
+)
+
+replace_http_by_https = get_string_replaces_function({'http:': 'https:'})
+
+
+def get_headers():
+    # user agent: https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Query_limits
+    return {'Accept': 'application/sparql-results+json', 'User-Agent': searx_useragent()}
+
+
+def get_label_for_entity(entity_id, language):
+    name = WIKIDATA_PROPERTIES.get(entity_id)
+    if name is None:
+        name = WIKIDATA_PROPERTIES.get((entity_id, language))
+    if name is None:
+        name = WIKIDATA_PROPERTIES.get((entity_id, language.split('-')[0]))
+    if name is None:
+        name = WIKIDATA_PROPERTIES.get((entity_id, 'en'))
+    if name is None:
+        name = entity_id
+    return name
+
+
+def send_wikidata_query(query, method='GET'):
+    if method == 'GET':
+        # query will be cached by wikidata
+        http_response = get(SPARQL_ENDPOINT_URL + '?' + urlencode({'query': query}), headers=get_headers())
+    else:
+        # query won't be cached by wikidata
+        http_response = post(SPARQL_ENDPOINT_URL, data={'query': query}, headers=get_headers())
+    if http_response.status_code != 200:
+        logger.debug('SPARQL endpoint error %s', http_response.content.decode())
+    logger.debug('request time %s', str(http_response.elapsed))
+    http_response.raise_for_status()
+    return loads(http_response.content.decode())
+
+
+def request(query, params):
+
+    eng_tag, _wiki_netloc = get_wiki_params(params['searxng_locale'], traits)
+    query, attributes = get_query(query, eng_tag)
+    logger.debug("request --> language %s // len(attributes): %s", eng_tag, len(attributes))
+
+    params['method'] = 'POST'
+    params['url'] = SPARQL_ENDPOINT_URL
+    params['data'] = {'query': query}
+    params['headers'] = get_headers()
+    params['language'] = eng_tag
+    params['attributes'] = attributes
+
+    return params
+
+
+def response(resp):
+
+    results = []
+    jsonresponse = loads(resp.content.decode())
+
+    language = resp.search_params['language']
+    attributes = resp.search_params['attributes']
+    logger.debug("request --> language %s // len(attributes): %s", language, len(attributes))
+
+    seen_entities = set()
+    for result in jsonresponse.get('results', {}).get('bindings', []):
+        attribute_result = {key: value['value'] for key, value in result.items()}
+        entity_url = attribute_result['item']
+        if entity_url not in seen_entities and entity_url not in DUMMY_ENTITY_URLS:
+            seen_entities.add(entity_url)
+            results += get_results(attribute_result, attributes, language)
+        else:
+            logger.debug('The SPARQL request returns duplicate entities: %s', str(attribute_result))
+
+    return results
+
+
+_IMG_SRC_DEFAULT_URL_PREFIX = "https://commons.wikimedia.org/wiki/Special:FilePath/"
+_IMG_SRC_NEW_URL_PREFIX = "https://upload.wikimedia.org/wikipedia/commons/thumb/"
+
+
+
[docs]def get_thumbnail(img_src):
+    """Get Thumbnail image from wikimedia commons
+
+    Images from commons.wikimedia.org are (HTTP) redirected to
+    upload.wikimedia.org.  The redirected URL can be calculated by this
+    function.
+
+    - https://stackoverflow.com/a/33691240
+
+    """
+    logger.debug('get_thumbnail(): %s', img_src)
+    if not img_src is None and _IMG_SRC_DEFAULT_URL_PREFIX in img_src.split()[0]:
+        img_src_name = unquote(img_src.replace(_IMG_SRC_DEFAULT_URL_PREFIX, "").split("?", 1)[0].replace("%20", "_"))
+        img_src_name_first = img_src_name
+        img_src_name_second = img_src_name
+
+        if ".svg" in img_src_name.split()[0]:
+            img_src_name_second = img_src_name + ".png"
+
+        img_src_size = img_src.replace(_IMG_SRC_DEFAULT_URL_PREFIX, "").split("?", 1)[1]
+        img_src_size = img_src_size[img_src_size.index("=") + 1 : img_src_size.index("&")]
+        img_src_name_md5 = md5(img_src_name.encode("utf-8")).hexdigest()
+        img_src = (
+            _IMG_SRC_NEW_URL_PREFIX
+            + img_src_name_md5[0]
+            + "/"
+            + img_src_name_md5[0:2]
+            + "/"
+            + img_src_name_first
+            + "/"
+            + img_src_size
+            + "px-"
+            + img_src_name_second
+        )
+        logger.debug('get_thumbnail() redirected: %s', img_src)
+
+    return img_src

+
+
+def get_results(attribute_result, attributes, language):
+    # pylint: disable=too-many-branches
+    results = []
+    infobox_title = attribute_result.get('itemLabel')
+    infobox_id = attribute_result['item']
+    infobox_id_lang = None
+    infobox_urls = []
+    infobox_attributes = []
+    infobox_content = attribute_result.get('itemDescription', [])
+    img_src = None
+    img_src_priority = 0
+
+    for attribute in attributes:
+        value = attribute.get_str(attribute_result, language)
+        if value is not None and value != '':
+            attribute_type = type(attribute)
+
+            if attribute_type in (WDURLAttribute, WDArticle):
+                # get_select() method : there is group_concat(distinct ...;separator=", ")
+                # split the value here
+                for url in value.split(', '):
+                    infobox_urls.append({'title': attribute.get_label(language), 'url': url, **attribute.kwargs})
+                    # "normal" results (not infobox) include official website and Wikipedia links.
+                    if attribute.kwargs.get('official') or attribute_type == WDArticle:
+                        results.append({'title': infobox_title, 'url': url, "content": infobox_content})
+                    # update the infobox_id with the wikipedia URL
+                    # first the local wikipedia URL, and as fallback the english wikipedia URL
+                    if attribute_type == WDArticle and (
+                        (attribute.language == 'en' and infobox_id_lang is None) or attribute.language != 'en'
+                    ):
+                        infobox_id_lang = attribute.language
+                        infobox_id = url
+            elif attribute_type == WDImageAttribute:
+                # this attribute is an image.
+                # replace the current image only the priority is lower
+                # (the infobox contain only one image).
+                if attribute.priority > img_src_priority:
+                    img_src = get_thumbnail(value)
+                    img_src_priority = attribute.priority
+            elif attribute_type == WDGeoAttribute:
+                # geocoordinate link
+                # use the area to get the OSM zoom
+                # Note: ignre the unit (must be km² otherwise the calculation is wrong)
+                # Should use normalized value p:P2046/psn:P2046/wikibase:quantityAmount
+                area = attribute_result.get('P2046')
+                osm_zoom = area_to_osm_zoom(area) if area else 19
+                url = attribute.get_geo_url(attribute_result, osm_zoom=osm_zoom)
+                if url:
+                    infobox_urls.append({'title': attribute.get_label(language), 'url': url, 'entity': attribute.name})
+            else:
+                infobox_attributes.append(
+                    {'label': attribute.get_label(language), 'value': value, 'entity': attribute.name}
+                )
+
+    if infobox_id:
+        infobox_id = replace_http_by_https(infobox_id)
+
+    # add the wikidata URL at the end
+    infobox_urls.append({'title': 'Wikidata', 'url': attribute_result['item']})
+
+    if img_src is None and len(infobox_attributes) == 0 and len(infobox_urls) == 1 and len(infobox_content) == 0:
+        results.append({'url': infobox_urls[0]['url'], 'title': infobox_title, 'content': infobox_content})
+    else:
+        results.append(
+            {
+                'infobox': infobox_title,
+                'id': infobox_id,
+                'content': infobox_content,
+                'img_src': img_src,
+                'urls': infobox_urls,
+                'attributes': infobox_attributes,
+            }
+        )
+    return results
+
+
+def get_query(query, language):
+    attributes = get_attributes(language)
+    select = [a.get_select() for a in attributes]
+    where = list(filter(lambda s: len(s) > 0, [a.get_where() for a in attributes]))
+    wikibase_label = list(filter(lambda s: len(s) > 0, [a.get_wikibase_label() for a in attributes]))
+    group_by = list(filter(lambda s: len(s) > 0, [a.get_group_by() for a in attributes]))
+    query = (
+        QUERY_TEMPLATE.replace('%QUERY%', sparql_string_escape(query))
+        .replace('%SELECT%', ' '.join(select))
+        .replace('%WHERE%', '\n  '.join(where))
+        .replace('%WIKIBASE_LABELS%', '\n      '.join(wikibase_label))
+        .replace('%GROUP_BY%', ' '.join(group_by))
+        .replace('%LANGUAGE%', language)
+    )
+    return query, attributes
+
+
+def get_attributes(language):
+    # pylint: disable=too-many-statements
+    attributes = []
+
+    def add_value(name):
+        attributes.append(WDAttribute(name))
+
+    def add_amount(name):
+        attributes.append(WDAmountAttribute(name))
+
+    def add_label(name):
+        attributes.append(WDLabelAttribute(name))
+
+    def add_url(name, url_id=None, **kwargs):
+        attributes.append(WDURLAttribute(name, url_id, kwargs))
+
+    def add_image(name, url_id=None, priority=1):
+        attributes.append(WDImageAttribute(name, url_id, priority))
+
+    def add_date(name):
+        attributes.append(WDDateAttribute(name))
+
+    # Dates
+    for p in [
+        'P571',  # inception date
+        'P576',  # dissolution date
+        'P580',  # start date
+        'P582',  # end date
+        'P569',  # date of birth
+        'P570',  # date of death
+        'P619',  # date of spacecraft launch
+        'P620',
+    ]:  # date of spacecraft landing
+        add_date(p)
+
+    for p in [
+        'P27',  # country of citizenship
+        'P495',  # country of origin
+        'P17',  # country
+        'P159',
+    ]:  # headquarters location
+        add_label(p)
+
+    # Places
+    for p in [
+        'P36',  # capital
+        'P35',  # head of state
+        'P6',  # head of government
+        'P122',  # basic form of government
+        'P37',
+    ]:  # official language
+        add_label(p)
+
+    add_value('P1082')  # population
+    add_amount('P2046')  # area
+    add_amount('P281')  # postal code
+    add_label('P38')  # currency
+    add_amount('P2048')  # height (building)
+
+    # Media
+    for p in [
+        'P400',  # platform (videogames, computing)
+        'P50',  # author
+        'P170',  # creator
+        'P57',  # director
+        'P175',  # performer
+        'P178',  # developer
+        'P162',  # producer
+        'P176',  # manufacturer
+        'P58',  # screenwriter
+        'P272',  # production company
+        'P264',  # record label
+        'P123',  # publisher
+        'P449',  # original network
+        'P750',  # distributed by
+        'P86',
+    ]:  # composer
+        add_label(p)
+
+    add_date('P577')  # publication date
+    add_label('P136')  # genre (music, film, artistic...)
+    add_label('P364')  # original language
+    add_value('P212')  # ISBN-13
+    add_value('P957')  # ISBN-10
+    add_label('P275')  # copyright license
+    add_label('P277')  # programming language
+    add_value('P348')  # version
+    add_label('P840')  # narrative location
+
+    # Languages
+    add_value('P1098')  # number of speakers
+    add_label('P282')  # writing system
+    add_label('P1018')  # language regulatory body
+    add_value('P218')  # language code (ISO 639-1)
+
+    # Other
+    add_label('P169')  # ceo
+    add_label('P112')  # founded by
+    add_label('P1454')  # legal form (company, organization)
+    add_label('P137')  # operator (service, facility, ...)
+    add_label('P1029')  # crew members (tripulation)
+    add_label('P225')  # taxon name
+    add_value('P274')  # chemical formula
+    add_label('P1346')  # winner (sports, contests, ...)
+    add_value('P1120')  # number of deaths
+    add_value('P498')  # currency code (ISO 4217)
+
+    # URL
+    add_url('P856', official=True)  # official website
+    attributes.append(WDArticle(language))  # wikipedia (user language)
+    if not language.startswith('en'):
+        attributes.append(WDArticle('en'))  # wikipedia (english)
+
+    add_url('P1324')  # source code repository
+    add_url('P1581')  # blog
+    add_url('P434', url_id='musicbrainz_artist')
+    add_url('P435', url_id='musicbrainz_work')
+    add_url('P436', url_id='musicbrainz_release_group')
+    add_url('P966', url_id='musicbrainz_label')
+    add_url('P345', url_id='imdb_id')
+    add_url('P2397', url_id='youtube_channel')
+    add_url('P1651', url_id='youtube_video')
+    add_url('P2002', url_id='twitter_profile')
+    add_url('P2013', url_id='facebook_profile')
+    add_url('P2003', url_id='instagram_profile')
+
+    # Map
+    attributes.append(WDGeoAttribute('P625'))
+
+    # Image
+    add_image('P15', priority=1, url_id='wikimedia_image')  # route map
+    add_image('P242', priority=2, url_id='wikimedia_image')  # locator map
+    add_image('P154', priority=3, url_id='wikimedia_image')  # logo
+    add_image('P18', priority=4, url_id='wikimedia_image')  # image
+    add_image('P41', priority=5, url_id='wikimedia_image')  # flag
+    add_image('P2716', priority=6, url_id='wikimedia_image')  # collage
+    add_image('P2910', priority=7, url_id='wikimedia_image')  # icon
+
+    return attributes
+
+
+class WDAttribute:
+    __slots__ = ('name',)
+
+    def __init__(self, name):
+        self.name = name
+
+    def get_select(self):
+        return '(group_concat(distinct ?{name};separator=", ") as ?{name}s)'.replace('{name}', self.name)
+
+    def get_label(self, language):
+        return get_label_for_entity(self.name, language)
+
+    def get_where(self):
+        return "OPTIONAL { ?item wdt:{name} ?{name} . }".replace('{name}', self.name)
+
+    def get_wikibase_label(self):
+        return ""
+
+    def get_group_by(self):
+        return ""
+
+    def get_str(self, result, language):  # pylint: disable=unused-argument
+        return result.get(self.name + 's')
+
+    def __repr__(self):
+        return '<' + str(type(self).__name__) + ':' + self.name + '>'
+
+
+class WDAmountAttribute(WDAttribute):
+    def get_select(self):
+        return '?{name} ?{name}Unit'.replace('{name}', self.name)
+
+    def get_where(self):
+        return """  OPTIONAL { ?item p:{name} ?{name}Node .
+    ?{name}Node rdf:type wikibase:BestRank ; ps:{name} ?{name} .
+    OPTIONAL { ?{name}Node psv:{name}/wikibase:quantityUnit ?{name}Unit. } }""".replace(
+            '{name}', self.name
+        )
+
+    def get_group_by(self):
+        return self.get_select()
+
+    def get_str(self, result, language):
+        value = result.get(self.name)
+        unit = result.get(self.name + "Unit")
+        if unit is not None:
+            unit = unit.replace('http://www.wikidata.org/entity/', '')
+            return value + " " + get_label_for_entity(unit, language)
+        return value
+
+
+class WDArticle(WDAttribute):
+
+    __slots__ = 'language', 'kwargs'
+
+    def __init__(self, language, kwargs=None):
+        super().__init__('wikipedia')
+        self.language = language
+        self.kwargs = kwargs or {}
+
+    def get_label(self, language):
+        # language parameter is ignored
+        return "Wikipedia ({language})".replace('{language}', self.language)
+
+    def get_select(self):
+        return "?article{language} ?articleName{language}".replace('{language}', self.language)
+
+    def get_where(self):
+        return """OPTIONAL { ?article{language} schema:about ?item ;
+             schema:inLanguage "{language}" ;
+             schema:isPartOf <https://{language}.wikipedia.org/> ;
+             schema:name ?articleName{language} . }""".replace(
+            '{language}', self.language
+        )
+
+    def get_group_by(self):
+        return self.get_select()
+
+    def get_str(self, result, language):
+        key = 'article{language}'.replace('{language}', self.language)
+        return result.get(key)
+
+
+class WDLabelAttribute(WDAttribute):
+    def get_select(self):
+        return '(group_concat(distinct ?{name}Label;separator=", ") as ?{name}Labels)'.replace('{name}', self.name)
+
+    def get_where(self):
+        return "OPTIONAL { ?item wdt:{name} ?{name} . }".replace('{name}', self.name)
+
+    def get_wikibase_label(self):
+        return "?{name} rdfs:label ?{name}Label .".replace('{name}', self.name)
+
+    def get_str(self, result, language):
+        return result.get(self.name + 'Labels')
+
+
+class WDURLAttribute(WDAttribute):
+
+    HTTP_WIKIMEDIA_IMAGE = 'http://commons.wikimedia.org/wiki/Special:FilePath/'
+
+    __slots__ = 'url_id', 'kwargs'
+
+    def __init__(self, name, url_id=None, kwargs=None):
+        super().__init__(name)
+        self.url_id = url_id
+        self.kwargs = kwargs
+
+    def get_str(self, result, language):
+        value = result.get(self.name + 's')
+        if self.url_id and value is not None and value != '':
+            value = value.split(',')[0]
+            url_id = self.url_id
+            if value.startswith(WDURLAttribute.HTTP_WIKIMEDIA_IMAGE):
+                value = value[len(WDURLAttribute.HTTP_WIKIMEDIA_IMAGE) :]
+                url_id = 'wikimedia_image'
+            return get_external_url(url_id, value)
+        return value
+
+
+class WDGeoAttribute(WDAttribute):
+    def get_label(self, language):
+        return "OpenStreetMap"
+
+    def get_select(self):
+        return "?{name}Lat ?{name}Long".replace('{name}', self.name)
+
+    def get_where(self):
+        return """OPTIONAL { ?item p:{name}/psv:{name} [
+    wikibase:geoLatitude ?{name}Lat ;
+    wikibase:geoLongitude ?{name}Long ] }""".replace(
+            '{name}', self.name
+        )
+
+    def get_group_by(self):
+        return self.get_select()
+
+    def get_str(self, result, language):
+        latitude = result.get(self.name + 'Lat')
+        longitude = result.get(self.name + 'Long')
+        if latitude and longitude:
+            return latitude + ' ' + longitude
+        return None
+
+    def get_geo_url(self, result, osm_zoom=19):
+        latitude = result.get(self.name + 'Lat')
+        longitude = result.get(self.name + 'Long')
+        if latitude and longitude:
+            return get_earth_coordinates_url(latitude, longitude, osm_zoom)
+        return None
+
+
+class WDImageAttribute(WDURLAttribute):
+
+    __slots__ = ('priority',)
+
+    def __init__(self, name, url_id=None, priority=100):
+        super().__init__(name, url_id)
+        self.priority = priority
+
+
+class WDDateAttribute(WDAttribute):
+    def get_select(self):
+        return '?{name} ?{name}timePrecision ?{name}timeZone ?{name}timeCalendar'.replace('{name}', self.name)
+
+    def get_where(self):
+        # To remove duplicate, add
+        # FILTER NOT EXISTS { ?item p:{name}/psv:{name}/wikibase:timeValue ?{name}bis FILTER (?{name}bis < ?{name}) }
+        # this filter is too slow, so the response function ignore duplicate results
+        # (see the seen_entities variable)
+        return """OPTIONAL { ?item p:{name}/psv:{name} [
+    wikibase:timeValue ?{name} ;
+    wikibase:timePrecision ?{name}timePrecision ;
+    wikibase:timeTimezone ?{name}timeZone ;
+    wikibase:timeCalendarModel ?{name}timeCalendar ] . }
+    hint:Prior hint:rangeSafe true;""".replace(
+            '{name}', self.name
+        )
+
+    def get_group_by(self):
+        return self.get_select()
+
+    def format_8(self, value, locale):  # pylint: disable=unused-argument
+        # precision: less than a year
+        return value
+
+    def format_9(self, value, locale):
+        year = int(value)
+        # precision: year
+        if year < 1584:
+            if year < 0:
+                return str(year - 1)
+            return str(year)
+        timestamp = isoparse(value)
+        return format_date(timestamp, format='yyyy', locale=locale)
+
+    def format_10(self, value, locale):
+        # precision: month
+        timestamp = isoparse(value)
+        return format_date(timestamp, format='MMMM y', locale=locale)
+
+    def format_11(self, value, locale):
+        # precision: day
+        timestamp = isoparse(value)
+        return format_date(timestamp, format='full', locale=locale)
+
+    def format_13(self, value, locale):
+        timestamp = isoparse(value)
+        # precision: minute
+        return (
+            get_datetime_format(format, locale=locale)
+            .replace("'", "")
+            .replace('{0}', format_time(timestamp, 'full', tzinfo=None, locale=locale))
+            .replace('{1}', format_date(timestamp, 'short', locale=locale))
+        )
+
+    def format_14(self, value, locale):
+        # precision: second.
+        return format_datetime(isoparse(value), format='full', locale=locale)
+
+    DATE_FORMAT = {
+        '0': ('format_8', 1000000000),
+        '1': ('format_8', 100000000),
+        '2': ('format_8', 10000000),
+        '3': ('format_8', 1000000),
+        '4': ('format_8', 100000),
+        '5': ('format_8', 10000),
+        '6': ('format_8', 1000),
+        '7': ('format_8', 100),
+        '8': ('format_8', 10),
+        '9': ('format_9', 1),  # year
+        '10': ('format_10', 1),  # month
+        '11': ('format_11', 0),  # day
+        '12': ('format_13', 0),  # hour (not supported by babel, display minute)
+        '13': ('format_13', 0),  # minute
+        '14': ('format_14', 0),  # second
+    }
+
+    def get_str(self, result, language):
+        value = result.get(self.name)
+        if value == '' or value is None:
+            return None
+        precision = result.get(self.name + 'timePrecision')
+        date_format = WDDateAttribute.DATE_FORMAT.get(precision)
+        if date_format is not None:
+            format_method = getattr(self, date_format[0])
+            precision = date_format[1]
+            try:
+                if precision >= 1:
+                    t = value.split('-')
+                    if value.startswith('-'):
+                        value = '-' + t[1]
+                    else:
+                        value = t[0]
+                return format_method(value, language)
+            except Exception:  # pylint: disable=broad-except
+                return value
+        return value
+
+
+def debug_explain_wikidata_query(query, method='GET'):
+    if method == 'GET':
+        http_response = get(SPARQL_EXPLAIN_URL + '&' + urlencode({'query': query}), headers=get_headers())
+    else:
+        http_response = post(SPARQL_EXPLAIN_URL, data={'query': query}, headers=get_headers())
+    http_response.raise_for_status()
+    return http_response.content
+
+
+def init(engine_settings=None):  # pylint: disable=unused-argument
+    # WIKIDATA_PROPERTIES : add unit symbols
+    WIKIDATA_PROPERTIES.update(WIKIDATA_UNITS)
+
+    # WIKIDATA_PROPERTIES : add property labels
+    wikidata_property_names = []
+    for attribute in get_attributes('en'):
+        if type(attribute) in (WDAttribute, WDAmountAttribute, WDURLAttribute, WDDateAttribute, WDLabelAttribute):
+            if attribute.name not in WIKIDATA_PROPERTIES:
+                wikidata_property_names.append("wd:" + attribute.name)
+    query = QUERY_PROPERTY_NAMES.replace('%ATTRIBUTES%', " ".join(wikidata_property_names))
+    jsonresponse = send_wikidata_query(query)
+    for result in jsonresponse.get('results', {}).get('bindings', {}):
+        name = result['name']['value']
+        lang = result['name']['xml:lang']
+        entity_id = result['item']['value'].replace('http://www.wikidata.org/entity/', '')
+        WIKIDATA_PROPERTIES[(entity_id, lang)] = name.capitalize()
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Uses languages evaluated from :py:obj:`wikipedia.fetch_wikimedia_traits
+    <searx.engines.wikipedia.fetch_wikimedia_traits>` and removes
+
+    - ``traits.custom['wiki_netloc']``: wikidata does not have net-locations for
+      the languages and the list of all
+
+    - ``traits.custom['WIKIPEDIA_LANGUAGES']``: not used in the wikipedia engine
+
+    """
+
+    fetch_wikimedia_traits(engine_traits)
+    engine_traits.custom['wiki_netloc'] = {}
+    engine_traits.custom['WIKIPEDIA_LANGUAGES'] = []

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/wikipedia.html b/_modules/searx/engines/wikipedia.html new file mode 100644 index 000000000..770182d2d --- /dev/null +++ b/_modules/searx/engines/wikipedia.html @@ -0,0 +1,429 @@ + + + + + + + + searx.engines.wikipedia — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.wikipedia

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""This module implements the Wikipedia engine.  Some of this implementations
+are shared by other engines:
+
+- :ref:`wikidata engine`
+
+The list of supported languages is :py:obj:`fetched <fetch_wikimedia_traits>` from
+the article linked by :py:obj:`list_of_wikipedias`.
+
+Unlike traditional search engines, wikipedia does not support one Wikipedia for
+all languages, but there is one Wikipedia for each supported language. Some of
+these Wikipedias have a LanguageConverter_ enabled
+(:py:obj:`rest_v1_summary_url`).
+
+A LanguageConverter_ (LC) is a system based on language variants that
+automatically converts the content of a page into a different variant. A variant
+is mostly the same language in a different script.
+
+- `Wikipedias in multiple writing systems`_
+- `Automatic conversion between traditional and simplified Chinese characters`_
+
+PR-2554_:
+  The Wikipedia link returned by the API is still the same in all cases
+  (`https://zh.wikipedia.org/wiki/出租車`_) but if your browser's
+  ``Accept-Language`` is set to any of ``zh``, ``zh-CN``, ``zh-TW``, ``zh-HK``
+  or .. Wikipedia's LC automatically returns the desired script in their
+  web-page.
+
+  - You can test the API here: https://reqbin.com/gesg2kvx
+
+.. _https://zh.wikipedia.org/wiki/出租車:
+   https://zh.wikipedia.org/wiki/%E5%87%BA%E7%A7%9F%E8%BB%8A
+
+To support Wikipedia's LanguageConverter_, a SearXNG request to Wikipedia uses
+:py:obj:`get_wiki_params` and :py:obj:`wiki_lc_locale_variants' in the
+:py:obj:`fetch_wikimedia_traits` function.
+
+To test in SearXNG, query for ``!wp 出租車`` with each of the available Chinese
+options:
+
+- ``!wp 出租車 :zh``    should show 出租車
+- ``!wp 出租車 :zh-CN`` should show 出租车
+- ``!wp 出租車 :zh-TW`` should show 計程車
+- ``!wp 出租車 :zh-HK`` should show 的士
+- ``!wp 出租車 :zh-SG`` should show 德士
+
+.. _LanguageConverter:
+   https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter
+.. _Wikipedias in multiple writing systems:
+   https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems
+.. _Automatic conversion between traditional and simplified Chinese characters:
+   https://en.wikipedia.org/wiki/Chinese_Wikipedia#Automatic_conversion_between_traditional_and_simplified_Chinese_characters
+.. _PR-2554: https://github.com/searx/searx/pull/2554
+
+"""
+
+import urllib.parse
+import babel
+
+from lxml import html
+
+from searx import utils
+from searx import network as _network
+from searx import locales
+from searx.enginelib.traits import EngineTraits
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://www.wikipedia.org/',
+    "wikidata_id": 'Q52',
+    "official_api_documentation": 'https://en.wikipedia.org/api/',
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+send_accept_language_header = True
+"""The HTTP ``Accept-Language`` header is needed for wikis where
+LanguageConverter_ is enabled."""
+
+list_of_wikipedias = 'https://meta.wikimedia.org/wiki/List_of_Wikipedias'
+"""`List of all wikipedias <https://meta.wikimedia.org/wiki/List_of_Wikipedias>`_
+"""
+
+wikipedia_article_depth = 'https://meta.wikimedia.org/wiki/Wikipedia_article_depth'
+"""The *editing depth* of Wikipedia is one of several possible rough indicators
+of the encyclopedia's collaborative quality, showing how frequently its articles
+are updated.  The measurement of depth was introduced after some limitations of
+the classic measurement of article count were realized.
+"""
+
+rest_v1_summary_url = 'https://{wiki_netloc}/api/rest_v1/page/summary/{title}'
+"""
+`wikipedia rest_v1 summary API`_:
+  The summary response includes an extract of the first paragraph of the page in
+  plain text and HTML as well as the type of page. This is useful for page
+  previews (fka. Hovercards, aka. Popups) on the web and link previews in the
+  apps.
+
+HTTP ``Accept-Language`` header (:py:obj:`send_accept_language_header`):
+  The desired language variant code for wikis where LanguageConverter_ is
+  enabled.
+
+.. _wikipedia rest_v1 summary API:
+   https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_summary__title_
+
+"""
+
+wiki_lc_locale_variants = {
+    "zh": (
+        "zh-CN",
+        "zh-HK",
+        "zh-MO",
+        "zh-MY",
+        "zh-SG",
+        "zh-TW",
+    ),
+    "zh-classical": ("zh-classical",),
+}
+"""Mapping rule of the LanguageConverter_ to map a language and its variants to
+a Locale (used in the HTTP ``Accept-Language`` header). For example see `LC
+Chinese`_.
+
+.. _LC Chinese:
+   https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems#Chinese
+"""
+
+wikipedia_script_variants = {
+    "zh": (
+        "zh_Hant",
+        "zh_Hans",
+    )
+}
+
+
+
[docs]def get_wiki_params(sxng_locale, eng_traits):
+    """Returns the Wikipedia language tag and the netloc that fits to the
+    ``sxng_locale``.  To support LanguageConverter_ this function rates a locale
+    (region) higher than a language (compare :py:obj:`wiki_lc_locale_variants`).
+
+    """
+    eng_tag = eng_traits.get_region(sxng_locale, eng_traits.get_language(sxng_locale, 'en'))
+    wiki_netloc = eng_traits.custom['wiki_netloc'].get(eng_tag, 'en.wikipedia.org')
+    return eng_tag, wiki_netloc

+
+
+
[docs]def request(query, params):
+    """Assemble a request (`wikipedia rest_v1 summary API`_)."""
+    if query.islower():
+        query = query.title()
+
+    _eng_tag, wiki_netloc = get_wiki_params(params['searxng_locale'], traits)
+    title = urllib.parse.quote(query)
+    params['url'] = rest_v1_summary_url.format(wiki_netloc=wiki_netloc, title=title)
+
+    params['raise_for_httperror'] = False
+    params['soft_max_redirects'] = 2
+
+    return params

+
+
+# get response from search-request
+def response(resp):
+
+    results = []
+    if resp.status_code == 404:
+        return []
+    if resp.status_code == 400:
+        try:
+            api_result = resp.json()
+        except Exception:  # pylint: disable=broad-except
+            pass
+        else:
+            if (
+                api_result['type'] == 'https://mediawiki.org/wiki/HyperSwitch/errors/bad_request'
+                and api_result['detail'] == 'title-invalid-characters'
+            ):
+                return []
+
+    _network.raise_for_httperror(resp)
+
+    api_result = resp.json()
+    title = utils.html_to_text(api_result.get('titles', {}).get('display') or api_result.get('title'))
+    wikipedia_link = api_result['content_urls']['desktop']['page']
+    results.append({'url': wikipedia_link, 'title': title, 'content': api_result.get('description', '')})
+
+    if api_result.get('type') == 'standard':
+        results.append(
+            {
+                'infobox': title,
+                'id': wikipedia_link,
+                'content': api_result.get('extract', ''),
+                'img_src': api_result.get('thumbnail', {}).get('source'),
+                'urls': [{'title': 'Wikipedia', 'url': wikipedia_link}],
+            }
+        )
+
+    return results
+
+
+# Nonstandard language codes
+#
+# These Wikipedias use language codes that do not conform to the ISO 639
+# standard (which is how wiki subdomains are chosen nowadays).
+
+lang_map = locales.LOCALE_BEST_MATCH.copy()
+lang_map.update(
+    {
+        'be-tarask': 'bel',
+        'ak': 'aka',
+        'als': 'gsw',
+        'bat-smg': 'sgs',
+        'cbk-zam': 'cbk',
+        'fiu-vro': 'vro',
+        'map-bms': 'map',
+        'no': 'nb-NO',
+        'nrm': 'nrf',
+        'roa-rup': 'rup',
+        'nds-nl': 'nds',
+        #'simple: – invented code used for the Simple English Wikipedia (not the official IETF code en-simple)
+        'zh-min-nan': 'nan',
+        'zh-yue': 'yue',
+        'an': 'arg',
+    }
+)
+
+
+def fetch_traits(engine_traits: EngineTraits):
+    fetch_wikimedia_traits(engine_traits)
+    print("WIKIPEDIA_LANGUAGES: %s" % len(engine_traits.custom['WIKIPEDIA_LANGUAGES']))
+
+
+
[docs]def fetch_wikimedia_traits(engine_traits: EngineTraits):
+    """Fetch languages from Wikipedia.  Not all languages from the
+    :py:obj:`list_of_wikipedias` are supported by SearXNG locales, only those
+    known from :py:obj:`searx.locales.LOCALE_NAMES` or those with a minimal
+    :py:obj:`editing depth <wikipedia_article_depth>`.
+
+    The location of the Wikipedia address of a language is mapped in a
+    :py:obj:`custom field <searx.enginelib.traits.EngineTraits.custom>`
+    (``wiki_netloc``).  Here is a reduced example:
+
+    .. code:: python
+
+       traits.custom['wiki_netloc'] = {
+           "en": "en.wikipedia.org",
+           ..
+           "gsw": "als.wikipedia.org",
+           ..
+           "zh": "zh.wikipedia.org",
+           "zh-classical": "zh-classical.wikipedia.org"
+       }
+    """
+    # pylint: disable=too-many-branches
+    engine_traits.custom['wiki_netloc'] = {}
+    engine_traits.custom['WIKIPEDIA_LANGUAGES'] = []
+
+    # insert alias to map from a script or region to a wikipedia variant
+
+    for eng_tag, sxng_tag_list in wikipedia_script_variants.items():
+        for sxng_tag in sxng_tag_list:
+            engine_traits.languages[sxng_tag] = eng_tag
+    for eng_tag, sxng_tag_list in wiki_lc_locale_variants.items():
+        for sxng_tag in sxng_tag_list:
+            engine_traits.regions[sxng_tag] = eng_tag
+
+    resp = _network.get(list_of_wikipedias)
+    if not resp.ok:
+        print("ERROR: response from Wikipedia is not OK.")
+
+    dom = html.fromstring(resp.text)
+    for row in dom.xpath('//table[contains(@class,"sortable")]//tbody/tr'):
+
+        cols = row.xpath('./td')
+        if not cols:
+            continue
+        cols = [c.text_content().strip() for c in cols]
+
+        depth = float(cols[11].replace('-', '0').replace(',', ''))
+        articles = int(cols[4].replace(',', '').replace(',', ''))
+
+        eng_tag = cols[3]
+        wiki_url = row.xpath('./td[4]/a/@href')[0]
+        wiki_url = urllib.parse.urlparse(wiki_url)
+
+        try:
+            sxng_tag = locales.language_tag(babel.Locale.parse(lang_map.get(eng_tag, eng_tag), sep='-'))
+        except babel.UnknownLocaleError:
+            # print("ERROR: %s [%s] is unknown by babel" % (cols[0], eng_tag))
+            continue
+        finally:
+            engine_traits.custom['WIKIPEDIA_LANGUAGES'].append(eng_tag)
+
+        if sxng_tag not in locales.LOCALE_NAMES:
+
+            if articles < 10000:
+                # exclude languages with too few articles
+                continue
+
+            if int(depth) < 20:
+                # Rough indicator of a Wikipedia’s quality, showing how
+                # frequently its articles are updated.
+                continue
+
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+
+        engine_traits.languages[sxng_tag] = eng_tag
+        engine_traits.custom['wiki_netloc'][eng_tag] = wiki_url.netloc
+
+    engine_traits.custom['WIKIPEDIA_LANGUAGES'].sort()

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/xpath.html b/_modules/searx/engines/xpath.html new file mode 100644 index 000000000..d2ad92ee9 --- /dev/null +++ b/_modules/searx/engines/xpath.html @@ -0,0 +1,423 @@ + + + + + + + + searx.engines.xpath — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.xpath

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""The XPath engine is a *generic* engine with which it is possible to configure
+engines in the settings.
+
+.. _XPath selector: https://quickref.me/xpath.html#xpath-selectors
+
+Configuration
+=============
+
+Request:
+
+- :py:obj:`search_url`
+- :py:obj:`lang_all`
+- :py:obj:`soft_max_redirects`
+- :py:obj:`cookies`
+- :py:obj:`headers`
+
+Paging:
+
+- :py:obj:`paging`
+- :py:obj:`page_size`
+- :py:obj:`first_page_num`
+
+Time Range:
+
+- :py:obj:`time_range_support`
+- :py:obj:`time_range_url`
+- :py:obj:`time_range_map`
+
+Safe-Search:
+
+- :py:obj:`safe_search_support`
+- :py:obj:`safe_search_map`
+
+Response:
+
+- :py:obj:`no_result_for_http_status`
+
+`XPath selector`_:
+
+- :py:obj:`results_xpath`
+- :py:obj:`url_xpath`
+- :py:obj:`title_xpath`
+- :py:obj:`content_xpath`
+- :py:obj:`thumbnail_xpath`
+- :py:obj:`suggestion_xpath`
+
+
+Example
+=======
+
+Here is a simple example of a XPath engine configured in the :ref:`settings
+engine` section, further read :ref:`engines-dev`.
+
+.. code:: yaml
+
+  - name : bitbucket
+    engine : xpath
+    paging : True
+    search_url : https://bitbucket.org/repo/all/{pageno}?name={query}
+    url_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]/@href
+    title_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]
+    content_xpath : //article[@class="repo-summary"]/p
+
+Implementations
+===============
+
+"""
+
+from urllib.parse import urlencode
+
+from lxml import html
+from searx.utils import extract_text, extract_url, eval_xpath, eval_xpath_list
+from searx.network import raise_for_httperror
+
+search_url = None
+"""
+Search URL of the engine.  Example::
+
+    https://example.org/?search={query}&page={pageno}{time_range}{safe_search}
+
+Replacements are:
+
+``{query}``:
+  Search terms from user.
+
+``{pageno}``:
+  Page number if engine supports pagging :py:obj:`paging`
+
+``{lang}``:
+  ISO 639-1 language code (en, de, fr ..)
+
+``{time_range}``:
+  :py:obj:`URL parameter <time_range_url>` if engine :py:obj:`supports time
+  range <time_range_support>`.  The value for the parameter is taken from
+  :py:obj:`time_range_map`.
+
+``{safe_search}``:
+  Safe-search :py:obj:`URL parameter <safe_search_map>` if engine
+  :py:obj:`supports safe-search <safe_search_support>`.  The ``{safe_search}``
+  replacement is taken from the :py:obj:`safes_search_map`.  Filter results::
+
+      0: none, 1: moderate, 2:strict
+
+  If not supported, the URL parameter is an empty string.
+
+"""
+
+lang_all = 'en'
+'''Replacement ``{lang}`` in :py:obj:`search_url` if language ``all`` is
+selected.
+'''
+
+no_result_for_http_status = []
+'''Return empty result for these HTTP status codes instead of throwing an error.
+
+.. code:: yaml
+
+    no_result_for_http_status: []
+'''
+
+soft_max_redirects = 0
+'''Maximum redirects, soft limit. Record an error but don't stop the engine'''
+
+results_xpath = ''
+'''`XPath selector`_ for the list of result items'''
+
+url_xpath = None
+'''`XPath selector`_ of result's ``url``.'''
+
+content_xpath = None
+'''`XPath selector`_ of result's ``content``.'''
+
+title_xpath = None
+'''`XPath selector`_ of result's ``title``.'''
+
+thumbnail_xpath = False
+'''`XPath selector`_ of result's ``img_src``.'''
+
+suggestion_xpath = ''
+'''`XPath selector`_ of result's ``suggestion``.'''
+
+cached_xpath = ''
+cached_url = ''
+
+cookies = {}
+'''Some engines might offer different result based on cookies.
+Possible use-case: To set safesearch cookie.'''
+
+headers = {}
+'''Some engines might offer different result based headers.  Possible use-case:
+To set header to moderate.'''
+
+paging = False
+'''Engine supports paging [True or False].'''
+
+page_size = 1
+'''Number of results on each page.  Only needed if the site requires not a page
+number, but an offset.'''
+
+first_page_num = 1
+'''Number of the first page (usually 0 or 1).'''
+
+time_range_support = False
+'''Engine supports search time range.'''
+
+time_range_url = '&hours={time_range_val}'
+'''Time range URL parameter in the in :py:obj:`search_url`.  If no time range is
+requested by the user, the URL parameter is an empty string.  The
+``{time_range_val}`` replacement is taken from the :py:obj:`time_range_map`.
+
+.. code:: yaml
+
+    time_range_url : '&days={time_range_val}'
+'''
+
+time_range_map = {
+    'day': 24,
+    'week': 24 * 7,
+    'month': 24 * 30,
+    'year': 24 * 365,
+}
+'''Maps time range value from user to ``{time_range_val}`` in
+:py:obj:`time_range_url`.
+
+.. code:: yaml
+
+    time_range_map:
+      day: 1
+      week: 7
+      month: 30
+      year: 365
+'''
+
+safe_search_support = False
+'''Engine supports safe-search.'''
+
+safe_search_map = {0: '&filter=none', 1: '&filter=moderate', 2: '&filter=strict'}
+'''Maps safe-search value to ``{safe_search}`` in :py:obj:`search_url`.
+
+.. code:: yaml
+
+    safesearch: true
+    safes_search_map:
+      0: '&filter=none'
+      1: '&filter=moderate'
+      2: '&filter=strict'
+
+'''
+
+
+
[docs]def request(query, params):
+    '''Build request parameters (see :ref:`engine request`).'''
+    lang = lang_all
+    if params['language'] != 'all':
+        lang = params['language'][:2]
+
+    time_range = ''
+    if params.get('time_range'):
+        time_range_val = time_range_map.get(params.get('time_range'))
+        time_range = time_range_url.format(time_range_val=time_range_val)
+
+    safe_search = ''
+    if params['safesearch']:
+        safe_search = safe_search_map[params['safesearch']]
+
+    fargs = {
+        'query': urlencode({'q': query})[2:],
+        'lang': lang,
+        'pageno': (params['pageno'] - 1) * page_size + first_page_num,
+        'time_range': time_range,
+        'safe_search': safe_search,
+    }
+
+    params['cookies'].update(cookies)
+    params['headers'].update(headers)
+
+    params['url'] = search_url.format(**fargs)
+    params['soft_max_redirects'] = soft_max_redirects
+
+    params['raise_for_httperror'] = False
+
+    return params

+
+
+
[docs]def response(resp):  # pylint: disable=too-many-branches
+    '''Scrap *results* from the response (see :ref:`engine results`).'''
+    if no_result_for_http_status and resp.status_code in no_result_for_http_status:
+        return []
+
+    raise_for_httperror(resp)
+
+    results = []
+    dom = html.fromstring(resp.text)
+    is_onion = 'onions' in categories
+
+    if results_xpath:
+        for result in eval_xpath_list(dom, results_xpath):
+
+            url = extract_url(eval_xpath_list(result, url_xpath, min_len=1), search_url)
+            title = extract_text(eval_xpath_list(result, title_xpath, min_len=1))
+            content = extract_text(eval_xpath_list(result, content_xpath))
+            tmp_result = {'url': url, 'title': title, 'content': content}
+
+            # add thumbnail if available
+            if thumbnail_xpath:
+                thumbnail_xpath_result = eval_xpath_list(result, thumbnail_xpath)
+                if len(thumbnail_xpath_result) > 0:
+                    tmp_result['img_src'] = extract_url(thumbnail_xpath_result, search_url)
+
+            # add alternative cached url if available
+            if cached_xpath:
+                tmp_result['cached_url'] = cached_url + extract_text(eval_xpath_list(result, cached_xpath, min_len=1))
+
+            if is_onion:
+                tmp_result['is_onion'] = True
+
+            results.append(tmp_result)
+
+    else:
+        if cached_xpath:
+            for url, title, content, cached in zip(
+                (extract_url(x, search_url) for x in eval_xpath_list(dom, url_xpath)),
+                map(extract_text, eval_xpath_list(dom, title_xpath)),
+                map(extract_text, eval_xpath_list(dom, content_xpath)),
+                map(extract_text, eval_xpath_list(dom, cached_xpath)),
+            ):
+                results.append(
+                    {
+                        'url': url,
+                        'title': title,
+                        'content': content,
+                        'cached_url': cached_url + cached,
+                        'is_onion': is_onion,
+                    }
+                )
+        else:
+            for url, title, content in zip(
+                (extract_url(x, search_url) for x in eval_xpath_list(dom, url_xpath)),
+                map(extract_text, eval_xpath_list(dom, title_xpath)),
+                map(extract_text, eval_xpath_list(dom, content_xpath)),
+            ):
+                results.append({'url': url, 'title': title, 'content': content, 'is_onion': is_onion})
+
+    if suggestion_xpath:
+        for suggestion in eval_xpath(dom, suggestion_xpath):
+            results.append({'suggestion': extract_text(suggestion)})
+
+    logger.debug("found %s results", len(results))
+    return results

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/yahoo.html b/_modules/searx/engines/yahoo.html new file mode 100644 index 000000000..45f19f848 --- /dev/null +++ b/_modules/searx/engines/yahoo.html @@ -0,0 +1,300 @@ + + + + + + + + searx.engines.yahoo — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.yahoo

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Yahoo Search (Web)
+
+Languages are supported by mapping the language to a domain.  If domain is not
+found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used.
+
+"""
+
+from urllib.parse import (
+    unquote,
+    urlencode,
+)
+from lxml import html
+
+from searx.utils import (
+    eval_xpath_getindex,
+    eval_xpath_list,
+    extract_text,
+)
+from searx.enginelib.traits import EngineTraits
+
+traits: EngineTraits
+
+# about
+about = {
+    "website": 'https://search.yahoo.com/',
+    "wikidata_id": None,
+    "official_api_documentation": 'https://developer.yahoo.com/api/',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+time_range_support = True
+# send_accept_language_header = True
+
+time_range_dict = {
+    'day': ('1d', 'd'),
+    'week': ('1w', 'w'),
+    'month': ('1m', 'm'),
+}
+
+lang2domain = {
+    'zh_chs': 'hk.search.yahoo.com',
+    'zh_cht': 'tw.search.yahoo.com',
+    'any': 'search.yahoo.com',
+    'en': 'search.yahoo.com',
+    'bg': 'search.yahoo.com',
+    'cs': 'search.yahoo.com',
+    'da': 'search.yahoo.com',
+    'el': 'search.yahoo.com',
+    'et': 'search.yahoo.com',
+    'he': 'search.yahoo.com',
+    'hr': 'search.yahoo.com',
+    'ja': 'search.yahoo.com',
+    'ko': 'search.yahoo.com',
+    'sk': 'search.yahoo.com',
+    'sl': 'search.yahoo.com',
+}
+"""Map language to domain"""
+
+locale_aliases = {
+    'zh': 'zh_Hans',
+    'zh-HK': 'zh_Hans',
+    'zh-CN': 'zh_Hans',  # dead since 2015 / routed to hk.search.yahoo.com
+    'zh-TW': 'zh_Hant',
+}
+
+
+
[docs]def request(query, params):
+    """build request"""
+
+    lang = locale_aliases.get(params['language'], None)
+    if not lang:
+        lang = params['language'].split('-')[0]
+    lang = traits.get_language(lang, traits.all_locale)
+
+    offset = (params['pageno'] - 1) * 7 + 1
+    age, btf = time_range_dict.get(params['time_range'], ('', ''))
+
+    args = urlencode(
+        {
+            'p': query,
+            'ei': 'UTF-8',
+            'fl': 1,
+            'vl': 'lang_' + lang,
+            'btf': btf,
+            'fr2': 'time',
+            'age': age,
+            'b': offset,
+            'xargs': 0,
+        }
+    )
+
+    domain = lang2domain.get(lang, '%s.search.yahoo.com' % lang)
+    params['url'] = 'https://%s/search?%s' % (domain, args)
+    return params

+
+
+
[docs]def parse_url(url_string):
+    """remove yahoo-specific tracking-url"""
+
+    endings = ['/RS', '/RK']
+    endpositions = []
+    start = url_string.find('http', url_string.find('/RU=') + 1)
+
+    for ending in endings:
+        endpos = url_string.rfind(ending)
+        if endpos > -1:
+            endpositions.append(endpos)
+
+    if start == 0 or len(endpositions) == 0:
+        return url_string
+
+    end = min(endpositions)
+    return unquote(url_string[start:end])

+
+
+
[docs]def response(resp):
+    """parse response"""
+
+    results = []
+    dom = html.fromstring(resp.text)
+
+    # parse results
+    for result in eval_xpath_list(dom, '//div[contains(@class,"algo-sr")]'):
+        url = eval_xpath_getindex(result, './/h3/a/@href', 0, default=None)
+        if url is None:
+            continue
+        url = parse_url(url)
+
+        title = eval_xpath_getindex(result, './/h3/a', 0, default=None)
+        if title is None:
+            continue
+        offset = len(extract_text(title.xpath('span')))
+        title = extract_text(title)[offset:]
+
+        content = eval_xpath_getindex(result, './/div[contains(@class, "compText")]', 0, default='')
+        content = extract_text(content, allow_none=True)
+
+        # append result
+        results.append({'url': url, 'title': title, 'content': content})
+
+    for suggestion in eval_xpath_list(dom, '//div[contains(@class, "AlsoTry")]//table//a'):
+        # append suggestion
+        results.append({'suggestion': extract_text(suggestion)})
+
+    return results

+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages from yahoo"""
+
+    # pylint: disable=import-outside-toplevel
+    import babel
+    from searx import network
+    from searx.locales import language_tag
+
+    engine_traits.all_locale = 'any'
+
+    resp = network.get('https://search.yahoo.com/preferences/languages')
+    if not resp.ok:
+        print("ERROR: response from peertube is not OK.")
+
+    dom = html.fromstring(resp.text)
+    offset = len('lang_')
+
+    eng2sxng = {'zh_chs': 'zh_Hans', 'zh_cht': 'zh_Hant'}
+
+    for val in eval_xpath_list(dom, '//div[contains(@class, "lang-item")]/input/@value'):
+        eng_tag = val[offset:]
+
+        try:
+            sxng_tag = language_tag(babel.Locale.parse(eng2sxng.get(eng_tag, eng_tag)))
+        except babel.UnknownLocaleError:
+            print('ERROR: unknown language --> %s' % eng_tag)
+            continue
+
+        conflict = engine_traits.languages.get(sxng_tag)
+        if conflict:
+            if conflict != eng_tag:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
+            continue
+        engine_traits.languages[sxng_tag] = eng_tag

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/engines/zlibrary.html b/_modules/searx/engines/zlibrary.html new file mode 100644 index 000000000..e50c803bc --- /dev/null +++ b/_modules/searx/engines/zlibrary.html @@ -0,0 +1,333 @@ + + + + + + + + searx.engines.zlibrary — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.engines.zlibrary

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""`Z-Library`_ (abbreviated as z-lib, formerly BookFinder) is a shadow library
+project for file-sharing access to scholarly journal articles, academic texts
+and general-interest books.  It began as a mirror of Library Genesis, from which
+most of its books originate.
+
+.. _Z-Library: https://zlibrary-global.se/
+
+Configuration
+=============
+
+The engine has the following additional settings:
+
+- :py:obj:`zlib_year_from`
+- :py:obj:`zlib_year_to`
+- :py:obj:`zlib_ext`
+
+With this options a SearXNG maintainer is able to configure **additional**
+engines for specific searches in Z-Library.  For example a engine to search
+only for EPUB from 2010 to 2020.
+
+.. code:: yaml
+
+   - name: z-library 2010s epub
+     engine: zlibrary
+     shortcut: zlib2010s
+     zlib_year_from: '2010'
+     zlib_year_to: '2020'
+     zlib_ext: 'EPUB'
+
+Implementations
+===============
+
+"""
+from __future__ import annotations
+from typing import TYPE_CHECKING
+from typing import List, Dict, Any, Optional
+from datetime import datetime
+from urllib.parse import quote
+from lxml import html
+from flask_babel import gettext
+
+from searx.utils import extract_text, eval_xpath, eval_xpath_list
+from searx.enginelib.traits import EngineTraits
+from searx.data import ENGINE_TRAITS
+
+if TYPE_CHECKING:
+    import httpx
+    import logging
+
+    logger: logging.Logger
+
+# about
+about: Dict[str, Any] = {
+    "website": "https://zlibrary-global.se",
+    "wikidata_id": "Q104863992",
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": "HTML",
+}
+
+categories: List[str] = ["files"]
+paging: bool = True
+base_url: str = "https://zlibrary-global.se"
+
+zlib_year_from: str = ""
+"""Filter z-library's results by year from. E.g '2010'.
+"""
+
+zlib_year_to: str = ""
+"""Filter z-library's results by year to. E.g. '2010'.
+"""
+
+zlib_ext: str = ""
+"""Filter z-library's results by a file ending. Common filters for example are
+``PDF`` and ``EPUB``.
+"""
+
+
+
[docs]def init(engine_settings=None) -> None:  # pylint: disable=unused-argument
+    """Check of engine's settings."""
+    traits: EngineTraits = EngineTraits(**ENGINE_TRAITS["z-library"])
+
+    if zlib_ext and zlib_ext not in traits.custom["ext"]:
+        raise ValueError(f"invalid setting ext: {zlib_ext}")
+    if zlib_year_from and zlib_year_from not in traits.custom["year_from"]:
+        raise ValueError(f"invalid setting year_from: {zlib_year_from}")
+    if zlib_year_to and zlib_year_to not in traits.custom["year_to"]:
+        raise ValueError(f"invalid setting year_to: {zlib_year_to}")

+
+
+def request(query: str, params: Dict[str, Any]) -> Dict[str, Any]:
+    lang: str = traits.get_language(params["language"], traits.all_locale)  # type: ignore
+    search_url: str = (
+        base_url
+        + "/s/{search_query}/?page={pageno}"
+        + "&yearFrom={zlib_year_from}"
+        + "&yearTo={zlib_year_to}"
+        + "&languages[]={lang}"
+        + "&extensions[]={zlib_ext}"
+    )
+    params["url"] = search_url.format(
+        search_query=quote(query),
+        pageno=params["pageno"],
+        lang=lang,
+        zlib_year_from=zlib_year_from,
+        zlib_year_to=zlib_year_to,
+        zlib_ext=zlib_ext,
+    )
+    return params
+
+
+def response(resp: httpx.Response) -> List[Dict[str, Any]]:
+    results: List[Dict[str, Any]] = []
+    dom = html.fromstring(resp.text)
+
+    for item in dom.xpath('//div[@id="searchResultBox"]//div[contains(@class, "resItemBox")]'):
+        results.append(_parse_result(item))
+
+    return results
+
+
+def _text(item, selector: str) -> str | None:
+    return extract_text(eval_xpath(item, selector))
+
+
+i18n_language = gettext("Language")
+i18n_book_rating = gettext("Book rating")
+i18n_file_quality = gettext("File quality")
+
+
+def _parse_result(item) -> Dict[str, Any]:
+
+    author_elements = eval_xpath_list(item, './/div[@class="authors"]//a[@itemprop="author"]')
+
+    result = {
+        "template": "paper.html",
+        "url": base_url + item.xpath('(.//a[starts-with(@href, "/book/")])[1]/@href')[0],
+        "title": _text(item, './/*[@itemprop="name"]'),
+        "authors": [extract_text(author) for author in author_elements],
+        "publisher": _text(item, './/a[@title="Publisher"]'),
+        "type": _text(item, './/div[contains(@class, "property__file")]//div[contains(@class, "property_value")]'),
+        "img_src": _text(item, './/img[contains(@class, "cover")]/@data-src'),
+    }
+
+    year = _text(item, './/div[contains(@class, "property_year")]//div[contains(@class, "property_value")]')
+    if year:
+        result["publishedDate"] = datetime.strptime(year, '%Y')
+
+    content = []
+    language = _text(item, './/div[contains(@class, "property_language")]//div[contains(@class, "property_value")]')
+    if language:
+        content.append(f"{i18n_language}: {language.capitalize()}")
+    book_rating = _text(item, './/span[contains(@class, "book-rating-interest-score")]')
+    if book_rating and float(book_rating):
+        content.append(f"{i18n_book_rating}: {book_rating}")
+    file_quality = _text(item, './/span[contains(@class, "book-rating-quality-score")]')
+    if file_quality and float(file_quality):
+        content.append(f"{i18n_file_quality}: {file_quality}")
+    result["content"] = " | ".join(content)
+
+    return result
+
+
+
[docs]def fetch_traits(engine_traits: EngineTraits) -> None:
+    """Fetch languages and other search arguments from zlibrary's search form."""
+    # pylint: disable=import-outside-toplevel
+
+    import babel
+    from searx.network import get  # see https://github.com/searxng/searxng/issues/762
+    from searx.locales import language_tag
+
+    engine_traits.all_locale = ""
+    engine_traits.custom["ext"] = []
+    engine_traits.custom["year_from"] = []
+    engine_traits.custom["year_to"] = []
+
+    resp = get(base_url)
+    if not resp.ok:  # type: ignore
+        raise RuntimeError("Response from zlibrary's search page is not OK.")
+    dom = html.fromstring(resp.text)  # type: ignore
+
+    for year in eval_xpath_list(dom, "//div[@id='advSearch-noJS']//select[@id='sf_yearFrom']/option"):
+        engine_traits.custom["year_from"].append(year.get("value"))
+
+    for year in eval_xpath_list(dom, "//div[@id='advSearch-noJS']//select[@id='sf_yearTo']/option"):
+        engine_traits.custom["year_to"].append(year.get("value"))
+
+    for ext in eval_xpath_list(dom, "//div[@id='advSearch-noJS']//select[@id='sf_extensions']/option"):
+        value: Optional[str] = ext.get("value")
+        if value is None:
+            value = ""
+        engine_traits.custom["ext"].append(value)
+
+    # Handle languages
+    # Z-library uses English names for languages, so we need to map them to their respective locales
+    language_name_locale_map: Dict[str, babel.Locale] = {}
+    for locale in babel.core.localedata.locale_identifiers():  # type: ignore
+        # Create a Locale object for the current locale
+        loc = babel.Locale.parse(locale)
+        language_name_locale_map[loc.english_name.lower()] = loc  # type: ignore
+
+    for x in eval_xpath_list(dom, "//div[@id='advSearch-noJS']//select[@id='sf_languages']/option"):
+        eng_lang = x.get("value")
+        if eng_lang is None:
+            continue
+        try:
+            locale = language_name_locale_map[eng_lang.lower()]
+        except KeyError:
+            # silently ignore unknown languages
+            # print("ERROR: %s is unknown by babel" % (eng_lang))
+            continue
+        sxng_lang = language_tag(locale)
+        conflict = engine_traits.languages.get(sxng_lang)
+        if conflict:
+            if conflict != eng_lang:
+                print("CONFLICT: babel %s --> %s, %s" % (sxng_lang, conflict, eng_lang))
+            continue
+        engine_traits.languages[sxng_lang] = eng_lang

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/exceptions.html b/_modules/searx/exceptions.html new file mode 100644 index 000000000..e7a71e26f --- /dev/null +++ b/_modules/searx/exceptions.html @@ -0,0 +1,226 @@ + + + + + + + + searx.exceptions — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.exceptions

+# -*- coding: utf-8 -*-
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Exception types raised by SearXNG modules.
+"""
+
+from typing import Optional, Union
+
+
+
[docs]class SearxException(Exception):
+    """Base SearXNG exception."""

+
+
+
[docs]class SearxParameterException(SearxException):
+    """Raised when query miss a required paramater"""
+
+    def __init__(self, name, value):
+        if value == '' or value is None:
+            message = 'Empty ' + name + ' parameter'
+        else:
+            message = 'Invalid value "' + value + '" for parameter ' + name
+        super().__init__(message)
+        self.message = message
+        self.parameter_name = name
+        self.parameter_value = value

+
+
+
[docs]class SearxSettingsException(SearxException):
+    """Error while loading the settings"""
+
+    def __init__(self, message: Union[str, Exception], filename: Optional[str]):
+        super().__init__(message)
+        self.message = message
+        self.filename = filename

+
+
+
[docs]class SearxEngineException(SearxException):
+    """Error inside an engine"""

+
+
+
[docs]class SearxXPathSyntaxException(SearxEngineException):
+    """Syntax error in a XPATH"""
+
+    def __init__(self, xpath_spec, message):
+        super().__init__(str(xpath_spec) + " " + message)
+        self.message = message
+        # str(xpath_spec) to deal with str and XPath instance
+        self.xpath_str = str(xpath_spec)

+
+
+
[docs]class SearxEngineResponseException(SearxEngineException):
+    """Impossible to parse the result of an engine"""

+
+
+
[docs]class SearxEngineAPIException(SearxEngineResponseException):
+    """The website has returned an application error"""

+
+
+
[docs]class SearxEngineAccessDeniedException(SearxEngineResponseException):
+    """The website is blocking the access"""
+
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineAccessDenied"
+    """This settings contains the default suspended time (default 86400 sec / 1
+    day)."""
+
+    def __init__(self, suspended_time: int = None, message: str = 'Access denied'):
+        """Generic exception to raise when an engine denies access to the results.
+
+        :param suspended_time: How long the engine is going to be suspended in
+            second. Defaults to None.
+        :type suspended_time: int, None
+        :param message: Internal message.  Defaults to ``Access denied``
+        :type message: str
+        """
+        suspended_time = suspended_time or self._get_default_suspended_time()
+        super().__init__(message + ', suspended_time=' + str(suspended_time))
+        self.suspended_time = suspended_time
+        self.message = message
+
+    def _get_default_suspended_time(self):
+        from searx import get_setting  # pylint: disable=C0415
+
+        return get_setting(self.SUSPEND_TIME_SETTING)

+
+
+
[docs]class SearxEngineCaptchaException(SearxEngineAccessDeniedException):
+    """The website has returned a CAPTCHA."""
+
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineCaptcha"
+    """This settings contains the default suspended time (default 86400 sec / 1
+    day)."""
+
+    def __init__(self, suspended_time=None, message='CAPTCHA'):
+        super().__init__(message=message, suspended_time=suspended_time)

+
+
+
[docs]class SearxEngineTooManyRequestsException(SearxEngineAccessDeniedException):
+    """The website has returned a Too Many Request status code
+
+    By default, searx stops sending requests to this engine for 1 hour.
+    """
+
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineTooManyRequests"
+    """This settings contains the default suspended time (default 3660 sec / 1
+    hour)."""
+
+    def __init__(self, suspended_time=None, message='Too many request'):
+        super().__init__(message=message, suspended_time=suspended_time)

+
+
+
[docs]class SearxEngineXPathException(SearxEngineResponseException):
+    """Error while getting the result of an XPath expression"""
+
+    def __init__(self, xpath_spec, message):
+        super().__init__(str(xpath_spec) + " " + message)
+        self.message = message
+        # str(xpath_spec) to deal with str and XPath instance
+        self.xpath_str = str(xpath_spec)

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/infopage.html b/_modules/searx/infopage.html new file mode 100644 index 000000000..25617d7e9 --- /dev/null +++ b/_modules/searx/infopage.html @@ -0,0 +1,295 @@ + + + + + + + + searx.infopage — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.infopage

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+# pyright: basic
+"""Render SearXNG instance documentation.
+
+Usage in a Flask app route:
+
+.. code:: python
+
+  from searx import infopage
+
+  _INFO_PAGES = infopage.InfoPageSet(infopage.MistletoePage)
+
+  @app.route('/info/<pagename>', methods=['GET'])
+  def info(pagename):
+
+      locale = request.preferences.get_value('locale')
+      page = _INFO_PAGES.get_page(pagename, locale)
+
+"""
+
+__all__ = ['InfoPage', 'InfoPageSet']
+
+import os
+import os.path
+import logging
+import typing
+
+import urllib.parse
+import jinja2
+from flask.helpers import url_for
+from markdown_it import MarkdownIt
+
+from .. import get_setting
+from ..compat import cached_property
+from ..version import GIT_URL
+from ..locales import LOCALE_NAMES
+
+
+logger = logging.getLogger('searx.infopage')
+_INFO_FOLDER = os.path.abspath(os.path.dirname(__file__))
+
+
+
[docs]class InfoPage:
+    """A page of the :py:obj:`online documentation <InfoPageSet>`."""
+
+    def __init__(self, fname):
+        self.fname = fname
+
+    @cached_property
+    def raw_content(self):
+        """Raw content of the page (without any jinja rendering)"""
+        with open(self.fname, 'r', encoding='utf-8') as f:
+            return f.read()
+
+    @cached_property
+    def content(self):
+        """Content of the page (rendered in a Jinja conntext)"""
+        ctx = self.get_ctx()
+        template = jinja2.Environment().from_string(self.raw_content)
+        return template.render(**ctx)
+
+    @cached_property
+    def title(self):
+        """Title of the content (without any markup)"""
+        t = ""
+        for l in self.raw_content.split('\n'):
+            if l.startswith('# '):
+                t = l.strip('# ')
+        return t
+
+    @cached_property
+    def html(self):
+        """Render Markdown (CommonMark_) to HTML by using markdown-it-py_.
+
+        .. _CommonMark: https://commonmark.org/
+        .. _markdown-it-py: https://github.com/executablebooks/markdown-it-py
+
+        """
+        return (
+            MarkdownIt("commonmark", {"typographer": True}).enable(["replacements", "smartquotes"]).render(self.content)
+        )
+
+
[docs]    def get_ctx(self):
+        """Jinja context to render :py:obj:`InfoPage.content`"""
+
+        def _md_link(name, url):
+            url = url_for(url, _external=True)
+            return "[%s](%s)" % (name, url)
+
+        def _md_search(query):
+            url = '%s?q=%s' % (url_for('search', _external=True), urllib.parse.quote(query))
+            return '[%s](%s)' % (query, url)
+
+        ctx = {}
+        ctx['GIT_URL'] = GIT_URL
+        ctx['get_setting'] = get_setting
+        ctx['link'] = _md_link
+        ctx['search'] = _md_search
+
+        return ctx

+
+    def __repr__(self):
+        return f'<{self.__class__.__name__} fname={self.fname!r}>'

+
+
+
[docs]class InfoPageSet:  # pylint: disable=too-few-public-methods
+    """Cached rendering of the online documentation a SearXNG instance has.
+
+    :param page_class: render online documentation by :py:obj:`InfoPage` parser.
+    :type page_class: :py:obj:`InfoPage`
+
+    :param info_folder: information directory
+    :type info_folder: str
+    """
+
+    def __init__(
+        self, page_class: typing.Optional[typing.Type[InfoPage]] = None, info_folder: typing.Optional[str] = None
+    ):
+        self.page_class = page_class or InfoPage
+        self.folder: str = info_folder or _INFO_FOLDER
+        """location of the Markdwon files"""
+
+        self.CACHE: typing.Dict[tuple, typing.Optional[InfoPage]] = {}
+
+        self.locale_default: str = 'en'
+        """default language"""
+
+        self.locales: typing.List[str] = [
+            locale.replace('_', '-') for locale in os.listdir(_INFO_FOLDER) if locale.replace('_', '-') in LOCALE_NAMES
+        ]
+        """list of supported languages (aka locales)"""
+
+        self.toc: typing.List[str] = [
+            'search-syntax',
+            'about',
+            'donate',
+        ]
+        """list of articles in the online documentation"""
+
+
[docs]    def get_page(self, pagename: str, locale: typing.Optional[str] = None):
+        """Return ``pagename`` instance of :py:obj:`InfoPage`
+
+        :param pagename: name of the page, a value from :py:obj:`InfoPageSet.toc`
+        :type pagename: str
+
+        :param locale: language of the page, e.g. ``en``, ``zh_Hans_CN``
+                       (default: :py:obj:`InfoPageSet.i18n_origin`)
+        :type locale: str
+
+        """
+        locale = locale or self.locale_default
+
+        if pagename not in self.toc:
+            return None
+        if locale not in self.locales:
+            return None
+
+        cache_key = (pagename, locale)
+
+        if cache_key in self.CACHE:
+            return self.CACHE[cache_key]
+
+        # not yet instantiated
+
+        fname = os.path.join(self.folder, locale.replace('-', '_'), pagename) + '.md'
+        if not os.path.exists(fname):
+            logger.info('file %s does not exists', fname)
+            self.CACHE[cache_key] = None
+            return None
+
+        page = self.page_class(fname)
+        self.CACHE[cache_key] = page
+        return page

+
+
[docs]    def iter_pages(self, locale: typing.Optional[str] = None, fallback_to_default=False):
+        """Iterate over all pages of the TOC"""
+        locale = locale or self.locale_default
+        for page_name in self.toc:
+            page_locale = locale
+            page = self.get_page(page_name, locale)
+            if fallback_to_default and page is None:
+                page_locale = self.locale_default
+                page = self.get_page(page_name, self.locale_default)
+            if page is not None:
+                # page is None if the page was deleted by the administrator
+                yield page_name, page_locale, page

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/locales.html b/_modules/searx/locales.html new file mode 100644 index 000000000..bc9a0b1fa --- /dev/null +++ b/_modules/searx/locales.html @@ -0,0 +1,579 @@ + + + + + + + + searx.locales — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.locales

+# -*- coding: utf-8 -*-
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Initialize :py:obj:`LOCALE_NAMES`, :py:obj:`RTL_LOCALES`.
+"""
+
+from typing import Set, Optional, List
+import os
+import pathlib
+
+import babel
+from babel.support import Translations
+import babel.languages
+import babel.core
+import flask_babel
+import flask
+from flask.ctx import has_request_context
+from searx import logger
+
+logger = logger.getChild('locales')
+
+
+# safe before monkey patching flask_babel.get_translations
+_flask_babel_get_translations = flask_babel.get_translations
+
+LOCALE_NAMES = {}
+"""Mapping of locales and their description.  Locales e.g. 'fr' or 'pt-BR' (see
+:py:obj:`locales_initialize`).
+
+:meta hide-value:
+"""
+
+RTL_LOCALES: Set[str] = set()
+"""List of *Right-To-Left* locales e.g. 'he' or 'fa-IR' (see
+:py:obj:`locales_initialize`)."""
+
+ADDITIONAL_TRANSLATIONS = {
+    "dv": "ދިވެހި (Dhivehi)",
+    "oc": "Occitan",
+    "szl": "Ślōnski (Silesian)",
+    "pap": "Papiamento",
+}
+"""Additional languages SearXNG has translations for but not supported by
+python-babel (see :py:obj:`locales_initialize`)."""
+
+LOCALE_BEST_MATCH = {
+    "dv": "si",
+    "oc": 'fr-FR',
+    "szl": "pl",
+    "nl-BE": "nl",
+    "zh-HK": "zh-Hant-TW",
+    "pap": "pt-BR",
+}
+"""Map a locale we do not have a translations for to a locale we have a
+translation for. By example: use Taiwan version of the translation for Hong
+Kong."""
+
+
+def localeselector():
+    locale = 'en'
+    if has_request_context():
+        value = flask.request.preferences.get_value('locale')
+        if value:
+            locale = value
+
+    # first, set the language that is not supported by babel
+    if locale in ADDITIONAL_TRANSLATIONS:
+        flask.request.form['use-translation'] = locale
+
+    # second, map locale to a value python-babel supports
+    locale = LOCALE_BEST_MATCH.get(locale, locale)
+
+    if locale == '':
+        # if there is an error loading the preferences
+        # the locale is going to be ''
+        locale = 'en'
+
+    # babel uses underscore instead of hyphen.
+    locale = locale.replace('-', '_')
+    return locale
+
+
+
[docs]def get_translations():
+    """Monkey patch of :py:obj:`flask_babel.get_translations`"""
+    if has_request_context():
+        use_translation = flask.request.form.get('use-translation')
+        if use_translation in ADDITIONAL_TRANSLATIONS:
+            babel_ext = flask_babel.current_app.extensions['babel']
+            return Translations.load(babel_ext.translation_directories[0], use_translation)
+    return _flask_babel_get_translations()

+
+
+
[docs]def get_locale_descr(locale, locale_name):
+    """Get locale name e.g. 'Français - fr' or 'Português (Brasil) - pt-BR'
+
+    :param locale: instance of :py:class:`Locale`
+    :param locale_name: name e.g. 'fr'  or 'pt_BR' (delimiter is *underscore*)
+    """
+
+    native_language, native_territory = _get_locale_descr(locale, locale_name)
+    english_language, english_territory = _get_locale_descr(locale, 'en')
+
+    if native_territory == english_territory:
+        english_territory = None
+
+    if not native_territory and not english_territory:
+        if native_language == english_language:
+            return native_language
+        return native_language + ' (' + english_language + ')'
+
+    result = native_language + ', ' + native_territory + ' (' + english_language
+    if english_territory:
+        return result + ', ' + english_territory + ')'
+    return result + ')'

+
+
+def _get_locale_descr(locale, language_code):
+    language_name = locale.get_language_name(language_code).capitalize()
+    if language_name and ('a' <= language_name[0] <= 'z'):
+        language_name = language_name.capitalize()
+    terrirtory_name = locale.get_territory_name(language_code)
+    return language_name, terrirtory_name
+
+
+
[docs]def locales_initialize(directory=None):
+    """Initialize locales environment of the SearXNG session.
+
+    - monkey patch :py:obj:`flask_babel.get_translations` by :py:obj:`get_translations`
+    - init global names :py:obj:`LOCALE_NAMES`, :py:obj:`RTL_LOCALES`
+    """
+
+    directory = directory or pathlib.Path(__file__).parent / 'translations'
+    logger.debug("locales_initialize: %s", directory)
+    flask_babel.get_translations = get_translations
+
+    for tag, descr in ADDITIONAL_TRANSLATIONS.items():
+        locale = babel.Locale.parse(LOCALE_BEST_MATCH[tag], sep='-')
+        LOCALE_NAMES[tag] = descr
+        if locale.text_direction == 'rtl':
+            RTL_LOCALES.add(tag)
+
+    for tag in LOCALE_BEST_MATCH:
+        descr = LOCALE_NAMES.get(tag)
+        if not descr:
+            locale = babel.Locale.parse(tag, sep='-')
+            LOCALE_NAMES[tag] = get_locale_descr(locale, tag.replace('-', '_'))
+            if locale.text_direction == 'rtl':
+                RTL_LOCALES.add(tag)
+
+    for dirname in sorted(os.listdir(directory)):
+        # Based on https://flask-babel.tkte.ch/_modules/flask_babel.html#Babel.list_translations
+        if not os.path.isdir(os.path.join(directory, dirname, 'LC_MESSAGES')):
+            continue
+        tag = dirname.replace('_', '-')
+        descr = LOCALE_NAMES.get(tag)
+        if not descr:
+            locale = babel.Locale.parse(dirname)
+            LOCALE_NAMES[tag] = get_locale_descr(locale, dirname)
+            if locale.text_direction == 'rtl':
+                RTL_LOCALES.add(tag)

+
+
+
[docs]def region_tag(locale: babel.Locale) -> str:
+    """Returns SearXNG's region tag from the locale (e.g. zh-TW , en-US)."""
+    if not locale.territory:
+        raise ValueError('%s missed a territory')
+    return locale.language + '-' + locale.territory

+
+
+
[docs]def language_tag(locale: babel.Locale) -> str:
+    """Returns SearXNG's language tag from the locale and if exits, the tag
+    includes the script name (e.g. en, zh_Hant).
+    """
+    sxng_lang = locale.language
+    if locale.script:
+        sxng_lang += '_' + locale.script
+    return sxng_lang

+
+
+
[docs]def get_locale(locale_tag: str) -> Optional[babel.Locale]:
+    """Returns a :py:obj:`babel.Locale` object parsed from argument
+    ``locale_tag``"""
+    try:
+        locale = babel.Locale.parse(locale_tag, sep='-')
+        return locale
+
+    except babel.core.UnknownLocaleError:
+        return None

+
+
+
[docs]def get_offical_locales(
+    territory: str, languages=None, regional: bool = False, de_facto: bool = True
+) -> Set[babel.Locale]:
+    """Returns a list of :py:obj:`babel.Locale` with languages from
+    :py:obj:`babel.languages.get_official_languages`.
+
+    :param territory: The territory (country or region) code.
+
+    :param languages: A list of language codes the languages from
+      :py:obj:`babel.languages.get_official_languages` should be in
+      (intersection).  If this argument is ``None``, all official languages in
+      this territory are used.
+
+    :param regional: If the regional flag is set, then languages which are
+      regionally official are also returned.
+
+    :param de_facto: If the de_facto flag is set to `False`, then languages
+      which are “de facto” official are not returned.
+
+    """
+    ret_val = set()
+    o_languages = babel.languages.get_official_languages(territory, regional=regional, de_facto=de_facto)
+
+    if languages:
+        languages = [l.lower() for l in languages]
+        o_languages = set(l for l in o_languages if l.lower() in languages)
+
+    for lang in o_languages:
+        try:
+            locale = babel.Locale.parse(lang + '_' + territory)
+            ret_val.add(locale)
+        except babel.UnknownLocaleError:
+            continue
+
+    return ret_val

+
+
+
[docs]def get_engine_locale(searxng_locale, engine_locales, default=None):
+    """Return engine's language (aka locale) string that best fits to argument
+    ``searxng_locale``.
+
+    Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
+    corresponding *engine locales*::
+
+      <engine>: {
+          # SearXNG string : engine-string
+          'ca-ES'          : 'ca_ES',
+          'fr-BE'          : 'fr_BE',
+          'fr-CA'          : 'fr_CA',
+          'fr-CH'          : 'fr_CH',
+          'fr'             : 'fr_FR',
+          ...
+          'pl-PL'          : 'pl_PL',
+          'pt-PT'          : 'pt_PT'
+          ..
+          'zh'             : 'zh'
+          'zh_Hans'        : 'zh'
+          'zh_Hant'        : 'zh_TW'
+      }
+
+    .. hint::
+
+       The *SearXNG locale* string has to be known by babel!
+
+    If there is no direct 1:1 mapping, this functions tries to narrow down
+    engine's language (locale).  If no value can be determined by these
+    approximation attempts the ``default`` value is returned.
+
+    Assumptions:
+
+    A. When user select a language the results should be optimized according to
+       the selected language.
+
+    B. When user select a language and a territory the results should be
+       optimized with first priority on terrirtory and second on language.
+
+    First approximation rule (*by territory*):
+
+      When the user selects a locale with terrirtory (and a language), the
+      territory has priority over the language.  If any of the offical languages
+      in the terrirtory is supported by the engine (``engine_locales``) it will
+      be used.
+
+    Second approximation rule (*by language*):
+
+      If "First approximation rule" brings no result or the user selects only a
+      language without a terrirtory.  Check in which territories the language
+      has an offical status and if one of these territories is supported by the
+      engine.
+
+    """
+    # pylint: disable=too-many-branches, too-many-return-statements
+
+    engine_locale = engine_locales.get(searxng_locale)
+
+    if engine_locale is not None:
+        # There was a 1:1 mapping (e.g. a region "fr-BE --> fr_BE" or a language
+        # "zh --> zh"), no need to narrow language-script nor territory.
+        return engine_locale
+
+    try:
+        locale = babel.Locale.parse(searxng_locale, sep='-')
+    except babel.core.UnknownLocaleError:
+        try:
+            locale = babel.Locale.parse(searxng_locale.split('-')[0])
+        except babel.core.UnknownLocaleError:
+            return default
+
+    searxng_lang = language_tag(locale)
+    engine_locale = engine_locales.get(searxng_lang)
+    if engine_locale is not None:
+        # There was a 1:1 mapping (e.g. "zh-HK --> zh_Hant" or "zh-CN --> zh_Hans")
+        return engine_locale
+
+    # SearXNG's selected locale is not supported by the engine ..
+
+    if locale.territory:
+        # Try to narrow by *offical* languages in the territory (??-XX).
+
+        for official_language in babel.languages.get_official_languages(locale.territory, de_facto=True):
+            searxng_locale = official_language + '-' + locale.territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+    # Engine does not support one of the offical languages in the territory or
+    # there is only a language selected without a territory.
+
+    # Now lets have a look if the searxng_lang (the language selected by the
+    # user) is a offical language in other territories.  If so, check if
+    # engine does support the searxng_lang in this other territory.
+
+    if locale.language:
+
+        terr_lang_dict = {}
+        for territory, langs in babel.core.get_global("territory_languages").items():
+            if not langs.get(searxng_lang, {}).get('official_status'):
+                continue
+            terr_lang_dict[territory] = langs.get(searxng_lang)
+
+        # first: check fr-FR, de-DE .. is supported by the engine
+        # exception: 'en' --> 'en-US'
+
+        territory = locale.language.upper()
+        if territory == 'EN':
+            territory = 'US'
+
+        if terr_lang_dict.get(territory):
+            searxng_locale = locale.language + '-' + territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+        # second: sort by population_percent and take first match
+
+        # drawback of "population percent": if there is a terrirtory with a
+        #   small number of people (e.g 100) but the majority speaks the
+        #   language, then the percentage migth be 100% (--> 100 people) but in
+        #   a different terrirtory with more people (e.g. 10.000) where only 10%
+        #   speak the language the total amount of speaker is higher (--> 200
+        #   people).
+        #
+        #   By example: The population of Saint-Martin is 33.000, of which 100%
+        #   speak French, but this is less than the 30% of the approximately 2.5
+        #   million Belgian citizens
+        #
+        #   - 'fr-MF', 'population_percent': 100.0, 'official_status': 'official'
+        #   - 'fr-BE', 'population_percent': 38.0, 'official_status': 'official'
+
+        terr_lang_list = []
+        for k, v in terr_lang_dict.items():
+            terr_lang_list.append((k, v))
+
+        for territory, _lang in sorted(terr_lang_list, key=lambda item: item[1]['population_percent'], reverse=True):
+            searxng_locale = locale.language + '-' + territory
+            engine_locale = engine_locales.get(searxng_locale)
+            if engine_locale is not None:
+                return engine_locale
+
+    # No luck: narrow by "language from territory" and "territory from language"
+    # does not fit to a locale supported by the engine.
+
+    if engine_locale is None:
+        engine_locale = default
+
+    return default

+
+
+
[docs]def match_locale(searxng_locale: str, locale_tag_list: List[str], fallback: Optional[str] = None) -> Optional[str]:
+    """Return tag from ``locale_tag_list`` that best fits to ``searxng_locale``.
+
+    :param str searxng_locale: SearXNG's internal representation of locale (de,
+        de-DE, fr-BE, zh, zh-CN, zh-TW ..).
+
+    :param list locale_tag_list: The list of locale tags to select from
+
+    :param str fallback: fallback locale tag (if unset --> ``None``)
+
+    The rules to find a match are implemented in :py:obj:`get_engine_locale`,
+    the ``engine_locales`` is build up by :py:obj:`build_engine_locales`.
+
+    .. hint::
+
+       The *SearXNG locale* string and the members of ``locale_tag_list`` has to
+       be known by babel!  The :py:obj:`ADDITIONAL_TRANSLATIONS` are used in the
+       UI and are not known by babel --> will be ignored.
+    """
+
+    # searxng_locale = 'es'
+    # locale_tag_list = ['es-AR', 'es-ES', 'es-MX']
+
+    if not searxng_locale:
+        return fallback
+
+    locale = get_locale(searxng_locale)
+    if locale is None:
+        return fallback
+
+    # normalize to a SearXNG locale that can be passed to get_engine_locale
+
+    searxng_locale = language_tag(locale)
+    if locale.territory:
+        searxng_locale = region_tag(locale)
+
+    # clean up locale_tag_list
+
+    tag_list = []
+    for tag in locale_tag_list:
+        if tag in ('all', 'auto') or tag in ADDITIONAL_TRANSLATIONS:
+            continue
+        tag_list.append(tag)
+
+    # emulate fetch_traits
+    engine_locales = build_engine_locales(tag_list)
+    return get_engine_locale(searxng_locale, engine_locales, default=fallback)

+
+
+
[docs]def build_engine_locales(tag_list: List[str]):
+    """From a list of locale tags a dictionary is build that can be passed by
+    argument ``engine_locales`` to :py:obj:`get_engine_locale`.  This function
+    is mainly used by :py:obj:`match_locale` and is similar to what the
+    ``fetch_traits(..)`` function of engines do.
+
+    If there are territory codes in the ``tag_list`` that have a *script code*
+    additional keys are added to the returned dictionary.
+
+    .. code:: python
+
+       >>> import locales
+       >>> engine_locales = locales.build_engine_locales(['en', 'en-US', 'zh', 'zh-CN', 'zh-TW'])
+       >>> engine_locales
+       {
+           'en': 'en', 'en-US': 'en-US',
+           'zh': 'zh', 'zh-CN': 'zh-CN', 'zh_Hans': 'zh-CN',
+           'zh-TW': 'zh-TW', 'zh_Hant': 'zh-TW'
+       }
+       >>> get_engine_locale('zh-Hans', engine_locales)
+       'zh-CN'
+
+    This function is a good example to understand the language/region model
+    of SearXNG:
+
+      SearXNG only distinguishes between **search languages** and **search
+      regions**, by adding the *script-tags*, languages with *script-tags* can
+      be assigned to the **regions** that SearXNG supports.
+
+    """
+    engine_locales = {}
+
+    for tag in tag_list:
+        locale = get_locale(tag)
+        if locale is None:
+            logger.warning("build_engine_locales: skip locale tag %s / unknown by babel", tag)
+            continue
+        if locale.territory:
+            engine_locales[region_tag(locale)] = tag
+            if locale.script:
+                engine_locales[language_tag(locale)] = tag
+        else:
+            engine_locales[language_tag(locale)] = tag
+    return engine_locales

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/redislib.html b/_modules/searx/redislib.html new file mode 100644 index 000000000..b8b82df60 --- /dev/null +++ b/_modules/searx/redislib.html @@ -0,0 +1,349 @@ + + + + + + + + searx.redislib — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.redislib

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""A collection of convenient functions and redis/lua scripts.
+
+This code was partial inspired by the `Bullet-Proofing Lua Scripts in RedisPy`_
+article.
+
+.. _Bullet-Proofing Lua Scripts in RedisPy:
+   https://redis.com/blog/bullet-proofing-lua-scripts-in-redispy/
+
+"""
+
+import hmac
+
+from searx import get_setting
+
+LUA_SCRIPT_STORAGE = {}
+"""A global dictionary to cache client's ``Script`` objects, used by
+:py:obj:`lua_script_storage`"""
+
+
+
[docs]def lua_script_storage(client, script):
+    """Returns a redis :py:obj:`Script
+    <redis.commands.core.CoreCommands.register_script>` instance.
+
+    Due to performance reason the ``Script`` object is instantiated only once
+    for a client (``client.register_script(..)``) and is cached in
+    :py:obj:`LUA_SCRIPT_STORAGE`.
+
+    """
+
+    # redis connection can be closed, lets use the id() of the redis connector
+    # as key in the script-storage:
+    client_id = id(client)
+
+    if LUA_SCRIPT_STORAGE.get(client_id) is None:
+        LUA_SCRIPT_STORAGE[client_id] = {}
+
+    if LUA_SCRIPT_STORAGE[client_id].get(script) is None:
+        LUA_SCRIPT_STORAGE[client_id][script] = client.register_script(script)
+
+    return LUA_SCRIPT_STORAGE[client_id][script]

+
+
+PURGE_BY_PREFIX = """
+local prefix = tostring(ARGV[1])
+for i, name in ipairs(redis.call('KEYS', prefix .. '*')) do
+    redis.call('EXPIRE', name, 0)
+end
+"""
+
+
+
[docs]def purge_by_prefix(client, prefix: str = "SearXNG_"):
+    """Purge all keys with ``prefix`` from database.
+
+    Queries all keys in the database by the given prefix and set expire time to
+    zero.  The default prefix will drop all keys which has been set by SearXNG
+    (drops SearXNG schema entirely from database).
+
+    The implementation is the lua script from string :py:obj:`PURGE_BY_PREFIX`.
+    The lua script uses EXPIRE_ instead of DEL_: if there are a lot keys to
+    delete and/or their values are big, `DEL` could take more time and blocks
+    the command loop while `EXPIRE` turns back immediate.
+
+    :param prefix: prefix of the key to delete (default: ``SearXNG_``)
+    :type name: str
+
+    .. _EXPIRE: https://redis.io/commands/expire/
+    .. _DEL: https://redis.io/commands/del/
+
+    """
+    script = lua_script_storage(client, PURGE_BY_PREFIX)
+    script(args=[prefix])

+
+
+
[docs]def secret_hash(name: str):
+    """Creates a hash of the ``name``.
+
+    Combines argument ``name`` with the ``secret_key`` from :ref:`settings
+    server`.  This function can be used to get a more anonymised name of a Redis
+    KEY.
+
+    :param name: the name to create a secret hash for
+    :type name: str
+    """
+    m = hmac.new(bytes(name, encoding='utf-8'), digestmod='sha256')
+    m.update(bytes(get_setting('server.secret_key'), encoding='utf-8'))
+    return m.hexdigest()

+
+
+INCR_COUNTER = """
+local limit = tonumber(ARGV[1])
+local expire = tonumber(ARGV[2])
+local c_name = KEYS[1]
+
+local c = redis.call('GET', c_name)
+
+if not c then
+    c = redis.call('INCR', c_name)
+    if expire > 0 then
+        redis.call('EXPIRE', c_name, expire)
+    end
+else
+    c = tonumber(c)
+    if limit == 0 or c < limit then
+       c = redis.call('INCR', c_name)
+    end
+end
+return c
+"""
+
+
+
[docs]def incr_counter(client, name: str, limit: int = 0, expire: int = 0):
+    """Increment a counter and return the new value.
+
+    If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
+    created with initial value 1 returned.  The replacement ``<name>`` is a
+    *secret hash* of the value from argument ``name`` (see
+    :py:func:`secret_hash`).
+
+    The implementation of the redis counter is the lua script from string
+    :py:obj:`INCR_COUNTER`.
+
+    :param name: name of the counter
+    :type name: str
+
+    :param expire: live-time of the counter in seconds (default ``None`` means
+      infinite).
+    :type expire: int / see EXPIRE_
+
+    :param limit: limit where the counter stops to increment (default ``None``)
+    :type limit: int / limit is 2^64 see INCR_
+
+    :return: value of the incremented counter
+    :type return: int
+
+    .. _EXPIRE: https://redis.io/commands/expire/
+    .. _INCR: https://redis.io/commands/incr/
+
+    A simple demo of a counter with expire time and limit::
+
+      >>> for i in range(6):
+      ...   i, incr_counter(client, "foo", 3, 5) # max 3, duration 5 sec
+      ...   time.sleep(1) # from the third call on max has been reached
+      ...
+      (0, 1)
+      (1, 2)
+      (2, 3)
+      (3, 3)
+      (4, 3)
+      (5, 1)
+
+    """
+    script = lua_script_storage(client, INCR_COUNTER)
+    name = "SearXNG_counter_" + secret_hash(name)
+    c = script(args=[limit, expire], keys=[name])
+    return c

+
+
+
[docs]def drop_counter(client, name):
+    """Drop counter with redis key ``SearXNG_counter_<name>``
+
+    The replacement ``<name>`` is a *secret hash* of the value from argument
+    ``name`` (see :py:func:`incr_counter` and :py:func:`incr_sliding_window`).
+    """
+    name = "SearXNG_counter_" + secret_hash(name)
+    client.delete(name)

+
+
+INCR_SLIDING_WINDOW = """
+local expire = tonumber(ARGV[1])
+local name = KEYS[1]
+local current_time = redis.call('TIME')
+
+redis.call('ZREMRANGEBYSCORE', name, 0, current_time[1] - expire)
+redis.call('ZADD', name, current_time[1], current_time[1] .. current_time[2])
+local result = redis.call('ZCOUNT', name, 0, current_time[1] + 1)
+redis.call('EXPIRE', name, expire)
+return result
+"""
+
+
+
[docs]def incr_sliding_window(client, name: str, duration: int):
+    """Increment a sliding-window counter and return the new value.
+
+    If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
+    created with initial value 1 returned.  The replacement ``<name>`` is a
+    *secret hash* of the value from argument ``name`` (see
+    :py:func:`secret_hash`).
+
+    :param name: name of the counter
+    :type name: str
+
+    :param duration: live-time of the sliding window in seconds
+    :typeduration: int
+
+    :return: value of the incremented counter
+    :type return: int
+
+    The implementation of the redis counter is the lua script from string
+    :py:obj:`INCR_SLIDING_WINDOW`.  The lua script uses `sorted sets in Redis`_
+    to implement a sliding window for the redis key ``SearXNG_counter_<name>``
+    (ZADD_).  The current TIME_ is used to score the items in the sorted set and
+    the time window is moved by removing items with a score lower current time
+    minus *duration* time (ZREMRANGEBYSCORE_).
+
+    The EXPIRE_ time (the duration of the sliding window) is refreshed on each
+    call (incrementation) and if there is no call in this duration, the sorted
+    set expires from the redis DB.
+
+    The return value is the amount of items in the sorted set (ZCOUNT_), what
+    means the number of calls in the sliding window.
+
+    .. _Sorted sets in Redis:
+       https://redis.com/ebook/part-1-getting-started/chapter-1-getting-to-know-redis/1-2-what-redis-data-structures-look-like/1-2-5-sorted-sets-in-redis/
+    .. _TIME: https://redis.io/commands/time/
+    .. _ZADD: https://redis.io/commands/zadd/
+    .. _EXPIRE: https://redis.io/commands/expire/
+    .. _ZREMRANGEBYSCORE: https://redis.io/commands/zremrangebyscore/
+    .. _ZCOUNT: https://redis.io/commands/zcount/
+
+    A simple demo of the sliding window::
+
+      >>> for i in range(5):
+      ...   incr_sliding_window(client, "foo", 3) # duration 3 sec
+      ...   time.sleep(1) # from the third call (second) on the window is moved
+      ...
+      1
+      2
+      3
+      3
+      3
+      >>> time.sleep(3)  # wait until expire
+      >>> incr_sliding_window(client, "foo", 3)
+      1
+
+    """
+    script = lua_script_storage(client, INCR_SLIDING_WINDOW)
+    name = "SearXNG_counter_" + secret_hash(name)
+    c = script(args=[duration], keys=[name])
+    return c

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search.html b/_modules/searx/search.html new file mode 100644 index 000000000..987d9acb9 --- /dev/null +++ b/_modules/searx/search.html @@ -0,0 +1,322 @@ + + + + + + + + searx.search — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+# pylint: disable=missing-module-docstring, too-few-public-methods
+
+import threading
+from copy import copy
+from timeit import default_timer
+from uuid import uuid4
+
+import flask
+from flask import copy_current_request_context
+import babel
+
+from searx import settings
+from searx.answerers import ask
+from searx.external_bang import get_bang_url
+from searx.results import ResultContainer
+from searx import logger
+from searx.plugins import plugins
+from searx.search.models import EngineRef, SearchQuery
+from searx.engines import load_engines
+from searx.network import initialize as initialize_network, check_network_configuration
+from searx.metrics import initialize as initialize_metrics, counter_inc, histogram_observe_time
+from searx.search.processors import PROCESSORS, initialize as initialize_processors
+from searx.search.checker import initialize as initialize_checker
+
+
+logger = logger.getChild('search')
+
+
+def initialize(settings_engines=None, enable_checker=False, check_network=False, enable_metrics=True):
+    settings_engines = settings_engines or settings['engines']
+    load_engines(settings_engines)
+    initialize_network(settings_engines, settings['outgoing'])
+    if check_network:
+        check_network_configuration()
+    initialize_metrics([engine['name'] for engine in settings_engines], enable_metrics)
+    initialize_processors(settings_engines)
+    if enable_checker:
+        initialize_checker()
+
+
+
+
+
+
[docs]class SearchWithPlugins(Search):
+    """Inherit from the Search class, add calls to the plugins."""
+
+    __slots__ = 'ordered_plugin_list', 'request'
+
+    def __init__(self, search_query: SearchQuery, ordered_plugin_list, request: flask.Request):
+        super().__init__(search_query)
+        self.ordered_plugin_list = ordered_plugin_list
+        self.result_container.on_result = self._on_result
+        # pylint: disable=line-too-long
+        # get the "real" request to use it outside the Flask context.
+        # see
+        # * https://github.com/pallets/flask/blob/d01d26e5210e3ee4cbbdef12f05c886e08e92852/src/flask/globals.py#L55
+        # * https://github.com/pallets/werkzeug/blob/3c5d3c9bd0d9ce64590f0af8997a38f3823b368d/src/werkzeug/local.py#L548-L559
+        # * https://werkzeug.palletsprojects.com/en/2.0.x/local/#werkzeug.local.LocalProxy._get_current_object
+        # pylint: enable=line-too-long
+        self.request = request._get_current_object()
+
+    def _on_result(self, result):
+        return plugins.call(self.ordered_plugin_list, 'on_result', self.request, self, result)
+
+
[docs]    def search(self) -> ResultContainer:
+        if plugins.call(self.ordered_plugin_list, 'pre_search', self.request, self):
+            super().search()
+
+        plugins.call(self.ordered_plugin_list, 'post_search', self.request, self)
+
+        self.result_container.close()
+
+        return self.result_container

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/models.html b/_modules/searx/search/models.html new file mode 100644 index 000000000..d4d036fea --- /dev/null +++ b/_modules/searx/search/models.html @@ -0,0 +1,236 @@ + + + + + + + + searx.search.models — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.models

+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+import typing
+import babel
+
+
+
[docs]class EngineRef:
+    """Reference by names to an engine and category"""
+
+    __slots__ = 'name', 'category'
+
+    def __init__(self, name: str, category: str):
+        self.name = name
+        self.category = category
+
+    def __repr__(self):
+        return "EngineRef({!r}, {!r})".format(self.name, self.category)
+
+    def __eq__(self, other):
+        return self.name == other.name and self.category == other.category
+
+    def __hash__(self):
+        return hash((self.name, self.category))

+
+
+
[docs]class SearchQuery:
+    """container for all the search parameters (query, language, etc...)"""
+
+    __slots__ = (
+        'query',
+        'engineref_list',
+        'lang',
+        'locale',
+        'safesearch',
+        'pageno',
+        'time_range',
+        'timeout_limit',
+        'external_bang',
+        'engine_data',
+    )
+
+    def __init__(
+        self,
+        query: str,
+        engineref_list: typing.List[EngineRef],
+        lang: str = 'all',
+        safesearch: int = 0,
+        pageno: int = 1,
+        time_range: typing.Optional[str] = None,
+        timeout_limit: typing.Optional[float] = None,
+        external_bang: typing.Optional[str] = None,
+        engine_data: typing.Optional[typing.Dict[str, str]] = None,
+    ):
+        self.query = query
+        self.engineref_list = engineref_list
+        self.lang = lang
+        self.safesearch = safesearch
+        self.pageno = pageno
+        self.time_range = time_range
+        self.timeout_limit = timeout_limit
+        self.external_bang = external_bang
+        self.engine_data = engine_data or {}
+
+        self.locale = None
+        if self.lang:
+            try:
+                self.locale = babel.Locale.parse(self.lang, sep='-')
+            except babel.core.UnknownLocaleError:
+                pass
+
+    @property
+    def categories(self):
+        return list(set(map(lambda engineref: engineref.category, self.engineref_list)))
+
+    def __repr__(self):
+        return "SearchQuery({!r}, {!r}, {!r}, {!r}, {!r}, {!r}, {!r}, {!r})".format(
+            self.query,
+            self.engineref_list,
+            self.lang,
+            self.safesearch,
+            self.pageno,
+            self.time_range,
+            self.timeout_limit,
+            self.external_bang,
+        )
+
+    def __eq__(self, other):
+        return (
+            self.query == other.query
+            and self.engineref_list == other.engineref_list
+            and self.lang == other.lang
+            and self.safesearch == other.safesearch
+            and self.pageno == other.pageno
+            and self.time_range == other.time_range
+            and self.timeout_limit == other.timeout_limit
+            and self.external_bang == other.external_bang
+        )
+
+    def __hash__(self):
+        return hash(
+            (
+                self.query,
+                tuple(self.engineref_list),
+                self.lang,
+                self.safesearch,
+                self.pageno,
+                self.time_range,
+                self.timeout_limit,
+                self.external_bang,
+            )
+        )
+
+    def __copy__(self):
+        return SearchQuery(
+            self.query,
+            self.engineref_list,
+            self.lang,
+            self.safesearch,
+            self.pageno,
+            self.time_range,
+            self.timeout_limit,
+            self.external_bang,
+            self.engine_data,
+        )

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/abstract.html b/_modules/searx/search/processors/abstract.html new file mode 100644 index 000000000..8cf18c55f --- /dev/null +++ b/_modules/searx/search/processors/abstract.html @@ -0,0 +1,303 @@ + + + + + + + + searx.search.processors.abstract — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.abstract

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+
+"""Abstract base classes for engine request processores.
+
+"""
+
+import threading
+from abc import abstractmethod, ABC
+from timeit import default_timer
+from typing import Dict, Union
+
+from searx import settings, logger
+from searx.engines import engines
+from searx.network import get_time_for_thread, get_network
+from searx.metrics import histogram_observe, counter_inc, count_exception, count_error
+from searx.exceptions import SearxEngineAccessDeniedException, SearxEngineResponseException
+from searx.utils import get_engine_from_settings
+
+logger = logger.getChild('searx.search.processor')
+SUSPENDED_STATUS: Dict[Union[int, str], 'SuspendedStatus'] = {}
+
+
+
[docs]class SuspendedStatus:
+    """Class to handle suspend state."""
+
+    __slots__ = 'suspend_end_time', 'suspend_reason', 'continuous_errors', 'lock'
+
+    def __init__(self):
+        self.lock = threading.Lock()
+        self.continuous_errors = 0
+        self.suspend_end_time = 0
+        self.suspend_reason = None
+
+    @property
+    def is_suspended(self):
+        return self.suspend_end_time >= default_timer()
+
+    def suspend(self, suspended_time, suspend_reason):
+        with self.lock:
+            # update continuous_errors / suspend_end_time
+            self.continuous_errors += 1
+            if suspended_time is None:
+                suspended_time = min(
+                    settings['search']['max_ban_time_on_fail'],
+                    self.continuous_errors * settings['search']['ban_time_on_fail'],
+                )
+            self.suspend_end_time = default_timer() + suspended_time
+            self.suspend_reason = suspend_reason
+            logger.debug('Suspend for %i seconds', suspended_time)
+
+    def resume(self):
+        with self.lock:
+            # reset the suspend variables
+            self.continuous_errors = 0
+            self.suspend_end_time = 0
+            self.suspend_reason = None

+
+
+
[docs]class EngineProcessor(ABC):
+    """Base classes used for all types of reqest processores."""
+
+    __slots__ = 'engine', 'engine_name', 'lock', 'suspended_status', 'logger'
+
+    def __init__(self, engine, engine_name: str):
+        self.engine = engine
+        self.engine_name = engine_name
+        self.logger = engines[engine_name].logger
+        key = get_network(self.engine_name)
+        key = id(key) if key else self.engine_name
+        self.suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())
+
+    def initialize(self):
+        try:
+            self.engine.init(get_engine_from_settings(self.engine_name))
+        except SearxEngineResponseException as exc:
+            self.logger.warning('Fail to initialize // %s', exc)
+        except Exception:  # pylint: disable=broad-except
+            self.logger.exception('Fail to initialize')
+        else:
+            self.logger.debug('Initialized')
+
+    @property
+    def has_initialize_function(self):
+        return hasattr(self.engine, 'init')
+
+    def handle_exception(self, result_container, exception_or_message, suspend=False):
+        # update result_container
+        if isinstance(exception_or_message, BaseException):
+            exception_class = exception_or_message.__class__
+            module_name = getattr(exception_class, '__module__', 'builtins')
+            module_name = '' if module_name == 'builtins' else module_name + '.'
+            error_message = module_name + exception_class.__qualname__
+        else:
+            error_message = exception_or_message
+        result_container.add_unresponsive_engine(self.engine_name, error_message)
+        # metrics
+        counter_inc('engine', self.engine_name, 'search', 'count', 'error')
+        if isinstance(exception_or_message, BaseException):
+            count_exception(self.engine_name, exception_or_message)
+        else:
+            count_error(self.engine_name, exception_or_message)
+        # suspend the engine ?
+        if suspend:
+            suspended_time = None
+            if isinstance(exception_or_message, SearxEngineAccessDeniedException):
+                suspended_time = exception_or_message.suspended_time
+            self.suspended_status.suspend(suspended_time, error_message)  # pylint: disable=no-member
+
+    def _extend_container_basic(self, result_container, start_time, search_results):
+        # update result_container
+        result_container.extend(self.engine_name, search_results)
+        engine_time = default_timer() - start_time
+        page_load_time = get_time_for_thread()
+        result_container.add_timing(self.engine_name, engine_time, page_load_time)
+        # metrics
+        counter_inc('engine', self.engine_name, 'search', 'count', 'successful')
+        histogram_observe(engine_time, 'engine', self.engine_name, 'time', 'total')
+        if page_load_time is not None:
+            histogram_observe(page_load_time, 'engine', self.engine_name, 'time', 'http')
+
+    def extend_container(self, result_container, start_time, search_results):
+        if getattr(threading.current_thread(), '_timeout', False):
+            # the main thread is not waiting anymore
+            self.handle_exception(result_container, 'timeout', None)
+        else:
+            # check if the engine accepted the request
+            if search_results is not None:
+                self._extend_container_basic(result_container, start_time, search_results)
+            self.suspended_status.resume()
+
+    def extend_container_if_suspended(self, result_container):
+        if self.suspended_status.is_suspended:
+            result_container.add_unresponsive_engine(
+                self.engine_name, self.suspended_status.suspend_reason, suspended=True
+            )
+            return True
+        return False
+
+
[docs]    def get_params(self, search_query, engine_category):
+        """Returns a set of (see :ref:`request params <engine request arguments>`) or
+        ``None`` if request is not supported.
+
+        Not supported conditions (``None`` is returned):
+
+        - A page-number > 1 when engine does not support paging.
+        - A time range when the engine does not support time range.
+        """
+        # if paging is not supported, skip
+        if search_query.pageno > 1 and not self.engine.paging:
+            return None
+
+        # if time_range is not supported, skip
+        if search_query.time_range and not self.engine.time_range_support:
+            return None
+
+        params = {}
+        params['category'] = engine_category
+        params['pageno'] = search_query.pageno
+        params['safesearch'] = search_query.safesearch
+        params['time_range'] = search_query.time_range
+        params['engine_data'] = search_query.engine_data.get(self.engine_name, {})
+        params['searxng_locale'] = search_query.lang
+
+        # deprecated / vintage --> use params['searxng_locale']
+        #
+        # Conditions related to engine's traits are implemented in engine.traits
+        # module. Don't do 'locale' decissions here in the abstract layer of the
+        # search processor, just pass the value from user's choice unchanged to
+        # the engine request.
+
+        if hasattr(self.engine, 'language') and self.engine.language:
+            params['language'] = self.engine.language
+        else:
+            params['language'] = search_query.lang
+
+        return params

+
+    @abstractmethod
+    def search(self, query, params, result_container, start_time, timeout_limit):
+        pass
+
+    def get_tests(self):
+        tests = getattr(self.engine, 'tests', None)
+        if tests is None:
+            tests = getattr(self.engine, 'additional_tests', {})
+            tests.update(self.get_default_tests())
+        return tests
+
+    def get_default_tests(self):
+        return {}

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/offline.html b/_modules/searx/search/processors/offline.html new file mode 100644 index 000000000..8a9c67f48 --- /dev/null +++ b/_modules/searx/search/processors/offline.html @@ -0,0 +1,140 @@ + + + + + + + + searx.search.processors.offline — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.offline

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+
+"""Processores for engine-type: ``offline``
+
+"""
+
+from .abstract import EngineProcessor
+
+
+
[docs]class OfflineProcessor(EngineProcessor):
+    """Processor class used by ``offline`` engines"""
+
+    engine_type = 'offline'
+
+    def _search_basic(self, query, params):
+        return self.engine.search(query, params)
+
+    def search(self, query, params, result_container, start_time, timeout_limit):
+        try:
+            search_results = self._search_basic(query, params)
+            self.extend_container(result_container, start_time, search_results)
+        except ValueError as e:
+            # do not record the error
+            self.logger.exception('engine {0} : invalid input : {1}'.format(self.engine_name, e))
+        except Exception as e:  # pylint: disable=broad-except
+            self.handle_exception(result_container, e)
+            self.logger.exception('engine {0} : exception : {1}'.format(self.engine_name, e))

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/online.html b/_modules/searx/search/processors/online.html new file mode 100644 index 000000000..40c6466a5 --- /dev/null +++ b/_modules/searx/search/processors/online.html @@ -0,0 +1,350 @@ + + + + + + + + searx.search.processors.online — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.online

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+
+"""Processores for engine-type: ``online``
+
+"""
+# pylint: disable=use-dict-literal
+
+from timeit import default_timer
+import asyncio
+import ssl
+import httpx
+
+import searx.network
+from searx.utils import gen_useragent
+from searx.exceptions import (
+    SearxEngineAccessDeniedException,
+    SearxEngineCaptchaException,
+    SearxEngineTooManyRequestsException,
+)
+from searx.metrics.error_recorder import count_error
+from .abstract import EngineProcessor
+
+
+
[docs]def default_request_params():
+    """Default request parameters for ``online`` engines."""
+    return {
+        # fmt: off
+        'method': 'GET',
+        'headers': {},
+        'data': {},
+        'url': '',
+        'cookies': {},
+        'auth': None
+        # fmt: on
+    }

+
+
+
[docs]class OnlineProcessor(EngineProcessor):
+    """Processor class for ``online`` engines."""
+
+    engine_type = 'online'
+
+    def initialize(self):
+        # set timeout for all HTTP requests
+        searx.network.set_timeout_for_thread(self.engine.timeout, start_time=default_timer())
+        # reset the HTTP total time
+        searx.network.reset_time_for_thread()
+        # set the network
+        searx.network.set_context_network_name(self.engine_name)
+        super().initialize()
+
+
[docs]    def get_params(self, search_query, engine_category):
+        """Returns a set of :ref:`request params <engine request online>` or ``None``
+        if request is not supported.
+        """
+        params = super().get_params(search_query, engine_category)
+        if params is None:
+            return None
+
+        # add default params
+        params.update(default_request_params())
+
+        # add an user agent
+        params['headers']['User-Agent'] = gen_useragent()
+
+        # add Accept-Language header
+        if self.engine.send_accept_language_header and search_query.locale:
+            ac_lang = search_query.locale.language
+            if search_query.locale.territory:
+                ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
+                    search_query.locale.language,
+                    search_query.locale.territory,
+                    search_query.locale.language,
+                )
+            params['headers']['Accept-Language'] = ac_lang
+
+        self.logger.debug('HTTP Accept-Language: %s', params['headers'].get('Accept-Language', ''))
+        return params

+
+    def _send_http_request(self, params):
+        # create dictionary which contain all
+        # information about the request
+        request_args = dict(headers=params['headers'], cookies=params['cookies'], auth=params['auth'])
+
+        # verify
+        # if not None, it overrides the verify value defined in the network.
+        # use False to accept any server certificate
+        # use a path to file to specify a server certificate
+        verify = params.get('verify')
+        if verify is not None:
+            request_args['verify'] = params['verify']
+
+        # max_redirects
+        max_redirects = params.get('max_redirects')
+        if max_redirects:
+            request_args['max_redirects'] = max_redirects
+
+        # allow_redirects
+        if 'allow_redirects' in params:
+            request_args['allow_redirects'] = params['allow_redirects']
+
+        # soft_max_redirects
+        soft_max_redirects = params.get('soft_max_redirects', max_redirects or 0)
+
+        # raise_for_status
+        request_args['raise_for_httperror'] = params.get('raise_for_httperror', True)
+
+        # specific type of request (GET or POST)
+        if params['method'] == 'GET':
+            req = searx.network.get
+        else:
+            req = searx.network.post
+
+        request_args['data'] = params['data']
+
+        # send the request
+        response = req(params['url'], **request_args)
+
+        # check soft limit of the redirect count
+        if len(response.history) > soft_max_redirects:
+            # unexpected redirect : record an error
+            # but the engine might still return valid results.
+            status_code = str(response.status_code or '')
+            reason = response.reason_phrase or ''
+            hostname = response.url.host
+            count_error(
+                self.engine_name,
+                '{} redirects, maximum: {}'.format(len(response.history), soft_max_redirects),
+                (status_code, reason, hostname),
+                secondary=True,
+            )
+
+        return response
+
+    def _search_basic(self, query, params):
+        # update request parameters dependent on
+        # search-engine (contained in engines folder)
+        self.engine.request(query, params)
+
+        # ignoring empty urls
+        if params['url'] is None:
+            return None
+
+        if not params['url']:
+            return None
+
+        # send request
+        response = self._send_http_request(params)
+
+        # parse the response
+        response.search_params = params
+        return self.engine.response(response)
+
+    def search(self, query, params, result_container, start_time, timeout_limit):
+        # set timeout for all HTTP requests
+        searx.network.set_timeout_for_thread(timeout_limit, start_time=start_time)
+        # reset the HTTP total time
+        searx.network.reset_time_for_thread()
+        # set the network
+        searx.network.set_context_network_name(self.engine_name)
+
+        try:
+            # send requests and parse the results
+            search_results = self._search_basic(query, params)
+            self.extend_container(result_container, start_time, search_results)
+        except ssl.SSLError as e:
+            # requests timeout (connect or read)
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.error("SSLError {}, verify={}".format(e, searx.network.get_network(self.engine_name).verify))
+        except (httpx.TimeoutException, asyncio.TimeoutError) as e:
+            # requests timeout (connect or read)
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.error(
+                "HTTP requests timeout (search duration : {0} s, timeout: {1} s) : {2}".format(
+                    default_timer() - start_time, timeout_limit, e.__class__.__name__
+                )
+            )
+        except (httpx.HTTPError, httpx.StreamError) as e:
+            # other requests exception
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.exception(
+                "requests exception (search duration : {0} s, timeout: {1} s) : {2}".format(
+                    default_timer() - start_time, timeout_limit, e
+                )
+            )
+        except SearxEngineCaptchaException as e:
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.exception('CAPTCHA')
+        except SearxEngineTooManyRequestsException as e:
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.exception('Too many requests')
+        except SearxEngineAccessDeniedException as e:
+            self.handle_exception(result_container, e, suspend=True)
+            self.logger.exception('Searx is blocked')
+        except Exception as e:  # pylint: disable=broad-except
+            self.handle_exception(result_container, e)
+            self.logger.exception('exception : {0}'.format(e))
+
+    def get_default_tests(self):
+        tests = {}
+
+        tests['simple'] = {
+            'matrix': {'query': ('life', 'computer')},
+            'result_container': ['not_empty'],
+        }
+
+        if getattr(self.engine, 'paging', False):
+            tests['paging'] = {
+                'matrix': {'query': 'time', 'pageno': (1, 2, 3)},
+                'result_container': ['not_empty'],
+                'test': ['unique_results'],
+            }
+            if 'general' in self.engine.categories:
+                # avoid documentation about HTML tags (<time> and <input type="time">)
+                tests['paging']['matrix']['query'] = 'news'
+
+        if getattr(self.engine, 'time_range', False):
+            tests['time_range'] = {
+                'matrix': {'query': 'news', 'time_range': (None, 'day')},
+                'result_container': ['not_empty'],
+                'test': ['unique_results'],
+            }
+
+        if getattr(self.engine, 'traits', False):
+            tests['lang_fr'] = {
+                'matrix': {'query': 'paris', 'lang': 'fr'},
+                'result_container': ['not_empty', ('has_language', 'fr')],
+            }
+            tests['lang_en'] = {
+                'matrix': {'query': 'paris', 'lang': 'en'},
+                'result_container': ['not_empty', ('has_language', 'en')],
+            }
+
+        if getattr(self.engine, 'safesearch', False):
+            tests['safesearch'] = {'matrix': {'query': 'porn', 'safesearch': (0, 2)}, 'test': ['unique_results']}
+
+        return tests

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/online_currency.html b/_modules/searx/search/processors/online_currency.html new file mode 100644 index 000000000..f80fd2c47 --- /dev/null +++ b/_modules/searx/search/processors/online_currency.html @@ -0,0 +1,187 @@ + + + + + + + + searx.search.processors.online_currency — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.online_currency

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Processores for engine-type: ``online_currency``
+
+"""
+
+import unicodedata
+import re
+
+from searx.data import CURRENCIES
+from .online import OnlineProcessor
+
+parser_re = re.compile('.*?(\\d+(?:\\.\\d+)?) ([^.0-9]+) (?:in|to) ([^.0-9]+)', re.I)
+
+
+def normalize_name(name):
+    name = name.lower().replace('-', ' ').rstrip('s')
+    name = re.sub(' +', ' ', name)
+    return unicodedata.normalize('NFKD', name).lower()
+
+
+def name_to_iso4217(name):
+    name = normalize_name(name)
+    currency = CURRENCIES['names'].get(name, [name])
+    if isinstance(currency, str):
+        return currency
+    return currency[0]
+
+
+def iso4217_to_name(iso4217, language):
+    return CURRENCIES['iso4217'].get(iso4217, {}).get(language, iso4217)
+
+
+
[docs]class OnlineCurrencyProcessor(OnlineProcessor):
+
+    """Processor class used by ``online_currency`` engines."""
+
+    engine_type = 'online_currency'
+
+
[docs]    def get_params(self, search_query, engine_category):
+        """Returns a set of :ref:`request params <engine request online_currency>`
+        or ``None`` if search query does not match to :py:obj:`parser_re`."""
+
+        params = super().get_params(search_query, engine_category)
+        if params is None:
+            return None
+
+        m = parser_re.match(search_query.query)
+        if not m:
+            return None
+
+        amount_str, from_currency, to_currency = m.groups()
+        try:
+            amount = float(amount_str)
+        except ValueError:
+            return None
+        from_currency = name_to_iso4217(from_currency.strip())
+        to_currency = name_to_iso4217(to_currency.strip())
+
+        params['amount'] = amount
+        params['from'] = from_currency
+        params['to'] = to_currency
+        params['from_name'] = iso4217_to_name(from_currency, 'en')
+        params['to_name'] = iso4217_to_name(to_currency, 'en')
+        return params

+
+    def get_default_tests(self):
+        tests = {}
+
+        tests['currency'] = {
+            'matrix': {'query': '1337 usd in rmb'},
+            'result_container': ['has_answer'],
+        }
+
+        return tests

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/online_dictionary.html b/_modules/searx/search/processors/online_dictionary.html new file mode 100644 index 000000000..f7bf8fbd8 --- /dev/null +++ b/_modules/searx/search/processors/online_dictionary.html @@ -0,0 +1,173 @@ + + + + + + + + searx.search.processors.online_dictionary — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.online_dictionary

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Processores for engine-type: ``online_dictionary``
+
+"""
+
+import re
+
+from searx.utils import is_valid_lang
+from .online import OnlineProcessor
+
+parser_re = re.compile('.*?([a-z]+)-([a-z]+) (.+)$', re.I)
+
+
+
[docs]class OnlineDictionaryProcessor(OnlineProcessor):
+    """Processor class used by ``online_dictionary`` engines."""
+
+    engine_type = 'online_dictionary'
+
+
[docs]    def get_params(self, search_query, engine_category):
+        """Returns a set of :ref:`request params <engine request online_dictionary>` or
+        ``None`` if search query does not match to :py:obj:`parser_re`.
+        """
+        params = super().get_params(search_query, engine_category)
+        if params is None:
+            return None
+
+        m = parser_re.match(search_query.query)
+        if not m:
+            return None
+
+        from_lang, to_lang, query = m.groups()
+
+        from_lang = is_valid_lang(from_lang)
+        to_lang = is_valid_lang(to_lang)
+
+        if not from_lang or not to_lang:
+            return None
+
+        params['from_lang'] = from_lang
+        params['to_lang'] = to_lang
+        params['query'] = query
+
+        return params

+
+    def get_default_tests(self):
+        tests = {}
+
+        if getattr(self.engine, 'paging', False):
+            tests['translation_paging'] = {
+                'matrix': {'query': 'en-es house', 'pageno': (1, 2, 3)},
+                'result_container': ['not_empty', ('one_title_contains', 'house')],
+                'test': ['unique_results'],
+            }
+        else:
+            tests['translation'] = {
+                'matrix': {'query': 'en-es house'},
+                'result_container': ['not_empty', ('one_title_contains', 'house')],
+            }
+
+        return tests

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/search/processors/online_url_search.html b/_modules/searx/search/processors/online_url_search.html new file mode 100644 index 000000000..21ee3118d --- /dev/null +++ b/_modules/searx/search/processors/online_url_search.html @@ -0,0 +1,158 @@ + + + + + + + + searx.search.processors.online_url_search — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.search.processors.online_url_search

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Processores for engine-type: ``online_url_search``
+
+"""
+
+import re
+from .online import OnlineProcessor
+
+re_search_urls = {
+    'http': re.compile(r'https?:\/\/[^ ]*'),
+    'ftp': re.compile(r'ftps?:\/\/[^ ]*'),
+    'data:image': re.compile('data:image/[^; ]*;base64,[^ ]*'),
+}
+
+
+
[docs]class OnlineUrlSearchProcessor(OnlineProcessor):
+    """Processor class used by ``online_url_search`` engines."""
+
+    engine_type = 'online_url_search'
+
+
[docs]    def get_params(self, search_query, engine_category):
+        """Returns a set of :ref:`request params <engine request online>` or ``None`` if
+        search query does not match to :py:obj:`re_search_urls`.
+        """
+
+        params = super().get_params(search_query, engine_category)
+        if params is None:
+            return None
+
+        url_match = False
+        search_urls = {}
+
+        for k, v in re_search_urls.items():
+            m = v.search(search_query.query)
+            v = None
+            if m:
+                url_match = True
+                v = m[0]
+            search_urls[k] = v
+
+        if not url_match:
+            return None
+
+        params['search_urls'] = search_urls
+        return params

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searx/utils.html b/_modules/searx/utils.html new file mode 100644 index 000000000..c2fe358d1 --- /dev/null +++ b/_modules/searx/utils.html @@ -0,0 +1,731 @@ + + + + + + + + searx.utils — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searx.utils

+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+# pyright: basic
+"""Utility functions for the engines
+
+"""
+import re
+import importlib
+import importlib.util
+import types
+
+from typing import Optional, Union, Any, Set, List, Dict, MutableMapping, Tuple, Callable
+from numbers import Number
+from os.path import splitext, join
+from random import choice
+from html.parser import HTMLParser
+from urllib.parse import urljoin, urlparse
+
+from lxml import html
+from lxml.etree import ElementBase, XPath, XPathError, XPathSyntaxError, _ElementStringResult, _ElementUnicodeResult
+
+from searx import settings
+from searx.data import USER_AGENTS, data_dir
+from searx.version import VERSION_TAG
+from searx.sxng_locales import sxng_locales
+from searx.exceptions import SearxXPathSyntaxException, SearxEngineXPathException
+from searx import logger
+
+
+logger = logger.getChild('utils')
+
+XPathSpecType = Union[str, XPath]
+
+_BLOCKED_TAGS = ('script', 'style')
+
+_ECMA_UNESCAPE4_RE = re.compile(r'%u([0-9a-fA-F]{4})', re.UNICODE)
+_ECMA_UNESCAPE2_RE = re.compile(r'%([0-9a-fA-F]{2})', re.UNICODE)
+
+_STORAGE_UNIT_VALUE: Dict[str, int] = {
+    'TB': 1024 * 1024 * 1024 * 1024,
+    'GB': 1024 * 1024 * 1024,
+    'MB': 1024 * 1024,
+    'TiB': 1000 * 1000 * 1000 * 1000,
+    'MiB': 1000 * 1000,
+    'KiB': 1000,
+}
+
+_XPATH_CACHE: Dict[str, XPath] = {}
+_LANG_TO_LC_CACHE: Dict[str, Dict[str, str]] = {}
+
+_FASTTEXT_MODEL: Optional["fasttext.FastText._FastText"] = None
+"""fasttext model to predict laguage of a search term"""
+
+SEARCH_LANGUAGE_CODES = frozenset([searxng_locale[0].split('-')[0] for searxng_locale in sxng_locales])
+"""Languages supported by most searxng engines (:py:obj:`searx.sxng_locales.sxng_locales`)."""
+
+
+class _NotSetClass:  # pylint: disable=too-few-public-methods
+    """Internal class for this module, do not create instance of this class.
+    Replace the None value, allow explicitly pass None as a function argument"""
+
+
+_NOTSET = _NotSetClass()
+
+
+
[docs]def searx_useragent() -> str:
+    """Return the searx User Agent"""
+    return 'searx/{searx_version} {suffix}'.format(
+        searx_version=VERSION_TAG, suffix=settings['outgoing']['useragent_suffix']
+    ).strip()

+
+
+
[docs]def gen_useragent(os_string: Optional[str] = None) -> str:
+    """Return a random browser User Agent
+
+    See searx/data/useragents.json
+    """
+    return USER_AGENTS['ua'].format(os=os_string or choice(USER_AGENTS['os']), version=choice(USER_AGENTS['versions']))

+
+
+class _HTMLTextExtractorException(Exception):
+    """Internal exception raised when the HTML is invalid"""
+
+
+class _HTMLTextExtractor(HTMLParser):  # pylint: disable=W0223  # (see https://bugs.python.org/issue31844)
+    """Internal class to extract text from HTML"""
+
+    def __init__(self):
+        HTMLParser.__init__(self)
+        self.result = []
+        self.tags = []
+
+    def handle_starttag(self, tag, attrs):
+        self.tags.append(tag)
+        if tag == 'br':
+            self.result.append(' ')
+
+    def handle_endtag(self, tag):
+        if not self.tags:
+            return
+
+        if tag != self.tags[-1]:
+            raise _HTMLTextExtractorException()
+
+        self.tags.pop()
+
+    def is_valid_tag(self):
+        return not self.tags or self.tags[-1] not in _BLOCKED_TAGS
+
+    def handle_data(self, data):
+        if not self.is_valid_tag():
+            return
+        self.result.append(data)
+
+    def handle_charref(self, name):
+        if not self.is_valid_tag():
+            return
+        if name[0] in ('x', 'X'):
+            codepoint = int(name[1:], 16)
+        else:
+            codepoint = int(name)
+        self.result.append(chr(codepoint))
+
+    def handle_entityref(self, name):
+        if not self.is_valid_tag():
+            return
+        # codepoint = htmlentitydefs.name2codepoint[name]
+        # self.result.append(chr(codepoint))
+        self.result.append(name)
+
+    def get_text(self):
+        return ''.join(self.result).strip()
+
+
+
[docs]def html_to_text(html_str: str) -> str:
+    """Extract text from a HTML string
+
+    Args:
+        * html_str (str): string HTML
+
+    Returns:
+        * str: extracted text
+
+    Examples:
+        >>> html_to_text('Example <span id="42">#2</span>')
+        'Example #2'
+
+        >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
+        'Example'
+    """
+    html_str = html_str.replace('\n', ' ').replace('\r', ' ')
+    html_str = ' '.join(html_str.split())
+    s = _HTMLTextExtractor()
+    try:
+        s.feed(html_str)
+    except _HTMLTextExtractorException:
+        logger.debug("HTMLTextExtractor: invalid HTML\n%s", html_str)
+    return s.get_text()

+
+
+
[docs]def extract_text(xpath_results, allow_none: bool = False) -> Optional[str]:
+    """Extract text from a lxml result
+
+    * if xpath_results is list, extract the text from each result and concat the list
+    * if xpath_results is a xml element, extract all the text node from it
+      ( text_content() method from lxml )
+    * if xpath_results is a string element, then it's already done
+    """
+    if isinstance(xpath_results, list):
+        # it's list of result : concat everything using recursive call
+        result = ''
+        for e in xpath_results:
+            result = result + (extract_text(e) or '')
+        return result.strip()
+    if isinstance(xpath_results, ElementBase):
+        # it's a element
+        text: str = html.tostring(xpath_results, encoding='unicode', method='text', with_tail=False)
+        text = text.strip().replace('\n', ' ')
+        return ' '.join(text.split())
+    if isinstance(xpath_results, (_ElementStringResult, _ElementUnicodeResult, str, Number, bool)):
+        return str(xpath_results)
+    if xpath_results is None and allow_none:
+        return None
+    if xpath_results is None and not allow_none:
+        raise ValueError('extract_text(None, allow_none=False)')
+    raise ValueError('unsupported type')

+
+
+
[docs]def normalize_url(url: str, base_url: str) -> str:
+    """Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path
+
+    Args:
+        * url (str): Relative URL
+        * base_url (str): Base URL, it must be an absolute URL.
+
+    Example:
+        >>> normalize_url('https://example.com', 'http://example.com/')
+        'https://example.com/'
+        >>> normalize_url('//example.com', 'http://example.com/')
+        'http://example.com/'
+        >>> normalize_url('//example.com', 'https://example.com/')
+        'https://example.com/'
+        >>> normalize_url('/path?a=1', 'https://example.com')
+        'https://example.com/path?a=1'
+        >>> normalize_url('', 'https://example.com')
+        'https://example.com/'
+        >>> normalize_url('/test', '/path')
+        raise ValueError
+
+    Raises:
+        * lxml.etree.ParserError
+
+    Returns:
+        * str: normalized URL
+    """
+    if url.startswith('//'):
+        # add http or https to this kind of url //example.com/
+        parsed_search_url = urlparse(base_url)
+        url = '{0}:{1}'.format(parsed_search_url.scheme or 'http', url)
+    elif url.startswith('/'):
+        # fix relative url to the search engine
+        url = urljoin(base_url, url)
+
+    # fix relative urls that fall through the crack
+    if '://' not in url:
+        url = urljoin(base_url, url)
+
+    parsed_url = urlparse(url)
+
+    # add a / at this end of the url if there is no path
+    if not parsed_url.netloc:
+        raise ValueError('Cannot parse url')
+    if not parsed_url.path:
+        url += '/'
+
+    return url

+
+
+
[docs]def extract_url(xpath_results, base_url) -> str:
+    """Extract and normalize URL from lxml Element
+
+    Args:
+        * xpath_results (Union[List[html.HtmlElement], html.HtmlElement]): lxml Element(s)
+        * base_url (str): Base URL
+
+    Example:
+        >>> def f(s, search_url):
+        >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
+        >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
+        'https://example.com/'
+        >>> f('https://example.com', 'http://example.com/')
+        'https://example.com/'
+        >>> f('//example.com', 'http://example.com/')
+        'http://example.com/'
+        >>> f('//example.com', 'https://example.com/')
+        'https://example.com/'
+        >>> f('/path?a=1', 'https://example.com')
+        'https://example.com/path?a=1'
+        >>> f('', 'https://example.com')
+        raise lxml.etree.ParserError
+        >>> searx.utils.extract_url([], 'https://example.com')
+        raise ValueError
+
+    Raises:
+        * ValueError
+        * lxml.etree.ParserError
+
+    Returns:
+        * str: normalized URL
+    """
+    if xpath_results == []:
+        raise ValueError('Empty url resultset')
+
+    url = extract_text(xpath_results)
+    if url:
+        return normalize_url(url, base_url)
+    raise ValueError('URL not found')

+
+
+
[docs]def dict_subset(dictionary: MutableMapping, properties: Set[str]) -> Dict:
+    """Extract a subset of a dict
+
+    Examples:
+        >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
+        {'A': 'a', 'C': 'c'}
+        >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
+        {'A': 'a'}
+    """
+    return {k: dictionary[k] for k in properties if k in dictionary}

+
+
+
[docs]def get_torrent_size(filesize: str, filesize_multiplier: str) -> Optional[int]:
+    """
+
+    Args:
+        * filesize (str): size
+        * filesize_multiplier (str): TB, GB, .... TiB, GiB...
+
+    Returns:
+        * int: number of bytes
+
+    Example:
+        >>> get_torrent_size('5', 'GB')
+        5368709120
+        >>> get_torrent_size('3.14', 'MiB')
+        3140000
+    """
+    try:
+        multiplier = _STORAGE_UNIT_VALUE.get(filesize_multiplier, 1)
+        return int(float(filesize) * multiplier)
+    except ValueError:
+        return None

+
+
+
[docs]def convert_str_to_int(number_str: str) -> int:
+    """Convert number_str to int or 0 if number_str is not a number."""
+    if number_str.isdigit():
+        return int(number_str)
+    return 0

+
+
+
[docs]def int_or_zero(num: Union[List[str], str]) -> int:
+    """Convert num to int or 0. num can be either a str or a list.
+    If num is a list, the first element is converted to int (or return 0 if the list is empty).
+    If num is a str, see convert_str_to_int
+    """
+    if isinstance(num, list):
+        if len(num) < 1:
+            return 0
+        num = num[0]
+    return convert_str_to_int(num)

+
+
+
[docs]def is_valid_lang(lang) -> Optional[Tuple[bool, str, str]]:
+    """Return language code and name if lang describe a language.
+
+    Examples:
+        >>> is_valid_lang('zz')
+        None
+        >>> is_valid_lang('uk')
+        (True, 'uk', 'ukrainian')
+        >>> is_valid_lang(b'uk')
+        (True, 'uk', 'ukrainian')
+        >>> is_valid_lang('en')
+        (True, 'en', 'english')
+        >>> searx.utils.is_valid_lang('Español')
+        (True, 'es', 'spanish')
+        >>> searx.utils.is_valid_lang('Spanish')
+        (True, 'es', 'spanish')
+    """
+    if isinstance(lang, bytes):
+        lang = lang.decode()
+    is_abbr = len(lang) == 2
+    lang = lang.lower()
+    if is_abbr:
+        for l in sxng_locales:
+            if l[0][:2] == lang:
+                return (True, l[0][:2], l[3].lower())
+        return None
+    for l in sxng_locales:
+        if l[1].lower() == lang or l[3].lower() == lang:
+            return (True, l[0][:2], l[3].lower())
+    return None

+
+
+def load_module(filename: str, module_dir: str) -> types.ModuleType:
+    modname = splitext(filename)[0]
+    modpath = join(module_dir, filename)
+    # and https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
+    spec = importlib.util.spec_from_file_location(modname, modpath)
+    if not spec:
+        raise ValueError(f"Error loading '{modpath}' module")
+    module = importlib.util.module_from_spec(spec)
+    if not spec.loader:
+        raise ValueError(f"Error loading '{modpath}' module")
+    spec.loader.exec_module(module)
+    return module
+
+
+
[docs]def to_string(obj: Any) -> str:
+    """Convert obj to its string representation."""
+    if isinstance(obj, str):
+        return obj
+    if hasattr(obj, '__str__'):
+        return str(obj)
+    return repr(obj)

+
+
+
[docs]def ecma_unescape(string: str) -> str:
+    """Python implementation of the unescape javascript function
+
+    https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string
+    https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape
+
+    Examples:
+        >>> ecma_unescape('%u5409')
+        '吉'
+        >>> ecma_unescape('%20')
+        ' '
+        >>> ecma_unescape('%F3')
+        'ó'
+    """
+    # "%u5409" becomes "吉"
+    string = _ECMA_UNESCAPE4_RE.sub(lambda e: chr(int(e.group(1), 16)), string)
+    # "%20" becomes " ", "%F3" becomes "ó"
+    string = _ECMA_UNESCAPE2_RE.sub(lambda e: chr(int(e.group(1), 16)), string)
+    return string

+
+
+def get_string_replaces_function(replaces: Dict[str, str]) -> Callable[[str], str]:
+    rep = {re.escape(k): v for k, v in replaces.items()}
+    pattern = re.compile("|".join(rep.keys()))
+
+    def func(text):
+        return pattern.sub(lambda m: rep[re.escape(m.group(0))], text)
+
+    return func
+
+
+
[docs]def get_engine_from_settings(name: str) -> Dict:
+    """Return engine configuration from settings.yml of a given engine name"""
+
+    if 'engines' not in settings:
+        return {}
+
+    for engine in settings['engines']:
+        if 'name' not in engine:
+            continue
+        if name == engine['name']:
+            return engine
+
+    return {}

+
+
+
[docs]def get_xpath(xpath_spec: XPathSpecType) -> XPath:
+    """Return cached compiled XPath
+
+    There is no thread lock.
+    Worst case scenario, xpath_str is compiled more than one time.
+
+    Args:
+        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
+
+    Returns:
+        * result (bool, float, list, str): Results.
+
+    Raises:
+        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
+        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
+    """
+    if isinstance(xpath_spec, str):
+        result = _XPATH_CACHE.get(xpath_spec, None)
+        if result is None:
+            try:
+                result = XPath(xpath_spec)
+            except XPathSyntaxError as e:
+                raise SearxXPathSyntaxException(xpath_spec, str(e.msg)) from e
+            _XPATH_CACHE[xpath_spec] = result
+        return result
+
+    if isinstance(xpath_spec, XPath):
+        return xpath_spec
+
+    raise TypeError('xpath_spec must be either a str or a lxml.etree.XPath')

+
+
+
[docs]def eval_xpath(element: ElementBase, xpath_spec: XPathSpecType):
+    """Equivalent of element.xpath(xpath_str) but compile xpath_str once for all.
+    See https://lxml.de/xpathxslt.html#xpath-return-values
+
+    Args:
+        * element (ElementBase): [description]
+        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
+
+    Returns:
+        * result (bool, float, list, str): Results.
+
+    Raises:
+        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
+        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
+        * SearxEngineXPathException: Raise when the XPath can't be evaluated.
+    """
+    xpath = get_xpath(xpath_spec)
+    try:
+        return xpath(element)
+    except XPathError as e:
+        arg = ' '.join([str(i) for i in e.args])
+        raise SearxEngineXPathException(xpath_spec, arg) from e

+
+
+
[docs]def eval_xpath_list(element: ElementBase, xpath_spec: XPathSpecType, min_len: Optional[int] = None):
+    """Same as eval_xpath, check if the result is a list
+
+    Args:
+        * element (ElementBase): [description]
+        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
+        * min_len (int, optional): [description]. Defaults to None.
+
+    Raises:
+        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
+        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
+        * SearxEngineXPathException: raise if the result is not a list
+
+    Returns:
+        * result (bool, float, list, str): Results.
+    """
+    result = eval_xpath(element, xpath_spec)
+    if not isinstance(result, list):
+        raise SearxEngineXPathException(xpath_spec, 'the result is not a list')
+    if min_len is not None and min_len > len(result):
+        raise SearxEngineXPathException(xpath_spec, 'len(xpath_str) < ' + str(min_len))
+    return result

+
+
+
[docs]def eval_xpath_getindex(elements: ElementBase, xpath_spec: XPathSpecType, index: int, default=_NOTSET):
+    """Call eval_xpath_list then get one element using the index parameter.
+    If the index does not exist, either aise an exception is default is not set,
+    other return the default value (can be None).
+
+    Args:
+        * elements (ElementBase): lxml element to apply the xpath.
+        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath.
+        * index (int): index to get
+        * default (Object, optional): Defaults if index doesn't exist.
+
+    Raises:
+        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
+        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
+        * SearxEngineXPathException: if the index is not found. Also see eval_xpath.
+
+    Returns:
+        * result (bool, float, list, str): Results.
+    """
+    result = eval_xpath_list(elements, xpath_spec)
+    if -len(result) <= index < len(result):
+        return result[index]
+    if default == _NOTSET:
+        # raise an SearxEngineXPathException instead of IndexError
+        # to record xpath_spec
+        raise SearxEngineXPathException(xpath_spec, 'index ' + str(index) + ' not found')
+    return default

+
+
+def _get_fasttext_model() -> "fasttext.FastText._FastText":
+    global _FASTTEXT_MODEL  # pylint: disable=global-statement
+    if _FASTTEXT_MODEL is None:
+        import fasttext  # pylint: disable=import-outside-toplevel
+
+        # Monkey patch: prevent fasttext from showing a (useless) warning when loading a model.
+        fasttext.FastText.eprint = lambda x: None
+        _FASTTEXT_MODEL = fasttext.load_model(str(data_dir / 'lid.176.ftz'))
+    return _FASTTEXT_MODEL
+
+
+
[docs]def detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) -> Optional[str]:
+    """Detect the language of the ``text`` parameter.
+
+    :param str text: The string whose language is to be detected.
+
+    :param float threshold: Threshold filters the returned labels by a threshold
+        on probability.  A choice of 0.3 will return labels with at least 0.3
+        probability.
+
+    :param bool only_search_languages: If ``True``, returns only supported
+        SearXNG search languages.  see :py:obj:`searx.languages`
+
+    :rtype: str, None
+    :returns:
+        The detected language code or ``None``. See below.
+
+    :raises ValueError: If ``text`` is not a string.
+
+    The language detection is done by using `a fork`_ of the fastText_ library
+    (`python fasttext`_). fastText_ distributes the `language identification
+    model`_, for reference:
+
+    - `FastText.zip: Compressing text classification models`_
+    - `Bag of Tricks for Efficient Text Classification`_
+
+    The `language identification model`_ support the language codes
+    (ISO-639-3)::
+
+        af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs
+        bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es
+        et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia
+        id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li
+        lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah
+        nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru
+        rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl
+        tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh
+
+    By using ``only_search_languages=True`` the `language identification model`_
+    is harmonized with the SearXNG's language (locale) model.  General
+    conditions of SearXNG's locale model are:
+
+    a. SearXNG's locale of a query is passed to the
+       :py:obj:`searx.locales.get_engine_locale` to get a language and/or region
+       code that is used by an engine.
+
+    b. Most of SearXNG's engines do not support all the languages from `language
+       identification model`_ and there is also a discrepancy in the ISO-639-3
+       (fastext) and ISO-639-2 (SearXNG)handling.  Further more, in SearXNG the
+       locales like ``zh-TH`` (``zh-CN``) are mapped to ``zh_Hant``
+       (``zh_Hans``) while the `language identification model`_ reduce both to
+       ``zh``.
+
+    .. _a fork: https://github.com/searxng/fasttext-predict
+    .. _fastText: https://fasttext.cc/
+    .. _python fasttext: https://pypi.org/project/fasttext/
+    .. _language identification model: https://fasttext.cc/docs/en/language-identification.html
+    .. _Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759
+    .. _`FastText.zip: Compressing text classification models`: https://arxiv.org/abs/1612.03651
+
+    """
+    if not isinstance(text, str):
+        raise ValueError('text must a str')
+    r = _get_fasttext_model().predict(text.replace('\n', ' '), k=1, threshold=threshold)
+    if isinstance(r, tuple) and len(r) == 2 and len(r[0]) > 0 and len(r[1]) > 0:
+        language = r[0][0].split('__label__')[1]
+        if only_search_languages and language not in SEARCH_LANGUAGE_CODES:
+            return None
+        return language
+    return None

+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searxng_extra/standalone_searx.html b/_modules/searxng_extra/standalone_searx.html new file mode 100644 index 000000000..469b43f48 --- /dev/null +++ b/_modules/searxng_extra/standalone_searx.html @@ -0,0 +1,288 @@ + + + + + + + + searxng_extra.standalone_searx — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searxng_extra.standalone_searx

+#!/usr/bin/env python
+# lint: pylint
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# (C) Copyright Contributors to the SearXNG project.
+
+"""Script to run SearXNG from terminal.
+
+  DON'T USE THIS SCRIPT!!
+
+.. danger::
+
+   Be warned, using the ``standalone_searx.py`` won't give you privacy!
+
+   On the contrary, this script behaves like a SearXNG server: your IP is
+   exposed and tracked by all active engines (google, bing, qwant, ... ), with
+   every query!
+
+.. note::
+
+   This is an old and grumpy hack / SearXNG is a Flask application with
+   client/server structure, which can't be turned into a command line tool the
+   way it was done here.
+
+Getting categories without initiate the engine will only return `['general']`
+
+>>> import searx.engines
+... list(searx.engines.categories.keys())
+['general']
+>>> import searx.search
+... searx.search.initialize()
+... list(searx.engines.categories.keys())
+['general', 'it', 'science', 'images', 'news', 'videos', 'music', 'files', 'social media', 'map']
+
+Example to use this script:
+
+.. code::  bash
+
+    $ python3 searxng_extra/standalone_searx.py rain
+
+"""  # pylint: disable=line-too-long
+
+import argparse
+import sys
+from datetime import datetime
+from json import dumps
+from typing import Any, Dict, List, Optional
+
+import searx
+import searx.preferences
+import searx.query
+import searx.search
+import searx.webadapter
+
+EngineCategoriesVar = Optional[List[str]]
+
+
+
[docs]def get_search_query(
+    args: argparse.Namespace, engine_categories: EngineCategoriesVar = None
+) -> searx.search.SearchQuery:
+    """Get  search results for the query"""
+    if engine_categories is None:
+        engine_categories = list(searx.engines.categories.keys())
+    try:
+        category = args.category.decode('utf-8')
+    except AttributeError:
+        category = args.category
+    form = {
+        "q": args.query,
+        "categories": category,
+        "pageno": str(args.pageno),
+        "language": args.lang,
+        "time_range": args.timerange,
+    }
+    preferences = searx.preferences.Preferences(['simple'], engine_categories, searx.engines.engines, [])
+    preferences.key_value_settings['safesearch'].parse(args.safesearch)
+
+    search_query = searx.webadapter.get_search_query_from_webapp(preferences, form)[0]
+    return search_query

+
+
+
[docs]def no_parsed_url(results: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    """Remove parsed url from dict."""
+    for result in results:
+        del result['parsed_url']
+    return results

+
+
+
[docs]def json_serial(obj: Any) -> Any:
+    """JSON serializer for objects not serializable by default json code.
+
+    :raise TypeError: raised when **obj** is not serializable
+    """
+    if isinstance(obj, datetime):
+        serial = obj.isoformat()
+        return serial
+    if isinstance(obj, bytes):
+        return obj.decode('utf8')
+    if isinstance(obj, set):
+        return list(obj)
+    raise TypeError("Type ({}) not serializable".format(type(obj)))

+
+
+
[docs]def to_dict(search_query: searx.search.SearchQuery) -> Dict[str, Any]:
+    """Get result from parsed arguments."""
+    result_container = searx.search.Search(search_query).search()
+    result_container_json = {
+        "search": {
+            "q": search_query.query,
+            "pageno": search_query.pageno,
+            "lang": search_query.lang,
+            "safesearch": search_query.safesearch,
+            "timerange": search_query.time_range,
+        },
+        "results": no_parsed_url(result_container.get_ordered_results()),
+        "infoboxes": result_container.infoboxes,
+        "suggestions": list(result_container.suggestions),
+        "answers": list(result_container.answers),
+        "paging": result_container.paging,
+        "number_of_results": result_container.number_of_results,
+    }
+    return result_container_json

+
+
+
[docs]def parse_argument(
+    args: Optional[List[str]] = None, category_choices: EngineCategoriesVar = None
+) -> argparse.Namespace:
+    """Parse command line.
+
+    :raise SystemExit: Query argument required on `args`
+
+    Examples:
+
+    >>> import importlib
+    ... # load module
+    ... spec = importlib.util.spec_from_file_location(
+    ...     'utils.standalone_searx', 'utils/standalone_searx.py')
+    ... sas = importlib.util.module_from_spec(spec)
+    ... spec.loader.exec_module(sas)
+    ... sas.parse_argument()
+    usage: ptipython [-h] [--category [{general}]] [--lang [LANG]] [--pageno [PAGENO]] [--safesearch [{0,1,2}]] [--timerange [{day,week,month,year}]]
+                     query
+    SystemExit: 2
+    >>> sas.parse_argument(['rain'])
+    Namespace(category='general', lang='all', pageno=1, query='rain', safesearch='0', timerange=None)
+    """  # noqa: E501
+    if not category_choices:
+        category_choices = list(searx.engines.categories.keys())
+    parser = argparse.ArgumentParser(description='Standalone searx.')
+    parser.add_argument('query', type=str, help='Text query')
+    parser.add_argument(
+        '--category', type=str, nargs='?', choices=category_choices, default='general', help='Search category'
+    )
+    parser.add_argument('--lang', type=str, nargs='?', default='all', help='Search language')
+    parser.add_argument('--pageno', type=int, nargs='?', default=1, help='Page number starting from 1')
+    parser.add_argument(
+        '--safesearch',
+        type=str,
+        nargs='?',
+        choices=['0', '1', '2'],
+        default='0',
+        help='Safe content filter from none to strict',
+    )
+    parser.add_argument(
+        '--timerange', type=str, nargs='?', choices=['day', 'week', 'month', 'year'], help='Filter by time range'
+    )
+    return parser.parse_args(args)

+
+
+if __name__ == '__main__':
+    settings_engines = searx.settings['engines']
+    searx.search.load_engines(settings_engines)
+    engine_cs = list(searx.engines.categories.keys())
+    prog_args = parse_argument(category_choices=engine_cs)
+    searx.search.initialize_network(settings_engines, searx.settings['outgoing'])
+    searx.search.check_network_configuration()
+    searx.search.initialize_metrics([engine['name'] for engine in settings_engines])
+    searx.search.initialize_processors(settings_engines)
+    search_q = get_search_query(prog_args, engine_categories=engine_cs)
+    res_dict = to_dict(search_q)
+    sys.stdout.write(dumps(res_dict, sort_keys=True, indent=4, ensure_ascii=False, default=json_serial))
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searxng_extra/update/update_engine_descriptions.html b/_modules/searxng_extra/update/update_engine_descriptions.html new file mode 100644 index 000000000..76414a241 --- /dev/null +++ b/_modules/searxng_extra/update/update_engine_descriptions.html @@ -0,0 +1,478 @@ + + + + + + + + searxng_extra.update.update_engine_descriptions — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searxng_extra.update.update_engine_descriptions

+#!/usr/bin/env python
+# lint: pylint
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+"""Fetch website description from websites and from
+:origin:`searx/engines/wikidata.py` engine.
+
+Output file: :origin:`searx/data/engine_descriptions.json`.
+
+"""
+
+# pylint: disable=invalid-name, global-statement
+
+import json
+from urllib.parse import urlparse
+from os.path import join
+
+from lxml.html import fromstring
+
+from searx.engines import wikidata, set_loggers
+from searx.utils import extract_text, searx_useragent
+from searx.locales import LOCALE_NAMES, locales_initialize, match_locale
+from searx import searx_dir
+from searx.utils import gen_useragent, detect_language
+import searx.search
+import searx.network
+
+set_loggers(wikidata, 'wikidata')
+locales_initialize()
+
+# you can run the query in https://query.wikidata.org
+# replace %IDS% by Wikidata entities separated by spaces with the prefix wd:
+# for example wd:Q182496 wd:Q1540899
+# replace %LANGUAGES_SPARQL% by languages
+SPARQL_WIKIPEDIA_ARTICLE = """
+SELECT DISTINCT ?item ?name ?article ?lang
+WHERE {
+  hint:Query hint:optimizer "None".
+  VALUES ?item { %IDS% }
+  ?article schema:about ?item ;
+              schema:inLanguage ?lang ;
+              schema:name ?name ;
+              schema:isPartOf [ wikibase:wikiGroup "wikipedia" ] .
+  FILTER(?lang in (%LANGUAGES_SPARQL%)) .
+  FILTER (!CONTAINS(?name, ':')) .
+}
+ORDER BY ?item ?lang
+"""
+
+SPARQL_DESCRIPTION = """
+SELECT DISTINCT ?item ?itemDescription
+WHERE {
+  VALUES ?item { %IDS% }
+  ?item schema:description ?itemDescription .
+  FILTER (lang(?itemDescription) in (%LANGUAGES_SPARQL%))
+}
+ORDER BY ?itemLang
+"""
+
+NOT_A_DESCRIPTION = [
+    'web site',
+    'site web',
+    'komputa serĉilo',
+    'interreta serĉilo',
+    'bilaketa motor',
+    'web search engine',
+    'wikimedia täpsustuslehekülg',
+]
+
+SKIP_ENGINE_SOURCE = [
+    # fmt: off
+    ('gitlab', 'wikidata')
+    # descriptions are about wikipedia disambiguation pages
+    # fmt: on
+]
+
+WIKIPEDIA_LANGUAGES = {}
+LANGUAGES_SPARQL = ''
+IDS = None
+WIKIPEDIA_LANGUAGE_VARIANTS = {'zh_Hant': 'zh-tw'}
+
+
+descriptions = {}
+wd_to_engine_name = {}
+
+
+def normalize_description(description):
+    for c in [chr(c) for c in range(0, 31)]:
+        description = description.replace(c, ' ')
+    description = ' '.join(description.strip().split())
+    return description
+
+
+def update_description(engine_name, lang, description, source, replace=True):
+    if not isinstance(description, str):
+        return
+    description = normalize_description(description)
+    if description.lower() == engine_name.lower():
+        return
+    if description.lower() in NOT_A_DESCRIPTION:
+        return
+    if (engine_name, source) in SKIP_ENGINE_SOURCE:
+        return
+    if ' ' not in description:
+        # skip unique word description (like "website")
+        return
+    if replace or lang not in descriptions[engine_name]:
+        descriptions[engine_name][lang] = [description, source]
+
+
+def get_wikipedia_summary(wikipedia_url, searxng_locale):
+    # get the REST API URL from the HTML URL
+
+    # Headers
+    headers = {'User-Agent': searx_useragent()}
+
+    if searxng_locale in WIKIPEDIA_LANGUAGE_VARIANTS:
+        headers['Accept-Language'] = WIKIPEDIA_LANGUAGE_VARIANTS.get(searxng_locale)
+
+    # URL path : from HTML URL to REST API URL
+    parsed_url = urlparse(wikipedia_url)
+    # remove the /wiki/ prefix
+    article_name = parsed_url.path.split('/wiki/')[1]
+    # article_name is already encoded but not the / which is required for the REST API call
+    encoded_article_name = article_name.replace('/', '%2F')
+    path = '/api/rest_v1/page/summary/' + encoded_article_name
+    wikipedia_rest_url = parsed_url._replace(path=path).geturl()
+    try:
+        response = searx.network.get(wikipedia_rest_url, headers=headers, timeout=10)
+        response.raise_for_status()
+    except Exception as e:  # pylint: disable=broad-except
+        print("     ", wikipedia_url, e)
+        return None
+    api_result = json.loads(response.text)
+    return api_result.get('extract')
+
+
+def get_website_description(url, lang1, lang2=None):
+    headers = {
+        'User-Agent': gen_useragent(),
+        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
+        'DNT': '1',
+        'Upgrade-Insecure-Requests': '1',
+        'Sec-GPC': '1',
+        'Cache-Control': 'max-age=0',
+    }
+    if lang1 is not None:
+        lang_list = [lang1]
+        if lang2 is not None:
+            lang_list.append(lang2)
+        headers['Accept-Language'] = f'{",".join(lang_list)};q=0.8'
+    try:
+        response = searx.network.get(url, headers=headers, timeout=10)
+        response.raise_for_status()
+    except Exception:  # pylint: disable=broad-except
+        return (None, None)
+
+    try:
+        html = fromstring(response.text)
+    except ValueError:
+        html = fromstring(response.content)
+
+    description = extract_text(html.xpath('/html/head/meta[@name="description"]/@content'))
+    if not description:
+        description = extract_text(html.xpath('/html/head/meta[@property="og:description"]/@content'))
+    if not description:
+        description = extract_text(html.xpath('/html/head/title'))
+    lang = extract_text(html.xpath('/html/@lang'))
+    if lang is None and len(lang1) > 0:
+        lang = lang1
+    lang = detect_language(description) or lang or 'en'
+    lang = lang.split('_')[0]
+    lang = lang.split('-')[0]
+    return (lang, description)
+
+
+def initialize():
+    global IDS, LANGUAGES_SPARQL
+    searx.search.initialize()
+    wikipedia_engine = searx.engines.engines['wikipedia']
+
+    locale2lang = {'nl-BE': 'nl'}
+    for sxng_ui_lang in LOCALE_NAMES:
+
+        sxng_ui_alias = locale2lang.get(sxng_ui_lang, sxng_ui_lang)
+        wiki_lang = None
+
+        if sxng_ui_alias in wikipedia_engine.traits.custom['WIKIPEDIA_LANGUAGES']:
+            wiki_lang = sxng_ui_alias
+        if not wiki_lang:
+            wiki_lang = wikipedia_engine.traits.get_language(sxng_ui_alias)
+        if not wiki_lang:
+            print(f"WIKIPEDIA_LANGUAGES missing {sxng_ui_lang}")
+            continue
+        WIKIPEDIA_LANGUAGES[sxng_ui_lang] = wiki_lang
+
+    LANGUAGES_SPARQL = ', '.join(f"'{l}'" for l in set(WIKIPEDIA_LANGUAGES.values()))
+    for engine_name, engine in searx.engines.engines.items():
+        descriptions[engine_name] = {}
+        wikidata_id = getattr(engine, "about", {}).get('wikidata_id')
+        if wikidata_id is not None:
+            wd_to_engine_name.setdefault(wikidata_id, set()).add(engine_name)
+
+    IDS = ' '.join(list(map(lambda wd_id: 'wd:' + wd_id, wd_to_engine_name.keys())))
+
+
+def fetch_wikidata_descriptions():
+    print('Fetching wikidata descriptions')
+    searx.network.set_timeout_for_thread(60)
+    result = wikidata.send_wikidata_query(
+        SPARQL_DESCRIPTION.replace('%IDS%', IDS).replace('%LANGUAGES_SPARQL%', LANGUAGES_SPARQL)
+    )
+    if result is not None:
+        for binding in result['results']['bindings']:
+            wikidata_id = binding['item']['value'].replace('http://www.wikidata.org/entity/', '')
+            wikidata_lang = binding['itemDescription']['xml:lang']
+            desc = binding['itemDescription']['value']
+            for engine_name in wd_to_engine_name[wikidata_id]:
+                for searxng_locale in LOCALE_NAMES:
+                    if WIKIPEDIA_LANGUAGES[searxng_locale] != wikidata_lang:
+                        continue
+                    print(
+                        f"    engine: {engine_name:20} / wikidata_lang: {wikidata_lang:5}",
+                        f"/ len(wikidata_desc): {len(desc)}",
+                    )
+                    update_description(engine_name, searxng_locale, desc, 'wikidata')
+
+
+def fetch_wikipedia_descriptions():
+    print('Fetching wikipedia descriptions')
+    result = wikidata.send_wikidata_query(
+        SPARQL_WIKIPEDIA_ARTICLE.replace('%IDS%', IDS).replace('%LANGUAGES_SPARQL%', LANGUAGES_SPARQL)
+    )
+    if result is not None:
+        for binding in result['results']['bindings']:
+            wikidata_id = binding['item']['value'].replace('http://www.wikidata.org/entity/', '')
+            wikidata_lang = binding['name']['xml:lang']
+            wikipedia_url = binding['article']['value']  # for example the URL https://de.wikipedia.org/wiki/PubMed
+            for engine_name in wd_to_engine_name[wikidata_id]:
+                for searxng_locale in LOCALE_NAMES:
+                    if WIKIPEDIA_LANGUAGES[searxng_locale] != wikidata_lang:
+                        continue
+                    desc = get_wikipedia_summary(wikipedia_url, searxng_locale)
+                    if not desc:
+                        continue
+                    print(
+                        f"    engine: {engine_name:20} / wikidata_lang: {wikidata_lang:5}",
+                        f"/ len(wikipedia_desc): {len(desc)}",
+                    )
+                    update_description(engine_name, searxng_locale, desc, 'wikipedia')
+
+
+def normalize_url(url):
+    url = url.replace('{language}', 'en')
+    url = urlparse(url)._replace(path='/', params='', query='', fragment='').geturl()
+    url = url.replace('https://api.', 'https://')
+    return url
+
+
+def fetch_website_description(engine_name, website):
+    print(f"- fetch website descr: {engine_name} / {website}")
+    default_lang, default_description = get_website_description(website, None, None)
+
+    if default_lang is None or default_description is None:
+        # the front page can't be fetched: skip this engine
+        return
+
+    # to specify an order in where the most common languages are in front of the
+    # language list ..
+    languages = ['en', 'es', 'pt', 'ru', 'tr', 'fr']
+    languages = languages + [l for l in LOCALE_NAMES if l not in languages]
+
+    previous_matched_lang = None
+    previous_count = 0
+
+    for lang in languages:
+
+        if lang in descriptions[engine_name]:
+            continue
+
+        fetched_lang, desc = get_website_description(website, lang, WIKIPEDIA_LANGUAGES[lang])
+        if fetched_lang is None or desc is None:
+            continue
+
+        # check if desc changed with the different lang values
+
+        if fetched_lang == previous_matched_lang:
+            previous_count += 1
+            if previous_count == 6:
+                # the website has returned the same description for 6 different languages in Accept-Language header
+                # stop now
+                break
+        else:
+            previous_matched_lang = fetched_lang
+            previous_count = 0
+
+        # Don't trust in the value of fetched_lang, some websites return
+        # for some inappropriate values, by example bing-images::
+        #
+        #   requested lang: zh-Hans-CN / fetched lang: ceb / desc: 查看根据您的兴趣量身定制的提要
+        #
+        # The lang ceb is "Cebuano" but the description is given in zh-Hans-CN
+
+        print(
+            f"    engine: {engine_name:20} / requested lang:{lang:7}"
+            f" / fetched lang: {fetched_lang:7} / len(desc): {len(desc)}"
+        )
+
+        matched_lang = match_locale(fetched_lang, LOCALE_NAMES.keys(), fallback=lang)
+        update_description(engine_name, matched_lang, desc, website, replace=False)
+
+
+def fetch_website_descriptions():
+    print('Fetching website descriptions')
+    for engine_name, engine in searx.engines.engines.items():
+        website = getattr(engine, "about", {}).get('website')
+        if website is None and hasattr(engine, "search_url"):
+            website = normalize_url(getattr(engine, "search_url"))
+        if website is None and hasattr(engine, "base_url"):
+            website = normalize_url(getattr(engine, "base_url"))
+        if website is not None:
+            fetch_website_description(engine_name, website)
+
+
+def get_engine_descriptions_filename():
+    return join(join(searx_dir, "data"), "engine_descriptions.json")
+
+
+
[docs]def get_output():
+    """
+    From descriptions[engine][language] = [description, source]
+    To
+
+    * output[language][engine] = description_and_source
+    * description_and_source can be:
+       * [description, source]
+       * description (if source = "wikipedia")
+       * [f"engine:lang", "ref"] (reference to another existing description)
+    """
+    output = {locale: {} for locale in LOCALE_NAMES}
+
+    seen_descriptions = {}
+
+    for engine_name, lang_descriptions in descriptions.items():
+        for language, description in lang_descriptions.items():
+            if description[0] in seen_descriptions:
+                ref = seen_descriptions[description[0]]
+                description = [f'{ref[0]}:{ref[1]}', 'ref']
+            else:
+                seen_descriptions[description[0]] = (engine_name, language)
+                if description[1] == 'wikipedia':
+                    description = description[0]
+            output.setdefault(language, {}).setdefault(engine_name, description)
+
+    return output

+
+
+def main():
+    initialize()
+    fetch_wikidata_descriptions()
+    fetch_wikipedia_descriptions()
+    fetch_website_descriptions()
+
+    output = get_output()
+    with open(get_engine_descriptions_filename(), 'w', encoding='utf8') as f:
+        f.write(json.dumps(output, indent=1, separators=(',', ':'), ensure_ascii=False))
+
+
+if __name__ == "__main__":
+    main()
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searxng_extra/update/update_engine_traits.html b/_modules/searxng_extra/update/update_engine_traits.html new file mode 100644 index 000000000..d964a8292 --- /dev/null +++ b/_modules/searxng_extra/update/update_engine_traits.html @@ -0,0 +1,306 @@ + + + + + + + + searxng_extra.update.update_engine_traits — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searxng_extra.update.update_engine_traits

+#!/usr/bin/env python
+# lint: pylint
+# SPDX-License-Identifier: AGPL-3.0-or-later
+"""Update :py:obj:`searx.enginelib.traits.EngineTraitsMap` and :origin:`searx/languages.py`
+
+:py:obj:`searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE`:
+  Persistence of engines traits, fetched from the engines.
+
+:origin:`searx/languages.py`
+  Is generated  from intersecting each engine's supported traits.
+
+The script :origin:`searxng_extra/update/update_engine_traits.py` is called in
+the :origin:`CI Update data ... <.github/workflows/data-update.yml>`
+
+"""
+
+# pylint: disable=invalid-name
+from unicodedata import lookup
+from pathlib import Path
+from pprint import pformat
+import babel
+
+from searx import settings, searx_dir
+from searx import network
+from searx.engines import load_engines
+from searx.enginelib.traits import EngineTraitsMap
+
+# Output files.
+languages_file = Path(searx_dir) / 'sxng_locales.py'
+languages_file_header = """\
+# -*- coding: utf-8 -*-
+'''List of SearXNG's locale codes.
+
+This file is generated automatically by::
+
+   ./manage pyenv.cmd searxng_extra/update/update_engine_traits.py
+'''
+
+sxng_locales = (
+"""
+languages_file_footer = """,
+)
+'''
+A list of five-digit tuples:
+
+0. SearXNG's internal locale tag (a language or region tag)
+1. Name of the language (:py:obj:`babel.core.Locale.get_language_name`)
+2. For region tags the name of the region (:py:obj:`babel.core.Locale.get_territory_name`).
+   Empty string for language tags.
+3. English language name (from :py:obj:`babel.core.Locale.english_name`)
+4. Unicode flag (emoji) that fits to SearXNG's internal region tag. Languages
+   are represented by a globe (\U0001F310)
+
+.. code:: python
+
+   ('en',    'English', '',              'English', '\U0001f310'),
+   ('en-CA', 'English', 'Canada',        'English', '\U0001f1e8\U0001f1e6'),
+   ('en-US', 'English', 'United States', 'English', '\U0001f1fa\U0001f1f8'),
+   ..
+   ('fr',    'Français', '',             'French',  '\U0001f310'),
+   ('fr-BE', 'Français', 'Belgique',     'French',  '\U0001f1e7\U0001f1ea'),
+   ('fr-CA', 'Français', 'Canada',       'French',  '\U0001f1e8\U0001f1e6'),
+
+:meta hide-value:
+'''
+"""
+
+
+lang2emoji = {
+    'ha': '\U0001F1F3\U0001F1EA',  # Hausa / Niger
+    'bs': '\U0001F1E7\U0001F1E6',  # Bosnian / Bosnia & Herzegovina
+    'jp': '\U0001F1EF\U0001F1F5',  # Japanese
+    'ua': '\U0001F1FA\U0001F1E6',  # Ukrainian
+    'he': '\U0001F1EE\U0001F1F1',  # Hebrew
+}
+
+
+def main():
+    load_engines(settings['engines'])
+    # traits_map = EngineTraitsMap.from_data()
+    traits_map = fetch_traits_map()
+    sxng_tag_list = filter_locales(traits_map)
+    write_languages_file(sxng_tag_list)
+
+
+
[docs]def fetch_traits_map():
+    """Fetchs supported languages for each engine and writes json file with those."""
+    network.set_timeout_for_thread(10.0)
+
+    def log(msg):
+        print(msg)
+
+    traits_map = EngineTraitsMap.fetch_traits(log=log)
+    print("fetched properties from %s engines" % len(traits_map))
+    print("write json file: %s" % traits_map.ENGINE_TRAITS_FILE)
+    traits_map.save_data()
+    return traits_map

+
+
+
[docs]def filter_locales(traits_map: EngineTraitsMap):
+    """Filter language & region tags by a threshold."""
+
+    min_eng_per_region = 11
+    min_eng_per_lang = 13
+
+    _ = {}
+    for eng in traits_map.values():
+        for reg in eng.regions.keys():
+            _[reg] = _.get(reg, 0) + 1
+
+    regions = set(k for k, v in _.items() if v >= min_eng_per_region)
+    lang_from_region = set(k.split('-')[0] for k in regions)
+
+    _ = {}
+    for eng in traits_map.values():
+        for lang in eng.languages.keys():
+            # ignore script types like zh_Hant, zh_Hans or sr_Latin, pa_Arab (they
+            # already counted by existence of 'zh' or 'sr', 'pa')
+            if '_' in lang:
+                # print("ignore %s" % lang)
+                continue
+            _[lang] = _.get(lang, 0) + 1
+
+    languages = set(k for k, v in _.items() if v >= min_eng_per_lang)
+
+    sxng_tag_list = set()
+    sxng_tag_list.update(regions)
+    sxng_tag_list.update(lang_from_region)
+    sxng_tag_list.update(languages)
+
+    return sxng_tag_list

+
+
+def write_languages_file(sxng_tag_list):
+
+    language_codes = []
+
+    for sxng_tag in sorted(sxng_tag_list):
+        sxng_locale: babel.Locale = babel.Locale.parse(sxng_tag, sep='-')
+
+        flag = get_unicode_flag(sxng_locale) or ''
+
+        item = (
+            sxng_tag,
+            sxng_locale.get_language_name().title(),
+            sxng_locale.get_territory_name() or '',
+            sxng_locale.english_name.split(' (')[0],
+            UnicodeEscape(flag),
+        )
+
+        language_codes.append(item)
+
+    language_codes = tuple(language_codes)
+
+    with open(languages_file, 'w', encoding='utf-8') as new_file:
+        file_content = "{header} {language_codes}{footer}".format(
+            header=languages_file_header,
+            language_codes=pformat(language_codes, width=120, indent=4)[1:-1],
+            footer=languages_file_footer,
+        )
+        new_file.write(file_content)
+        new_file.close()
+
+
+
[docs]class UnicodeEscape(str):
+    """Escape unicode string in :py:obj:`pprint.pformat`"""
+
+    def __repr__(self):
+        return "'" + "".join([chr(c) for c in self.encode('unicode-escape')]) + "'"

+
+
+
[docs]def get_unicode_flag(locale: babel.Locale):
+    """Determine a unicode flag (emoji) that fits to the ``locale``"""
+
+    emoji = lang2emoji.get(locale.language)
+    if emoji:
+        return emoji
+
+    if not locale.territory:
+        return '\U0001F310'
+
+    emoji = lang2emoji.get(locale.territory.lower())
+    if emoji:
+        return emoji
+
+    try:
+        c1 = lookup('REGIONAL INDICATOR SYMBOL LETTER ' + locale.territory[0])
+        c2 = lookup('REGIONAL INDICATOR SYMBOL LETTER ' + locale.territory[1])
+        # print("OK   : %s --> %s%s" % (locale, c1, c2))
+    except KeyError as exc:
+        print("ERROR: %s --> %s" % (locale, exc))
+        return None
+
+    return c1 + c2

+
+
+if __name__ == "__main__":
+    main()
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_modules/searxng_extra/update/update_external_bangs.html b/_modules/searxng_extra/update/update_external_bangs.html new file mode 100644 index 000000000..a69eb1b0b --- /dev/null +++ b/_modules/searxng_extra/update/update_external_bangs.html @@ -0,0 +1,271 @@ + + + + + + + + searxng_extra.update.update_external_bangs — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +

Source code for searxng_extra.update.update_external_bangs

+#!/usr/bin/env python
+# lint: pylint
+# SPDX-License-Identifier: AGPL-3.0-or-later
+"""Update :origin:`searx/data/external_bangs.json` using the duckduckgo bangs
+(:origin:`CI Update data ... <.github/workflows/data-update.yml>`).
+
+https://duckduckgo.com/newbang loads:
+
+* a javascript which provides the bang version ( https://duckduckgo.com/bv1.js )
+* a JSON file which contains the bangs ( https://duckduckgo.com/bang.v260.js for example )
+
+This script loads the javascript, then the bangs.
+
+The javascript URL may change in the future ( for example
+https://duckduckgo.com/bv2.js ), but most probably it will requires to update
+RE_BANG_VERSION
+
+"""
+# pylint: disable=C0116
+
+import json
+import re
+from os.path import join
+
+import httpx
+
+from searx import searx_dir  # pylint: disable=E0401 C0413
+from searx.external_bang import LEAF_KEY
+
+# from https://duckduckgo.com/newbang
+URL_BV1 = 'https://duckduckgo.com/bv1.js'
+RE_BANG_VERSION = re.compile(r'\/bang\.v([0-9]+)\.js')
+HTTPS_COLON = 'https:'
+HTTP_COLON = 'http:'
+
+
+def get_bang_url():
+    response = httpx.get(URL_BV1)
+    response.raise_for_status()
+
+    r = RE_BANG_VERSION.findall(response.text)
+    return f'https://duckduckgo.com/bang.v{r[0]}.js', r[0]
+
+
+def fetch_ddg_bangs(url):
+    response = httpx.get(url)
+    response.raise_for_status()
+    return json.loads(response.content.decode())
+
+
+
[docs]def merge_when_no_leaf(node):
+    """Minimize the number of nodes
+
+    ``A -> B -> C``
+
+    - ``B`` is child of ``A``
+    - ``C`` is child of ``B``
+
+    If there are no ``C`` equals to ``<LEAF_KEY>``, then each ``C`` are merged
+    into ``A``.  For example (5 nodes)::
+
+      d -> d -> g -> <LEAF_KEY> (ddg)
+        -> i -> g -> <LEAF_KEY> (dig)
+
+    becomes (3 noodes)::
+
+      d -> dg -> <LEAF_KEY>
+        -> ig -> <LEAF_KEY>
+
+    """
+    restart = False
+    if not isinstance(node, dict):
+        return
+
+    # create a copy of the keys so node can be modified
+    keys = list(node.keys())
+
+    for key in keys:
+        if key == LEAF_KEY:
+            continue
+
+        value = node[key]
+        value_keys = list(value.keys())
+        if LEAF_KEY not in value_keys:
+            for value_key in value_keys:
+                node[key + value_key] = value[value_key]
+                merge_when_no_leaf(node[key + value_key])
+            del node[key]
+            restart = True
+        else:
+            merge_when_no_leaf(value)
+
+    if restart:
+        merge_when_no_leaf(node)

+
+
+def optimize_leaf(parent, parent_key, node):
+    if not isinstance(node, dict):
+        return
+
+    if len(node) == 1 and LEAF_KEY in node and parent is not None:
+        parent[parent_key] = node[LEAF_KEY]
+    else:
+        for key, value in node.items():
+            optimize_leaf(node, key, value)
+
+
+def parse_ddg_bangs(ddg_bangs):
+    bang_trie = {}
+    bang_urls = {}
+
+    for bang_definition in ddg_bangs:
+        # bang_list
+        bang_url = bang_definition['u']
+        if '{{{s}}}' not in bang_url:
+            # ignore invalid bang
+            continue
+
+        bang_url = bang_url.replace('{{{s}}}', chr(2))
+
+        # only for the https protocol: "https://example.com" becomes "//example.com"
+        if bang_url.startswith(HTTPS_COLON + '//'):
+            bang_url = bang_url[len(HTTPS_COLON) :]
+
+        #
+        if bang_url.startswith(HTTP_COLON + '//') and bang_url[len(HTTP_COLON) :] in bang_urls:
+            # if the bang_url uses the http:// protocol, and the same URL exists in https://
+            # then reuse the https:// bang definition. (written //example.com)
+            bang_def_output = bang_urls[bang_url[len(HTTP_COLON) :]]
+        else:
+            # normal use case : new http:// URL or https:// URL (without "https:", see above)
+            bang_rank = str(bang_definition['r'])
+            bang_def_output = bang_url + chr(1) + bang_rank
+            bang_def_output = bang_urls.setdefault(bang_url, bang_def_output)
+
+        bang_urls[bang_url] = bang_def_output
+
+        # bang name
+        bang = bang_definition['t']
+
+        # bang_trie
+        t = bang_trie
+        for bang_letter in bang:
+            t = t.setdefault(bang_letter, {})
+        t = t.setdefault(LEAF_KEY, bang_def_output)
+
+    # optimize the trie
+    merge_when_no_leaf(bang_trie)
+    optimize_leaf(None, None, bang_trie)
+
+    return bang_trie
+
+
+def get_bangs_filename():
+    return join(join(searx_dir, "data"), "external_bangs.json")
+
+
+if __name__ == '__main__':
+    bangs_url, bangs_version = get_bang_url()
+    print(f'fetch bangs from {bangs_url}')
+    output = {'version': bangs_version, 'trie': parse_ddg_bangs(fetch_ddg_bangs(bangs_url))}
+    with open(get_bangs_filename(), 'w', encoding="utf8") as fp:
+        json.dump(output, fp, ensure_ascii=False, indent=4)
+
+ +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/_sources/admin/answer-captcha.rst.txt b/_sources/admin/answer-captcha.rst.txt new file mode 100644 index 000000000..7ae29be0c --- /dev/null +++ b/_sources/admin/answer-captcha.rst.txt @@ -0,0 +1,69 @@ +=============================== +Answer CAPTCHA from server's IP +=============================== + +With a SSH tunnel we can send requests from server's IP and solve a CAPTCHA that +blocks requests from this IP. If your SearXNG instance is hosted at +``example.org`` and your login is ``user`` you can setup a proxy simply by +:man:`ssh`: + +.. code:: bash + + # SOCKS server: socks://127.0.0.1:8080 + + $ ssh -q -N -D 8080 user@example.org + +The ``socks://localhost:8080`` from above can be tested by: + +.. tabs:: + + .. group-tab:: server's IP + + .. code:: bash + + $ curl -x socks://127.0.0.1:8080 http://ipecho.net/plain + n.n.n.n + + .. group-tab:: desktop's IP + + .. code:: bash + + $ curl http://ipecho.net/plain + x.x.x.x + +In the settings of the WEB browser open the *"Network Settings"* and setup a +proxy on ``SOCKS5 127.0.0.1:8080`` (see screenshot below). In the WEB browser +check the IP from the server is used: + +- http://ipecho.net/plain + +Now open the search engine that blocks requests from your server's IP. If you +have `issues with the qwant engine +`__, +solve the CAPTCHA from `qwant.com `__. + +----- + +.. tabs:: + + .. group-tab:: Firefox + + .. kernel-figure:: answer-captcha/ffox-setting-proxy-socks.png + :alt: FFox proxy on SOCKS5, 127.0.0.1:8080 + + Firefox's network settings + + +.. admonition:: :man:`ssh` manual: + + -D [bind_address:]port + Specifies a local “dynamic” application-level port forwarding. This works + by allocating a socket to listen to port on the local side .. Whenever a + connection is made to this port, the connection is forwarded over the + secure channel, and the application protocol is then used to determine + where to connect to from the remote machine .. ssh will act as a SOCKS + server .. + + -N + Do not execute a remote command. This is useful for just forwarding ports. + diff --git a/_sources/admin/api.rst.txt b/_sources/admin/api.rst.txt new file mode 100644 index 000000000..7bf465404 --- /dev/null +++ b/_sources/admin/api.rst.txt @@ -0,0 +1,96 @@ +.. _adminapi: + +================== +Administration API +================== + +Get configuration data +====================== + +.. code:: http + + GET /config HTTP/1.1 + +Sample response +--------------- + +.. code:: json + + { + "autocomplete": "", + "categories": [ + "map", + "it", + "images", + ], + "default_locale": "", + "default_theme": "simple", + "engines": [ + { + "categories": [ + "map" + ], + "enabled": true, + "name": "openstreetmap", + "shortcut": "osm" + }, + { + "categories": [ + "it" + ], + "enabled": true, + "name": "arch linux wiki", + "shortcut": "al" + }, + { + "categories": [ + "images" + ], + "enabled": true, + "name": "google images", + "shortcut": "goi" + }, + { + "categories": [ + "it" + ], + "enabled": false, + "name": "bitbucket", + "shortcut": "bb" + }, + ], + "instance_name": "searx", + "locales": { + "de": "Deutsch (German)", + "en": "English", + "eo": "Esperanto (Esperanto)", + }, + "plugins": [ + { + "enabled": true, + "name": "HTTPS rewrite" + }, + { + "enabled": false, + "name": "Vim-like hotkeys" + } + ], + "safe_search": 0 + } + + +Embed search bar +================ + +The search bar can be embedded into websites. Just paste the example into the +HTML of the site. URL of the SearXNG instance and values are customizable. + +.. code:: html + +
+ + + + + + diff --git a/_sources/admin/architecture.rst.txt b/_sources/admin/architecture.rst.txt new file mode 100644 index 000000000..d0d40715d --- /dev/null +++ b/_sources/admin/architecture.rst.txt @@ -0,0 +1,38 @@ +.. _architecture: + +============ +Architecture +============ + +.. sidebar:: Further reading + + - Reverse Proxy: :ref:`Apache ` & :ref:`nginx ` + - uWSGI: :ref:`searxng uwsgi` + - SearXNG: :ref:`installation basic` + +Herein you will find some hints and suggestions about typical architectures of +SearXNG infrastructures. + +.. _architecture uWSGI: + +uWSGI Setup +=========== + +We start with a *reference* setup for public SearXNG instances which can be build +up and maintained by the scripts from our :ref:`toolboxing`. + +.. _arch public: + +.. kernel-figure:: arch_public.dot + :alt: arch_public.dot + + Reference architecture of a public SearXNG setup. + +The reference installation activates ``server.limiter``, ``server.image_proxy`` +and ``ui.static_use_hash`` (:origin:`/etc/searxng/settings.yml +`) + +.. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # preferences: diff --git a/_sources/admin/buildhosts.rst.txt b/_sources/admin/buildhosts.rst.txt new file mode 100644 index 000000000..b7f317f74 --- /dev/null +++ b/_sources/admin/buildhosts.rst.txt @@ -0,0 +1,163 @@ +.. _buildhosts: + +========== +Buildhosts +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +To get best results from build, it's recommend to install additional packages on +build hosts (see :ref:`searxng.sh`). + +.. _searxng.sh install buildhost: + +Build and Development tools +=========================== + +To Install tools used by build and development tasks in once: + +.. tabs:: + + .. group-tab:: SearXNG's development tools + + .. code:: sh + + $ sudo -H ./utils/searxng.sh install buildhost + +This will install packages needed by SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +and packages needed to build docuemtation and run tests: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START build-packages + :end-before: END build-packages + +.. _docs build: + +Build docs +========== + +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org +.. _XeTeX: https://tug.org/xetex/ +.. _dvisvgm: https://dvisvgm.de/ + +.. sidebar:: Sphinx build needs + + - ImageMagick_ + - Graphviz_ + - XeTeX_ + - dvisvgm_ + +Most of the sphinx requirements are installed from :origin:`setup.py` and the +docs can be build from scratch with ``make docs.html``. For better math and +image processing additional packages are needed. The XeTeX_ needed not only for +PDF creation, it's also needed for :ref:`math` when HTML output is build. + +To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered +as images (``sphinx.ext.imgmath`` extension). + +Here is the extract from the :origin:`docs/conf.py` file, setting math renderer +to ``imgmath``: + +.. literalinclude:: ../conf.py + :language: python + :start-after: # sphinx.ext.imgmath setup + :end-before: # sphinx.ext.imgmath setup END + +If your docs build (``make docs.html``) shows warnings like this:: + + WARNING: dot(1) not found, for better output quality install \ + graphviz from https://www.graphviz.org + .. + WARNING: LaTeX command 'latex' cannot be run (needed for math \ + display), check the imgmath_latex setting + +you need to install additional packages on your build host, to get better HTML +output (:ref:`install buildhost `). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S graphviz imagemagick texlive-bin extra/librsvg + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools + + +For PDF output you also need: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + $ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu + + .. group-tab:: Arch Linux + + .. code:: sh + + $ sudo pacman -S texlive-core texlive-latexextra ttf-dejavu + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + $ sudo dnf install \ + texlive-collection-fontsrecommended texlive-collection-latex \ + dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts + +.. _sh lint: + +Lint shell scripts +================== + +.. _ShellCheck: https://github.com/koalaman/shellcheck + +To lint shell scripts we use ShellCheck_ - a shell script static analysis tool +(:ref:`install buildhost `). + +.. SNIP sh lint requirements + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install shellcheck + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S shellcheck + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install ShellCheck + +.. SNAP sh lint requirements diff --git a/_sources/admin/index.rst.txt b/_sources/admin/index.rst.txt new file mode 100644 index 000000000..583b6b428 --- /dev/null +++ b/_sources/admin/index.rst.txt @@ -0,0 +1,21 @@ +=========================== +Administrator documentation +=========================== + +.. toctree:: + :maxdepth: 2 + + settings/index + installation + installation-docker + installation-scripts + installation-searxng + installation-uwsgi + installation-nginx + installation-apache + update-searxng + answer-captcha + api + architecture + plugins + buildhosts diff --git a/_sources/admin/installation-apache.rst.txt b/_sources/admin/installation-apache.rst.txt new file mode 100644 index 000000000..9fbb187c5 --- /dev/null +++ b/_sources/admin/installation-apache.rst.txt @@ -0,0 +1,388 @@ +.. _installation apache: + +====== +Apache +====== + +.. _Apache: https://httpd.apache.org/ +.. _Apache Debian: + https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x): +.. _apache2.README.Debian: + https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian +.. _Apache Arch Linux: + https://wiki.archlinux.org/index.php/Apache_HTTP_Server +.. _Apache Fedora: + https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html +.. _Apache directives: + https://httpd.apache.org/docs/trunk/mod/directives.html +.. _Getting Started: + https://httpd.apache.org/docs/current/en/getting-started.html +.. _Terms Used to Describe Directives: + https://httpd.apache.org/docs/current/en/mod/directive-dict.html +.. _Configuration Files: + https://httpd.apache.org/docs/current/en/configuring.html +.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost +.. _LoadModule: + https://httpd.apache.org/docs/mod/mod_so.html#loadmodule +.. _IncludeOptional: + https://httpd.apache.org/docs/mod/core.html#includeoptional +.. _DocumentRoot: + https://httpd.apache.org/docs/trunk/mod/core.html#documentroot +.. _Location: + https://httpd.apache.org/docs/trunk/mod/core.html#location +.. _uWSGI Apache support: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html +.. _mod_proxy_uwsgi: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi +.. _mod_proxy_http: + https://httpd.apache.org/docs/current/mod/mod_proxy_http.html +.. _mod_proxy: + https://httpd.apache.org/docs/current/mod/mod_proxy.html + + +This section explains how to set up a SearXNG instance using the HTTP server Apache_. +If you did use the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site ` using +:ref:`searxng.sh `: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install apache + +If you have special interests or problems with setting up Apache, the following +section might give you some guidance. + + +.. sidebar:: further read + + - `Apache Arch Linux`_ + - `Apache Debian`_ + - `apache2.README.Debian`_ + - `Apache Fedora`_ + - `Apache directives`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The Apache HTTP server +====================== + +If Apache_ is not installed, install it now. If apache_ is new to you, the +`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe +Directives`_ documentation gives first orientation. There is also a list of +`Apache directives`_ *to keep in the pocket*. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install apache2 + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H pacman -S apache + sudo -H systemctl enable httpd + sudo -H systemctl start http + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H dnf install httpd + sudo -H systemctl enable httpd + sudo -H systemctl start httpd + +Now at http://localhost you should see some kind of *Welcome* or *Test* page. +How this default site is configured, depends on the linux distribution +(compare `Apache directives`_). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/apache2/sites-enabled/000-default.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot /var/www/html + + And the *welcome* page is the HTML file at ``/var/www/html/index.html``. + + .. group-tab:: Arch Linux + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot "/srv/http" + + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + + + The *welcome* page of Arch Linux is a page showing the directory located + at ``DocumentRoot``. This *directory* page is generated by the Module + `mod_autoindex `_: + + .. code:: apache + + LoadModule autoindex_module modules/mod_autoindex.so + ... + Include conf/extra/httpd-autoindex.conf + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the ``DocumentRoot`` directive: + + .. code:: apache + + DocumentRoot "/var/www/html" + ... + + AllowOverride None + # Allow open access: + Require all granted + + + On fresh installations, the ``/var/www`` is empty and the *default + welcome page* is shown, the configuration is located at:: + + less /etc/httpd/conf.d/welcome.conf + + +.. _Debian's Apache layout: + +Debian's Apache layout +---------------------- + +Be aware, Debian's Apache layout is quite different from the standard Apache +configuration. For details look at the apache2.README.Debian_ +(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on +Debian: + +* :man:`apache2ctl`: Apache HTTP server control interface +* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules +* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations +* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites + +.. _apache modules: + +Apache modules +-------------- + +To load additional modules, in most distributions you have to un-comment the +lines with the corresponding LoadModule_ directive, except in :ref:`Debian's +Apache layout`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + :ref:`Debian's Apache layout` uses :man:`a2enmod` and :man:`a2dismod` to + activate or disable modules: + + .. code:: bash + + sudo -H a2enmod ssl + sudo -H a2enmod headers + sudo -H a2enmod proxy + sudo -H a2enmod proxy_http + sudo -H a2enmod proxy_uwsgi + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + +.. _apache sites: + +Apache sites +------------ + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + In :ref:`Debian's Apache layout` you create a ``searxng.conf`` with the + ```` directive and save this file in the *sites + available* folder at ``/etc/apache2/sites-available``. To enable the + ``searxng.conf`` use :man:`a2ensite`: + + .. code:: bash + + sudo -H a2ensite searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + +.. _apache searxng site: + +Apache's SearXNG site +===================== + +.. _mod_uwsgi: https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi + +.. sidebar:: uWSGI + + Use mod_proxy_uwsgi_ / don't use the old mod_uwsgi_ anymore. + +To proxy the incoming requests to the SearXNG instance Apache needs the +mod_proxy_ module (:ref:`apache modules`). + +.. sidebar:: HTTP headers + + With ProxyPreserveHost_ the incoming ``Host`` header is passed to the proxied + host. + +Depending on what your SearXNG installation is listening on, you need a http +mod_proxy_http_) or socket (mod_proxy_uwsgi_) communication to upstream. + +The :ref:`installation scripts` installs the :ref:`reference setup +` and a :ref:`uwsgi setup` that listens on a socket by default. +You can install and activate your own ``searxng.conf`` like shown in +:ref:`apache sites`. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache socket + :end-before: END apache socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache http + :end-before: END apache http + +.. _restart apache: + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart apache2 + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H touch /etc/uwsgi.d/searxng.ini + + +disable logs +============ + +For better privacy you can disable Apache logs. In the examples above activate +one of the lines and `restart apache`_: + +.. code:: apache + + SetEnvIf Request_URI "/searxng" dontlog + # CustomLog /dev/null combined env=dontlog + +The ``CustomLog`` directive disables logs for the entire (virtual) server, use it +when the URL of the service does not have a path component (``/searxng``), so when +SearXNG is located at root (``/``). diff --git a/_sources/admin/installation-docker.rst.txt b/_sources/admin/installation-docker.rst.txt new file mode 100644 index 000000000..eb52c5828 --- /dev/null +++ b/_sources/admin/installation-docker.rst.txt @@ -0,0 +1,194 @@ +.. _installation docker: + +================ +Docker Container +================ + +.. _ENTRYPOINT: https://docs.docker.com/engine/reference/builder/#entrypoint +.. _searxng/searxng @dockerhub: https://hub.docker.com/r/searxng/searxng +.. _searxng-docker: https://github.com/searxng/searxng-docker +.. _[caddy]: https://hub.docker.com/_/caddy +.. _Redis: https://redis.io/ + +---- + +.. sidebar:: info + + - `searxng/searxng @dockerhub`_ + - :origin:`Dockerfile` + - `Docker overview `_ + - `Docker Cheat Sheet `_ + - `Alpine Linux `_ + `(wiki) `__ + `apt packages `_ + - Alpine's ``/bin/sh`` is :man:`dash` + +**If you intend to create a public instance using Docker, use our well maintained +docker container** + +- `searxng/searxng @dockerhub`_. + +.. sidebar:: hint + + The rest of this article is of interest only to those who want to create and + maintain their own Docker images. + +The sources are hosted at searxng-docker_ and the container includes: + +- a HTTPS reverse proxy `[caddy]`_ and +- a Redis_ DB + +The `default SearXNG setup `_ +of this container: + +- enables :ref:`limiter ` to protect against bots +- enables :ref:`image proxy ` for better privacy +- enables :ref:`cache busting ` to save bandwith + +---- + + +Get Docker +========== + +If you plan to build and maintain a docker image by yourself, make sure you have +`Docker installed `_. On Linux don't +forget to add your user to the docker group (log out and log back in so that +your group membership is re-evaluated): + +.. code:: sh + + $ sudo usermod -a -G docker $USER + + +searxng/searxng +=============== + +.. sidebar:: ``docker run`` + + - `-\-rm `__ + automatically clean up when container exits + - `-d `__ start + detached container + - `-v `__ + mount volume ``HOST:CONTAINER`` + +The docker image is based on :origin:`Dockerfile` and available from +`searxng/searxng @dockerhub`_. Using the docker image is quite easy, for +instance you can pull the `searxng/searxng @dockerhub`_ image and deploy a local +instance using `docker run `_: + +.. code:: sh + + $ mkdir my-instance + $ cd my-instance + $ export PORT=8080 + $ docker pull searxng/searxng + $ docker run --rm \ + -d -p ${PORT}:8080 \ + -v "${PWD}/searxng:/etc/searxng" \ + -e "BASE_URL=http://localhost:$PORT/" \ + -e "INSTANCE_NAME=my-instance" \ + searxng/searxng + 2f998.... # container's ID + +Open your WEB browser and visit the URL: + +.. code:: sh + + $ xdg-open "http://localhost:$PORT" + +Inside ``${PWD}/searxng``, you will find ``settings.yml`` and ``uwsgi.ini``. You +can modify these files according to your needs and restart the Docker image. + +.. code:: sh + + $ docker container restart 2f998 + +Use command ``container ls`` to list running containers, add flag `-a +`__ to list +exited containers also. With ``container stop`` a running container can be +stoped. To get rid of a container use ``container rm``: + +.. code:: sh + + $ docker container ls + CONTAINER ID IMAGE COMMAND CREATED ... + 2f998d725993 searxng/searxng "/sbin/tini -- /usr/…" 7 minutes ago ... + + $ docker container stop 2f998 + $ docker container rm 2f998 + +.. sidebar:: Warning + + This might remove all docker items, not only those from SearXNG. + +If you won't use docker anymore and want to get rid of all containers & images +use the following *prune* command: + +.. code:: sh + + $ docker stop $(docker ps -aq) # stop all containers + $ docker system prune # make some housekeeping + $ docker rmi -f $(docker images -q) # drop all images + + +shell inside container +---------------------- + +.. sidebar:: Bashism + + - `A tale of two shells: bash or dash `_ + - `How to make bash scripts work in dash `_ + - `Checking for Bashisms `_ + +Like in many other distributions, Alpine's `/bin/sh +`__ is :man:`dash`. Dash is meant to be +`POSIX-compliant `__. +Compared to debian, in the Alpine image :man:`bash` is not installed. The +:origin:`dockerfiles/docker-entrypoint.sh` script is checked *against dash* +(``make tests.shell``). + +To open a shell inside the container: + +.. code:: sh + + $ docker exec -it 2f998 sh + + +Build the image +=============== + +It's also possible to build SearXNG from the embedded :origin:`Dockerfile`:: + + $ git clone https://github.com/searxng/searxng.git + $ cd searxng + $ make docker.build + ... + Successfully built 49586c016434 + Successfully tagged searxng/searxng:latest + Successfully tagged searxng/searxng:1.0.0-209-9c823800-dirty + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + searxng/searxng 1.0.0-209-9c823800-dirty 49586c016434 13 minutes ago 308MB + searxng/searxng latest 49586c016434 13 minutes ago 308MB + alpine 3.13 6dbb9cc54074 3 weeks ago 5.61MB + + +Command line +============ + +.. sidebar:: docker run + + Use flags ``-it`` for `interactive processes + `__. + +In the :origin:`Dockerfile` the ENTRYPOINT_ is defined as +:origin:`dockerfiles/docker-entrypoint.sh` + +.. code:: sh + + docker run --rm -it searxng/searxng -h + +.. program-output:: ../dockerfiles/docker-entrypoint.sh -h diff --git a/_sources/admin/installation-nginx.rst.txt b/_sources/admin/installation-nginx.rst.txt new file mode 100644 index 000000000..f95354b53 --- /dev/null +++ b/_sources/admin/installation-nginx.rst.txt @@ -0,0 +1,252 @@ +.. _installation nginx: + +===== +NGINX +===== + +.. _nginx: + https://docs.nginx.com/nginx/admin-guide/ +.. _nginx server configuration: + https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers +.. _nginx beginners guide: + https://nginx.org/en/docs/beginners_guide.html +.. _Getting Started wiki: + https://www.nginx.com/resources/wiki/start/ +.. _uWSGI support from nginx: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html +.. _uwsgi_params: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx +.. _SCRIPT_NAME: + https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name + +This section explains how to set up a SearXNG instance using the HTTP server nginx_. +If you have used the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site ` using +:ref:`searxng.sh `: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install nginx + +If you have special interests or problems with setting up nginx, the following +section might give you some guidance. + + +.. sidebar:: further reading + + - nginx_ + - `nginx beginners guide`_ + - `nginx server configuration`_ + - `Getting Started wiki`_ + - `uWSGI support from nginx`_ + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The nginx HTTP server +===================== + +If nginx_ is not installed, install it now. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install nginx + + .. group-tab:: Arch Linux + + .. code-block:: sh + + sudo -H pacman -S nginx-mainline + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + sudo -H dnf install nginx + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + +Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you +see a *Fedora Webserver - Test Page*. The test page comes from the default +`nginx server configuration`_. How this default site is configured, +depends on the linux distribution: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/sites-enabled/*; + + .. group-tab:: Arch Linux + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is a configuration section named ``server``: + + .. code-block:: nginx + + server { + listen 80; + server_name localhost; + # ... + } + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/conf.d/*.conf; + + +.. _nginx searxng site: + +NGINX's SearXNG site +==================== + +Now you have to create a configuration file (``searxng.conf``) for the SearXNG +site. If nginx_ is new to you, the `nginx beginners guide`_ is a good starting +point and the `Getting Started wiki`_ is always a good resource *to keep in the +pocket*. + +Depending on what your SearXNG installation is listening on, you need a http or socket +communication to upstream. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx socket + :end-before: END nginx socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx http + :end-before: END nginx http + +The :ref:`installation scripts` installs the :ref:`reference setup +` and a :ref:`uwsgi setup` that listens on a socket by default. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + Create configuration at ``/etc/nginx/sites-available/`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/sites-available/searxng.conf \ + /etc/nginx/sites-enabled/searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/nginx/nginx.conf`` file, in the ``server`` section add a + `include `_ + directive: + + .. code:: nginx + + server { + # ... + include /etc/nginx/default.d/*.conf; + # ... + } + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.d + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``default.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/default.d/searxng.conf + + .. group-tab:: Fedora / RHEL + + Create a folder for the *available sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``conf.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/conf.d/searxng.conf + +Restart services: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H touch /etc/uwsgi.d/searxng.ini + + +Disable logs +============ + +For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``. + +.. code:: nginx + + http { + # ... + access_log /dev/null; + error_log /dev/null; + # ... + } diff --git a/_sources/admin/installation-scripts.rst.txt b/_sources/admin/installation-scripts.rst.txt new file mode 100644 index 000000000..2d43f5e37 --- /dev/null +++ b/_sources/admin/installation-scripts.rst.txt @@ -0,0 +1,62 @@ +.. _installation scripts: + +=================== +Installation Script +=================== + +.. sidebar:: Update the OS first! + + To avoid unwanted side effects, update your OS before installing SearXNG. + +The following will install a setup as shown in :ref:`the reference architecture +`. First you need to get a clone of the repository. The clone is only needed for +the installation procedure and some maintenance tasks. + +.. sidebar:: further read + + - :ref:`toolboxing` + +Jump to a folder that is readable by *others* and start to clone SearXNG, +alternatively you can create your own fork and clone from there. + +.. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: further read + + - :ref:`inspect searxng` + +To install a SearXNG :ref:`reference setup ` +including a :ref:`uWSGI setup ` as described in the +:ref:`installation basic` and in the :ref:`searxng uwsgi` section type: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install all + +.. attention:: + + For the installation procedure, use a *sudoer* login to run the scripts. If + you install from ``root``, take into account that the scripts are creating a + ``searxng`` user. In the installation procedure this new created user does + need to have read access to the cloned SearXNG repository, which is not the case if you clone + it into a folder below ``/root``! + +.. sidebar:: further read + + - :ref:`update searxng` + +.. _caddy: https://hub.docker.com/_/caddy + +When all services are installed and running fine, you can add SearXNG to your +HTTP server. We do not have any preferences regarding the HTTP server, you can use +whatever you prefer. + +We use caddy in our :ref:`docker image ` and we have +implemented installation procedures for: + +- :ref:`installation nginx` +- :ref:`installation apache` diff --git a/_sources/admin/installation-searxng.rst.txt b/_sources/admin/installation-searxng.rst.txt new file mode 100644 index 000000000..3efc48e8c --- /dev/null +++ b/_sources/admin/installation-searxng.rst.txt @@ -0,0 +1,132 @@ +.. _installation basic: + +========================= +Step by step installation +========================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +In this section we show the setup of a SearXNG instance that will be installed +by the :ref:`installation scripts`. + +.. _install packages: + +Install packages +================ + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +.. hint:: + + This installs also the packages needed by :ref:`searxng uwsgi` + +.. _create searxng user: + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create user + :end-before: END create user + +.. _searxng-src: + +Install SearXNG & dependencies +============================== + +Start a interactive shell from new created user and clone SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START clone searxng + :end-before: END clone searxng + +In the same shell create *virtualenv*: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create virtualenv + :end-before: END create virtualenv + +To install SearXNG's dependencies, exit the SearXNG *bash* session you opened above +and start a new one. Before installing, check if your *virtualenv* was sourced +from the login (*~/.profile*): + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START manage.sh update_packages + :end-before: END manage.sh update_packages + +.. tip:: + + Open a second terminal for the configuration tasks and leave the ``(searx)$`` + terminal open for the tasks below. + + +.. _use_default_settings.yml: + +Configuration +============= + +.. sidebar:: ``use_default_settings: True`` + + - :ref:`settings.yml` + - :ref:`settings location` + - :ref:`settings use_default_settings` + - :origin:`/etc/searxng/settings.yml ` + +To create a initial ``/etc/searxng/settings.yml`` we recommend to start with a +copy of the file :origin:`utils/templates/etc/searxng/settings.yml`. This setup +:ref:`use default settings ` from +:origin:`searx/settings.yml` and is shown in the tab *"Use default settings"* +below. This setup: + +- enables :ref:`limiter ` to protect against bots +- enables :ref:`image proxy ` for better privacy +- enables :ref:`cache busting ` to save bandwith + +Modify the ``/etc/searxng/settings.yml`` to your needs: + +.. tabs:: + + .. group-tab:: Use default settings + + .. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # hostname_replace: + + To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml` + + .. group-tab:: searx/settings.yml + + .. literalinclude:: ../../searx/settings.yml + :language: yaml + :end-before: # hostname_replace: + + To see the entire file jump to :origin:`searx/settings.yml` + +For a *minimal setup* you need to set ``server:secret_key``. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng config + :end-before: END searxng config + + +Check +===== + +To check your SearXNG setup, optional enable debugging and start the *webapp*. +SearXNG looks at the exported environment ``$SEARXNG_SETTINGS_PATH`` for a +configuration file. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START check searxng installation + :end-before: END check searxng installation + +If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the +debug option in ``settings.yml``. You can now exit SearXNG user bash session (enter exit +command twice). At this point SearXNG is not demonized; uwsgi allows this. + diff --git a/_sources/admin/installation-uwsgi.rst.txt b/_sources/admin/installation-uwsgi.rst.txt new file mode 100644 index 000000000..78da22f45 --- /dev/null +++ b/_sources/admin/installation-uwsgi.rst.txt @@ -0,0 +1,268 @@ +.. _searxng uwsgi: + +===== +uWSGI +===== + +.. sidebar:: further reading + + - `systemd.unit`_ + - `uWSGI Emperor`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html +.. _One service per app in systemd: + https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd +.. _uWSGI Emperor: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html +.. _uwsgi ini file: + https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files +.. _systemd unit template: + http://0pointer.de/blog/projects/instances.html + + +Origin uWSGI +============ + +.. _Tyrant mode: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html#tyrant-mode-secure-multi-user-hosting + +How uWSGI is implemented by distributors varies. The uWSGI project itself +recommends two methods: + +1. `systemd.unit`_ template file as described here `One service per app in systemd`_: + + There is one `systemd unit template`_ on the system installed and one `uwsgi + ini file`_ per uWSGI-app placed at dedicated locations. Take archlinux and a + ``searxng.ini`` as example:: + + systemd template unit: /usr/lib/systemd/system/uwsgi@.service + contains: [Service] + ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/%I.ini + + SearXNG application: /etc/uwsgi/searxng.ini + links to: /etc/uwsgi/apps-available/searxng.ini + + The SearXNG app (template ``/etc/uwsgi/%I.ini``) can be maintained as known + from common systemd units: + + .. code:: sh + + $ systemctl enable uwsgi@searxng + $ systemctl start uwsgi@searxng + $ systemctl restart uwsgi@searxng + $ systemctl stop uwsgi@searxng + +2. The `uWSGI Emperor`_ which fits for maintaining a large range of uwsgi + apps and there is a `Tyrant mode`_ to secure multi-user hosting. + + The Emperor mode is a special uWSGI instance that will monitor specific + events. The Emperor mode (the service) is started by a (common, not template) + systemd unit. + + The Emperor service will scan specific directories for `uwsgi ini file`_\s + (also know as *vassals*). If a *vassal* is added, removed or the timestamp is + modified, a corresponding action takes place: a new uWSGI instance is started, + reload or stopped. Take Fedora and a ``searxng.ini`` as example:: + + to install & start SearXNG instance create --> /etc/uwsgi.d/searxng.ini + to reload the instance edit timestamp --> touch /etc/uwsgi.d/searxng.ini + to stop instance remove ini --> rm /etc/uwsgi.d/searxng.ini + + +Distributors +============ + +The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors +mostly offer their users, even if they differ in the way they implement both +modes and their defaults. Another point they might differ in is the packaging of +plugins (if so, compare :ref:`install packages`) and what the default python +interpreter is (python2 vs. python3). + +While archlinux does not start a uWSGI service by default, Fedora (RHEL) starts +a Emperor in `Tyrant mode`_ by default (you should have read :ref:`uWSGI Tyrant +mode pitfalls`). Worth to know; debian (ubuntu) follow a complete different +approach, read see :ref:`Debian's uWSGI layout`. + +.. _Debian's uWSGI layout: + +Debian's uWSGI layout +--------------------- + +.. _uwsgi.README.Debian: + https://salsa.debian.org/uwsgi-team/uwsgi/-/raw/debian/latest/debian/uwsgi.README.Debian + +Be aware, Debian's uWSGI layout is quite different from the standard uWSGI +configuration. Your are familiar with :ref:`Debian's Apache layout`? .. they do a +similar thing for the uWSGI infrastructure. The folders are:: + + /etc/uwsgi/apps-available/ + /etc/uwsgi/apps-enabled/ + +The `uwsgi ini file`_ is enabled by a symbolic link:: + + ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/ + +More details can be found in the uwsgi.README.Debian_ +(``/usr/share/doc/uwsgi/README.Debian.gz``). Some commands you should know on +Debian: + +.. code:: none + + Commands recognized by init.d script + ==================================== + + You can issue to init.d script following commands: + * start | starts daemon + * stop | stops daemon + * reload | sends to daemon SIGHUP signal + * force-reload | sends to daemon SIGTERM signal + * restart | issues 'stop', then 'start' commands + * status | shows status of daemon instance (running/not running) + + 'status' command must be issued with exactly one argument: ''. + + Controlling specific instances of uWSGI + ======================================= + + You could control specific instance(s) by issuing: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi ... + + where: + * is one of 'start', 'stop' etc. + * is the name of configuration file (without extension) + + For example, this is how instance for /etc/uwsgi/apps-enabled/hello.xml is + started: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi start hello + + +.. _uWSGI maintenance: + +uWSGI maintenance +================= + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description ubuntu-20.04 + :end-before: END searxng uwsgi-description ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description arch + :end-before: END searxng uwsgi-description arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description fedora + :end-before: END searxng uwsgi-description fedora + + +.. _uwsgi setup: + +uWSGI setup +=========== + +Create the configuration ini-file according to your distribution and restart the +uwsgi application. As shown below, the :ref:`installation scripts` installs by +default: + +- a uWSGI setup that listens on a socket and +- enables :ref:`cache busting `. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini ubuntu-20.04 + :end-before: END searxng uwsgi-appini ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini arch + :end-before: END searxng uwsgi-appini arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini fedora + :end-before: END searxng uwsgi-appini fedora + + +.. _uWSGI Tyrant mode pitfalls: + +Pitfalls of the Tyrant mode +=========================== + +The implementation of the process owners and groups in the `Tyrant mode`_ is +somewhat unusual and requires special consideration. In `Tyrant mode`_ mode the +Emperor will run the vassal using the UID/GID of the vassal configuration file +(user and group of the app ``.ini`` file). + +.. _#2099@uWSGI: https://github.com/unbit/uwsgi/issues/2099 +.. _#752@uWSGI: https://github.com/unbit/uwsgi/pull/752 +.. _#2425uWSGI: https://github.com/unbit/uwsgi/issues/2425 + +Without option ``emperor-tyrant-initgroups=true`` in ``/etc/uwsgi.ini`` the +process won't get the additional groups, but this option is not available in +2.0.x branch (see `#2099@uWSGI`_) the feature `#752@uWSGI`_ has been merged (on +Oct. 2014) to the master branch of uWSGI but had never been released; the last +major release is from Dec. 2013, since the there had been only bugfix releases +(see `#2425uWSGI`_). To shorten up: + + **In Tyrant mode, there is no way to get additional groups, and the uWSGI + process misses additional permissions that may be needed.** + +For example on Fedora (RHEL): If you try to install a redis DB with socket +communication and you want to connect to it from the SearXNG uWSGI, you will see a +*Permission denied* in the log of your instance:: + + ERROR:searx.redisdb: [searxng (993)] can't connect redis DB ... + ERROR:searx.redisdb: Error 13 connecting to unix socket: /usr/local/searxng-redis/run/redis.sock. Permission denied. + ERROR:searx.plugins.limiter: init limiter DB failed!!! + +Even if your *searxng* user of the uWSGI process is added to additional groups +to give access to the socket from the redis DB:: + + $ groups searxng + searxng : searxng searxng-redis + +To see the effective groups of the uwsgi process, you have to look at the status +of the process, by example:: + + $ ps -aef | grep '/usr/sbin/uwsgi --ini searxng.ini' + searxng 93 92 0 12:43 ? 00:00:00 /usr/sbin/uwsgi --ini searxng.ini + searxng 186 93 0 12:44 ? 00:00:01 /usr/sbin/uwsgi --ini searxng.ini + +Here you can see that the additional "Groups" of PID 186 are unset (missing gid +of ``searxng-redis``):: + + $ cat /proc/186/task/186/status + ... + Uid: 993 993 993 993 + Gid: 993 993 993 993 + FDSize: 128 + Groups: + ... diff --git a/_sources/admin/installation.rst.txt b/_sources/admin/installation.rst.txt new file mode 100644 index 000000000..54d901f76 --- /dev/null +++ b/_sources/admin/installation.rst.txt @@ -0,0 +1,22 @@ +.. _installation: + +============ +Installation +============ + +*You're spoilt for choice*, choose your preferred method of installation. + +- :ref:`installation docker` +- :ref:`installation scripts` +- :ref:`installation basic` + +The :ref:`installation basic` is an excellent illustration of *how a SearXNG +instance is build up* (see :ref:`architecture uWSGI`). If you do not have any +special preferences, it's recommended to use the :ref:`installation docker` or the +:ref:`installation scripts`. + +.. attention:: + + SearXNG is growing rapidly, you should regularly read our :ref:`migrate and + stay tuned` section. If you want to upgrade an existing instance or migrate + from searx to SearXNG, you should read this section first! diff --git a/_sources/admin/plugins.rst.txt b/_sources/admin/plugins.rst.txt new file mode 100644 index 000000000..d97b3dada --- /dev/null +++ b/_sources/admin/plugins.rst.txt @@ -0,0 +1,39 @@ +.. _plugins generic: + +=============== +Plugins builtin +=============== + +.. sidebar:: Further reading .. + + - :ref:`dev plugin` + +Configuration defaults (at built time): + +:DO: Default on + +.. _configured plugins: + +.. jinja:: searx + + .. flat-table:: Plugins configured at built time (defaults) + :header-rows: 1 + :stub-columns: 1 + :widths: 3 1 9 + + * - Name + - DO + - Description + + JS & CSS dependencies + + {% for plgin in plugins %} + + * - {{plgin.name}} + - {{(plgin.default_on and "y") or ""}} + - {{plgin.description}} + + {% for dep in (plgin.js_dependencies + plgin.css_dependencies) %} + | ``{{dep}}`` {% endfor %} + + {% endfor %} diff --git a/_sources/admin/settings/index.rst.txt b/_sources/admin/settings/index.rst.txt new file mode 100644 index 000000000..005ee37e1 --- /dev/null +++ b/_sources/admin/settings/index.rst.txt @@ -0,0 +1,25 @@ +======== +Settings +======== + +.. sidebar:: Further reading .. + + - :ref:`engine settings` + - :ref:`engine file` + +.. toctree:: + :maxdepth: 2 + + settings + settings_engine + settings_brand + settings_general + settings_search + settings_server + settings_ui + settings_redis + settings_outgoing + settings_categories_as_tabs + + + diff --git a/_sources/admin/settings/settings.rst.txt b/_sources/admin/settings/settings.rst.txt new file mode 100644 index 000000000..9c6fb01be --- /dev/null +++ b/_sources/admin/settings/settings.rst.txt @@ -0,0 +1,117 @@ +.. _settings.yml: + +================ +``settings.yml`` +================ + +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + +.. sidebar:: Further reading .. + + - :ref:`use_default_settings.yml` + - :ref:`search API` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _settings location: + +settings.yml location +===================== + +The initial ``settings.yml`` we be load from these locations: + +1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable. +2. ``/etc/searxng/settings.yml`` + +If these files don't exist (or are empty or can't be read), SearXNG uses the +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. + + + +.. _settings use_default_settings: + +use_default_settings +==================== + +.. sidebar:: ``use_default_settings: true`` + + - :ref:`settings location` + - :ref:`use_default_settings.yml` + - :origin:`/etc/searxng/settings.yml ` + +The user defined ``settings.yml`` is loaded from the :ref:`settings location` +and can relied on the default configuration :origin:`searx/settings.yml` using: + + ``use_default_settings: true`` + +``server:`` + In the following example, the actual settings are the default settings defined + in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and + the ``bind_address``: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + bind_address: "0.0.0.0" + +``engines:`` + With ``use_default_settings: true``, each settings can be override in a + similar way, the ``engines`` section is merged according to the engine + ``name``. In this example, SearXNG will load all the default engines, will + enable the ``bing`` engine and define a :ref:`token ` for + the arch linux engine: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + - name: bing + disabled: false + + +``engines:`` / ``remove:`` + It is possible to remove some engines from the default settings. The following + example is similar to the above one, but SearXNG doesn't load the the google + engine: + + .. code:: yaml + + use_default_settings: + engines: + remove: + - google + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``keep_only:`` + As an alternative, it is possible to specify the engines to keep. In the + following example, SearXNG has only two engines: + + .. code:: yaml + + use_default_settings: + engines: + keep_only: + - google + - duckduckgo + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: google + tokens: ['$ecretValue'] + - name: duckduckgo + tokens: ['$ecretValue'] diff --git a/_sources/admin/settings/settings_brand.rst.txt b/_sources/admin/settings/settings_brand.rst.txt new file mode 100644 index 000000000..0f1a0d9a9 --- /dev/null +++ b/_sources/admin/settings/settings_brand.rst.txt @@ -0,0 +1,25 @@ +.. _settings brand: + +========== +``brand:`` +========== + +.. code:: yaml + + brand: + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://docs.searxng.org + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki + +``issue_url`` : + If you host your own issue tracker change this URL. + +``docs_url`` : + If you host your own documentation change this URL. + +``public_instances`` : + If you host your own https://searx.space change this URL. + +``wiki_url`` : + Link to your wiki (or ``false``) diff --git a/_sources/admin/settings/settings_categories_as_tabs.rst.txt b/_sources/admin/settings/settings_categories_as_tabs.rst.txt new file mode 100644 index 000000000..732d04678 --- /dev/null +++ b/_sources/admin/settings/settings_categories_as_tabs.rst.txt @@ -0,0 +1,31 @@ +.. _settings categories_as_tabs: + +======================= +``categories_as_tabs:`` +======================= + +A list of the categories that are displayed as tabs in the user interface. +Categories not listed here can still be searched with the :ref:`search-syntax`. + +.. code:: yaml + + categories_as_tabs: + general: + images: + videos: + news: + map: + music: + it: + science: + files: + social media: + +Engines are added to ``categories:`` (compare :ref:`engine categories`), the +categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If +there are no active engines in a category, the tab is not displayed (e.g. if a +user disables all engines in a category). + +On the preferences page (``/preferences``) -- under *engines* -- there is an +additional tab, called *other*. In this tab are all engines listed that are not +in one of the UI tabs (not included in ``categories_as_tabs``). diff --git a/_sources/admin/settings/settings_engine.rst.txt b/_sources/admin/settings/settings_engine.rst.txt new file mode 100644 index 000000000..78c400ccf --- /dev/null +++ b/_sources/admin/settings/settings_engine.rst.txt @@ -0,0 +1,244 @@ +.. _settings engine: + +=========== +``engine:`` +=========== + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`engines-dev` + +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + +.. hint:: + + A few more options are possible, but they are pretty specific to some + engines (:ref:`engine implementations`). + +.. code:: yaml + + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + send_accept_language_header: false + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weight: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + + # overwrite values from section 'outgoing:' + enable_http2: false + retries: 1 + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + using_tor_proxy: false + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 + + # other network settings + enable_http: false + retry_on_http_error: true # or 403 or [404, 429] + + +``name`` : + Name that will be used across SearXNG to define this engine. In settings, on + the result page... + +``engine`` : + Name of the python file used to handle requests and responses to and from this + search engine. + +``shortcut`` : + Code used to execute bang requests (in this case using ``!bi``) + +``base_url`` : optional + Part of the URL that should be stable across every request. Can be useful to + use multiple sites using only one engine, or updating the site URL without + touching at the code. + +``send_accept_language_header`` : + Several engines that support languages (or regions) deal with the HTTP header + ``Accept-Language`` to build a response that fits to the locale. When this + option is activated, the language (locale) that is selected by the user is used + to build and send a ``Accept-Language`` header in the request to the origin + search engine. + +.. _engine categories: + +``categories`` : optional + Specifies to which categories the engine should be added. Engines can be + assigned to multiple categories. + + Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the + UI. A search in a tab (in the UI) will query all engines that are active in + this tab. In the preferences page (``/preferences``) -- under *engines* -- + users can select what engine should be active when querying in this tab. + + Alternatively, :ref:`\!bang ` can be used to search all engines + in a category, regardless of whether they are active or not, or whether they + are in a tab of the UI or not. For example, ``!dictionaries`` can be used to + query all search engines in that category (group). + +``timeout`` : optional + Timeout of the search with the current search engine. Overwrites + ``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will + modify the global timeout of SearXNG.** + +``api_key`` : optional + In a few cases, using an API needs the use of a secret key. How to obtain them + is described in the file. + +``disabled`` : optional + To disable by default the engine, but not deleting it. It will allow the user + to manually activate it in the settings. + +``inactive``: optional + Remove the engine from the settings (*disabled & removed*). + +``language`` : optional + If you want to use another language for a specific engine, you can define it + by using the ISO code of language (and region), like ``fr``, ``en-US``, + ``de-DE``. + +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + +``weight`` : default ``1`` + Weighting of the results of this engine. + +``display_error_messages`` : default ``true`` + When an engine returns an error, the message is displayed on the user interface. + +``network`` : optional + Use the network configuration from another engine. + In addition, there are two default networks: + + - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) + - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) + +``enable_http`` : optional + Enable HTTP for this engine (by default only HTTPS is enabled). + +``retry_on_http_error`` : optional + Retry request on some HTTP status code. + + Example: + + * ``true`` : on HTTP status code between 400 and 599. + * ``403`` : on HTTP status code 403. + * ``[403, 429]``: on HTTP status code 403 and 429. + +``proxies`` : + Overwrites proxy settings from :ref:`settings outgoing`. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for this engine. The default is + taken from ``using_tor_proxy`` of the :ref:`settings outgoing`. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``max_keepalive_connection#s`` : + `Pool limit configuration`_, overwrites value ``pool_maxsize`` from + :ref:`settings outgoing` for this engine. + +``max_connections`` : + `Pool limit configuration`_, overwrites value ``pool_connections`` from + :ref:`settings outgoing` for this engine. + +``keepalive_expiry`` : + `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from + :ref:`settings outgoing` for this engine. + + +.. _private engines: + +Private Engines (``tokens``) +============================ + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through :ref:`offline engines`. Or they would +rather share engines only with their trusted friends or colleagues. + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +To solve this issue the concept of *private engines* exists. + +A new option was added to engines named `tokens`. It expects a list of strings. +If the user making a request presents one of the tokens of an engine, they can +access information about the engine and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name: arch linux wiki + engine: archlinux + shortcut: al + tokens: [ 'my-secret-token' ] + +Unless a user has configured the right token, the engine is going to be hidden +from him/her. It is not going to be included in the list of engines on the +Preferences page and in the output of `/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page under "Engine +tokens". The input expects a comma separated list of strings. + +The distribution of the tokens from the administrator to the users is not carved +in stone. As providing access to such engines implies that the admin knows and +trusts the user, we do not see necessary to come up with a strict process. +Instead, we would like to add guidelines to the documentation of the feature. + + +Example: Multilingual Search +============================ + +SearXNG does not support true multilingual search. You have to use the language +prefix in your search query when searching in a different language. + +But there is a workaround: By adding a new search engine with a different +language, SearXNG will search in your default and other language. + +Example configuration in settings.yml for a German and English speaker: + +.. code-block:: yaml + + search: + default_lang : "de" + ... + + engines: + - name : google english + engine : google + language : en + ... + +When searching, the default google engine will return German results and +"google english" will return English results. + diff --git a/_sources/admin/settings/settings_general.rst.txt b/_sources/admin/settings/settings_general.rst.txt new file mode 100644 index 000000000..02a2156b3 --- /dev/null +++ b/_sources/admin/settings/settings_general.rst.txt @@ -0,0 +1,34 @@ +.. _settings general: + +============ +``general:`` +============ + +.. code:: yaml + + general: + debug: false + instance_name: "SearXNG" + privacypolicy_url: false + donation_url: false + contact_url: false + enable_metrics: true + +``debug`` : ``$SEARXNG_DEBUG`` + Allow a more detailed log if you run SearXNG directly. Display *detailed* error + messages in the browser too, so this must be deactivated in production. + +``donation_url`` : + Set value to ``true`` to use your own donation page written in the + :ref:`searx/info/en/donate.md ` and use ``false`` to disable + the donation link altogether. + +``privacypolicy_url``: + Link to privacy policy. + +``contact_url``: + Contact ``mailto:`` address or WEB form. + +``enable_metrics``: + Enabled by default. Record various anonymous metrics available at ``/stats``, + ``/stats/errors`` and ``/preferences``. diff --git a/_sources/admin/settings/settings_outgoing.rst.txt b/_sources/admin/settings/settings_outgoing.rst.txt new file mode 100644 index 000000000..7d49ab789 --- /dev/null +++ b/_sources/admin/settings/settings_outgoing.rst.txt @@ -0,0 +1,110 @@ +.. _settings outgoing: + +============= +``outgoing:`` +============= + +Communication with search engines. + +.. code:: yaml + + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # information like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a custom server certificate + # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults + # and https://www.python-httpx.org/compatibility/#ssl-configuration + # verify: ~/.mitmproxy/mitmproxy-ca-cert.cer + # + # uncomment below section if you want to use a proxyq see: SOCKS proxies + # https://2.python-requests.org/en/latest/user/advanced/#proxies + # are also supported: see + # https://2.python-requests.org/en/latest/user/advanced/#socks + # + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # + # using_tor_proxy: true + # + # Extra seconds to add in order to account for the time taken by the proxy + # + # extra_proxy_timeout: 10.0 + # + +``request_timeout`` : + Global timeout of the requests made to others engines in seconds. A bigger + timeout will allow to wait for answers from slow engines, but in consequence + will slow SearXNG reactivity (the result page may take the time specified in the + timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`. + +``useragent_suffix`` : + Suffix to the user-agent SearXNG uses to send requests to others engines. If an + engine wish to block you, a contact info here may be useful to avoid that. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``pool_maxsize``: + Number of allowable keep-alive connections, or ``null`` to always allow. The + default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_. + +``pool_connections`` : + Maximum number of allowable connections, or ``null`` # for no limits. The + default is 100. See ``max_connections`` `Pool limit configuration`_. + +``keepalive_expiry`` : + Number of seconds to keep a connection in the pool. By default 5.0 seconds. + See ``keepalive_expiry`` `Pool limit configuration`_. + +.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying + +``proxies`` : + Define one or more proxies you wish to use, see `httpx proxies`_. + If there are more than one proxy for one protocol (http, https), + requests to the engines are distributed in a round-robin fashion. + +``source_ips`` : + If you use multiple network interfaces, define from which IP the requests must + be made. Example: + + * ``0.0.0.0`` any local IPv4 address. + * ``::`` any local IPv6 address. + * ``192.168.0.1`` + * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses + * ``fe80::60a2:1691:e5a2:ee1f`` + * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. + * ``[ 192.168.0.1, fe80::/126 ]`` + +``retries`` : + Number of retry in case of an HTTP error. On each retry, SearXNG uses an + different proxy and source ip. + +``enable_http2`` : + Enable by default. Set to ``false`` to disable HTTP/2. + +.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults +.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration + +``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR`` + Allow to specify a path to certificate. + see `httpx verification defaults`_. + + In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and + ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables. + see `httpx ssl configuration`_. + +``max_redirects`` : + 30 by default. Maximum redirect before it is an error. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for all engines. The default is + ``false`` and can be overwritten in the :ref:`settings engine` + + diff --git a/_sources/admin/settings/settings_redis.rst.txt b/_sources/admin/settings/settings_redis.rst.txt new file mode 100644 index 000000000..df5dc743f --- /dev/null +++ b/_sources/admin/settings/settings_redis.rst.txt @@ -0,0 +1,49 @@ +.. _settings redis: + +========== +``redis:`` +========== + +.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url + +A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you +will find a description to test your redis connection in SerXNG. When using +sockets, don't forget to check the access rights on the socket:: + + ls -la /usr/local/searxng-redis/run/redis.sock + srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock + +In this example read/write access is given to the *searxng-redis* group. To get +access rights to redis instance (the socket), your SearXNG (or even your +developer) account needs to be added to the *searxng-redis* group. + +``url`` : ``$SEARXNG_REDIS_URL`` + URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`:: + + redis://[[username]:[password]]@localhost:6379/0 + rediss://[[username]:[password]]@localhost:6379/0 + unix://[[username]:[password]]@/path/to/socket.sock?db=0 + +.. _Redis Developer Notes: + +Redis Developer Notes +===================== + +To set up a local redis instance, first set the socket path of the Redis DB +in your YAML setting: + +.. code:: yaml + + redis: + url: unix:///usr/local/searxng-redis/run/redis.sock?db=0 + +Then use the following commands to install the redis instance (:ref:`manage +redis.help`): + +.. code:: sh + + $ ./manage redis.build + $ sudo -H ./manage redis.install + $ sudo -H ./manage redis.addgrp "${USER}" + # don't forget to logout & login to get member of group + diff --git a/_sources/admin/settings/settings_search.rst.txt b/_sources/admin/settings/settings_search.rst.txt new file mode 100644 index 000000000..dcc0ff198 --- /dev/null +++ b/_sources/admin/settings/settings_search.rst.txt @@ -0,0 +1,97 @@ +.. _settings search: + +=========== +``search:`` +=========== + +.. code:: yaml + + search: + safe_search: 0 + autocomplete: "" + default_lang: "" + ban_time_on_fail: 5 + max_ban_time_on_fail: 120 + suspended_times: + SearxEngineAccessDenied: 86400 + SearxEngineCaptcha: 86400 + SearxEngineTooManyRequests: 3600 + cf_SearxEngineCaptcha: 1296000 + cf_SearxEngineAccessDenied: 86400 + recaptcha_SearxEngineCaptcha: 604800 + formats: + - html + +``safe_search``: + Filter results. + + - ``0``: None + - ``1``: Moderate + - ``2``: Strict + +``autocomplete``: + Existing autocomplete backends, leave blank to turn it off. + + - ``dbpedia`` + - ``duckduckgo`` + - ``google`` + - ``startpage`` + - ``swisscows`` + - ``qwant`` + - ``wikipedia`` + +``default_lang``: + Default search language - leave blank to detect from browser information or + use codes from :origin:`searx/languages.py`. + +``languages``: + List of available languages - leave unset to use all codes from + :origin:`searx/languages.py`. Otherwise list codes of available languages. + The ``all`` value is shown as the ``Default language`` in the user interface + (in most cases, it is meant to send the query without a language parameter ; + in some cases, it means the English language) Example: + + .. code:: yaml + + languages: + - all + - en + - en-US + - de + - it-IT + - fr + - fr-BE + +``ban_time_on_fail``: + Ban time in seconds after engine errors. + +``max_ban_time_on_fail``: + Max ban time in seconds after engine errors. + +``suspended_times``: + Engine suspension time after error (in seconds; set to 0 to disable) + + ``SearxEngineAccessDenied``: 86400 + For error "Access denied" and "HTTP error [402, 403]" + + ``SearxEngineCaptcha``: 86400 + For error "CAPTCHA" + + ``SearxEngineTooManyRequests``: 3600 + For error "Too many request" and "HTTP error 429" + + Cloudflare CAPTCHA: + - ``cf_SearxEngineCaptcha``: 1296000 + - ``cf_SearxEngineAccessDenied``: 86400 + + Google CAPTCHA: + - ``recaptcha_SearxEngineCaptcha``: 604800 + +``formats``: + Result formats available from web, remove format to deny access (use lower + case). + + - ``html`` + - ``csv`` + - ``json`` + - ``rss`` diff --git a/_sources/admin/settings/settings_server.rst.txt b/_sources/admin/settings/settings_server.rst.txt new file mode 100644 index 000000000..e2b4cb67d --- /dev/null +++ b/_sources/admin/settings/settings_server.rst.txt @@ -0,0 +1,54 @@ +.. _settings server: + +=========== +``server:`` +=========== + +.. code:: yaml + + server: + base_url: http://example.org/location # change this! + port: 8888 + bind_address: "127.0.0.1" + secret_key: "ultrasecretkey" # change this! + limiter: false + image_proxy: false + default_http_headers: + X-Content-Type-Options : nosniff + X-XSS-Protection : 1; mode=block + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer + + +``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv ` + The base URL where SearXNG is deployed. Used to create correct inbound links. + If you change the value, don't forget to rebuild instance's environment + (:ref:`utils/brand.env `) + +``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv ` + Port number and *bind address* of the SearXNG web application if you run it + directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG + services running behind a proxy and using socket communications. If you + change the value, don't forget to rebuild instance's environment + (:ref:`utils/brand.env `) + +``secret_key`` : ``$SEARXNG_SECRET`` + Used for cryptography purpose. + +.. _limiter: + +``limiter`` : + Rate limit the number of request on the instance, block some bots. The + :ref:`limiter src` requires a :ref:`settings redis` database. + +.. _image_proxy: + +``image_proxy`` : + Allow your instance of SearXNG of being able to proxy images. Uses memory space. + +.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers + +``default_http_headers`` : + Set additional HTTP headers, see `#755 `__ + diff --git a/_sources/admin/settings/settings_ui.rst.txt b/_sources/admin/settings/settings_ui.rst.txt new file mode 100644 index 000000000..e355199a2 --- /dev/null +++ b/_sources/admin/settings/settings_ui.rst.txt @@ -0,0 +1,62 @@ +.. _settings ui: + +======= +``ui:`` +======= + +.. _cache busting: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting + +.. code:: yaml + + ui: + static_use_hash: false + default_locale: "" + query_in_title: false + infinite_scroll: false + center_alignment: false + cache_url: https://web.archive.org/web/ + default_theme: simple + theme_args: + simple_style: auto + +.. _static_use_hash: + +``static_use_hash`` : + Enables `cache busting`_ of static files. + +``default_locale`` : + SearXNG interface language. If blank, the locale is detected by using the + browser language. If it doesn't work, or you are deploying a language + specific instance of searx, a locale can be defined using an ISO language + code, like ``fr``, ``en``, ``de``. + +``query_in_title`` : + When true, the result page's titles contains the query it decreases the + privacy, since the browser can records the page titles. + +``infinite_scroll``: + When true, automatically loads the next page when scrolling to bottom of the current page. + +``center_alignment`` : default ``false`` + When enabled, the results are centered instead of being in the left (or RTL) + side of the screen. This setting only affects the *desktop layout* + (:origin:`min-width: @tablet `) + +.. cache_url: + +``cache_url`` : ``https://web.archive.org/web/`` + URL prefix of the internet archive or cache, don't forgett trailing slash (if + needed). The default is https://web.archive.org/web/ alternatives are: + + - https://webcache.googleusercontent.com/search?q=cache: + - https://archive.today/ + +``default_theme`` : + Name of the theme you want to use by default on your SearXNG instance. + +``theme_args.simple_style``: + Style of simple theme: ``auto``, ``light``, ``dark`` + +``results_on_new_tab``: + Open result links in a new tab by default. diff --git a/_sources/admin/update-searxng.rst.txt b/_sources/admin/update-searxng.rst.txt new file mode 100644 index 000000000..27c0108d4 --- /dev/null +++ b/_sources/admin/update-searxng.rst.txt @@ -0,0 +1,138 @@ +.. _searxng maintenance: + +=================== +SearXNG maintenance +=================== + +.. sidebar:: further read + + - :ref:`toolboxing` + - :ref:`uWSGI maintenance` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _update searxng: + +How to update +============= + +How to update depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``update`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance update + +.. _inspect searxng: + +How to inspect & debug +====================== + +How to debug depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``inspect`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance inspect + +.. _migrate and stay tuned: + +Migrate and stay tuned! +======================= + +.. sidebar:: info + + - :pull:`1332` + - :pull:`456` + - :pull:`A comment about rolling release <446#issuecomment-954730358>` + +SearXNG is a *rolling release*; each commit to the master branch is a release. +SearXNG is growing rapidly, the services and opportunities are change every now +and then, to name just a few: + +- Bot protection has been switched from filtron to SearXNG's :ref:`limiter + `, this requires a :ref:`Redis ` database. + +- The image proxy morty is no longer needed, it has been replaced by the + :ref:`image proxy ` from SearXNG. + +- To save bandwith :ref:`cache busting ` has been implemented. + To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi + setup`. + +To stay tuned and get in use of the new features, instance maintainers have to +update the SearXNG code regularly (see :ref:`update searxng`). As the above +examples show, this is not always enough, sometimes services have to be set up +or reconfigured and sometimes services that are no longer needed should be +uninstalled. + +.. hint:: + + First of all: SearXNG is installed by the script :ref:`searxng.sh`. If you + have old filtron, morty or searx setup you should consider complete + uninstall/reinstall. + +Here you will find a list of changes that affect the infrastructure. Please +check to what extent it is necessary to update your installations: + +:pull:`1595`: ``[fix] uWSGI: increase buffer-size`` + Re-install uWSGI (:ref:`searxng.sh`) or fix your uWSGI ``searxng.ini`` + file manually. + + +remove obsolete services +------------------------ + +If your searx instance was installed *"Step by step"* or by the *"Installation +scripts"*, you need to undo the installation procedure completely. If you have +morty & filtron installed, it is recommended to uninstall these services also. +In case of scripts, to uninstall use the scripts from the origin you installed +searx from or try:: + + $ sudo -H ./utils/filtron.sh remove all + $ sudo -H ./utils/morty.sh remove all + $ sudo -H ./utils/searx.sh remove all + +.. hint:: + + If you are migrate from searx take into account that the ``.config.sh`` is no + longer used. + +If you upgrade from searx or from before :pull:`1332` has been merged and you +have filtron and/or morty installed, don't forget to remove HTTP sites. + +Apache:: + + $ sudo -H ./utils/filtron.sh apache remove + $ sudo -H ./utils/morty.sh apache remove + +nginx:: + + $ sudo -H ./utils/filtron.sh nginx remove + $ sudo -H ./utils/morty.sh nginx remove + + + +Check after Installation +------------------------ + +Once you have done your installation, you can run a SearXNG *check* procedure, +to see if there are some left overs. In this example there exists a *old* +``/etc/searx/settings.yml``:: + + $ sudo -H ./utils/searxng.sh instance check + + SearXNG checks + -------------- + ERROR: settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/ + INFO: [OK] (old) account 'searx' does not exists + INFO: [OK] (old) account 'filtron' does not exists + INFO: [OK] (old) account 'morty' does not exists + ... + INFO searx.redisdb : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock' + INFO searx.redisdb : connected to Redis diff --git a/_sources/dev/contribution_guide.rst.txt b/_sources/dev/contribution_guide.rst.txt new file mode 100644 index 000000000..bc4f6514d --- /dev/null +++ b/_sources/dev/contribution_guide.rst.txt @@ -0,0 +1,190 @@ +.. _how to contribute: + +================= +How to contribute +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Prime directives: Privacy, Hackability +====================================== + +SearXNG has two prime directives, **privacy-by-design and hackability** . The +hackability comes in three levels: + +- support of search engines +- plugins to alter search behaviour +- hacking SearXNG itself + +Note the lack of "world domination" among the directives. SearXNG has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. + +Privacy-by-design +----------------- + +SearXNG was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. + +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. + +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. + +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). + +Remember the other prime directive of SearXNG is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. + + *Happy hacking.* + +Code +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ +.. _Conventional Commits: https://www.conventionalcommits.org/ +.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages +.. _Structural split of changes: + https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes +.. _gitmoji: https://gitmoji.carloscuesta.me/ +.. _Semantic PR: https://github.com/zeke/semantic-pull-requests + +.. sidebar:: Create good commits! + + - `Structural split of changes`_ + - `Conventional Commits`_ + - `Git Commit Good Practice`_ + - some like to use: gitmoji_ + - not yet active: `Semantic PR`_ + +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8_ standards apply, except the convention of line length + - Maximum line length is 120 characters + +- The cardinal rule for creating good commits is to ensure there is only one + *logical change* per commit / read `Structural split of changes`_ + +- Check if your code breaks existing tests. If so, update the tests or fix your + code. + +- If your code can be unit-tested, add unit tests. + +- Add yourself to the :origin:`AUTHORS.rst` file. + +- Choose meaningful commit messages, read `Conventional Commits`_ + + .. code:: + + [optional scope]: + + [optional body] + + [optional footer(s)] + +- Create a pull request. + +For more help on getting started with SearXNG development, see :ref:`devquickstart`. + + +Translation +=========== + +Translation currently takes place on :ref:`weblate `. + + +.. _contrib docs: + +Documentation +============= + +.. _Sphinx: https://www.sphinx-doc.org +.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + +.. sidebar:: The reST sources + + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +our :ref:`makefile`. + +Here is an example which makes a complete rebuild: + +.. code:: sh + + $ make docs.clean docs.html + ... + The HTML pages are in dist/docs. + +.. _make docs.live: + +live build +---------- + +.. _sphinx-autobuild: + https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md + +.. sidebar:: docs.clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs.clean``). + +Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target ``docs.live`` builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed (:ref:`make docs.clean`). + +.. code:: sh + + $ make docs.live + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8000 + ... Start watching changes + +Live builds are implemented by sphinx-autobuild_. Use environment +``$(SPHINXOPTS)`` to pass arguments to the sphinx-autobuild_ command. Except +option ``--host`` (which is always set to ``0.0.0.0``) you can pass any +argument. E.g to find and use a free port, use: + +.. code:: sh + + $ SPHINXOPTS="--port 0" make docs.live + ... + ... Serving on http://0.0.0.0:50593 + ... + + +.. _deploy on github.io: + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make +docs.gh-pages`, which builds the documentation and runs all the needed git add, +commit and push: + +.. code:: sh + + $ make docs.clean docs.gh-pages + +.. attention:: + + If you are working in your own brand, don't forgett to adjust your + :ref:`settings brand`. diff --git a/_sources/dev/engines/demo/demo_offline.rst.txt b/_sources/dev/engines/demo/demo_offline.rst.txt new file mode 100644 index 000000000..1b4cb887f --- /dev/null +++ b/_sources/dev/engines/demo/demo_offline.rst.txt @@ -0,0 +1,14 @@ +.. _demo offline engine: + +=================== +Demo Offline Engine +=================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_offline + :members: + diff --git a/_sources/dev/engines/demo/demo_online.rst.txt b/_sources/dev/engines/demo/demo_online.rst.txt new file mode 100644 index 000000000..9b94207f7 --- /dev/null +++ b/_sources/dev/engines/demo/demo_online.rst.txt @@ -0,0 +1,14 @@ +.. _demo online engine: + +================== +Demo Online Engine +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_online + :members: + diff --git a/_sources/dev/engines/engine_overview.rst.txt b/_sources/dev/engines/engine_overview.rst.txt new file mode 100644 index 000000000..8052a8bea --- /dev/null +++ b/_sources/dev/engines/engine_overview.rst.txt @@ -0,0 +1,468 @@ +.. _engines-dev: + +=============== +Engine Overview +=============== + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`settings engine` + +SearXNG is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between SearXNG and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. _general engine configuration: + +General Engine Configuration +============================ + +It is required to tell SearXNG the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally +``settings.yml``). The arguments in the settings file override the ones in the +engine file. + +It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following: + +.. _engine file: + +Engine File +----------- + +.. table:: Common options in the engine module + :width: 100% + + ======================= =========== ======================================================== + argument type information + ======================= =========== ======================================================== + categories list categories, in which the engine is working + paging boolean support multiple pages + time_range_support boolean support search time range + engine_type str - ``online`` :ref:`[ref] ` by + default, other possibles values are: + - ``offline`` :ref:`[ref] ` + - ``online_dictionary`` :ref:`[ref] ` + - ``online_currency`` :ref:`[ref] ` + - ``online_url_search`` :ref:`[ref] ` + ======================= =========== ======================================================== + +.. _engine settings: + +Engine ``settings.yml`` +----------------------- + +For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. + +.. table:: Common options in the engine setup (``settings.yml``) + :width: 100% + + ======================= =========== ================================================== + argument type information + ======================= =========== ================================================== + name string name of search-engine + engine string name of searxng-engine (file name without ``.py``) + enable_http bool enable HTTP (by default only HTTPS is enabled). + shortcut string shortcut of search-engine + timeout string specific timeout for search-engine + display_error_messages boolean display error messages on the web UI + proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) + ======================= =========== ================================================== + +.. _engine overrides: + +Overrides +--------- + +A few of the options have default values in the namespace of engine's python +modul, but are often overwritten by the settings. If ``None`` is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +SearXNG will not start with that engine (global names with a leading underline can +be ``None``). + +Here is an very simple example of the global names in the namespace of engine's +module: + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + _non_overwritten_global = 'foo' + + +.. table:: The naming of overrides is arbitrary / recommended overrides are: + :width: 100% + + ======================= =========== =========================================== + argument type information + ======================= =========== =========================================== + base_url string base-url, can be overwritten to use same + engine on other URL + number_of_results int maximum number of results per request + language string ISO code of language and country like en_US + api_key string api-key if required by engine + ======================= =========== =========================================== + +.. _engine request: + +Making a Request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +.. _engine request arguments: + +Passed Arguments (request) +-------------------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +.. _engine request online: + +.. table:: If the ``engine_type`` is :py:obj:`online + ` + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + url str ``''`` + method str ``'GET'`` + headers set ``{}`` + data set ``{}`` + cookies set ``{}`` + verify bool ``True`` + headers.User-Agent str a random User-Agent + category str current category, like ``'general'`` + safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) + time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` + pageno int current pagenumber + searxng_locale str SearXNG's locale selected by user. Specific language code like + ``'en'``, ``'en-US'``, or ``'all'`` if unspecified. + ====================== ============== ======================================================================== + + +.. _engine request online_dictionary: + +.. table:: If the ``engine_type`` is :py:obj:`online_dictionary + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + from_lang str specific language code like ``'en_US'`` + to_lang str specific language code like ``'en_US'`` + query str the text query without the languages + ====================== ============== ======================================================================== + +.. _engine request online_currency: + +.. table:: If the ``engine_type`` is :py:obj:`online_currency + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + amount float the amount to convert + from str ISO 4217 code + to str ISO 4217 code + from_name str currency name + to_name str currency name + ====================== ============== ======================================================================== + +.. _engine request online_url_search: + +.. table:: If the ``engine_type`` is :py:obj:`online_url_search + `, + in addition to the :ref:`online ` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + search_url dict URLs from the search query: + + .. code:: python + + { + 'http': str, + 'ftp': str, + 'data:image': str + } + ====================== ============== ======================================================================== + +Specify Request +--------------- + +The function :py:func:`def request(query, params): +` always returns the ``params`` variable, the +following parameters can be used to specify a search request: + +.. table:: + :width: 100% + + =================== =========== ========================================================================== + argument type information + =================== =========== ========================================================================== + url str requested url + method str HTTP request method + headers set HTTP header information + data set HTTP data information + cookies set HTTP cookies + verify bool Performing SSL-Validity check + allow_redirects bool Follow redirects + max_redirects int maximum redirects, hard limit + soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine + raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 + =================== =========== ========================================================================== + + +.. _engine results: +.. _engine media types: + +Result Types (``template``) +=========================== + +Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as +:ref:`template default`, the parameter ``template`` must be set to the desired +type. + +.. _template default: + +``default`` +----------- + +.. table:: Parameter of the **default** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + ========================= ===================================================== + + +.. _template images: + +``images`` +---------- + +.. table:: Parameter of the **images** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``images.html`` + ========================= ===================================================== + url string, url to the result site + title string, title of the result *(partly implemented)* + content *(partly implemented)* + publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* + img\_src string, url to the result image + thumbnail\_src string, url to a small-preview image + ========================= ===================================================== + + +.. _template videos: + +``videos`` +---------- + +.. table:: Parameter of the **videos** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``videos.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content *(not implemented yet)* + publishedDate :py:class:`datetime.datetime`, time of publish + thumbnail string, url to a small-preview image + ========================= ===================================================== + + +.. _template torrent: + +``torrent`` +----------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +.. table:: Parameter of the **torrent** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``torrent.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* + seed int, number of seeder + leech int, number of leecher + filesize int, size of file in bytes + files int, number of files + magnetlink string, magnetlink_ of the result + torrentfile string, torrentfile of the result + ========================= ===================================================== + + +.. _template map: + +``map`` +------- + +.. table:: Parameter of the **map** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``map.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + latitude latitude of result (in decimal format) + longitude longitude of result (in decimal format) + boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) + geojson geojson of result (https://geojson.org/) + osm.type type of osm-object (if OSM-Result) + osm.id id of osm-object (if OSM-Result) + address.name name of object + address.road street name of object + address.house_number house number of object + address.locality city, place of object + address.postcode postcode of object + address.country country of object + ========================= ===================================================== + + +.. _template paper: + +``paper`` +--------- + +.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ +.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types + +.. list-table:: Parameter of the **paper** media type / + see `BibTeX field types`_ and `BibTeX format`_ + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``paper.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - comments + - :py:class:`str` + - free text display in italic below the content + + * - tags + - :py:class:`List `\ [\ :py:class:`str`\ ] + - free tag list + + * - publishedDate + - :py:class:`datetime ` + - last publication date + + * - type + - :py:class:`str` + - short description of medium type, e.g. *book*, *pdf* or *html* ... + + * - authors + - :py:class:`List `\ [\ :py:class:`str`\ ] + - list of authors of the work (authors with a "s") + + * - editor + - :py:class:`str` + - list of editors of a book + + * - publisher + - :py:class:`str` + - name of the publisher + + * - journal + - :py:class:`str` + - name of the journal or magazine the article was + published in + + * - volume + - :py:class:`str` + - volume number + + * - pages + - :py:class:`str` + - page range where the article is + + * - number + - :py:class:`str` + - number of the report or the issue number for a journal article + + * - doi + - :py:class:`str` + - DOI number (like ``10.1038/d41586-018-07848-2``) + + * - issn + - :py:class:`List `\ [\ :py:class:`str`\ ] + - ISSN number like ``1476-4687`` + + * - isbn + - :py:class:`List `\ [\ :py:class:`str`\ ] + - ISBN number like ``9780201896831`` + + * - pdf_url + - :py:class:`str` + - URL to the full article, the PDF version + + * - html_url + - :py:class:`str` + - URL to full article, HTML version + diff --git a/_sources/dev/engines/enginelib.rst.txt b/_sources/dev/engines/enginelib.rst.txt new file mode 100644 index 000000000..34e3250da --- /dev/null +++ b/_sources/dev/engines/enginelib.rst.txt @@ -0,0 +1,22 @@ +.. _searx.enginelib: + +============== +Engine Library +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.enginelib + :members: + +.. _searx.enginelib.traits: + + +Engine traits +============= + +.. automodule:: searx.enginelib.traits + :members: diff --git a/_sources/dev/engines/engines.rst.txt b/_sources/dev/engines/engines.rst.txt new file mode 100644 index 000000000..0f2cb1f22 --- /dev/null +++ b/_sources/dev/engines/engines.rst.txt @@ -0,0 +1,9 @@ +.. _searx.engines loader: + +======================== +SearXNG's engines loader +======================== + +.. automodule:: searx.engines + :members: + diff --git a/_sources/dev/engines/index.rst.txt b/_sources/dev/engines/index.rst.txt new file mode 100644 index 000000000..b5c3184e0 --- /dev/null +++ b/_sources/dev/engines/index.rst.txt @@ -0,0 +1,107 @@ +.. _engine implementations: + +====================== +Engine Implementations +====================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. toctree:: + :caption: Framework Components + :maxdepth: 2 + + enginelib + engines + engine_overview + + +Engine Types +============ + +The :py:obj:`engine_type ` of an engine +determines which :ref:`search processor ` is used by +the engine. + +In this section a list of the enignes that are documented is given, a complete +list of the engines can be found in the source under: :origin:`searx/engines`. + +.. _online engines: + +Online Engines +-------------- + +.. sidebar:: info + + - :py:obj:`processors.online ` + +.. toctree:: + :maxdepth: 1 + :glob: + + demo/demo_online + xpath + mediawiki + +.. toctree:: + :maxdepth: 1 + :glob: + + online/* + +.. _offline engines: + +Offline Engines +--------------- + +.. sidebar:: info + + - :py:obj:`processors.offline ` + +.. toctree:: + :maxdepth: 1 + :glob: + + offline_concept + demo/demo_offline + offline/* + +.. _online url search: + +Online URL Search +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_url_search ` + +.. toctree:: + :maxdepth: 1 + :glob: + + online_url_search/* + +.. _online currency: + +Online Currency +--------------- + +.. sidebar:: info + + - :py:obj:`processors.online_currency ` + +*no engine of this type is documented yet / comming soon* + +.. _online dictionary: + +Online Dictionary +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_dictionary ` + +*no engine of this type is documented yet / comming soon* diff --git a/_sources/dev/engines/mediawiki.rst.txt b/_sources/dev/engines/mediawiki.rst.txt new file mode 100644 index 000000000..ce708f95b --- /dev/null +++ b/_sources/dev/engines/mediawiki.rst.txt @@ -0,0 +1,13 @@ +.. _mediawiki engine: + +================ +MediaWiki Engine +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mediawiki + :members: diff --git a/_sources/dev/engines/offline/command-line-engines.rst.txt b/_sources/dev/engines/offline/command-line-engines.rst.txt new file mode 100644 index 000000000..0a80d698e --- /dev/null +++ b/_sources/dev/engines/offline/command-line-engines.rst.txt @@ -0,0 +1,23 @@ +.. _engine command: + +==================== +Command Line Engines +==================== + +.. sidebar:: info + + - :origin:`command.py ` + - :ref:`offline engines` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +.. automodule:: searx.engines.command + :members: diff --git a/_sources/dev/engines/offline/nosql-engines.rst.txt b/_sources/dev/engines/offline/nosql-engines.rst.txt new file mode 100644 index 000000000..76f5cfb61 --- /dev/null +++ b/_sources/dev/engines/offline/nosql-engines.rst.txt @@ -0,0 +1,97 @@ +.. _nosql engines: + +=============== +NoSQL databases +=============== + +.. sidebar:: further read + + - `NoSQL databases `_ + - `redis.io `_ + - `MongoDB `_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +The following `NoSQL databases`_ are supported: + +- :ref:`engine redis_server` +- :ref:`engine mongodb` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set various options and install +dependencies before using them. + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple ` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +Furthermore, if you do not wish to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +`NoSQL databases`_ are used for storing arbitrary data without first defining +their structure. + + +.. _engine redis_server: + +Redis Server +------------ + +.. _redis: https://github.com/andymccurdy/redis-py#installation + +.. sidebar:: info + + - ``pip install`` redis_ + - redis.io_ + - :origin:`redis_server.py ` + +.. automodule:: searx.engines.redis_server + :members: + + +.. _engine mongodb: + +MongoDB +------- + +.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation + +.. sidebar:: info + + - ``pip install`` pymongo_ + - MongoDB_ + - :origin:`mongodb.py ` + + +.. automodule:: searx.engines.mongodb + :members: + diff --git a/_sources/dev/engines/offline/search-indexer-engines.rst.txt b/_sources/dev/engines/offline/search-indexer-engines.rst.txt new file mode 100644 index 000000000..fa92798cb --- /dev/null +++ b/_sources/dev/engines/offline/search-indexer-engines.rst.txt @@ -0,0 +1,62 @@ +================= +Local Search APIs +================= + +.. sidebar:: further read + + - `Comparison to alternatives + `_ + +.. contents:: + :depth: 1 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to +enable HTTP requests by setting ``enable_http`` to ``True``. + +Furthermore, if you do not want to expose these engines on a public instance, +you can still add them and limit the access by setting ``tokens`` as described +in section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. automodule:: searx.engines.meilisearch + :members: + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. automodule:: searx.engines.elasticsearch + :members: + +.. _engine solr: + +Solr +==== + +.. automodule:: searx.engines.solr + :members: + diff --git a/_sources/dev/engines/offline/sql-engines.rst.txt b/_sources/dev/engines/offline/sql-engines.rst.txt new file mode 100644 index 000000000..f0f5add0b --- /dev/null +++ b/_sources/dev/engines/offline/sql-engines.rst.txt @@ -0,0 +1,121 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite `_ + - `PostgreSQL `_ + - `MySQL `_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + `_ of `NLnet Foundation `_. + +With the *SQL engines* you can bind SQL databases into SearXNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple ` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py ` + +.. automodule:: searx.engines.sqlite + :members: + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py ` + - ``pip install`` `psycopg2-binary `_ + +.. automodule:: searx.engines.postgresql + :members: + +.. _engine mysql_server: + +MySQL +----- + +.. sidebar:: info + + - :origin:`mysql_server.py ` + - ``pip install`` :pypi:`mysql-connector-python ` + + +.. automodule:: searx.engines.mysql_server + :members: + diff --git a/_sources/dev/engines/offline_concept.rst.txt b/_sources/dev/engines/offline_concept.rst.txt new file mode 100644 index 000000000..ddb34fc60 --- /dev/null +++ b/_sources/dev/engines/offline_concept.rst.txt @@ -0,0 +1,69 @@ +=============== +Offline Concept +=============== + +.. sidebar:: offline engines + + - :ref:`demo offline engine` + - :ref:`engine command` + - :ref:`sql engines` + - :ref:`nosql engines` + - :py:obj:`searx.search.processors.offline` + +To extend the functionality of SearXNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate. + +Offline engines can be configured, by adding those to the `engines` list of +:origin:`settings.yml `. An example skeleton for offline +engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py +`). + + +Programming Interface +===================== + +:py:func:`init(engine_settings=None) ` + All offline engines can have their own init function to setup the engine before + accepting requests. The function gets the settings from settings.yml as a + parameter. This function can be omitted, if there is no need to setup anything + in advance. + +:py:func:`search(query, params) ` + Each offline engine has a function named ``search``. This function is + responsible to perform a search and return the results in a presentable + format. (Where *presentable* means presentable by the selected result + template.) + + The return value is a list of results retrieved by the engine. + +Engine representation in ``/config`` + If an engine is offline, the attribute ``offline`` is set to ``True``. + +.. _offline requirements: + +Extra Dependencies +================== + +If an offline engine depends on an external tool, SearXNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance. + +If there is a need to install additional packages in *Python's Virtual +Environment* of your SearXNG instance you need to switch into the environment +(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Private engines (Security) +========================== + +To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the :ref:`private engines`. If a +query contains a valid token, then SearXNG performs the requested private +search. If not, requests from an offline engines return errors. + diff --git a/_sources/dev/engines/online/annas_archive.rst.txt b/_sources/dev/engines/online/annas_archive.rst.txt new file mode 100644 index 000000000..db88e5069 --- /dev/null +++ b/_sources/dev/engines/online/annas_archive.rst.txt @@ -0,0 +1,13 @@ +.. _annas_archive engine: + +============== +Anna's Archive +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.annas_archive + :members: diff --git a/_sources/dev/engines/online/archlinux.rst.txt b/_sources/dev/engines/online/archlinux.rst.txt new file mode 100644 index 000000000..834fffa43 --- /dev/null +++ b/_sources/dev/engines/online/archlinux.rst.txt @@ -0,0 +1,14 @@ +.. _archlinux engine: + +========== +Arch Linux +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.archlinux + :members: + diff --git a/_sources/dev/engines/online/bing.rst.txt b/_sources/dev/engines/online/bing.rst.txt new file mode 100644 index 000000000..19c31aa80 --- /dev/null +++ b/_sources/dev/engines/online/bing.rst.txt @@ -0,0 +1,43 @@ +.. _bing engines: + +============ +Bing Engines +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _bing web engine: + +Bing WEB +======== + +.. automodule:: searx.engines.bing + :members: + +.. _bing images engine: + +Bing Images +=========== + +.. automodule:: searx.engines.bing_images + :members: + +.. _bing videos engine: + +Bing Videos +=========== + +.. automodule:: searx.engines.bing_videos + :members: + +.. _bing news engine: + +Bing News +========= + +.. automodule:: searx.engines.bing_news + :members: diff --git a/_sources/dev/engines/online/brave.rst.txt b/_sources/dev/engines/online/brave.rst.txt new file mode 100644 index 000000000..a1c589b9d --- /dev/null +++ b/_sources/dev/engines/online/brave.rst.txt @@ -0,0 +1,13 @@ +.. _brave engine: + +============= +Brave Engines +============= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.brave + :members: diff --git a/_sources/dev/engines/online/bt4g.rst.txt b/_sources/dev/engines/online/bt4g.rst.txt new file mode 100644 index 000000000..980665204 --- /dev/null +++ b/_sources/dev/engines/online/bt4g.rst.txt @@ -0,0 +1,14 @@ +.. _bt4g engine: + +==== +BT4G +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.bt4g + :members: + diff --git a/_sources/dev/engines/online/dailymotion.rst.txt b/_sources/dev/engines/online/dailymotion.rst.txt new file mode 100644 index 000000000..c661172e5 --- /dev/null +++ b/_sources/dev/engines/online/dailymotion.rst.txt @@ -0,0 +1,13 @@ +.. _dailymotion engine: + +=========== +Dailymotion +=========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.dailymotion + :members: diff --git a/_sources/dev/engines/online/duckduckgo.rst.txt b/_sources/dev/engines/online/duckduckgo.rst.txt new file mode 100644 index 000000000..d0d18eb8a --- /dev/null +++ b/_sources/dev/engines/online/duckduckgo.rst.txt @@ -0,0 +1,22 @@ +.. _duckduckgo engines: + +================= +DukcDukGo Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.duckduckgo + :members: + +.. automodule:: searx.engines.duckduckgo_images + :members: + +.. automodule:: searx.engines.duckduckgo_definitions + :members: + +.. automodule:: searx.engines.duckduckgo_weather + :members: diff --git a/_sources/dev/engines/online/google.rst.txt b/_sources/dev/engines/online/google.rst.txt new file mode 100644 index 000000000..9085070bd --- /dev/null +++ b/_sources/dev/engines/online/google.rst.txt @@ -0,0 +1,76 @@ +.. _google engines: + +============== +Google Engines +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _google API: + +Google API +========== + +.. _Query Parameter Definitions: + https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions + +SearXNG's implementation of the Google API is mainly done in +:py:obj:`get_google_info `. + +For detailed description of the *REST-full* API see: `Query Parameter +Definitions`_. The linked API documentation can sometimes be helpful during +reverse engineering. However, we cannot use it in the freely accessible WEB +services; not all parameters can be applied and some engines are more *special* +than other (e.g. :ref:`google news engine`). + + +.. _google web engine: + +Google WEB +========== + +.. automodule:: searx.engines.google + :members: + +.. _google autocomplete: + +Google Autocomplete +==================== + +.. autofunction:: searx.autocomplete.google_complete + +.. _google images engine: + +Google Images +============= + +.. automodule:: searx.engines.google_images + :members: + +.. _google videos engine: + +Google Videos +============= + +.. automodule:: searx.engines.google_videos + :members: + +.. _google news engine: + +Google News +=========== + +.. automodule:: searx.engines.google_news + :members: + +.. _google scholar engine: + +Google Scholar +============== + +.. automodule:: searx.engines.google_scholar + :members: diff --git a/_sources/dev/engines/online/lemmy.rst.txt b/_sources/dev/engines/online/lemmy.rst.txt new file mode 100644 index 000000000..584246fd1 --- /dev/null +++ b/_sources/dev/engines/online/lemmy.rst.txt @@ -0,0 +1,13 @@ +.. _lemmy engine: + +===== +Lemmy +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.lemmy + :members: diff --git a/_sources/dev/engines/online/peertube.rst.txt b/_sources/dev/engines/online/peertube.rst.txt new file mode 100644 index 000000000..bedf055fb --- /dev/null +++ b/_sources/dev/engines/online/peertube.rst.txt @@ -0,0 +1,27 @@ +.. _peertube engines: + +================ +Peertube Engines +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _peertube video engine: + +Peertube Video +============== + +.. automodule:: searx.engines.peertube + :members: + +.. _sepiasearch engine: + +SepiaSearch +=========== + +.. automodule:: searx.engines.sepiasearch + :members: diff --git a/_sources/dev/engines/online/piped.rst.txt b/_sources/dev/engines/online/piped.rst.txt new file mode 100644 index 000000000..822981056 --- /dev/null +++ b/_sources/dev/engines/online/piped.rst.txt @@ -0,0 +1,13 @@ +.. _piped engine: + +===== +Piped +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.piped + :members: diff --git a/_sources/dev/engines/online/recoll.rst.txt b/_sources/dev/engines/online/recoll.rst.txt new file mode 100644 index 000000000..2f1a1e4df --- /dev/null +++ b/_sources/dev/engines/online/recoll.rst.txt @@ -0,0 +1,13 @@ +.. _engine recoll: + +============= +Recoll Engine +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.recoll + :members: diff --git a/_sources/dev/engines/online/startpage.rst.txt b/_sources/dev/engines/online/startpage.rst.txt new file mode 100644 index 000000000..89e3ad10b --- /dev/null +++ b/_sources/dev/engines/online/startpage.rst.txt @@ -0,0 +1,13 @@ +.. _startpage engines: + +================= +Startpage Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.startpage + :members: diff --git a/_sources/dev/engines/online/tagesschau.rst.txt b/_sources/dev/engines/online/tagesschau.rst.txt new file mode 100644 index 000000000..f850bf99b --- /dev/null +++ b/_sources/dev/engines/online/tagesschau.rst.txt @@ -0,0 +1,13 @@ +.. _tagesschau engine: + +============== +Tagesschau API +============== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tagesschau + :members: diff --git a/_sources/dev/engines/online/torznab.rst.txt b/_sources/dev/engines/online/torznab.rst.txt new file mode 100644 index 000000000..9056b60d0 --- /dev/null +++ b/_sources/dev/engines/online/torznab.rst.txt @@ -0,0 +1,13 @@ +.. _torznab engine: + +============== +Torznab WebAPI +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.torznab + :members: diff --git a/_sources/dev/engines/online/wikipedia.rst.txt b/_sources/dev/engines/online/wikipedia.rst.txt new file mode 100644 index 000000000..d4920f0f6 --- /dev/null +++ b/_sources/dev/engines/online/wikipedia.rst.txt @@ -0,0 +1,27 @@ +.. _wikimedia engines: + +========= +Wikimedia +========= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _wikipedia engine: + +Wikipedia +========= + +.. automodule:: searx.engines.wikipedia + :members: + +.. _wikidata engine: + +Wikidata +========= + +.. automodule:: searx.engines.wikidata + :members: diff --git a/_sources/dev/engines/online/yacy.rst.txt b/_sources/dev/engines/online/yacy.rst.txt new file mode 100644 index 000000000..9407aca80 --- /dev/null +++ b/_sources/dev/engines/online/yacy.rst.txt @@ -0,0 +1,13 @@ +.. _yacy engine: + +==== +Yacy +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yacy + :members: diff --git a/_sources/dev/engines/online/yahoo.rst.txt b/_sources/dev/engines/online/yahoo.rst.txt new file mode 100644 index 000000000..96c1e2774 --- /dev/null +++ b/_sources/dev/engines/online/yahoo.rst.txt @@ -0,0 +1,13 @@ +.. _yahoo engine: + +============ +Yahoo Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yahoo + :members: diff --git a/_sources/dev/engines/online/zlibrary.rst.txt b/_sources/dev/engines/online/zlibrary.rst.txt new file mode 100644 index 000000000..fb197abff --- /dev/null +++ b/_sources/dev/engines/online/zlibrary.rst.txt @@ -0,0 +1,13 @@ +.. _zlibrary engine: + +========= +Z-Library +========= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.zlibrary + :members: diff --git a/_sources/dev/engines/online_url_search/tineye.rst.txt b/_sources/dev/engines/online_url_search/tineye.rst.txt new file mode 100644 index 000000000..3f4db7e0e --- /dev/null +++ b/_sources/dev/engines/online_url_search/tineye.rst.txt @@ -0,0 +1,14 @@ +.. _tineye engine: + +====== +Tineye +====== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tineye + :members: + diff --git a/_sources/dev/engines/xpath.rst.txt b/_sources/dev/engines/xpath.rst.txt new file mode 100644 index 000000000..42c4d47b6 --- /dev/null +++ b/_sources/dev/engines/xpath.rst.txt @@ -0,0 +1,14 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.xpath + :members: + diff --git a/_sources/dev/index.rst.txt b/_sources/dev/index.rst.txt new file mode 100644 index 000000000..aa0667c58 --- /dev/null +++ b/_sources/dev/index.rst.txt @@ -0,0 +1,17 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 2 + + quickstart + contribution_guide + engines/index + search_api + plugins + translation + lxcdev + makefile + reST + searxng_extra/index diff --git a/_sources/dev/lxcdev.rst.txt b/_sources/dev/lxcdev.rst.txt new file mode 100644 index 000000000..854b193ff --- /dev/null +++ b/_sources/dev/lxcdev.rst.txt @@ -0,0 +1,438 @@ +.. _lxcdev: + +============================== +Developing in Linux Containers +============================== + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +In this article we will show, how you can make use of Linux Containers (LXC_) in +*distributed and heterogeneous development cycles* (TL;DR; jump to the +:ref:`lxcdev summary`). + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers. Readers + should have a serious meaning about the terms: *distributed*, *merge* and + *linux container*. + + **hint** + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Motivation +========== + +Most often in our development cycle, we edit the sources and run some test +and/or builds by using ``make`` :ref:`[ref] ` before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook. + + **The environment in which we run all our development processes matters!** + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer's desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer's desktop that would never +be set on a *production* system. + + **Linux Containers are isolate environments**, we use them to not mix up all + the prerequisites from various projects on developer's desktop. + +The scripts from :ref:`searx_utils` can divide in those to install and maintain +software + +- :ref:`searxng.sh` + +and the script + +- :ref:`lxc.sh` + +with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the: + +- :ref:`searxng lxc suite` + +.. _lxcdev install searxng: + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: The ``searxng-archlinux`` container + + is the base of all our exercises here. + +The :ref:`lxc-searxng.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. +For this blog post we exercise on a archlinux_ image. The container of this +image is named ``searxng-archlinux``. + +Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux + + +.. sidebar:: further read + + - :ref:`lxc.sh install suite` + - :ref:`installation nginx` + +To install the complete :ref:`SearXNG suite ` and the HTTP +proxy :ref:`installation nginx` into the archlinux container run: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + ... + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +.. sidebar:: Fully functional SearXNG suite + + From here on you have a fully functional SearXNG suite (including a + :ref:`redis db`). + +In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy. + +In the example above the SearXNG instance in the container is wrapped to +``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the +IP of your ``searxng-archlinux`` container is different to this example. To +test the instance in the conatiner from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container is done by the ``./utils/lxc.sh cmd +searxng-archlinux`` command: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]$ + +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + +Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. + +Another point we have to notice is that the service :ref:`SearXNG ` +runs under dedicated system user account with the same name (compare +:ref:`create searxng user`). To get a login shell from these accounts, simply +call: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + +The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system +user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* +``(searxng-pyenv)`` environment is activated. + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional SearXNG suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- SearXNG setup in: ``/etc/searxng/settings.yml`` +- SearXNG user's home: ``/usr/local/searxng`` +- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` +- SearXNG software in: ``/usr/local/searxng/searxng-src`` + +With the use of the :ref:`searxng.sh` the SearXNG service was installed as +:ref:`uWSGI application `. To maintain this service, we can use +``systemctl`` (compare :ref:`uWSGI maintenance`). + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng + +With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searxng/searxng-src/searx + virtualenv = /usr/local/searxng/searxng-pyenv + pythonpath = /usr/local/searxng/searxng-src + +If you have read the :ref:`Good to know` you remember, that each container +shares the root folder of the repository and the command ``utils/lxc.sh cmd`` +handles relative path names **transparent**. + +To wrap the SearXNG installation in the container into a developer one, we +simple have to create a smylink to the **transparent** reposetory from the +desktop. Now lets replace the repository at ``searxng-src`` in the container +with the working tree from outside of the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container): + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +To *inspect* the SearXNG instance (already described above): + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searx.sh inspect service + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service + +Run :ref:`makefile`, e.g. to test inside the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make test + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test + + + +To install all prerequisites needed for a :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh install buildhost + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost + + +To build the docs on a buildhost :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make docs.html + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html + + +.. _lxcdev summary: + +Summary +======= + +We build up a fully functional SearXNG suite in a archlinux container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + ... + Developer install? (wraps source from HOST into the running instance) [YES/no] + +To wrap the suite into a developer one answer ``YES`` (or press Enter). + +.. code:: text + + link SearXNG's sources to: /share/SearXNG + ========================================= + + mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" + ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" + ls -ld /usr/local/searxng/searxng-src + |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG + +On code modification the instance has to be restarted (see :ref:`uWSGI +maintenance`): + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + +To access HTTP from the desktop we installed nginx for the services inside the +container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +To get information about the SearxNG suite in the archlinux container we can +use: + +.. code:: text + + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux + [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] + [searxng-archlinux] uWSGI: + [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket + [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: + [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng + [searxng-archlinux] GIT_BRANCH : master + [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng + [searxng-archlinux] SEARXNG_PORT : 8888 + [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 + diff --git a/_sources/dev/makefile.rst.txt b/_sources/dev/makefile.rst.txt new file mode 100644 index 000000000..62ac2e905 --- /dev/null +++ b/_sources/dev/makefile.rst.txt @@ -0,0 +1,452 @@ +.. _makefile: + +======================= +Makefile & ``./manage`` +======================= + +.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction + +All relevant build and development tasks are implemented in the +:origin:`./manage ` script and for CI or IDE integration a small +:origin:`Makefile` wrapper is available. If you are not familiar with +Makefiles, we recommend to read gnu-make_ introduction. + +.. sidebar:: build environment + + Before looking deeper at the targets, first read about :ref:`make + install`. + + To install developer requirements follow :ref:`buildhosts`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +The usage is simple, just type ``make {target-name}`` to *build* a target. +Calling the ``help`` target gives a first overview (``make help``): + +.. tabs:: + + .. group-tab:: ``make`` + + .. program-output:: bash -c "cd ..; make --no-print-directory help" + + + .. group-tab:: ``./manage`` + + The Makefile targets are implemented for comfort, if you can do without + tab-completion and need to have a more granular control, use + :origin:`manage` without the Makefile wrappers. + + .. code:: sh + + $ ./manage help + +.. _make install: + +Python environment (``make install``) +===================================== + +.. sidebar:: activate environment + + ``source ./local/py3/bin/activate`` + +We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a ``make install`` to get a virtualenv with a +*developer install* of SearXNG (:origin:`setup.py`). :: + + $ cd ~/searxng-clone + $ make install + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env + +If you release ``make install`` multiple times the installation will only +rebuild if the sha256 sum of the *requirement files* fails. With other words: +the check fails if you edit the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`). :: + + $ make install + PYENV OK + PYENV [virtualenv] requirements.sha256 failed + [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt + [virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt + ... + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env + +.. sidebar:: drop environment + + To get rid of the existing environment before re-build use :ref:`clean target + ` first. + +If you think, something goes wrong with your ./local environment or you change +the :origin:`setup.py` file, you have to call :ref:`make clean`. + +.. _make buildenv: + +``make buildenv`` +================= + +Rebuild instance's environment with the modified settings from the +:ref:`settings brand` and :ref:`settings server` section of your +:ref:`settings.yml `. + + What is the :origin:`utils/brand.env` needed for and why do you need to rebuild + it if necessary? + + Short answer: :ref:`installation and maintenance ` + scripts are running outside of instance's runtime environment and need some + values defined in the runtime environment. + +All the SearXNG setups are centralized in the :ref:`settings.yml` file. This +setup is available as long we are in a *installed instance*. E.g. the +*installed instance* on the server or the *installed developer instance* at +``./local`` (the later one is created by a :ref:`make install ` or +:ref:`make run `). + +Tasks running outside of an *installed instance*, especially :ref:`installation +and maintenance ` tasks running at (pre-) installation time +do not have access to the SearXNG setup (from a *installed instance*). Those +tasks need a *build environment*. + +The ``make buildenv`` target will update the *build environment* in: + +- :origin:`utils/brand.env` + +Tasks running outside of an *installed instance*, need the following settings +from the YAML configuration: + +- ``SEARXNG_URL`` from :ref:`server.base_url ` (aka + ``PUBLIC_URL``) +- ``SEARXNG_BIND_ADDRESS`` from :ref:`server.bind_address ` +- ``SEARXNG_PORT`` from :ref:`server.port ` + +The ``GIT_URL`` and ``GIT_BRANCH`` in the origin:`utils/brand.env` file, are +readed from the git VCS and the branch that is checked out when ``make +buildenv`` command runs. + +.. _brand: + +**I would like to create my own brand, how should I proceed?** + +Create a remote branch (``example.org``), checkout the remote branch (on your +local developer desktop) and in the :origin:`searx/settings.yml` file in the +:ref:`settings server` section set ``base_url``. Run ``make buildenv`` and +create a commit for your brand. + +On your server you clone the branch (``example.org``) into your HOME folder +``~`` from where you run the :ref:`installation ` and +:ref:`maintenance ` task. + +To upgrade you brand, rebase on SearXNG's master branch (on your local +developer desktop), force push it to your remote branch. Go to your server, do +a force pull and run :ref:`sudo -H ./utils/searxng.sh instance update `. + +.. _make node.env: + +Node.js environment (``make node.env``) +======================================= + +.. _Node.js: https://nodejs.org/ +.. _nvm: https://github.com/nvm-sh +.. _npm: https://www.npmjs.com/ + +.. jinja:: searx + + Node.js_ version {{version.node}} or higher is required to build the themes. + If the requirement is not met, the build chain uses nvm_ (Node Version + Manager) to install latest LTS of Node.js_ locally: there is no need to + install nvm_ or npm_ on your system. + +To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`. + +.. _make nvm: + +NVM ``make nvm.install nvm.status`` +----------------------------------- + +Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_ +setup. + +.. tabs:: + + .. group-tab:: nvm.install + + .. code:: sh + + $ LANG=C make nvm.install + INFO: install (update) NVM at ./searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + || Cloning into './searxng/.nvm'... + INFO: checkout v0.39.4 + || HEAD is now at 8fbf8ab v0.39.4 + + .. group-tab:: nvm.status (ubu2004) + + Here is the output you will typically get on a Ubuntu 20.04 system which + serves only a `no longer active `_ + Release `Node.js v10.19.0 `_. + + .. code:: sh + + $ make nvm.status + INFO: Node.js is installed at /usr/bin/node + INFO: Node.js is version v10.19.0 + WARN: minimal Node.js version is 16.13.0 + INFO: npm is installed at /usr/bin/npm + INFO: npm is version 6.14.4 + WARN: NVM is not installed + +.. _make nvm.nodejs: + +``make nvm.nodejs`` +------------------- + +Install latest Node.js_ LTS locally (uses nvm_):: + + $ make nvm.nodejs + INFO: install (update) NVM at /share/searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + ... + Downloading and installing node v16.13.0... + ... + INFO: Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node + INFO: Node.js is version v16.13.0 + INFO: npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm + INFO: npm is version 8.1.0 + INFO: NVM is installed at searxng/.nvm + +.. _make run: + +``make run`` +============ + +To get up a running a developer instance simply call ``make run``. This enables +*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` +instance and opens the URL in your favorite WEB browser (:man:`xdg-open`):: + + $ make run + +Changes to theme's HTML templates (jinja2) are instant. Changes to the CSS & JS +sources of the theme need to be rebuild. You can do that by running:: + + $ make themes.all + +Alternatively to ``themes.all`` you can run *live builds* of the theme you are +modify (:ref:`make themes`):: + + $ LIVE_THEME=simple make run + +.. _make format.python: + +``make format.python`` +====================== + +Format Python sourcee code using `Black code style`_. See ``$BLACK_OPTIONS`` +and ``$BLACK_TARGETS`` in :origin:`Makefile`. + +.. attention:: + + We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0 + to 23.1.0`_ + +.. _Bump black from 22.12.0 to 23.1.0: + https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735 + +.. _Black code style: + https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html + +.. _make clean: + +``make clean`` +============== + +Drops all intermediate files, all builds, but keep sources untouched. Before +calling ``make clean`` stop all processes using the :ref:`make install` or +:ref:`make node.env`. :: + + $ make clean + CLEAN pyenv + PYENV [virtualenv] drop local/py3 + CLEAN docs -- build/docs dist/docs + CLEAN themes -- locally installed npm dependencies + ... + CLEAN test stuff + CLEAN common files + +.. _make docs: + +``make docs`` +============= + +Target ``docs`` builds the documentation: + +.. code:: bash + + $ make docs + HTML ./docs --> file:// + DOCS build build/docs/includes + ... + The HTML pages are in dist/docs. + +.. _make docs.clean: + +``make docs.clean docs.live`` +---------------------------------- + +We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / +Documentation ` section. If you want to edit the documentation +read our :ref:`make docs.live` section. If you are working in your own brand, +adjust your :ref:`settings brand`. + + +.. _make docs.gh-pages: + +``make docs.gh-pages`` +---------------------- + +To deploy on github.io first adjust your :ref:`settings brand`. For any +further read :ref:`deploy on github.io`. + +.. _make test: + +``make test`` +============= + +Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` +and ``test.robot``. You can run tests selective, e.g.:: + + $ make test.pep8 test.unit test.shell + TEST test.pep8 OK + ... + TEST test.unit OK + ... + TEST test.shell OK + +.. _make test.shell: + +``make test.shell`` +------------------- + +:ref:`sh lint` / if you have changed some bash scripting run this test before +commit. + +.. _make test.pylint: + +``make test.pylint`` +-------------------- + +.. _Pylint: https://www.pylint.org/ + +Pylint_ is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile used in the SearXNG project is +found in project's root folder :origin:`.pylintrc`. + +.. _make search.checker: + +``make search.checker.{engine name}`` +===================================== + +To check all engines:: + + make search.checker + +To check a engine with whitespace in the name like *google news* replace space +by underline:: + + make search.checker.google_news + +To see HTTP requests and more use SEARXNG_DEBUG:: + + make SEARXNG_DEBUG=1 search.checker.google_news + +.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection + +To filter out HTTP redirects (3xx_):: + + make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]" + ... + Engine google news Checking + https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + +.. _make themes: + +``make themes.*`` +================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the +theme(s). The ``./manage themes.*`` command line can be used to convenient run +common theme build tasks. + +.. program-output:: bash -c "cd ..; ./manage themes.help" + +To get live builds while modifying CSS & JS use (:ref:`make run`): + +.. code:: sh + + $ LIVE_THEME=simple make run + +.. _make static.build: + +``make static.build.*`` +======================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a +commit of) the static files. The ``./manage static.build..*`` command line +can be used to convenient run common build tasks of the satic files. + +.. program-output:: bash -c "cd ..; ./manage static.help" + + +.. _manage redis.help: + +``./manage redis.help`` +======================= + +The ``./manage redis.*`` command line can be used to convenient run common Redis +tasks (:ref:`Redis developer notes`). + +.. program-output:: bash -c "cd ..; ./manage redis.help" + + +.. _manage go.help: + +``./manage go.help`` +==================== + +The ``./manage go.*`` command line can be used to convenient run common `go +(wiki)`_ tasks. + +.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language) + +.. program-output:: bash -c "cd ..; ./manage go.help" diff --git a/_sources/dev/plugins.rst.txt b/_sources/dev/plugins.rst.txt new file mode 100644 index 000000000..fb3201e66 --- /dev/null +++ b/_sources/dev/plugins.rst.txt @@ -0,0 +1,106 @@ +.. _dev plugin: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins generic` + +Plugins can extend or replace functionality of various components of searx. + +Example plugin +============== + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, search): + search.result_container.suggestions.add('example') + return True + +External plugins +================ + +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +`__: + +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash -c + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + ... + - only_show_green_results + ... + + +Plugin entry points +=================== + +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. A plugin doesn't need to implement all the hooks. + + +.. py:function:: pre_search(request, search) -> bool + + Runs BEFORE the search request. + + `search.result_container` can be changed. + + Return a boolean: + + * True to continue the search + * False to stop the search + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :return: False to stop the search + :rtype: bool + + +.. py:function:: post_search(request, search) -> None + + Runs AFTER the search request. + + :param flask.request request: Flask request. + :param searx.search.SearchWithPlugins search: Context. + + +.. py:function:: on_result(request, search, result) -> bool + + Runs for each result of each engine. + + `result` can be changed. + + If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])` + + .. warning:: + `result["url"]` can be changed, but `result["parsed_url"]` must be updated too. + + Return a boolean: + + * True to keep the result + * False to remove the result + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :param typing.Dict result: Result, see - :ref:`engine results` + :return: True to keep the result + :rtype: bool diff --git a/_sources/dev/quickstart.rst.txt b/_sources/dev/quickstart.rst.txt new file mode 100644 index 000000000..9bee8f60c --- /dev/null +++ b/_sources/dev/quickstart.rst.txt @@ -0,0 +1,82 @@ +.. _devquickstart: + +====================== +Development Quickstart +====================== + +.. _npm: https://www.npmjs.com/ +.. _Node.js: https://nodejs.org/ + + +.. sidebar:: further read + + - :ref:`makefile` + - :ref:`buildhosts` + +SearXNG loves developers; Developers do not need to worry about tool chains, the +usual developer tasks can be comfortably executed via :ref:`make `. + +Don't hesitate, just clone SearXNG's sources and start hacking right now .. + +.. code:: bash + + git clone https://github.com/searxng/searxng.git searxng + +Here is how a minimal workflow looks like: + +1. *start* hacking +2. *run* your code: :ref:`make run` +3. *format & test* your code: :ref:`make format.python` and :ref:`make test` + +If you think at some point something fails, go back to *start*. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in *wild west* we have some directives, please pay attention +to our ":ref:`how to contribute`" guideline. + +.. sidebar:: further read + + - :ref:`make nvm` + - :ref:`make themes` + +If you implement themes, you will need to setup a :ref:`Node.js environment +`: ``make node.env`` + +Before you call *make run* (2.), you need to compile the modified styles and +JavaScript: ``make themes.all`` + +Alternatively you can also compile selective the theme you have modified, +e.g. the *simple* theme. + +.. code:: bash + + make themes.simple + +.. tip:: + + To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run`` + +.. sidebar:: further read + + - :ref:`make static.build` + +If you finished your *tests* you can start to commit your changes. To separate +the modified source code from the build products first run: + +.. code:: bash + + make static.build.restore + +This will restore the old build products and only your changes of the code +remain in the working tree which can now be added & commited. When all sources +are commited, you can commit the build products simply by: + +.. code:: bash + + make static.build.commit + +Commiting the build products should be the last step, just before you send us +your PR. There is also a make target to rewind this last build commit: + +.. code:: bash + + make static.build.drop diff --git a/_sources/dev/reST.rst.txt b/_sources/dev/reST.rst.txt new file mode 100644 index 000000000..47af0b130 --- /dev/null +++ b/_sources/dev/reST.rst.txt @@ -0,0 +1,1438 @@ +.. _reST primer: + +=========== +reST primer +=========== + +.. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +We at SearXNG are using reStructuredText (aka reST_) markup for all kind of +documentation. With the builders from the Sphinx_ project a HTML output is +generated and deployed at docs.searxng.org_. For build prerequisites read +:ref:`docs build`. + +.. _docs.searxng.org: https://docs.searxng.org/ + +The source files of SearXNG's documentation are located at :origin:`docs`. +Sphinx assumes source files to be encoded in UTF-8 by default. Run :ref:`make +docs.live ` to build HTML while editing. + +.. sidebar:: Further reading + + - Sphinx-Primer_ + - `Sphinx markup constructs`_ + - reST_, docutils_, `docutils FAQ`_ + - Sphinx_, `sphinx-doc FAQ`_ + - `sphinx config`_, doctree_ + - `sphinx cross references`_ + - linuxdoc_ + - intersphinx_ + - sphinx-jinja_ + - `Sphinx's autodoc`_ + - `Sphinx's Python domain`_, `Sphinx's C domain`_ + - SVG_, ImageMagick_ + - DOT_, `Graphviz's dot`_, Graphviz_ + + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +Sphinx_ and reST_ have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation `[kernel doc]`_. + +.. _[kernel doc]: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html + +.. sidebar:: Content matters + + The readability_ of the reST sources has its value, therefore we recommend to + make sparse usage of reST markup / .. content matters! + +**reST** is a plaintext markup language, its markup is *mostly* intuitive and +you will not need to learn much to produce well formed articles with. I use the +word *mostly*: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups). + +Soft skills +=========== + +Before going any deeper into the markup let's face on some **soft skills** a +trained author brings with, to reach a well feedback from readers: + +- Documentation is dedicated to an audience and answers questions from the + audience point of view. +- Don't detail things which are general knowledge from the audience point of + view. +- Limit the subject, use cross links for any further reading. + +To be more concrete what a *point of view* means. In the (:origin:`docs`) +folder we have three sections (and the *blog* folder), each dedicate to a +different group of audience. + +User's POV: :origin:`docs/user` + A typical user knows about search engines and might have heard about + meta crawlers and privacy. + +Admin's POV: :origin:`docs/admin` + A typical Admin knows about setting up services on a linux system, but he does + not know all the pros and cons of a SearXNG setup. + +Developer's POV: :origin:`docs/dev` + Depending on the readability_ of code, a typical developer is able to read and + understand source code. Describe what a item aims to do (e.g. a function). + If the chronological order matters, describe it. Name the *out-of-limits + conditions* and all the side effects a external developer will not know. + +.. _reST inline markup: + +Basic inline markup +=================== + +.. sidebar:: Inline markup + + - :ref:`reST roles` + - :ref:`reST smart ref` + +Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (``\*pointer``). + +.. table:: basic inline markup + :widths: 4 3 7 + + ================================================ ==================== ======================== + description rendered markup + ================================================ ==================== ======================== + one asterisk for emphasis *italics* ``*italics*`` + two asterisks for strong emphasis **boldface** ``**boldface**`` + backquotes for code samples and literals ``foo()`` ````foo()```` + quote asterisks or backquotes \*foo is a pointer ``\*foo is a pointer`` + ================================================ ==================== ======================== + +.. _reST basic structure: + +Basic article structure +======================= + +The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections. + +.. _reST template: + +reST template +------------- + +reST template for an simple article: + +.. code:: reST + + .. _doc refname: + + ============== + Document title + ============== + + Lorem ipsum dolor sit amet, consectetur adipisici elit .. Further read + :ref:`chapter refname`. + + .. _chapter refname: + + Chapter + ======= + + Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut + aliquid ex ea commodi consequat ... + + .. _section refname: + + Section + ------- + + lorem .. + + .. _subsection refname: + + Subsection + ~~~~~~~~~~ + + lorem .. + + +Headings +-------- + +#. title - with overline for document title: + + .. code:: reST + + ============== + Document title + ============== + + +#. chapter - with anchor named ``anchor name``: + + .. code:: reST + + .. _anchor name: + + Chapter + ======= + +#. section + + .. code:: reST + + Section + ------- + +#. subsection + + .. code:: reST + + Subsection + ~~~~~~~~~~ + + + +Anchors & Links +=============== + +.. _reST anchor: + +Anchors +------- + +.. _ref role: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref + +To refer a point in the documentation a anchor is needed. The :ref:`reST +template ` shows an example where a chapter titled *"Chapters"* +gets an anchor named ``chapter title``. Another example from *this* document, +where the anchor named ``reST anchor``: + +.. code:: reST + + .. _reST anchor: + + Anchors + ------- + + To refer a point in the documentation a anchor is needed ... + +To refer anchors use the `ref role`_ markup: + +.. code:: reST + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar `. + +.. admonition:: ``:ref:`` role + :class: rst-example + + Visist chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar `. + +.. _reST ordinary ref: + +Link ordinary URL +----------------- + +If you need to reference external URLs use *named* hyperlinks to maintain +readability of reST sources. Here is a example taken from *this* article: + +.. code:: reST + + .. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + `__. + +.. admonition:: Named hyperlink + :class: rst-example + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + `__. + + +.. _reST smart ref: + +Smart refs +---------- + +With the power of sphinx.ext.extlinks_ and intersphinx_ referencing external +content becomes smart. + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + refer ... rendered example markup + ========================== ================================== ==================================== + :rst:role:`rfc` :rfc:`822` ``:rfc:`822``` + :rst:role:`pep` :pep:`8` ``:pep:`8``` + sphinx.ext.extlinks_ + -------------------------------------------------------------------------------------------------- + project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines``` + to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` + files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` + pull request :pull:`4` ``:pull:`4``` + patch :patch:`af2cae6` ``:patch:`af2cae6``` + PyPi package :pypi:`searx` ``:pypi:`searx``` + manual page man :man:`bash` ``:man:`bash``` + intersphinx_ + -------------------------------------------------------------------------------------------------- + external anchor :ref:`python:and` ``:ref:`python:and``` + external doc anchor :doc:`jinja:templates` ``:doc:`jinja:templates``` + python code object :py:obj:`datetime.datetime` ``:py:obj:`datetime.datetime``` + flask code object :py:obj:`flask.Flask` ``:py:obj:`flask.Flask``` + ========================== ================================== ==================================== + + +Intersphinx is configured in :origin:`docs/conf.py`: + +.. code:: python + + intersphinx_mapping = { + "python": ("https://docs.python.org/3/", None), + "flask": ("https://flask.palletsprojects.com/", None), + "jinja": ("https://jinja.palletsprojects.com/", None), + "linuxdoc" : ("https://return42.github.io/linuxdoc/", None), + "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), + } + + +To list all anchors of the inventory (e.g. ``python``) use: + +.. code:: sh + + $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + ... + $ python -m sphinx.ext.intersphinx https://docs.searxng.org/objects.inv + ... + +Literal blocks +============== + +The simplest form of :duref:`literal-blocks` is a indented block introduced by +two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST +code` directive. To include literals from external files use +:rst:dir:`literalinclude` or :ref:`kernel-include ` +directive (latter one expands environment variables in the path name). + +.. _reST literal: + +``::`` +------ + +.. code:: reST + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + +.. admonition:: Literal block + :class: rst-example + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + + +.. _reST code: + +``code-block`` +-------------- + +.. _pygments: https://pygments.org/languages/ + +.. sidebar:: Syntax highlighting + + is handled by pygments_. + +The :rst:dir:`code-block` directive is a variant of the :dudir:`code` directive +with additional options. To learn more about code literals visit +:ref:`sphinx:code-examples`. + +.. code-block:: reST + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +.. code-block:: reST + +.. admonition:: Code block + :class: rst-example + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +Unicode substitution +==================== + +The :dudir:`unicode directive ` converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition. + +.. code-block:: reST + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + +.. admonition:: Unicode + :class: rst-example + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + + +.. _reST roles: + +Roles +===== + +.. sidebar:: Further reading + + - `Sphinx Roles`_ + - :doc:`sphinx:usage/restructuredtext/domains` + + +A *custom interpreted text role* (:duref:`ref `) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way. + +The general markup is one of: + +.. code:: reST + + :rolename:`ref-name` + :rolename:`ref text ` + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + role rendered example markup + ========================== ================================== ==================================== + :rst:role:`guilabel` :guilabel:`&Cancel` ``:guilabel:`&Cancel``` + :rst:role:`kbd` :kbd:`C-x C-f` ``:kbd:`C-x C-f``` + :rst:role:`menuselection` :menuselection:`Open --> File` ``:menuselection:`Open --> File``` + :rst:role:`download` :download:`this file ` ``:download:`this file ``` + math_ :math:`a^2 + b^2 = c^2` ``:math:`a^2 + b^2 = c^2``` + :rst:role:`ref` :ref:`svg image example` ``:ref:`svg image example``` + :rst:role:`command` :command:`ls -la` ``:command:`ls -la``` + :durole:`emphasis` :emphasis:`italic` ``:emphasis:`italic``` + :durole:`strong` :strong:`bold` ``:strong:`bold``` + :durole:`literal` :literal:`foo()` ``:literal:`foo()``` + :durole:`subscript` H\ :sub:`2`\ O ``H\ :sub:`2`\ O`` + :durole:`superscript` E = mc\ :sup:`2` ``E = mc\ :sup:`2``` + :durole:`title-reference` :title:`Time` ``:title:`Time``` + ========================== ================================== ==================================== + +Figures & Images +================ + +.. sidebar:: Image processing + + With the directives from :ref:`linuxdoc ` the build process + is flexible. To get best results in the generated output format, install + ImageMagick_ and Graphviz_. + +SearXNG's sphinx setup includes: :ref:`linuxdoc:kfigure`. Scalable here means; +scalable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With :ref:`linuxdoc:kfigure` the build process +continues and scales output quality in dependence of installed image processors. + +If you want to add an image, you should use the ``kernel-figure`` (inheritance +of :dudir:`figure`) and ``kernel-image`` (inheritance of :dudir:`image`) +directives. E.g. to insert a figure with a scalable image format use SVG +(:ref:`svg image example`): + +.. code:: reST + + .. _svg image example: + + .. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image + + To refer the figure, a caption block is needed: :ref:`svg image example`. + +.. _svg image example: + +.. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image. + +To refer the figure, a caption block is needed: :ref:`svg image example`. + +DOT files (aka Graphviz) +------------------------ + +With :ref:`linuxdoc:kernel-figure` reST support for **DOT** formatted files is +given. + +- `Graphviz's dot`_ +- DOT_ +- Graphviz_ + +A simple example is shown in :ref:`dot file example`: + +.. code:: reST + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. admonition:: hello.dot + :class: rst-example + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +``kernel-render`` DOT +--------------------- + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +:ref:`linuxdoc:kernel-render` directive. A simple example of embedded DOT_ is +shown in figure :ref:`dot render example`: + +.. code:: reST + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +Please note :ref:`build tools `. If Graphviz_ is +installed, you will see an vector image. If not, the raw markup is inserted as +*literal-block*. + +.. admonition:: kernel-render DOT + :class: rst-example + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +``kernel-render`` SVG +--------------------- + +A simple example of embedded SVG_ is shown in figure :ref:`svg render example`: + +.. code:: reST + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow +.. + + .. code:: xml + + + + + + + +.. admonition:: kernel-render SVG + :class: rst-example + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow + + + + + + + + + + +.. _reST lists: + +List markups +============ + +Bullet list +----------- + +List markup (:duref:`ref `) is simple: + +.. code:: reST + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + +.. admonition:: bullet list + :class: rst-example + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + + +Horizontal list +--------------- + +The :rst:dir:`.. hlist:: ` transforms a bullet list into a more compact +list. + +.. code:: reST + + .. hlist:: + + - first list item + - second list item + - third list item + ... + +.. admonition:: hlist + :class: rst-example + + .. hlist:: + + - first list item + - second list item + - third list item + - next list item + - next list item xxxx + - next list item yyyy + - next list item zzzz + + +Definition list +--------------- + +.. sidebar:: Note .. + + - the term cannot have more than one line of text + + - there is **no blank line between term and definition block** // this + distinguishes definition lists (:duref:`ref `) from block + quotes (:duref:`ref `). + +Each definition list (:duref:`ref `) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ' : ' (**space, colon, space**). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(*this distinguishes definition lists from block quotes*). Blank lines are +required before the first and after the last definition list item, but are +optional in-between. + +Definition lists are created as follows: + +.. code:: reST + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref `) and not as term's definition! + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + Definition 4. + +.. admonition:: definition list + :class: rst-example + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref `) and not as term's definition! + + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + + +Quoted paragraphs +----------------- + +Quoted paragraphs (:duref:`ref `) are created by just indenting +them more than the surrounding paragraphs. Line blocks (:duref:`ref +`) are a way of preserving line breaks: + +.. code:: reST + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. admonition:: Quoted paragraph and line block + :class: rst-example + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. _reST field list: + +Field Lists +----------- + +.. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + +.. sidebar:: bibliographic fields + + First lines fields are bibliographic fields, see `Sphinx Field Lists`_. + +Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this: + +.. code:: reST + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + +.. admonition:: Field List + :class: rst-example + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + + +They are commonly used in Python documentation: + +.. code:: python + + def my_function(my_arg, my_other_arg): + """A function just for me. + + :param my_arg: The first of my arguments. + :param my_other_arg: The second of my arguments. + + :returns: A message (just for me, of course). + """ + +Further list blocks +------------------- + +- field lists (:duref:`ref `, with caveats noted in + :ref:`reST field list`) +- option lists (:duref:`ref `) +- quoted literal blocks (:duref:`ref `) +- doctest blocks (:duref:`ref `) + + +Admonitions +=========== + +Sidebar +------- + +Sidebar is an eye catcher, often used for admonitions pointing further stuff or +site effects. Here is the source of the sidebar :ref:`on top of this page `. + +.. code:: reST + + .. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +Generic admonition +------------------ + +The generic :dudir:`admonition ` needs a title: + +.. code:: reST + + .. admonition:: generic admonition title + + lorem ipsum .. + + +.. admonition:: generic admonition title + + lorem ipsum .. + + +Specific admonitions +-------------------- + +Specific admonitions: :dudir:`hint`, :dudir:`note`, :dudir:`tip` :dudir:`attention`, +:dudir:`caution`, :dudir:`danger`, :dudir:`error`, , :dudir:`important`, and +:dudir:`warning` . + +.. code:: reST + + .. hint:: + + lorem ipsum .. + + .. note:: + + lorem ipsum .. + + .. warning:: + + lorem ipsum .. + + +.. hint:: + + lorem ipsum .. + +.. note:: + + lorem ipsum .. + +.. tip:: + + lorem ipsum .. + +.. attention:: + + lorem ipsum .. + +.. caution:: + + lorem ipsum .. + +.. danger:: + + lorem ipsum .. + +.. important:: + + lorem ipsum .. + +.. error:: + + lorem ipsum .. + +.. warning:: + + lorem ipsum .. + + +Tables +====== + +.. sidebar:: Nested tables + + Nested tables are ugly! Not all builder support nested tables, don't use + them! + +ASCII-art tables like :ref:`reST simple table` and :ref:`reST grid table` might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables. + + +.. sidebar:: List tables + + For meaningful patch and diff use :ref:`reST flat table`. + +Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers: + +* `Emacs Table Mode`_ +* `Online Tables Generator`_ + +.. _reST simple table: + +Simple tables +------------- + +:duref:`Simple tables ` allow *colspan* but not *rowspan*. If +your table need some metadata (e.g. a title) you need to add the ``.. table:: +directive`` :dudir:`(ref)
` in front and place the table in its body: + +.. code:: reST + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + +.. admonition:: Simple ASCII table + :class: rst-example + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + + + +.. _reST grid table: + +Grid tables +----------- + +:duref:`Grid tables ` allow colspan *colspan* and *rowspan*: + +.. code:: reST + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + +.. admonition:: ASCII grid table + :class: rst-example + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + + +.. _reST flat table: + +flat-table +---------- + +The ``flat-table`` is a further developed variant of the :ref:`list tables +`. It is a double-stage list similar to the +:dudir:`list-table` with some additional features: + +column-span: ``cspan`` + with the role ``cspan`` a cell can be extended through additional columns + +row-span: ``rspan`` + with the role ``rspan`` a cell can be extended through additional rows + +auto-span: + spans rightmost cell of a table row over the missing cells on the right side + of that table-row. With Option ``:fill-cells:`` this behavior can changed + from *auto span* to *auto fill*, which automatically inserts (empty) cells + instead of spanning the last cell. + +options: + :header-rows: [int] count of header rows + :stub-columns: [int] count of stub columns + :widths: [[int] [int] ... ] widths of columns + :fill-cells: instead of auto-span missing cells, insert missing cells + +roles: + :cspan: [int] additional columns (*morecols*) + :rspan: [int] additional rows (*morerows*) + +The example below shows how to use this markup. The first level of the staged +list is the *table-row*. In the *table-row* there is only one markup allowed, +the list of the cells in this *table-row*. Exception are *comments* ( ``..`` ) +and *targets* (e.g. a ref to :ref:`row 2 of table's body `). + +.. code:: reST + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +.. admonition:: List table + :class: rst-example + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +CSV table +--------- + +CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation. + +.. code:: reST + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 2 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Content of file ``csv_table.txt``: + +.. literalinclude:: csv_table.txt + +.. admonition:: CSV table + :class: rst-example + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 3 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Templating +========== + +.. sidebar:: Build environment + + All *generic-doc* tasks are running in the :ref:`make install`. + +Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with SearXNG modules installed). We use this e.g. to build chapter: +:ref:`configured engines`. Below the jinja directive from the +:origin:`docs/admin/engines.rst` is shown: + +.. literalinclude:: ../user/configured_engines.rst + :language: reST + :start-after: .. _configured engines: + +The context for the template is selected in the line ``.. jinja:: searx``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``searx`` context +contains the ``engines`` and ``plugins``. + +.. code:: py + + import searx.search + import searx.engines + import searx.plugins + searx.search.initialize() + jinja_contexts = { + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, + } + + +Tabbed views +============ + +.. _sphinx-tabs: https://github.com/djungelorm/sphinx-tabs +.. _basic-tabs: https://github.com/djungelorm/sphinx-tabs#basic-tabs +.. _group-tabs: https://github.com/djungelorm/sphinx-tabs#group-tabs +.. _code-tabs: https://github.com/djungelorm/sphinx-tabs#code-tabs + +With `sphinx-tabs`_ extension we have *tabbed views*. To provide installation +instructions with one tab per distribution we use the `group-tabs`_ directive, +others are basic-tabs_ and code-tabs_. Below a *group-tab* example from +:ref:`docs build` is shown: + +.. literalinclude:: ../admin/buildhosts.rst + :language: reST + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements + +.. _math: + +Math equations +============== + +.. _Mathematics: https://en.wikibooks.org/wiki/LaTeX/Mathematics +.. _amsmath user guide: + http://vesta.informatik.rwth-aachen.de/ftp/pub/mirror/ctan/macros/latex/required/amsmath/amsldoc.pdf + +.. sidebar:: About LaTeX + + - `amsmath user guide`_ + - Mathematics_ + - :ref:`docs build` + +The input language for mathematics is LaTeX markup using the :ctan:`amsmath` +package. + +To embed LaTeX markup in reST documents, use role :rst:role:`:math: ` for +inline and directive :rst:dir:`.. math:: ` for block markup. + +.. code:: reST + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + +.. admonition:: LaTeX math equation + :class: rst-example + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + + +The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` +(*displaystyle*) used in a inline markup or another fraction. + +.. code:: reST + + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + ``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + +.. admonition:: Line spacing + :class: rst-example + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + +.. _KISS: https://en.wikipedia.org/wiki/KISS_principle + +.. _readability: https://docs.python-guide.org/writing/style/ +.. _Sphinx-Primer: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _reST: https://docutils.sourceforge.io/rst.html +.. _Sphinx Roles: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html +.. _Sphinx: https://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx markup constructs: + https://www.sphinx-doc.org/en/stable/markup/index.html +.. _`sphinx cross references`: + https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations +.. _sphinx.ext.extlinks: + https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html +.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _Sphinx's Python domain: + https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain +.. _Sphinx's C domain: + https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs +.. _doctree: + https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases +.. _docutils: http://docutils.sourceforge.net/docs/index.html +.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html +.. _linuxdoc: https://return42.github.io/linuxdoc +.. _jinja: https://jinja.palletsprojects.com/ +.. _sphinx-jinja: https://github.com/tardyp/sphinx-jinja +.. _SVG: https://www.w3.org/TR/SVG11/expanded-toc.html +.. _DOT: https://graphviz.gitlab.io/_pages/doc/info/lang.html +.. _`Graphviz's dot`: https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org + +.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode +.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables +.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/_sources/dev/search_api.rst.txt b/_sources/dev/search_api.rst.txt new file mode 100644 index 000000000..2333474a9 --- /dev/null +++ b/_sources/dev/search_api.rst.txt @@ -0,0 +1,126 @@ +.. _search API: + +========== +Search API +========== + +The search supports both ``GET`` and ``POST``. + +Furthermore, two endpoints ``/`` and ``/search`` are available for querying. + + +``GET /`` + +``GET /search`` + +Parameters +========== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings.yml` + - :ref:`configured engines` + +``q`` : required + The search query. This string is passed to external search services. Thus, + SearXNG supports syntax of each search service. For example, ``site:github.com + SearXNG`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. + + See more at :ref:`search-syntax` + +``categories`` : optional + Comma separated list, specifies the active search categories (see + :ref:`configured engines`) + +``engines`` : optional + Comma separated list, specifies the active search engines (see + :ref:`configured engines`). + +``language`` : default from :ref:`settings search` + Code of the language. + +``pageno`` : default ``1`` + Search page number. + +``time_range`` : optional + [ ``day``, ``month``, ``year`` ] + + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. + +``format`` : optional + [ ``json``, ``csv``, ``rss`` ] + + Output format of results. Format needs to be activated in :ref:`settings + search`. + +``results_on_new_tab`` : default ``0`` + [ ``0``, ``1`` ] + + Open search results on new tab. + +``image_proxy`` : default from :ref:`settings server` + [ ``True``, ``False`` ] + + Proxy image results through SearXNG. + +``autocomplete`` : default from :ref:`settings search` + [ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia``, + ``swisscows``, ``qwant`` ] + + Service which completes words as you type. + +``safesearch`` : default from :ref:`settings search` + [ ``0``, ``1``, ``2`` ] + + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. + +``theme`` : default ``simple`` + [ ``simple`` ] + + Theme of instance. + + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on their instance. + See the available options in the preferences page of the instance. + +``enabled_plugins`` : optional + List of enabled plugins. + + :default: + ``Hash_plugin``, ``Search_on_category_select``, + ``Self_Information``, ``Tracker_URL_remover``, + ``Ahmia_blacklist`` + + :values: + .. enabled by default + + ``Hash_plugin``, ``Search_on_category_select``, + ``Self_Information``, ``Tracker_URL_remover``, + ``Ahmia_blacklist``, + + .. disabled by default + + ``Hostname_replace``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + +``disabled_plugins``: optional + List of disabled plugins. + + :default: + ``Hostname_replace``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + + :values: + see values from ``enabled_plugins`` + +``enabled_engines`` : optional : *all* :origin:`engines ` + List of enabled engines. + +``disabled_engines`` : optional : *all* :origin:`engines ` + List of disabled engines. + diff --git a/_sources/dev/searxng_extra/index.rst.txt b/_sources/dev/searxng_extra/index.rst.txt new file mode 100644 index 000000000..7bca3c0d4 --- /dev/null +++ b/_sources/dev/searxng_extra/index.rst.txt @@ -0,0 +1,14 @@ +.. _searxng_extra: + +============================= +Tooling box ``searxng_extra`` +============================= + +In the folder :origin:`searxng_extra/` we maintain some tools useful for CI and +developers. + +.. toctree:: + :maxdepth: 2 + + update + standalone_searx.py diff --git a/_sources/dev/searxng_extra/standalone_searx.py.rst.txt b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt new file mode 100644 index 000000000..7cbbccfde --- /dev/null +++ b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt @@ -0,0 +1,9 @@ + +.. _standalone_searx.py: + +===================================== +``searxng_extra/standalone_searx.py`` +===================================== + +.. automodule:: searxng_extra.standalone_searx + :members: diff --git a/_sources/dev/searxng_extra/update.rst.txt b/_sources/dev/searxng_extra/update.rst.txt new file mode 100644 index 000000000..a125303e0 --- /dev/null +++ b/_sources/dev/searxng_extra/update.rst.txt @@ -0,0 +1,88 @@ +========================= +``searxng_extra/update/`` +========================= + +:origin:`[source] ` + +Scripts to update static data in :origin:`searx/data/` + +.. _update_ahmia_blacklist.py: + +``update_ahmia_blacklist.py`` +============================= + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_ahmia_blacklist + :members: + + +``update_currencies.py`` +======================== + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_currencies + :members: + +``update_engine_descriptions.py`` +================================= + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_engine_descriptions + :members: + + +``update_external_bangs.py`` +============================ + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_external_bangs + :members: + + +``update_firefox_version.py`` +============================= + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_firefox_version + :members: + + +``update_engine_traits.py`` +=========================== + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_engine_traits + :members: + + +``update_osm_keys_tags.py`` +=========================== + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_osm_keys_tags + :members: + + +``update_pygments.py`` +====================== + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_pygments + :members: + + +``update_wikidata_units.py`` +============================ + +:origin:`[source] ` + +.. automodule:: searxng_extra.update.update_wikidata_units + :members: diff --git a/_sources/dev/translation.rst.txt b/_sources/dev/translation.rst.txt new file mode 100644 index 000000000..693f6d2da --- /dev/null +++ b/_sources/dev/translation.rst.txt @@ -0,0 +1,81 @@ +.. _translation: + +=========== +Translation +=========== + +.. _translate.codeberg.org: https://translate.codeberg.org/projects/searxng/ +.. _Weblate: https://docs.weblate.org +.. _translations branch: https://github.com/searxng/searxng/tree/translations +.. _orphan branch: https://git-scm.com/docs/git-checkout#Documentation/git-checkout.txt---orphanltnewbranchgt +.. _Weblate repository: https://translate.codeberg.org/projects/searxng/searxng/#repository +.. _wlc: https://docs.weblate.org/en/latest/wlc.html + +.. |translated| image:: https://translate.codeberg.org/widgets/searxng/-/searxng/svg-badge.svg + :target: https://translate.codeberg.org/projects/searxng/ + +.. sidebar:: |translated| + + - :ref:`searx.babel_extract` + - Weblate_ + - SearXNG `translations branch`_ + - SearXNG `Weblate repository`_ + - Weblate Client: wlc_ + - Babel Command-Line: `pybabel `_ + - `weblate workflow `_ + +Translation takes place on translate.codeberg.org_. + +Translations which has been added by translators on the translate.codeberg.org_ UI are +committed to Weblate's counterpart of the SearXNG *origin* repository which is +located at ``https://translate.codeberg.org/git/searxng/searxng``. + +There is no need to clone this repository, :ref:`SearXNG Weblate workflow` take +care of the synchronization with the *origin*. To avoid merging commits from +the counterpart directly on the ``master`` branch of *SearXNG origin*, a *pull +request* (PR) is created by this workflow. + +Weblate monitors the `translations branch`_, not the ``master`` branch. This +branch is an `orphan branch`_, decoupled from the master branch (we already know +orphan branches from the ``gh-pages``). The `translations branch`_ contains +only the + +- ``translation/messages.pot`` and the +- ``translation/*/messages.po`` files, nothing else. + + +.. _SearXNG Weblate workflow: + +.. figure:: translation.svg + + SearXNG's PR workflow to be in sync with Weblate + +Sync from *origin* to *weblate*: using ``make weblate.push.translations`` + For each commit on the ``master`` branch of SearXNG *origin* the GitHub job + :origin:`babel / Update translations branch + <.github/workflows/integration.yml>` checks for updated translations. + +Sync from *weblate* to *origin*: using ``make weblate.translations.commit`` + Every Friday, the GitHub workflow :origin:`babel / create PR for additons from + weblate <.github/workflows/translations-update.yml>` creates a PR with the + updated translation files: + + - ``translation/messages.pot``, + - ``translation/*/messages.po`` and + - ``translation/*/messages.mo`` + +wlc +=== + +.. _wlc configuration: https://docs.weblate.org/en/latest/wlc.html#wlc-config +.. _API key: https://translate.codeberg.org/accounts/profile/#api + +All weblate integration is done by GitHub workflows, but if you want to use wlc_, +copy this content into `wlc configuration`_ in your HOME ``~/.config/weblate`` + +.. code-block:: ini + + [keys] + https://translate.codeberg.org/api/ = APIKEY + +Replace ``APIKEY`` by your `API key`_. diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt new file mode 100644 index 000000000..3db54734a --- /dev/null +++ b/_sources/index.rst.txt @@ -0,0 +1,54 @@ +================== +Welcome to SearXNG +================== + + *Search without being tracked.* + +SearXNG is a free internet metasearch engine which aggregates results from more +than 70 search services. Users are neither tracked nor profiled. Additionally, +SearXNG can be used over Tor for online anonymity. + +Get started with SearXNG by using one of the instances listed at searx.space_. +If you don't trust anyone, you can set up your own, see :ref:`installation`. + +.. sidebar:: features + + - :ref:`self hosted ` + - :ref:`no user tracking / no profiling ` + - script & cookies are optional + - secure, encrypted connections + - :ref:`about 130 search engines ` + - `about 60 translations `_ + - about 100 `well maintained `__ instances on searx.space_ + - :ref:`easy integration of search engines ` + - professional development: `CI `_, + `quality assurance `_ & + `automated tested UI `_ + +.. sidebar:: be a part + + SearXNG is driven by an open community, come join us! Don't hesitate, no + need to be an *expert*, everyone can contribute: + + - `help to improve translations `_ + - `discuss with the community `_ + - report bugs & suggestions + - ... + +.. sidebar:: the origin + + SearXNG development has been started in the middle of 2021 as a fork of the + searx project. + + +.. toctree:: + :maxdepth: 2 + + user/index + own-instance + admin/index + dev/index + utils/index + src/index + +.. _searx.space: https://searx.space diff --git a/_sources/own-instance.rst.txt b/_sources/own-instance.rst.txt new file mode 100644 index 000000000..62099ce23 --- /dev/null +++ b/_sources/own-instance.rst.txt @@ -0,0 +1,86 @@ +=========================== +Why use a private instance? +=========================== + +.. sidebar:: Is it worth to run my own instance? + + \.\. is a common question among SearXNG users. Before answering this + question, see what options a SearXNG user has. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Public instances are open to everyone who has access to its URL. Usually, these +are operated by unknown parties (from the users' point of view). Private +instances can be used by a select group of people. It is for example a SearXNG of +group of friends or a company which can be accessed through VPN. Also it can be +single user one which runs on the user's laptop. + +To gain more insight on how these instances work let's dive into how SearXNG +protects its users. + +.. _SearXNG protect privacy: + +How does SearXNG protect privacy? +================================= + +SearXNG protects the privacy of its users in multiple ways regardless of the type +of the instance (private, public). Removal of private data from search requests +comes in three forms: + + 1. removal of private data from requests going to search services + 2. not forwarding anything from a third party services through search services + (e.g. advertisement) + 3. removal of private data from requests going to the result pages + +Removing private data means not sending cookies to external search engines and +generating a random browser profile for every request. Thus, it does not matter +if a public or private instance handles the request, because it is anonymized in +both cases. IP addresses will be the IP of the instance. But SearXNG can be +configured to use proxy or Tor. `Result proxy +`__ is supported, too. + +SearXNG does not serve ads or tracking content unlike most search services. So +private data is not forwarded to third parties who might monetize it. Besides +protecting users from search services, both referring page and search query are +hidden from visited result pages. + + +What are the consequences of using public instances? +---------------------------------------------------- + +If someone uses a public instance, they have to trust the administrator of that +instance. This means that the user of the public instance does not know whether +their requests are logged, aggregated and sent or sold to a third party. + +Also, public instances without proper protection are more vulnerable to abusing +the search service, In this case the external service in exchange returns +CAPTCHAs or bans the IP of the instance. Thus, search requests return less +results. + +I see. What about private instances? +------------------------------------ + +If users run their :ref:`own instances `, everything is in their +control: the source code, logging settings and private data. Unknown instance +administrators do not have to be trusted. + +Furthermore, as the default settings of their instance is editable, there is no +need to use cookies to tailor SearXNG to their needs. So preferences will not be +reset to defaults when clearing browser cookies. As settings are stored on +their computer, it will not be accessible to others as long as their computer is +not compromised. + +Conclusion +========== + +Always use an instance which is operated by people you trust. The privacy +features of SearXNG are available to users no matter what kind of instance they +use. + +If someone is on the go or just wants to try SearXNG for the first time public +instances are the best choices. Additionally, public instance are making a +world a better place, because those who cannot or do not want to run an +instance, have access to a privacy respecting search service. diff --git a/_sources/src/index.rst.txt b/_sources/src/index.rst.txt new file mode 100644 index 000000000..fd16e7635 --- /dev/null +++ b/_sources/src/index.rst.txt @@ -0,0 +1,13 @@ +=========== +Source-Code +=========== + +This is a partial documentation of our source code. We are not aiming to document +every item from the source code, but we will add documentation when requested. + + +.. toctree:: + :maxdepth: 2 + :glob: + + searx.* diff --git a/_sources/src/searx.babel_extract.rst.txt b/_sources/src/searx.babel_extract.rst.txt new file mode 100644 index 000000000..741d67fc1 --- /dev/null +++ b/_sources/src/searx.babel_extract.rst.txt @@ -0,0 +1,8 @@ +.. _searx.babel_extract: + +=============================== +Custom message extractor (i18n) +=============================== + +.. automodule:: searx.babel_extract + :members: diff --git a/_sources/src/searx.botdetection.rst.txt b/_sources/src/searx.botdetection.rst.txt new file mode 100644 index 000000000..48261ceac --- /dev/null +++ b/_sources/src/searx.botdetection.rst.txt @@ -0,0 +1,48 @@ +.. _botdetection: + +============= +Bot Detection +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.botdetection + :members: + +.. automodule:: searx.botdetection.limiter + :members: + +.. automodule:: searx.botdetection.ip_lists + :members: + + +Rate limit +========== + +.. automodule:: searx.botdetection.ip_limit + :members: + +.. automodule:: searx.botdetection.link_token + :members: + + +Probe HTTP headers +================== + +.. automodule:: searx.botdetection.http_accept + :members: + +.. automodule:: searx.botdetection.http_accept_encoding + :members: + +.. automodule:: searx.botdetection.http_accept_language + :members: + +.. automodule:: searx.botdetection.http_connection + :members: + +.. automodule:: searx.botdetection.http_user_agent + :members: diff --git a/_sources/src/searx.exceptions.rst.txt b/_sources/src/searx.exceptions.rst.txt new file mode 100644 index 000000000..72117e148 --- /dev/null +++ b/_sources/src/searx.exceptions.rst.txt @@ -0,0 +1,8 @@ +.. _searx.exceptions: + +================== +SearXNG Exceptions +================== + +.. automodule:: searx.exceptions + :members: diff --git a/_sources/src/searx.infopage.rst.txt b/_sources/src/searx.infopage.rst.txt new file mode 100644 index 000000000..243e5b0bd --- /dev/null +++ b/_sources/src/searx.infopage.rst.txt @@ -0,0 +1,8 @@ +.. _searx.infopage: + +================ +Online ``/info`` +================ + +.. automodule:: searx.infopage + :members: diff --git a/_sources/src/searx.locales.rst.txt b/_sources/src/searx.locales.rst.txt new file mode 100644 index 000000000..0de49a5e1 --- /dev/null +++ b/_sources/src/searx.locales.rst.txt @@ -0,0 +1,20 @@ +.. _searx.locales: + +======= +Locales +======= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.locales + :members: + + +SearXNG's locale codes +====================== + +.. automodule:: searx.sxng_locales + :members: diff --git a/_sources/src/searx.plugins.tor_check.rst.txt b/_sources/src/searx.plugins.tor_check.rst.txt new file mode 100644 index 000000000..905328ebf --- /dev/null +++ b/_sources/src/searx.plugins.tor_check.rst.txt @@ -0,0 +1,9 @@ +.. _tor check plugin: + +================ +Tor check plugin +================ + +.. automodule:: searx.plugins.tor_check + :members: + diff --git a/_sources/src/searx.redisdb.rst.txt b/_sources/src/searx.redisdb.rst.txt new file mode 100644 index 000000000..625378c91 --- /dev/null +++ b/_sources/src/searx.redisdb.rst.txt @@ -0,0 +1,8 @@ +.. _redis db: + +======== +Redis DB +======== + +.. automodule:: searx.redisdb + :members: diff --git a/_sources/src/searx.redislib.rst.txt b/_sources/src/searx.redislib.rst.txt new file mode 100644 index 000000000..b4604574c --- /dev/null +++ b/_sources/src/searx.redislib.rst.txt @@ -0,0 +1,8 @@ +.. _searx.redis: + +============= +Redis Library +============= + +.. automodule:: searx.redislib + :members: diff --git a/_sources/src/searx.search.processors.rst.txt b/_sources/src/searx.search.processors.rst.txt new file mode 100644 index 000000000..3bb4c44bc --- /dev/null +++ b/_sources/src/searx.search.processors.rst.txt @@ -0,0 +1,47 @@ +.. _searx.search.processors: + +================= +Search processors +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Abstract processor class +======================== + +.. automodule:: searx.search.processors.abstract + :members: + +Offline processor +================= + +.. automodule:: searx.search.processors.offline + :members: + +Online processor +================ + +.. automodule:: searx.search.processors.online + :members: + +Online currency processor +========================= + +.. automodule:: searx.search.processors.online_currency + :members: + +Online dictionary processor +=========================== + +.. automodule:: searx.search.processors.online_dictionary + :members: + +Online URL search processor +=========================== + +.. automodule:: searx.search.processors.online_url_search + :members: diff --git a/_sources/src/searx.search.rst.txt b/_sources/src/searx.search.rst.txt new file mode 100644 index 000000000..ad76d4183 --- /dev/null +++ b/_sources/src/searx.search.rst.txt @@ -0,0 +1,38 @@ +.. _searx.search: + +====== +Search +====== + +.. autoclass:: searx.search.EngineRef + :members: + +.. autoclass:: searx.search.SearchQuery + :members: + +.. autoclass:: searx.search.Search + + .. attribute:: search_query + :type: searx.search.SearchQuery + + .. attribute:: result_container + :type: searx.results.ResultContainer + + .. automethod:: search() -> searx.results.ResultContainer + +.. autoclass:: searx.search.SearchWithPlugins + :members: + + .. attribute:: search_query + :type: searx.search.SearchQuery + + .. attribute:: result_container + :type: searx.results.ResultContainer + + .. attribute:: ordered_plugin_list + :type: typing.List + + .. attribute:: request + :type: flask.request + + .. automethod:: search() -> searx.results.ResultContainer diff --git a/_sources/src/searx.utils.rst.txt b/_sources/src/searx.utils.rst.txt new file mode 100644 index 000000000..6496700c1 --- /dev/null +++ b/_sources/src/searx.utils.rst.txt @@ -0,0 +1,8 @@ +.. _searx.utils: + +================================= +Utility functions for the engines +================================= + +.. automodule:: searx.utils + :members: diff --git a/_sources/user/about.rst.txt b/_sources/user/about.rst.txt new file mode 100644 index 000000000..08f1a068e --- /dev/null +++ b/_sources/user/about.rst.txt @@ -0,0 +1,4 @@ +.. _about SearXNG: + +.. include:: about.md + :parser: myst_parser.sphinx_ diff --git a/_sources/user/configured_engines.rst.txt b/_sources/user/configured_engines.rst.txt new file mode 100644 index 000000000..c32a264ed --- /dev/null +++ b/_sources/user/configured_engines.rst.txt @@ -0,0 +1,89 @@ +.. _configured engines: + +================== +Configured Engines +================== + +.. sidebar:: Further reading .. + + - :ref:`settings categories_as_tabs` + - :ref:`engines-dev` + - :ref:`settings engine` + - :ref:`general engine configuration` + +.. jinja:: searx + + SearXNG supports {{engines | length}} search engines of which + {{enabled_engine_count}} are enabled by default. + + Engines can be assigned to multiple :ref:`categories `. + The UI displays the tabs that are configured in :ref:`categories_as_tabs + `. In addition to these UI categories (also + called *tabs*), engines can be queried by their name or the categories they + belong to, by using a :ref:`\!bing syntax `. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. jinja:: searx + + {% for category, engines in categories_as_tabs.items() %} + + tab ``!{{category.replace(' ', '_')}}`` + --------------------------------------- + + {% for group, group_bang, engines in engines | group_engines_in_tab %} + + {% if loop.length > 1 %} + {% if group_bang %}group ``{{group_bang}}``{% else %}{{group}}{% endif %} + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + {% endif %} + + .. flat-table:: + :header-rows: 2 + :stub-columns: 1 + :widths: 10 1 10 1 1 1 1 1 1 1 + + * - :cspan:`5` Engines configured by default (in :ref:`settings.yml `) + - :cspan:`3` :ref:`Supported features ` + + * - Name + - !bang + - Module + - Disabled + - Timeout + - Weight + - Paging + - Locale + - Safe search + - Time range + + {% for mod in engines %} + + * - `{{mod.name}} <{{mod.about and mod.about.website}}>`_ + {%- if mod.about and mod.about.language %} + ({{mod.about.language | upper}}) + {%- endif %} + - ``!{{mod.shortcut}}`` + - {%- if 'searx.engines.' + mod.__name__ in documented_modules %} + :py:mod:`~searx.engines.{{mod.__name__}}` + {%- else %} + :origin:`{{mod.__name__}} ` + {%- endif %} + - {{(mod.disabled and "y") or ""}} + - {{mod.timeout}} + - {{mod.weight or 1 }} + {% if mod.engine_type == 'online' %} + - {{(mod.paging and "y") or ""}} + - {{(mod.language_support and "y") or ""}} + - {{(mod.safesearch and "y") or ""}} + - {{(mod.time_range_support and "y") or ""}} + {% else %} + - :cspan:`3` not applicable ({{mod.engine_type}}) + {% endif %} + + {% endfor %} + {% endfor %} + {% endfor %} diff --git a/_sources/user/index.rst.txt b/_sources/user/index.rst.txt new file mode 100644 index 000000000..705fd3f9e --- /dev/null +++ b/_sources/user/index.rst.txt @@ -0,0 +1,15 @@ +================ +User information +================ + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. toctree:: + :maxdepth: 2 + + search-syntax + configured_engines + about diff --git a/_sources/user/search-syntax.rst.txt b/_sources/user/search-syntax.rst.txt new file mode 100644 index 000000000..3acddd6e3 --- /dev/null +++ b/_sources/user/search-syntax.rst.txt @@ -0,0 +1,4 @@ +.. _search-syntax: + +.. include:: search-syntax.md + :parser: myst_parser.sphinx_ diff --git a/_sources/utils/index.rst.txt b/_sources/utils/index.rst.txt new file mode 100644 index 000000000..b570b07e6 --- /dev/null +++ b/_sources/utils/index.rst.txt @@ -0,0 +1,31 @@ +.. _searx_utils: +.. _toolboxing: + +================== +DevOps tooling box +================== + +In the folder :origin:`utils/` we maintain some tools useful for administrators +and developers. + +.. toctree:: + :maxdepth: 2 + + searxng.sh + lxc.sh + +Common command environments +=========================== + +The scripts in our tooling box often dispose of common environments: + +.. _FORCE_TIMEOUT: + +``FORCE_TIMEOUT`` : environment + Sets timeout for interactive prompts. If you want to run a script in batch + job, with defaults choices, set ``FORCE_TIMEOUT=0``. By example; to install a + SearXNG server and nginx proxy on all containers of the :ref:`SearXNG suite + ` use:: + + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx diff --git a/_sources/utils/lxc.sh.rst.txt b/_sources/utils/lxc.sh.rst.txt new file mode 100644 index 000000000..43dc915ef --- /dev/null +++ b/_sources/utils/lxc.sh.rst.txt @@ -0,0 +1,295 @@ + +.. _snap: https://snapcraft.io +.. _snapcraft LXD: https://snapcraft.io/lxd +.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/ +.. _LXC: https://linuxcontainers.org/lxc/introduction/ +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _`LXD@github`: https://github.com/lxc/lxd + +.. _archlinux: https://www.archlinux.org/ + +.. _lxc.sh: + +================ +``utils/lxc.sh`` +================ + +With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of +containers, what we call the: *lxc suite*. The :ref:`lxc-searxng.env` is +loaded by default, every time you start the ``lxc.sh`` script (*you do not need +to care about*). + +.. sidebar:: further reading + + - snap_, `snapcraft LXD`_ + - LXC_, LXD_ + - `LXC/LXD Image Server`_ + - `LXD@github`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _lxd install: + +Install LXD +=========== + +Before you can start with containers, you need to install and initiate LXD_ +once:: + + $ snap install lxd + $ lxd init --auto + +To make use of the containers from the *SearXNG suite*, you have to build the +:ref:`LXC suite containers ` initial. But be warned, **this might +take some time**:: + + $ sudo -H ./utils/lxc.sh build + +.. sidebar:: hint + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + +A cup of coffee later, your LXC suite is build up and you can run whatever task +you want / in a selected or even in all :ref:`LXC suite containers `. + +.. _internet connectivity docker: + +Internet Connectivity & Docker +------------------------------ + +.. sidebar:: further read + + - `Docker blocking network of existing LXC containers `__ + - `Docker and IPtables (fralef.me) `__ + - `Docker and iptables (docker.com) `__ + +There is a conflict in the ``iptables`` setup of Docker & LXC. If you have +docker installed, you may find that the internet connectivity of your LXD +containers no longer work. + +Whenever docker is started (reboot) it sets the iptables policy for the +``FORWARD`` chain to ``DROP`` `[ref] +`__:: + + $ sudo -H iptables-save | grep FORWARD + :FORWARD ACCEPT [7048:7851230] + :FORWARD DROP [7048:7851230] + +A handy solution of this problem might be to reset the policy for the +``FORWARD`` chain after the network has been initialized. For this create a +file in the ``if-up`` section of the network (``/etc/network/if-up.d/iptable``) +and insert the following lines:: + + #!/bin/sh + iptables -F FORWARD + iptables -P FORWARD ACCEPT + +Don't forget to set the execution bit:: + + sudo chmod ugo+x /etc/network/if-up.d/iptable + +Reboot your system and check the iptables rules:: + + $ sudo -H iptables-save | grep FORWARD + :FORWARD ACCEPT [7048:7851230] + :FORWARD ACCEPT [7048:7851230] + + +.. _searxng lxc suite: + +SearXNG LXC suite +================= + +The intention of the *SearXNG LXC suite* is to build up a suite of containers +for development tasks or :ref:`buildhosts ` with a very +small set of simple commands. At the end of the ``--help`` output the SearXNG +suite from the :ref:`lxc-searxng.env` is introduced:: + + $ sudo -H ./utils/lxc.sh --help + ... + LXC suite: searxng + Suite includes installation of SearXNG + images: ubu2004 ubu2204 fedora35 archlinux + containers: searxng-ubu2004 searxng-ubu2204 searxng-fedora35 searxng-archlinux + +As shown above there are images and containers build up on this images. To show +more info about the containers in the *SearXNG LXC suite* call ``show suite``. +If this is the first time you make use of the SearXNG LXC suite, no containers +are installed and the output is:: + + $ sudo -H ./utils/lxc.sh show suite + + LXC suite (searxng-*) + ===================== + + +------+-------+------+------+------+-----------+ + | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | + +------+-------+------+------+------+-----------+ + + WARN: container searxng-ubu2004 does not yet exists + WARN: container searxng-ubu2204 does not yet exists + WARN: container searxng-fedora35 does not yet exists + WARN: container searxng-archlinux does not yet exists + +If you do not want to run a command or a build in all containers, **you can +build just one**. Here by example in the container that is build upon the +*archlinux* image:: + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd + +Otherwise, to apply a command to all containers you can use:: + + $ sudo -H ./utils/lxc.sh build + $ sudo -H ./utils/lxc.sh cmd -- ls -la . + +Running commands +---------------- + +**Inside containers, you can run scripts** from the :ref:`toolboxing` or run +what ever command you need. By example, to start a bash use:: + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]# + +.. _Good to know: + +Good to know +------------ + +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handle relative path names transparent**:: + + $ pwd + /share/SearXNG + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd + INFO: [searxng-archlinux] pwd + /share/SearXNG + +The path ``/share/SearXNG`` will be different on your HOST system. The commands +in the conatiner are executed by the ``root`` inside of the container. Compare +output of:: + + $ ls -li Makefile + 47712402 -rw-rw-r-- 1 markus markus 2923 Apr 19 13:52 Makefile + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ls -li Makefile + INFO: [searxng-archlinux] ls -li Makefile + 47712402 -rw-rw-r-- 1 root root 2923 Apr 19 11:52 Makefile + ... + +Since the path ``/share/SearXNG`` of the HOST system is wrapped into the +container under the same name, the shown ``Makefile`` (inode ``47712402``) in +the ouput is always the identical ``/share/SearXNG/Makefile`` from the HOST +system. In the example shown above the owner of the path in the container is +the ``root`` user of the conatiner (and the timezone in the container is +different to HOST system). + + +.. _lxc.sh install suite: + +Install suite +------------- + +.. sidebar:: further read + + - :ref:`working in containers` + - :ref:`FORCE_TIMEOUT ` + +To install the complete :ref:`SearXNG suite ` into **all** LXC_ +containers leave the container argument empty and run:: + + $ sudo -H ./utils/lxc.sh build + $ sudo -H ./utils/lxc.sh install suite + +To *build & install* suite only in one container you can use by example:: + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + +The command above installs a SearXNG suite (see :ref:`installation scripts`). +To :ref:`install a nginx ` reverse proxy (or alternatively +use :ref:`apache `):: + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +Same operation just in one container of the suite:: + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +The :ref:`FORCE_TIMEOUT ` environment is set to zero to run the +script without user interaction. + +To get the IP (URL) of the SearXNG service in the containers use ``show suite`` +command. To test instances from containers just open the URLs in your +WEB-Browser:: + + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + + [searxng-ubu2110] SEARXNG_URL : http://n.n.n.170/searxng + [searxng-ubu2004] SEARXNG_URL : http://n.n.n.160/searxng + [searxnggfedora35] SEARXNG_URL : http://n.n.n.150/searxng + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +Clean up +-------- + +If there comes the time you want to **get rid off all** the containers and +**clean up local images** just type:: + + $ sudo -H ./utils/lxc.sh remove + $ sudo -H ./utils/lxc.sh remove images + + +.. _Setup SearXNG buildhost: + +Setup SearXNG buildhost +======================= + +You can **install the SearXNG buildhost environment** into one or all containers. +The installation procedure to set up a :ref:`build host` takes its +time. Installation in all containers will take more time (time for another cup +of coffee). :: + + sudo -H ./utils/lxc.sh cmd -- ./utils/searxng.sh install buildhost + +To build (live) documentation inside a archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.clean docs.live + ... + [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 + +To get IP of the container and the port number *live docs* is listening:: + + $ sudo ./utils/lxc.sh show suite | grep docs.live + ... + [searxng-archlinux] INFO: (eth0) docs.live: http://n.n.n.140:8080/ + + +.. _lxc.sh help: + +Command Help +============ + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/lxc.sh --help + + +.. _lxc-searxng.env: + +SearXNG suite config +==================== + +The SearXNG suite is defined in the file :origin:`utils/lxc-searxng.env`: + +.. literalinclude:: ../../utils/lxc-searxng.env + :language: bash diff --git a/_sources/utils/searxng.sh.rst.txt b/_sources/utils/searxng.sh.rst.txt new file mode 100644 index 000000000..bedc1ba4c --- /dev/null +++ b/_sources/utils/searxng.sh.rst.txt @@ -0,0 +1,42 @@ + +.. _searxng.sh: + +==================== +``utils/searxng.sh`` +==================== + +To simplify the installation and maintenance of a SearXNG instance you can use the +script :origin:`utils/searxng.sh`. + +.. sidebar:: further reading + + - :ref:`architecture` + - :ref:`installation` + - :ref:`installation nginx` + - :ref:`installation apache` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Install +======= + +In most cases you will install SearXNG simply by running the command: + +.. code:: bash + + sudo -H ./utils/searx.sh install all + +The installation is described in chapter :ref:`installation basic`. + +.. _searxng.sh overview: + +Command Help +============ + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/searxng.sh --help diff --git a/_static/basic.css b/_static/basic.css new file mode 100644 index 000000000..cfc60b86c --- /dev/null +++ b/_static/basic.css @@ -0,0 +1,921 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2023 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} + +nav.contents, +aside.topic, +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ + +nav.contents, +aside.topic, +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +nav.contents > :last-child, +aside.topic > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +nav.contents::after, +aside.topic::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} + +aside.footnote > span, +div.citation > span { + float: left; +} +aside.footnote > span:last-of-type, +div.citation > span:last-of-type { + padding-right: 0.5em; +} +aside.footnote > p { + margin-left: 2em; +} +div.citation > p { + margin-left: 4em; +} +aside.footnote > p:last-of-type, +div.citation > p:last-of-type { + margin-bottom: 0em; +} +aside.footnote > p:last-of-type:after, +div.citation > p:last-of-type:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +.sig dd { + margin-top: 0px; + margin-bottom: 0px; +} + +.sig dl { + margin-top: 0px; + margin-bottom: 0px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +.translated { + background-color: rgba(207, 255, 207, 0.2) +} + +.untranslated { + background-color: rgba(255, 207, 207, 0.2) +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/_static/doctools.js b/_static/doctools.js new file mode 100644 index 000000000..d06a71d75 --- /dev/null +++ b/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2023 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/_static/documentation_options.js b/_static/documentation_options.js new file mode 100644 index 000000000..04d53f644 --- /dev/null +++ b/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: '2023.8.11+905ce2a6f', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/_static/file.png b/_static/file.png new file mode 100644 index 000000000..a858a410e Binary files /dev/null and b/_static/file.png differ diff --git a/_static/language_data.js b/_static/language_data.js new file mode 100644 index 000000000..250f5665f --- /dev/null +++ b/_static/language_data.js @@ -0,0 +1,199 @@ +/* + * language_data.js + * ~~~~~~~~~~~~~~~~ + * + * This script contains the language-specific data used by searchtools.js, + * namely the list of stopwords, stemmer, scorer and splitter. + * + * :copyright: Copyright 2007-2023 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +var stopwords = ["a", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "near", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with"]; + + +/* Non-minified version is copied as a separate JS file, is available */ + +/** + * Porter Stemmer + */ +var Stemmer = function() { + + var step2list = { + ational: 'ate', + tional: 'tion', + enci: 'ence', + anci: 'ance', + izer: 'ize', + bli: 'ble', + alli: 'al', + entli: 'ent', + eli: 'e', + ousli: 'ous', + ization: 'ize', + ation: 'ate', + ator: 'ate', + alism: 'al', + iveness: 'ive', + fulness: 'ful', + ousness: 'ous', + aliti: 'al', + iviti: 'ive', + biliti: 'ble', + logi: 'log' + }; + + var step3list = { + icate: 'ic', + ative: '', + alize: 'al', + iciti: 'ic', + ical: 'ic', + ful: '', + ness: '' + }; + + var c = "[^aeiou]"; // consonant + var v = "[aeiouy]"; // vowel + var C = c + "[^aeiouy]*"; // consonant sequence + var V = v + "[aeiou]*"; // vowel sequence + + var mgr0 = "^(" + C + ")?" + V + C; // [C]VC... is m>0 + var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; // [C]VC[V] is m=1 + var mgr1 = "^(" + C + ")?" + V + C + V + C; // [C]VCVC... is m>1 + var s_v = "^(" + C + ")?" + v; // vowel in stem + + this.stemWord = function (w) { + var stem; + var suffix; + var firstch; + var origword = w; + + if (w.length < 3) + return w; + + var re; + var re2; + var re3; + var re4; + + firstch = w.substr(0,1); + if (firstch == "y") + w = firstch.toUpperCase() + w.substr(1); + + // Step 1a + re = /^(.+?)(ss|i)es$/; + re2 = /^(.+?)([^s])s$/; + + if (re.test(w)) + w = w.replace(re,"$1$2"); + else if (re2.test(w)) + w = w.replace(re2,"$1$2"); + + // Step 1b + re = /^(.+?)eed$/; + re2 = /^(.+?)(ed|ing)$/; + if (re.test(w)) { + var fp = re.exec(w); + re = new RegExp(mgr0); + if (re.test(fp[1])) { + re = /.$/; + w = w.replace(re,""); + } + } + else if (re2.test(w)) { + var fp = re2.exec(w); + stem = fp[1]; + re2 = new RegExp(s_v); + if (re2.test(stem)) { + w = stem; + re2 = /(at|bl|iz)$/; + re3 = new RegExp("([^aeiouylsz])\\1$"); + re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re2.test(w)) + w = w + "e"; + else if (re3.test(w)) { + re = /.$/; + w = w.replace(re,""); + } + else if (re4.test(w)) + w = w + "e"; + } + } + + // Step 1c + re = /^(.+?)y$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(s_v); + if (re.test(stem)) + w = stem + "i"; + } + + // Step 2 + re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stem)) + w = stem + step2list[suffix]; + } + + // Step 3 + re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stem)) + w = stem + step3list[suffix]; + } + + // Step 4 + re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; + re2 = /^(.+?)(s|t)(ion)$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(mgr1); + if (re.test(stem)) + w = stem; + } + else if (re2.test(w)) { + var fp = re2.exec(w); + stem = fp[1] + fp[2]; + re2 = new RegExp(mgr1); + if (re2.test(stem)) + w = stem; + } + + // Step 5 + re = /^(.+?)e$/; + if (re.test(w)) { + var fp = re.exec(w); + stem = fp[1]; + re = new RegExp(mgr1); + re2 = new RegExp(meq1); + re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re.test(stem) || (re2.test(stem) && !(re3.test(stem)))) + w = stem; + } + re = /ll$/; + re2 = new RegExp(mgr1); + if (re.test(w) && re2.test(w)) { + re = /.$/; + w = w.replace(re,""); + } + + // and turn initial Y back to y + if (firstch == "y") + w = firstch.toLowerCase() + w.substr(1); + return w; + } +} + diff --git a/_static/minus.png b/_static/minus.png new file mode 100644 index 000000000..d96755fda Binary files /dev/null and b/_static/minus.png differ diff --git a/_static/plus.png b/_static/plus.png new file mode 100644 index 000000000..7107cec93 Binary files /dev/null and b/_static/plus.png differ diff --git a/_static/pocoo.css b/_static/pocoo.css new file mode 100644 index 000000000..4f14b31ee --- /dev/null +++ b/_static/pocoo.css @@ -0,0 +1,525 @@ +@import url("basic.css"); + +/* -- page layout --------------------------------------------------- */ + +body { + font-family: 'Garamond', 'Georgia', serif; + font-size: 17px; + background-color: #fff; + color: #3e4349; + margin: 0; + padding: 0; +} + +div.related { + max-width: 1140px; + margin: 10px auto; + + /* displayed on mobile */ + display: none; +} + +div.document { + max-width: 1140px; + margin: 10px auto; +} + +div.documentwrapper { + float: left; + width: 100%; +} + +div.bodywrapper { + margin: 0 0 0 220px; +} + +div.body { + min-width: initial; + max-width: initial; + padding: 0 30px; +} + +div.sphinxsidebarwrapper { + padding: 10px; +} + +div.sphinxsidebar { + width: 220px; + font-size: 14px; + line-height: 1.5; + color: #444; +} + +div.sphinxsidebar li { + overflow: hidden; + text-overflow: ellipsis; +} + +div.sphinxsidebar li:hover { + overflow: visible; +} + +div.sphinxsidebar a, +div.sphinxsidebar a code { + color: #444; + border-color: #444; +} + +div.sphinxsidebar a:hover { + background-color:#fff; +} + +div.sphinxsidebar p.logo { + margin: 0; + text-align: center; +} + +div.sphinxsidebar h3, +div.sphinxsidebar h4 { + font-size: 24px; + color: #444; +} + +div.sphinxsidebar p.logo a, +div.sphinxsidebar h3 a, +div.sphinxsidebar p.logo a:hover, +div.sphinxsidebar h3 a:hover { + border: none; +} + +div.sphinxsidebar p, +div.sphinxsidebar h3, +div.sphinxsidebar h4 { + margin: 10px 0; +} + +div.sphinxsidebar ul { + margin: 10px 0; + padding: 0; +} + +div.sphinxsidebar input { + border: 1px solid #999; + font-size: 1em; +} + +div.footer { + max-width: 1140px; + margin: 20px auto; + font-size: 14px; + text-align: right; + color: #888; +} + +div.footer a { + color: #888; + border-color: #888; +} + +/* -- quick search -------------------------------------------------- */ + +div.sphinxsidebar #searchbox form { + display: flex; +} + +div.sphinxsidebar #searchbox form > div { + display: flex; + flex: 1 1 auto; +} + +div.sphinxsidebar #searchbox input[type=text] { + flex: 1 1 auto; + width: 1% !important; +} + +div.sphinxsidebar #searchbox input[type=submit] { + border-left-width: 0; +} + +/* -- versions ------------------------------------------------------ */ + +div.sphinxsidebar ul.versions a.current { + font-style: italic; + border-bottom: 1px solid #000; + color: #000; +} + +div.sphinxsidebar ul.versions span.note { + color: #999; +} + +/* -- version warning ----------------------------------------------- */ + +p.version-warning { + top: 10px; + position: sticky; + + margin: 10px 0; + padding: 5px 10px; + border-radius: 4px; + + letter-spacing: 1px; + color: #fff; + text-shadow: 0 0 2px #000; + text-align: center; + + background: #d40 repeating-linear-gradient( + 135deg, + transparent, + transparent 56px, + rgba(255, 255, 255, 0.2) 56px, + rgba(255, 255, 255, 0.2) 112px + ); +} + +p.version-warning a { + color: #fff; + border-color: #fff; +} + +/* -- body styles --------------------------------------------------- */ + +a { + text-decoration: underline; + text-decoration-style: dotted; + text-decoration-color: #000; + text-decoration-thickness: 1px; +} + +a:hover { + text-decoration-style: solid; +} + +h1, h2, h3, h4, h5, h6 { + font-weight: normal; + margin: 30px 0 10px; + padding: 0; + color: black; +} + +div.body h1 { + font-size: 240%; +} + +div.body h2 { + font-size: 180%; +} + +div.body h3 { + font-size: 150%; +} + +div.body h4 { + font-size: 130%; +} + +div.body h5 { + font-size: 100%; +} + +div.body h6 { + font-size: 100%; +} + +div.body h1:first-of-type { + margin-top: 0; +} + +a.headerlink { + color: #ddd; + margin: 0 0.2em; + padding: 0 0.2em; + border: none; +} + +a.headerlink:hover { + color: #444; +} + +div.body p, +div.body dd, +div.body li { + line-height: 1.4; +} + +img.screenshot { + box-shadow: 2px 2px 4px #eee; +} + +hr { + border: 1px solid #999; +} + +blockquote { + margin: 0 0 0 30px; + padding: 0; +} + +ul, ol { + margin: 10px 0 10px 30px; + padding: 0; +} + +a.footnote-reference { + font-size: 0.7em; + vertical-align: top; +} + +/* -- admonitions --------------------------------------------------- */ + +div.admonition, +div.topic { + background-color: #fafafa; + margin: 10px -10px; + padding: 10px; + border-top: 1px solid #ccc; + border-right: none; + border-bottom: 1px solid #ccc; + border-left: none; +} + +div.admonition p.admonition-title, +div.topic p.topic-title { + font-weight: normal; + font-size: 24px; + margin: 0 0 10px 0; + padding: 0; + line-height: 1; + display: inline; +} + +p.admonition-title::after { + content: ":"; +} + +div.admonition p.last, +div.topic p:last-child { + margin-bottom: 0; +} + +div.danger, div.error { + background-color: #fff0f0; + border-color: #ffb0b0; +} + +div.seealso { + background-color: #fffff0; + border-color: #f0f0a8; +} + +/* -- changelog ----------------------------------------------------- */ + +details.changelog summary { + cursor: pointer; + font-style: italic; + margin-bottom: 10px; +} + +/* -- search highlight ---------------------------------------------- */ + +dt:target, +.footnote:target, +span.highlighted { + background-color: #ffdf80; +} + +rect.highlighted { + fill: #ffdf80; +} + +/* -- code displays ------------------------------------------------- */ + +pre, code { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; + font-size: 0.9em; +} + +pre { + margin: 0; + padding: 0; + line-height: 1.3; +} + +div.literal-block-wrapper { + padding: 10px 0 0; +} + +div.code-block-caption { + padding: 0; +} + +div.highlight, div.literal-block-wrapper div.highlight { + margin: 10px -10px; + padding: 10px; +} + +code { + color: #222; + background: #e8eff0; +} + +/* -- tables -------------------------------------------------------- */ + +table.docutils { + border: 1px solid #888; + box-shadow: 2px 2px 4px #eee; +} + +table.docutils td, +table.docutils th { + border: 1px solid #888; + padding: 0.25em 0.7em; +} + +table.field-list, +table.footnote { + border: none; + box-shadow: none; +} + +table.footnote { + margin: 15px 0; + width: 100%; + border: 1px solid #eee; + background-color: #fafafa; + font-size: 0.9em; +} + +table.footnote + table.footnote { + margin-top: -15px; + border-top: none; +} + +table.field-list th { + padding: 0 0.8em 0 0; +} + +table.field-list td { + padding: 0; +} + +table.footnote td.label { + width: 0; + padding: 0.3em 0 0.3em 0.5em; +} + +table.footnote td { + padding: 0.3em 0.5em; +} + +/* -- responsive screen --------------------------------------------- */ + +@media screen and (max-width: 1139px) { + p.version-warning { + margin: 10px; + } + + div.footer { + margin: 20px 10px; + } +} + +/* -- small screen -------------------------------------------------- */ + +@media screen and (max-width: 767px) { + body { + padding: 0 20px; + } + + div.related { + display: block; + } + + p.version-warning { + margin: 10px 0; + } + + div.documentwrapper { + float: none; + } + + div.bodywrapper { + margin: 0; + } + + div.body { + min-height: 0; + padding: 0; + } + + div.sphinxsidebar { + float: none; + width: 100%; + margin: 0 -20px -10px; + padding: 0 20px; + background-color: #333; + color: #ccc; + } + + div.sphinxsidebar a, + div.sphinxsidebar a code, + div.sphinxsidebar h3, + div.sphinxsidebar h4, + div.footer a { + color: #ccc; + border-color: #ccc; + } + + div.sphinxsidebar p.logo { + display: none; + } + + div.sphinxsidebar ul.versions a.current { + border-bottom-color: #fff; + color: #fff; + } + + div.footer { + text-align: left; + margin: 0 -20px; + padding: 20px; + background-color: #333; + color: #ccc; + } +} + +/* https://github.com/twbs/bootstrap/blob + /0e8831505ac845f3102fa2c5996a7141c9ab01ee + /scss/mixins/_screen-reader.scss */ +.hide-header > h1:first-child { + position: absolute; + width: 1px; + height: 1px; + padding: 0; + overflow: hidden; + clip: rect(0, 0, 0, 0); + white-space: nowrap; + border: 0; +} + +/* -- sphinx-tabs -------------------------------------------------- */ + +.sphinx-tabs { + margin-bottom: 0; +} + +.sphinx-tabs .ui.menu { + font-family: 'Garamond', 'Georgia', serif !important; +} + +.sphinx-tabs .ui.attached.menu { + border-bottom: none +} + +.sphinx-tabs .ui.tabular.menu .item { + border-bottom: 2px solid transparent; + border-left: none; + border-right: none; + border-top: none; + padding: .3em 0.6em; +} + +.sphinx-tabs .ui.attached.segment, .ui.segment { + border: 0; + padding: 0; +} diff --git a/_static/pygments.css b/_static/pygments.css new file mode 100644 index 000000000..57c7df37b --- /dev/null +++ b/_static/pygments.css @@ -0,0 +1,84 @@ +pre { line-height: 125%; } +td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; } +span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; } +td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } +span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; } +.highlight .hll { background-color: #ffffcc } +.highlight { background: #f8f8f8; } +.highlight .c { color: #8f5902; font-style: italic } /* Comment */ +.highlight .err { color: #a40000; border: 1px solid #ef2929 } /* Error */ +.highlight .g { color: #000000 } /* Generic */ +.highlight .k { color: #004461; font-weight: bold } /* Keyword */ +.highlight .l { color: #000000 } /* Literal */ +.highlight .n { color: #000000 } /* Name */ +.highlight .o { color: #582800 } /* Operator */ +.highlight .x { color: #000000 } /* Other */ +.highlight .p { color: #000000; font-weight: bold } /* Punctuation */ +.highlight .ch { color: #8f5902; font-style: italic } /* Comment.Hashbang */ +.highlight .cm { color: #8f5902; font-style: italic } /* Comment.Multiline */ +.highlight .cp { color: #8f5902 } /* Comment.Preproc */ +.highlight .cpf { color: #8f5902; font-style: italic } /* Comment.PreprocFile */ +.highlight .c1 { color: #8f5902; font-style: italic } /* Comment.Single */ +.highlight .cs { color: #8f5902; font-style: italic } /* Comment.Special */ +.highlight .gd { color: #a40000 } /* Generic.Deleted */ +.highlight .ge { color: #000000; font-style: italic } /* Generic.Emph */ +.highlight .ges { color: #000000 } /* Generic.EmphStrong */ +.highlight .gr { color: #ef2929 } /* Generic.Error */ +.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ +.highlight .gi { color: #00A000 } /* Generic.Inserted */ +.highlight .go { color: #888888 } /* Generic.Output */ +.highlight .gp { color: #745334 } /* Generic.Prompt */ +.highlight .gs { color: #000000; font-weight: bold } /* Generic.Strong */ +.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ +.highlight .gt { color: #a40000; font-weight: bold } /* Generic.Traceback */ +.highlight .kc { color: #004461; font-weight: bold } /* Keyword.Constant */ +.highlight .kd { color: #004461; font-weight: bold } /* Keyword.Declaration */ +.highlight .kn { color: #004461; font-weight: bold } /* Keyword.Namespace */ +.highlight .kp { color: #004461; font-weight: bold } /* Keyword.Pseudo */ +.highlight .kr { color: #004461; font-weight: bold } /* Keyword.Reserved */ +.highlight .kt { color: #004461; font-weight: bold } /* Keyword.Type */ +.highlight .ld { color: #000000 } /* Literal.Date */ +.highlight .m { color: #990000 } /* Literal.Number */ +.highlight .s { color: #4e9a06 } /* Literal.String */ +.highlight .na { color: #c4a000 } /* Name.Attribute */ +.highlight .nb { color: #004461 } /* Name.Builtin */ +.highlight .nc { color: #000000 } /* Name.Class */ +.highlight .no { color: #000000 } /* Name.Constant */ +.highlight .nd { color: #888888 } /* Name.Decorator */ +.highlight .ni { color: #ce5c00 } /* Name.Entity */ +.highlight .ne { color: #cc0000; font-weight: bold } /* Name.Exception */ +.highlight .nf { color: #000000 } /* Name.Function */ +.highlight .nl { color: #f57900 } /* Name.Label */ +.highlight .nn { color: #000000 } /* Name.Namespace */ +.highlight .nx { color: #000000 } /* Name.Other */ +.highlight .py { color: #000000 } /* Name.Property */ +.highlight .nt { color: #004461; font-weight: bold } /* Name.Tag */ +.highlight .nv { color: #000000 } /* Name.Variable */ +.highlight .ow { color: #004461; font-weight: bold } /* Operator.Word */ +.highlight .pm { color: #000000; font-weight: bold } /* Punctuation.Marker */ +.highlight .w { color: #f8f8f8; text-decoration: underline } /* Text.Whitespace */ +.highlight .mb { color: #990000 } /* Literal.Number.Bin */ +.highlight .mf { color: #990000 } /* Literal.Number.Float */ +.highlight .mh { color: #990000 } /* Literal.Number.Hex */ +.highlight .mi { color: #990000 } /* Literal.Number.Integer */ +.highlight .mo { color: #990000 } /* Literal.Number.Oct */ +.highlight .sa { color: #4e9a06 } /* Literal.String.Affix */ +.highlight .sb { color: #4e9a06 } /* Literal.String.Backtick */ +.highlight .sc { color: #4e9a06 } /* Literal.String.Char */ +.highlight .dl { color: #4e9a06 } /* Literal.String.Delimiter */ +.highlight .sd { color: #8f5902; font-style: italic } /* Literal.String.Doc */ +.highlight .s2 { color: #4e9a06 } /* Literal.String.Double */ +.highlight .se { color: #4e9a06 } /* Literal.String.Escape */ +.highlight .sh { color: #4e9a06 } /* Literal.String.Heredoc */ +.highlight .si { color: #4e9a06 } /* Literal.String.Interpol */ +.highlight .sx { color: #4e9a06 } /* Literal.String.Other */ +.highlight .sr { color: #4e9a06 } /* Literal.String.Regex */ +.highlight .s1 { color: #4e9a06 } /* Literal.String.Single */ +.highlight .ss { color: #4e9a06 } /* Literal.String.Symbol */ +.highlight .bp { color: #3465a4 } /* Name.Builtin.Pseudo */ +.highlight .fm { color: #000000 } /* Name.Function.Magic */ +.highlight .vc { color: #000000 } /* Name.Variable.Class */ +.highlight .vg { color: #000000 } /* Name.Variable.Global */ +.highlight .vi { color: #000000 } /* Name.Variable.Instance */ +.highlight .vm { color: #000000 } /* Name.Variable.Magic */ +.highlight .il { color: #990000 } /* Literal.Number.Integer.Long */ \ No newline at end of file diff --git a/_static/searchtools.js b/_static/searchtools.js new file mode 100644 index 000000000..97d56a74d --- /dev/null +++ b/_static/searchtools.js @@ -0,0 +1,566 @@ +/* + * searchtools.js + * ~~~~~~~~~~~~~~~~ + * + * Sphinx JavaScript utilities for the full-text search. + * + * :copyright: Copyright 2007-2023 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +/** + * Simple result scoring code. + */ +if (typeof Scorer === "undefined") { + var Scorer = { + // Implement the following function to further tweak the score for each result + // The function takes a result array [docname, title, anchor, descr, score, filename] + // and returns the new score. + /* + score: result => { + const [docname, title, anchor, descr, score, filename] = result + return score + }, + */ + + // query matches the full name of an object + objNameMatch: 11, + // or matches in the last dotted part of the object name + objPartialMatch: 6, + // Additive scores depending on the priority of the object + objPrio: { + 0: 15, // used to be importantResults + 1: 5, // used to be objectResults + 2: -5, // used to be unimportantResults + }, + // Used when the priority is not in the mapping. + objPrioDefault: 0, + + // query found in title + title: 15, + partialTitle: 7, + // query found in terms + term: 5, + partialTerm: 2, + }; +} + +const _removeChildren = (element) => { + while (element && element.lastChild) element.removeChild(element.lastChild); +}; + +/** + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#escaping + */ +const _escapeRegExp = (string) => + string.replace(/[.*+\-?^${}()|[\]\\]/g, "\\$&"); // $& means the whole matched string + +const _displayItem = (item, searchTerms) => { + const docBuilder = DOCUMENTATION_OPTIONS.BUILDER; + const docUrlRoot = DOCUMENTATION_OPTIONS.URL_ROOT; + const docFileSuffix = DOCUMENTATION_OPTIONS.FILE_SUFFIX; + const docLinkSuffix = DOCUMENTATION_OPTIONS.LINK_SUFFIX; + const showSearchSummary = DOCUMENTATION_OPTIONS.SHOW_SEARCH_SUMMARY; + + const [docName, title, anchor, descr, score, _filename] = item; + + let listItem = document.createElement("li"); + let requestUrl; + let linkUrl; + if (docBuilder === "dirhtml") { + // dirhtml builder + let dirname = docName + "/"; + if (dirname.match(/\/index\/$/)) + dirname = dirname.substring(0, dirname.length - 6); + else if (dirname === "index/") dirname = ""; + requestUrl = docUrlRoot + dirname; + linkUrl = requestUrl; + } else { + // normal html builders + requestUrl = docUrlRoot + docName + docFileSuffix; + linkUrl = docName + docLinkSuffix; + } + let linkEl = listItem.appendChild(document.createElement("a")); + linkEl.href = linkUrl + anchor; + linkEl.dataset.score = score; + linkEl.innerHTML = title; + if (descr) + listItem.appendChild(document.createElement("span")).innerHTML = + " (" + descr + ")"; + else if (showSearchSummary) + fetch(requestUrl) + .then((responseData) => responseData.text()) + .then((data) => { + if (data) + listItem.appendChild( + Search.makeSearchSummary(data, searchTerms) + ); + }); + Search.output.appendChild(listItem); +}; +const _finishSearch = (resultCount) => { + Search.stopPulse(); + Search.title.innerText = _("Search Results"); + if (!resultCount) + Search.status.innerText = Documentation.gettext( + "Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories." + ); + else + Search.status.innerText = _( + `Search finished, found ${resultCount} page(s) matching the search query.` + ); +}; +const _displayNextItem = ( + results, + resultCount, + searchTerms +) => { + // results left, load the summary and display it + // this is intended to be dynamic (don't sub resultsCount) + if (results.length) { + _displayItem(results.pop(), searchTerms); + setTimeout( + () => _displayNextItem(results, resultCount, searchTerms), + 5 + ); + } + // search finished, update title and status message + else _finishSearch(resultCount); +}; + +/** + * Default splitQuery function. Can be overridden in ``sphinx.search`` with a + * custom function per language. + * + * The regular expression works by splitting the string on consecutive characters + * that are not Unicode letters, numbers, underscores, or emoji characters. + * This is the same as ``\W+`` in Python, preserving the surrogate pair area. + */ +if (typeof splitQuery === "undefined") { + var splitQuery = (query) => query + .split(/[^\p{Letter}\p{Number}_\p{Emoji_Presentation}]+/gu) + .filter(term => term) // remove remaining empty strings +} + +/** + * Search Module + */ +const Search = { + _index: null, + _queued_query: null, + _pulse_status: -1, + + htmlToText: (htmlString) => { + const htmlElement = new DOMParser().parseFromString(htmlString, 'text/html'); + htmlElement.querySelectorAll(".headerlink").forEach((el) => { el.remove() }); + const docContent = htmlElement.querySelector('[role="main"]'); + if (docContent !== undefined) return docContent.textContent; + console.warn( + "Content block not found. Sphinx search tries to obtain it via '[role=main]'. Could you check your theme or template." + ); + return ""; + }, + + init: () => { + const query = new URLSearchParams(window.location.search).get("q"); + document + .querySelectorAll('input[name="q"]') + .forEach((el) => (el.value = query)); + if (query) Search.performSearch(query); + }, + + loadIndex: (url) => + (document.body.appendChild(document.createElement("script")).src = url), + + setIndex: (index) => { + Search._index = index; + if (Search._queued_query !== null) { + const query = Search._queued_query; + Search._queued_query = null; + Search.query(query); + } + }, + + hasIndex: () => Search._index !== null, + + deferQuery: (query) => (Search._queued_query = query), + + stopPulse: () => (Search._pulse_status = -1), + + startPulse: () => { + if (Search._pulse_status >= 0) return; + + const pulse = () => { + Search._pulse_status = (Search._pulse_status + 1) % 4; + Search.dots.innerText = ".".repeat(Search._pulse_status); + if (Search._pulse_status >= 0) window.setTimeout(pulse, 500); + }; + pulse(); + }, + + /** + * perform a search for something (or wait until index is loaded) + */ + performSearch: (query) => { + // create the required interface elements + const searchText = document.createElement("h2"); + searchText.textContent = _("Searching"); + const searchSummary = document.createElement("p"); + searchSummary.classList.add("search-summary"); + searchSummary.innerText = ""; + const searchList = document.createElement("ul"); + searchList.classList.add("search"); + + const out = document.getElementById("search-results"); + Search.title = out.appendChild(searchText); + Search.dots = Search.title.appendChild(document.createElement("span")); + Search.status = out.appendChild(searchSummary); + Search.output = out.appendChild(searchList); + + const searchProgress = document.getElementById("search-progress"); + // Some themes don't use the search progress node + if (searchProgress) { + searchProgress.innerText = _("Preparing search..."); + } + Search.startPulse(); + + // index already loaded, the browser was quick! + if (Search.hasIndex()) Search.query(query); + else Search.deferQuery(query); + }, + + /** + * execute search (requires search index to be loaded) + */ + query: (query) => { + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const titles = Search._index.titles; + const allTitles = Search._index.alltitles; + const indexEntries = Search._index.indexentries; + + // stem the search terms and add them to the correct list + const stemmer = new Stemmer(); + const searchTerms = new Set(); + const excludedTerms = new Set(); + const highlightTerms = new Set(); + const objectTerms = new Set(splitQuery(query.toLowerCase().trim())); + splitQuery(query.trim()).forEach((queryTerm) => { + const queryTermLower = queryTerm.toLowerCase(); + + // maybe skip this "word" + // stopwords array is from language_data.js + if ( + stopwords.indexOf(queryTermLower) !== -1 || + queryTerm.match(/^\d+$/) + ) + return; + + // stem the word + let word = stemmer.stemWord(queryTermLower); + // select the correct list + if (word[0] === "-") excludedTerms.add(word.substr(1)); + else { + searchTerms.add(word); + highlightTerms.add(queryTermLower); + } + }); + + if (SPHINX_HIGHLIGHT_ENABLED) { // set in sphinx_highlight.js + localStorage.setItem("sphinx_highlight_terms", [...highlightTerms].join(" ")) + } + + // console.debug("SEARCH: searching for:"); + // console.info("required: ", [...searchTerms]); + // console.info("excluded: ", [...excludedTerms]); + + // array of [docname, title, anchor, descr, score, filename] + let results = []; + _removeChildren(document.getElementById("search-progress")); + + const queryLower = query.toLowerCase(); + for (const [title, foundTitles] of Object.entries(allTitles)) { + if (title.toLowerCase().includes(queryLower) && (queryLower.length >= title.length/2)) { + for (const [file, id] of foundTitles) { + let score = Math.round(100 * queryLower.length / title.length) + results.push([ + docNames[file], + titles[file] !== title ? `${titles[file]} > ${title}` : title, + id !== null ? "#" + id : "", + null, + score, + filenames[file], + ]); + } + } + } + + // search for explicit entries in index directives + for (const [entry, foundEntries] of Object.entries(indexEntries)) { + if (entry.includes(queryLower) && (queryLower.length >= entry.length/2)) { + for (const [file, id] of foundEntries) { + let score = Math.round(100 * queryLower.length / entry.length) + results.push([ + docNames[file], + titles[file], + id ? "#" + id : "", + null, + score, + filenames[file], + ]); + } + } + } + + // lookup as object + objectTerms.forEach((term) => + results.push(...Search.performObjectSearch(term, objectTerms)) + ); + + // lookup as search terms in fulltext + results.push(...Search.performTermsSearch(searchTerms, excludedTerms)); + + // let the scorer override scores with a custom scoring function + if (Scorer.score) results.forEach((item) => (item[4] = Scorer.score(item))); + + // now sort the results by score (in opposite order of appearance, since the + // display function below uses pop() to retrieve items) and then + // alphabetically + results.sort((a, b) => { + const leftScore = a[4]; + const rightScore = b[4]; + if (leftScore === rightScore) { + // same score: sort alphabetically + const leftTitle = a[1].toLowerCase(); + const rightTitle = b[1].toLowerCase(); + if (leftTitle === rightTitle) return 0; + return leftTitle > rightTitle ? -1 : 1; // inverted is intentional + } + return leftScore > rightScore ? 1 : -1; + }); + + // remove duplicate search results + // note the reversing of results, so that in the case of duplicates, the highest-scoring entry is kept + let seen = new Set(); + results = results.reverse().reduce((acc, result) => { + let resultStr = result.slice(0, 4).concat([result[5]]).map(v => String(v)).join(','); + if (!seen.has(resultStr)) { + acc.push(result); + seen.add(resultStr); + } + return acc; + }, []); + + results = results.reverse(); + + // for debugging + //Search.lastresults = results.slice(); // a copy + // console.info("search results:", Search.lastresults); + + // print the results + _displayNextItem(results, results.length, searchTerms); + }, + + /** + * search for object names + */ + performObjectSearch: (object, objectTerms) => { + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const objects = Search._index.objects; + const objNames = Search._index.objnames; + const titles = Search._index.titles; + + const results = []; + + const objectSearchCallback = (prefix, match) => { + const name = match[4] + const fullname = (prefix ? prefix + "." : "") + name; + const fullnameLower = fullname.toLowerCase(); + if (fullnameLower.indexOf(object) < 0) return; + + let score = 0; + const parts = fullnameLower.split("."); + + // check for different match types: exact matches of full name or + // "last name" (i.e. last dotted part) + if (fullnameLower === object || parts.slice(-1)[0] === object) + score += Scorer.objNameMatch; + else if (parts.slice(-1)[0].indexOf(object) > -1) + score += Scorer.objPartialMatch; // matches in last name + + const objName = objNames[match[1]][2]; + const title = titles[match[0]]; + + // If more than one term searched for, we require other words to be + // found in the name/title/description + const otherTerms = new Set(objectTerms); + otherTerms.delete(object); + if (otherTerms.size > 0) { + const haystack = `${prefix} ${name} ${objName} ${title}`.toLowerCase(); + if ( + [...otherTerms].some((otherTerm) => haystack.indexOf(otherTerm) < 0) + ) + return; + } + + let anchor = match[3]; + if (anchor === "") anchor = fullname; + else if (anchor === "-") anchor = objNames[match[1]][1] + "-" + fullname; + + const descr = objName + _(", in ") + title; + + // add custom score for some objects according to scorer + if (Scorer.objPrio.hasOwnProperty(match[2])) + score += Scorer.objPrio[match[2]]; + else score += Scorer.objPrioDefault; + + results.push([ + docNames[match[0]], + fullname, + "#" + anchor, + descr, + score, + filenames[match[0]], + ]); + }; + Object.keys(objects).forEach((prefix) => + objects[prefix].forEach((array) => + objectSearchCallback(prefix, array) + ) + ); + return results; + }, + + /** + * search for full-text terms in the index + */ + performTermsSearch: (searchTerms, excludedTerms) => { + // prepare search + const terms = Search._index.terms; + const titleTerms = Search._index.titleterms; + const filenames = Search._index.filenames; + const docNames = Search._index.docnames; + const titles = Search._index.titles; + + const scoreMap = new Map(); + const fileMap = new Map(); + + // perform the search on the required terms + searchTerms.forEach((word) => { + const files = []; + const arr = [ + { files: terms[word], score: Scorer.term }, + { files: titleTerms[word], score: Scorer.title }, + ]; + // add support for partial matches + if (word.length > 2) { + const escapedWord = _escapeRegExp(word); + Object.keys(terms).forEach((term) => { + if (term.match(escapedWord) && !terms[word]) + arr.push({ files: terms[term], score: Scorer.partialTerm }); + }); + Object.keys(titleTerms).forEach((term) => { + if (term.match(escapedWord) && !titleTerms[word]) + arr.push({ files: titleTerms[word], score: Scorer.partialTitle }); + }); + } + + // no match but word was a required one + if (arr.every((record) => record.files === undefined)) return; + + // found search word in contents + arr.forEach((record) => { + if (record.files === undefined) return; + + let recordFiles = record.files; + if (recordFiles.length === undefined) recordFiles = [recordFiles]; + files.push(...recordFiles); + + // set score for the word in each file + recordFiles.forEach((file) => { + if (!scoreMap.has(file)) scoreMap.set(file, {}); + scoreMap.get(file)[word] = record.score; + }); + }); + + // create the mapping + files.forEach((file) => { + if (fileMap.has(file) && fileMap.get(file).indexOf(word) === -1) + fileMap.get(file).push(word); + else fileMap.set(file, [word]); + }); + }); + + // now check if the files don't contain excluded terms + const results = []; + for (const [file, wordList] of fileMap) { + // check if all requirements are matched + + // as search terms with length < 3 are discarded + const filteredTermCount = [...searchTerms].filter( + (term) => term.length > 2 + ).length; + if ( + wordList.length !== searchTerms.size && + wordList.length !== filteredTermCount + ) + continue; + + // ensure that none of the excluded terms is in the search result + if ( + [...excludedTerms].some( + (term) => + terms[term] === file || + titleTerms[term] === file || + (terms[term] || []).includes(file) || + (titleTerms[term] || []).includes(file) + ) + ) + break; + + // select one (max) score for the file. + const score = Math.max(...wordList.map((w) => scoreMap.get(file)[w])); + // add result to the result list + results.push([ + docNames[file], + titles[file], + "", + null, + score, + filenames[file], + ]); + } + return results; + }, + + /** + * helper function to return a node containing the + * search summary for a given text. keywords is a list + * of stemmed words. + */ + makeSearchSummary: (htmlText, keywords) => { + const text = Search.htmlToText(htmlText); + if (text === "") return null; + + const textLower = text.toLowerCase(); + const actualStartPosition = [...keywords] + .map((k) => textLower.indexOf(k.toLowerCase())) + .filter((i) => i > -1) + .slice(-1)[0]; + const startWithContext = Math.max(actualStartPosition - 120, 0); + + const top = startWithContext === 0 ? "" : "..."; + const tail = startWithContext + 240 < text.length ? "..." : ""; + + let summary = document.createElement("p"); + summary.classList.add("context"); + summary.textContent = top + text.substr(startWithContext, 240).trim() + tail; + + return summary; + }, +}; + +_ready(Search.init); diff --git a/_static/searxng-wordmark.svg b/_static/searxng-wordmark.svg new file mode 100644 index 000000000..b94fe3728 --- /dev/null +++ b/_static/searxng-wordmark.svg @@ -0,0 +1,56 @@ + + + + + + + image/svg+xml + + + + + + + + + + + + diff --git a/_static/searxng.css b/_static/searxng.css new file mode 100644 index 000000000..a214bc9d1 --- /dev/null +++ b/_static/searxng.css @@ -0,0 +1,151 @@ +@import url("pocoo.css"); + +a, a.reference, a.footnote-reference { + color: #004b6b; + border-color: #004b6b; +} + +a:hover { + color: #6d4100; + border-color: #6d4100; +} + +p.version-warning { + background-color: #004b6b; +} + +aside.sidebar { + background-color: whitesmoke; + border-color: lightsteelblue; + border-radius: 3pt; +} + +div.sphinxsidebar p.caption { + display: none; +} + +p.sidebar-title, .sidebar p { + margin: 6pt; +} + +.sidebar li, +.hlist li { + list-style-type: disclosure-closed; +} + +.sphinxsidebar .current > a { + font-weight: bold; +} + +/* admonitions +*/ + +div.admonition, div.topic, nav.contents, div.toctree-wrapper { + background-color: #fafafa; + margin: 8px 0px; + padding: 1em; + border-radius: 3pt 0 0 3pt; + border-top: none; + border-right: none; + border-bottom: none; + border-left: 5pt solid #ccc; + list-style-type: disclosure-closed; +} + +div.toctree-wrapper p.caption { + font-weight: normal; + font-size: 24px; + margin: 0 0 10px 0; + padding: 0; + line-height: 1; + display: inline; +} + +p.admonition-title:after { + content: none; +} + +.admonition.hint { border-color: #416dc0b0; } +.admonition.note { border-color: #6c856cb0; } +.admonition.tip { border-color: #85c5c2b0; } +.admonition.attention { border-color: #ecec97b0; } +.admonition.caution { border-color: #a6c677b0; } +.admonition.danger { border-color: #d46262b0; } +.admonition.important { border-color: #dfa3a3b0; } +.admonition.error { border-color: red; } +.admonition.warning { border-color: darkred; } + +.admonition.admonition-generic-admonition-title { + border-color: #416dc0b0; +} + + +/* admonitions with (rendered) reST markup examples (:class: rst-example) + * + * .. admonition:: title of the example + * :class: rst-example + * .... +*/ + +div.rst-example { + background-color: inherit; + margin: 0; + border-top: none; + border-right: 1px solid #ccc; + border-bottom: none; + border-left: none; + border-radius: none; + padding: 0; +} + +div.rst-example > p.admonition-title { + font-family: Sans Serif; + font-style: italic; + font-size: 0.8em; + display: block; + border-bottom: 1px solid #ccc; + padding: 0.5em 1em; + text-align: right; +} + +/* code block in figures + */ + +div.highlight pre { + text-align: left; +} + +/* Table theme +*/ + +thead, tfoot { + background-color: #fff; +} + +th:hover, td:hover { + background-color: #ffc; +} + +thead th, tfoot th, tfoot td, tbody th { + background-color: #fffaef; +} + +tbody tr:nth-child(odd) { + background-color: #fff; +} + +tbody tr:nth-child(even) { + background-color: #fafafa; +} + +caption { + font-family: Sans Serif; + padding: 0.5em; + margin: 0.5em 0 0.5em 0; + caption-side: top; + text-align: left; +} + +div.sphinx-tabs { + clear: both; +} diff --git a/_static/sphinx_highlight.js b/_static/sphinx_highlight.js new file mode 100644 index 000000000..aae669d7e --- /dev/null +++ b/_static/sphinx_highlight.js @@ -0,0 +1,144 @@ +/* Highlighting utilities for Sphinx HTML documentation. */ +"use strict"; + +const SPHINX_HIGHLIGHT_ENABLED = true + +/** + * highlight a given string on a node by wrapping it in + * span elements with the given class name. + */ +const _highlight = (node, addItems, text, className) => { + if (node.nodeType === Node.TEXT_NODE) { + const val = node.nodeValue; + const parent = node.parentNode; + const pos = val.toLowerCase().indexOf(text); + if ( + pos >= 0 && + !parent.classList.contains(className) && + !parent.classList.contains("nohighlight") + ) { + let span; + + const closestNode = parent.closest("body, svg, foreignObject"); + const isInSVG = closestNode && closestNode.matches("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.classList.add(className); + } + + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + parent.insertBefore( + span, + parent.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling + ) + ); + node.nodeValue = val.substr(0, pos); + + if (isInSVG) { + const rect = document.createElementNS( + "http://www.w3.org/2000/svg", + "rect" + ); + const bbox = parent.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute("class", className); + addItems.push({ parent: parent, target: rect }); + } + } + } else if (node.matches && !node.matches("button, select, textarea")) { + node.childNodes.forEach((el) => _highlight(el, addItems, text, className)); + } +}; +const _highlightText = (thisNode, text, className) => { + let addItems = []; + _highlight(thisNode, addItems, text, className); + addItems.forEach((obj) => + obj.parent.insertAdjacentElement("beforebegin", obj.target) + ); +}; + +/** + * Small JavaScript module for the documentation. + */ +const SphinxHighlight = { + + /** + * highlight the search words provided in localstorage in the text + */ + highlightSearchWords: () => { + if (!SPHINX_HIGHLIGHT_ENABLED) return; // bail if no highlight + + // get and clear terms from localstorage + const url = new URL(window.location); + const highlight = + localStorage.getItem("sphinx_highlight_terms") + || url.searchParams.get("highlight") + || ""; + localStorage.removeItem("sphinx_highlight_terms") + url.searchParams.delete("highlight"); + window.history.replaceState({}, "", url); + + // get individual terms from highlight string + const terms = highlight.toLowerCase().split(/\s+/).filter(x => x); + if (terms.length === 0) return; // nothing to do + + // There should never be more than one element matching "div.body" + const divBody = document.querySelectorAll("div.body"); + const body = divBody.length ? divBody[0] : document.querySelector("body"); + window.setTimeout(() => { + terms.forEach((term) => _highlightText(body, term, "highlighted")); + }, 10); + + const searchBox = document.getElementById("searchbox"); + if (searchBox === null) return; + searchBox.appendChild( + document + .createRange() + .createContextualFragment( + '

' + + '' + + _("Hide Search Matches") + + "

" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(SphinxHighlight.highlightSearchWords); +_ready(SphinxHighlight.initEscapeListener); diff --git a/_static/tabs.css b/_static/tabs.css new file mode 100644 index 000000000..957ba60d6 --- /dev/null +++ b/_static/tabs.css @@ -0,0 +1,89 @@ +.sphinx-tabs { + margin-bottom: 1rem; +} + +[role="tablist"] { + border-bottom: 1px solid #a0b3bf; +} + +.sphinx-tabs-tab { + position: relative; + font-family: Lato,'Helvetica Neue',Arial,Helvetica,sans-serif; + color: #1D5C87; + line-height: 24px; + margin: 0; + font-size: 16px; + font-weight: 400; + background-color: rgba(255, 255, 255, 0); + border-radius: 5px 5px 0 0; + border: 0; + padding: 1rem 1.5rem; + margin-bottom: 0; +} + +.sphinx-tabs-tab[aria-selected="true"] { + font-weight: 700; + border: 1px solid #a0b3bf; + border-bottom: 1px solid white; + margin: -1px; + background-color: white; +} + +.sphinx-tabs-tab:focus { + z-index: 1; + outline-offset: 1px; +} + +.sphinx-tabs-panel { + position: relative; + padding: 1rem; + border: 1px solid #a0b3bf; + margin: 0px -1px -1px -1px; + border-radius: 0 0 5px 5px; + border-top: 0; + background: white; +} + +.sphinx-tabs-panel.code-tab { + padding: 0.4rem; +} + +.sphinx-tab img { + margin-bottom: 24 px; +} + +/* Dark theme preference styling */ + +@media (prefers-color-scheme: dark) { + body[data-theme="auto"] .sphinx-tabs-panel { + color: white; + background-color: rgb(50, 50, 50); + } + + body[data-theme="auto"] .sphinx-tabs-tab { + color: white; + background-color: rgba(255, 255, 255, 0.05); + } + + body[data-theme="auto"] .sphinx-tabs-tab[aria-selected="true"] { + border-bottom: 1px solid rgb(50, 50, 50); + background-color: rgb(50, 50, 50); + } +} + +/* Explicit dark theme styling */ + +body[data-theme="dark"] .sphinx-tabs-panel { + color: white; + background-color: rgb(50, 50, 50); +} + +body[data-theme="dark"] .sphinx-tabs-tab { + color: white; + background-color: rgba(255, 255, 255, 0.05); +} + +body[data-theme="dark"] .sphinx-tabs-tab[aria-selected="true"] { + border-bottom: 2px solid rgb(50, 50, 50); + background-color: rgb(50, 50, 50); +} diff --git a/_static/tabs.js b/_static/tabs.js new file mode 100644 index 000000000..48dc303c8 --- /dev/null +++ b/_static/tabs.js @@ -0,0 +1,145 @@ +try { + var session = window.sessionStorage || {}; +} catch (e) { + var session = {}; +} + +window.addEventListener("DOMContentLoaded", () => { + const allTabs = document.querySelectorAll('.sphinx-tabs-tab'); + const tabLists = document.querySelectorAll('[role="tablist"]'); + + allTabs.forEach(tab => { + tab.addEventListener("click", changeTabs); + }); + + tabLists.forEach(tabList => { + tabList.addEventListener("keydown", keyTabs); + }); + + // Restore group tab selection from session + const lastSelected = session.getItem('sphinx-tabs-last-selected'); + if (lastSelected != null) selectNamedTabs(lastSelected); +}); + +/** + * Key focus left and right between sibling elements using arrows + * @param {Node} e the element in focus when key was pressed + */ +function keyTabs(e) { + const tab = e.target; + let nextTab = null; + if (e.keyCode === 39 || e.keyCode === 37) { + tab.setAttribute("tabindex", -1); + // Move right + if (e.keyCode === 39) { + nextTab = tab.nextElementSibling; + if (nextTab === null) { + nextTab = tab.parentNode.firstElementChild; + } + // Move left + } else if (e.keyCode === 37) { + nextTab = tab.previousElementSibling; + if (nextTab === null) { + nextTab = tab.parentNode.lastElementChild; + } + } + } + + if (nextTab !== null) { + nextTab.setAttribute("tabindex", 0); + nextTab.focus(); + } +} + +/** + * Select or deselect clicked tab. If a group tab + * is selected, also select tab in other tabLists. + * @param {Node} e the element that was clicked + */ +function changeTabs(e) { + // Use this instead of the element that was clicked, in case it's a child + const notSelected = this.getAttribute("aria-selected") === "false"; + const positionBefore = this.parentNode.getBoundingClientRect().top; + const notClosable = !this.parentNode.classList.contains("closeable"); + + deselectTabList(this); + + if (notSelected || notClosable) { + selectTab(this); + const name = this.getAttribute("name"); + selectNamedTabs(name, this.id); + + if (this.classList.contains("group-tab")) { + // Persist during session + session.setItem('sphinx-tabs-last-selected', name); + } + } + + const positionAfter = this.parentNode.getBoundingClientRect().top; + const positionDelta = positionAfter - positionBefore; + // Scroll to offset content resizing + window.scrollTo(0, window.scrollY + positionDelta); +} + +/** + * Select tab and show associated panel. + * @param {Node} tab tab to select + */ +function selectTab(tab) { + tab.setAttribute("aria-selected", true); + + // Show the associated panel + document + .getElementById(tab.getAttribute("aria-controls")) + .removeAttribute("hidden"); +} + +/** + * Hide the panels associated with all tabs within the + * tablist containing this tab. + * @param {Node} tab a tab within the tablist to deselect + */ +function deselectTabList(tab) { + const parent = tab.parentNode; + const grandparent = parent.parentNode; + + Array.from(parent.children) + .forEach(t => t.setAttribute("aria-selected", false)); + + Array.from(grandparent.children) + .slice(1) // Skip tablist + .forEach(panel => panel.setAttribute("hidden", true)); +} + +/** + * Select grouped tabs with the same name, but no the tab + * with the given id. + * @param {Node} name name of grouped tab to be selected + * @param {Node} clickedId id of clicked tab + */ +function selectNamedTabs(name, clickedId=null) { + const groupedTabs = document.querySelectorAll(`.sphinx-tabs-tab[name="${name}"]`); + const tabLists = Array.from(groupedTabs).map(tab => tab.parentNode); + + tabLists + .forEach(tabList => { + // Don't want to change the tabList containing the clicked tab + const clickedTab = tabList.querySelector(`[id="${clickedId}"]`); + if (clickedTab === null ) { + // Select first tab with matching name + const tab = tabList.querySelector(`.sphinx-tabs-tab[name="${name}"]`); + deselectTabList(tab); + selectTab(tab); + } + }) +} + +if (typeof exports === 'undefined') { + exports = {}; +} + +exports.keyTabs = keyTabs; +exports.changeTabs = changeTabs; +exports.selectTab = selectTab; +exports.deselectTabList = deselectTabList; +exports.selectNamedTabs = selectNamedTabs; diff --git a/_static/version_warning_offset.js b/_static/version_warning_offset.js new file mode 100644 index 000000000..c7f9f49b5 --- /dev/null +++ b/_static/version_warning_offset.js @@ -0,0 +1,40 @@ +/* +When showing the sticky version warning, the warning will cover the +scroll target when navigating to #id hash locations. Take over scrolling +to adjust the position to account for the height of the warning. +*/ +$(() => { + const versionWarning = $('.version-warning') + + // Skip if there is no version warning, regular browser behavior is + // fine in that case. + if (versionWarning.length) { + const height = versionWarning.outerHeight(true) + const target = $(':target') + + // Adjust position when the initial link has a hash. + if (target.length) { + // Use absolute scrollTo instead of relative scrollBy to avoid + // scrolling when the viewport is already at the bottom of the + // document and has space. + const y = target.offset().top - height + // Delayed because the initial browser scroll doesn't seem to + // happen until after the document ready event, so scrolling + // immediately will be overridden. + setTimeout(() => scrollTo(0, y), 100) + } + + // Listen to clicks on hash anchors. + $('a[href^="#"]').on('click', e => { + // Stop default scroll. Also stops the automatic URL hash update. + e.preventDefault() + // Get the id to scroll to and set the URL hash manually. + const id = $(e.currentTarget).attr('href').substring(1) + location.hash = id + // Use getElementById since the hash may have dots in it. + const target = $(document.getElementById(id)) + // Scroll to top of target with space for the version warning. + scrollTo(0, target.offset().top - height) + }) + } +}) diff --git a/admin/answer-captcha.html b/admin/answer-captcha.html new file mode 100644 index 000000000..281243572 --- /dev/null +++ b/admin/answer-captcha.html @@ -0,0 +1,196 @@ + + + + + + + + Answer CAPTCHA from server’s IP — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Answer CAPTCHA from server’s IP

+

With a SSH tunnel we can send requests from server’s IP and solve a CAPTCHA that +blocks requests from this IP. If your SearXNG instance is hosted at +example.org and your login is user you can setup a proxy simply by +ssh:

+
# SOCKS server: socks://127.0.0.1:8080
+
+$ ssh -q -N -D 8080 user@example.org
+
+
+

The socks://localhost:8080 from above can be tested by:

+
+
$ curl -x socks://127.0.0.1:8080 http://ipecho.net/plain
+n.n.n.n
+
+
+
+

In the settings of the WEB browser open the “Network Settings” and setup a +proxy on SOCKS5 127.0.0.1:8080 (see screenshot below). In the WEB browser +check the IP from the server is used:

+ +

Now open the search engine that blocks requests from your server’s IP. If you +have issues with the qwant engine, +solve the CAPTCHA from qwant.com.

+
+
+
+FFox proxy on SOCKS5, 127.0.0.1:8080 +
+

Fig. 1 Firefox’s network settings

+
+
+
+
+

ssh manual:

+
+
-D [bind_address:]port

Specifies a local “dynamic” application-level port forwarding. This works +by allocating a socket to listen to port on the local side .. Whenever a +connection is made to this port, the connection is forwarded over the +secure channel, and the application protocol is then used to determine +where to connect to from the remote machine .. ssh will act as a SOCKS +server ..

+
+
+
+
-N
+

Do not execute a remote command. This is useful for just forwarding ports.

+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/api.html b/admin/api.html new file mode 100644 index 000000000..c18918ba8 --- /dev/null +++ b/admin/api.html @@ -0,0 +1,226 @@ + + + + + + + + Administration API — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Administration API

+
+

Get configuration data

+
GET /config  HTTP/1.1
+
+
+
+

Sample response

+
{
+  "autocomplete": "",
+  "categories": [
+    "map",
+    "it",
+    "images",
+  ],
+  "default_locale": "",
+  "default_theme": "simple",
+  "engines": [
+    {
+      "categories": [
+        "map"
+      ],
+      "enabled": true,
+      "name": "openstreetmap",
+      "shortcut": "osm"
+    },
+    {
+      "categories": [
+        "it"
+      ],
+      "enabled": true,
+      "name": "arch linux wiki",
+      "shortcut": "al"
+    },
+    {
+      "categories": [
+        "images"
+      ],
+      "enabled": true,
+      "name": "google images",
+      "shortcut": "goi"
+    },
+    {
+      "categories": [
+        "it"
+      ],
+      "enabled": false,
+      "name": "bitbucket",
+      "shortcut": "bb"
+    },
+  ],
+  "instance_name": "searx",
+  "locales": {
+    "de": "Deutsch (German)",
+    "en": "English",
+    "eo": "Esperanto (Esperanto)",
+  },
+  "plugins": [
+    {
+      "enabled": true,
+      "name": "HTTPS rewrite"
+    },
+    {
+      "enabled": false,
+      "name": "Vim-like hotkeys"
+    }
+  ],
+  "safe_search": 0
+}
+
+
+
+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/architecture.html b/admin/architecture.html new file mode 100644 index 000000000..11dfce900 --- /dev/null +++ b/admin/architecture.html @@ -0,0 +1,191 @@ + + + + + + + + Architecture — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Architecture

+ +

Herein you will find some hints and suggestions about typical architectures of +SearXNG infrastructures.

+
+

uWSGI Setup

+

We start with a reference setup for public SearXNG instances which can be build +up and maintained by the scripts from our DevOps tooling box.

+
+arch_public.dot
+

Fig. 2 Reference architecture of a public SearXNG setup.

+
+
+

The reference installation activates server.limiter, server.image_proxy +and ui.static_use_hash (/etc/searxng/settings.yml)

+
# SearXNG settings
+
+use_default_settings: true
+
+general:
+  debug: false
+  instance_name: "SearXNG"
+
+search:
+  safe_search: 2
+  autocomplete: 'duckduckgo'
+
+server:
+  # Is overwritten by ${SEARXNG_SECRET}
+  secret_key: "ultrasecretkey"
+  limiter: true
+  image_proxy: true
+  # public URL of the instance, to ensure correct inbound links. Is overwritten
+  # by ${SEARXNG_URL}.
+  # base_url: http://example.com/location
+
+redis:
+  # URL to connect redis database. Is overwritten by ${SEARXNG_REDIS_URL}.
+  url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+ui:
+  static_use_hash: true
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/buildhosts.html b/admin/buildhosts.html new file mode 100644 index 000000000..812ad79b6 --- /dev/null +++ b/admin/buildhosts.html @@ -0,0 +1,277 @@ + + + + + + + + Buildhosts — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Buildhosts

+ +

To get best results from build, it’s recommend to install additional packages on +build hosts (see utils/searxng.sh).

+
+

Build and Development tools

+

To Install tools used by build and development tasks in once:

+
+
$ sudo -H ./utils/searxng.sh install buildhost
+
+
+
+

This will install packages needed by SearXNG:

+
+
$ sudo -H apt-get install -y \
+    python3-dev python3-babel python3-venv \
+    uwsgi uwsgi-plugin-python3 \
+    git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev
+
+
+
+

and packages needed to build docuemtation and run tests:

+
+
$ sudo -H apt-get install -y \
+    firefox graphviz imagemagick texlive-xetex librsvg2-bin \
+    texlive-latex-recommended texlive-extra-utils fonts-dejavu \
+    latexmk shellcheck
+
+
+
+
+
+

Build docs

+ +

Most of the sphinx requirements are installed from git://setup.py and the +docs can be build from scratch with make docs.html. For better math and +image processing additional packages are needed. The XeTeX needed not only for +PDF creation, it’s also needed for Math equations when HTML output is build.

+

To be able to do Math support for HTML outputs in Sphinx without CDNs, the math are rendered +as images (sphinx.ext.imgmath extension).

+

Here is the extract from the git://docs/conf.py file, setting math renderer +to imgmath:

+
html_math_renderer = 'imgmath'
+imgmath_image_format = 'svg'
+imgmath_font_size = 14
+
+
+

If your docs build (make docs.html) shows warnings like this:

+
WARNING: dot(1) not found, for better output quality install \
+         graphviz from https://www.graphviz.org
+..
+WARNING: LaTeX command 'latex' cannot be run (needed for math \
+         display), check the imgmath_latex setting
+
+
+

you need to install additional packages on your build host, to get better HTML +output (install buildhost).

+
+
$ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin
+
+
+
+

For PDF output you also need:

+
+
$ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu
+
+
+
+
+
+

Lint shell scripts

+

To lint shell scripts we use ShellCheck - a shell script static analysis tool +(install buildhost).

+
+
$ sudo apt install shellcheck
+
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/index.html b/admin/index.html new file mode 100644 index 000000000..802b96721 --- /dev/null +++ b/admin/index.html @@ -0,0 +1,213 @@ + + + + + + + + Administrator documentation — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Administrator documentation

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-apache.html b/admin/installation-apache.html new file mode 100644 index 000000000..1521cf6d0 --- /dev/null +++ b/admin/installation-apache.html @@ -0,0 +1,471 @@ + + + + + + + + Apache — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Apache

+

This section explains how to set up a SearXNG instance using the HTTP server Apache. +If you did use the Installation Script and do not have any special preferences +you can install the SearXNG site using +searxng.sh:

+
$ sudo -H ./utils/searxng.sh install apache
+
+
+

If you have special interests or problems with setting up Apache, the following +section might give you some guidance.

+ + +
+

The Apache HTTP server

+

If Apache is not installed, install it now. If apache is new to you, the +Getting Started, Configuration Files and Terms Used to Describe +Directives documentation gives first orientation. There is also a list of +Apache directives to keep in the pocket.

+
+
sudo -H apt-get install apache2
+
+
+
+

Now at http://localhost you should see some kind of Welcome or Test page. +How this default site is configured, depends on the linux distribution +(compare Apache directives).

+
+
less /etc/apache2/sites-enabled/000-default.conf
+
+
+

In this file, there is a line setting the DocumentRoot directive:

+
DocumentRoot /var/www/html
+
+
+

And the welcome page is the HTML file at /var/www/html/index.html.

+
+
+

Debian’s Apache layout

+

Be aware, Debian’s Apache layout is quite different from the standard Apache +configuration. For details look at the apache2.README.Debian +(/usr/share/doc/apache2/README.Debian.gz). Some commands you should know on +Debian:

+ +
+
+

Apache modules

+

To load additional modules, in most distributions you have to un-comment the +lines with the corresponding LoadModule directive, except in Debian’s Apache layout.

+
+

Debian’s Apache layout uses a2enmod and a2dismod to +activate or disable modules:

+
sudo -H a2enmod ssl
+sudo -H a2enmod headers
+sudo -H a2enmod proxy
+sudo -H a2enmod proxy_http
+sudo -H a2enmod proxy_uwsgi
+
+
+
+
+
+

Apache sites

+
+

In Debian’s Apache layout you create a searxng.conf with the +<Location /searxng > directive and save this file in the sites +available folder at /etc/apache2/sites-available. To enable the +searxng.conf use a2ensite:

+
sudo -H a2ensite searxng.conf
+
+
+
+
+
+
+

Apache’s SearXNG site

+ +

To proxy the incoming requests to the SearXNG instance Apache needs the +mod_proxy module (Apache modules).

+ +

Depending on what your SearXNG installation is listening on, you need a http +mod_proxy_http) or socket (mod_proxy_uwsgi) communication to upstream.

+

The Installation Script installs the reference setup and a uWSGI setup that listens on a socket by default. +You can install and activate your own searxng.conf like shown in +Apache sites.

+
+
# -*- coding: utf-8; mode: apache -*-
+
+LoadModule ssl_module           /usr/lib/apache2/modules/mod_ssl.so
+LoadModule headers_module       /usr/lib/apache2/modules/mod_headers.so
+LoadModule proxy_module         /usr/lib/apache2/modules/mod_proxy.so
+LoadModule proxy_uwsgi_module   /usr/lib/apache2/modules/mod_proxy_uwsgi.so
+# LoadModule setenvif_module      /usr/lib/apache2/modules/mod_setenvif.so
+#
+# SetEnvIf Request_URI /searxng dontlog
+# CustomLog /dev/null combined env=dontlog
+
+<Location /searxng>
+
+    Require all granted
+    Order deny,allow
+    Deny from all
+    # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
+    Allow from all
+
+    # add the trailing slash
+    RedirectMatch  308 /searxng$ /searxng/
+
+    ProxyPreserveHost On
+    ProxyPass unix:/usr/local/searxng/run/socket|uwsgi://uwsgi-uds-searxng/
+
+    # see flaskfix.py
+    RequestHeader set X-Scheme %{REQUEST_SCHEME}s
+    RequestHeader set X-Script-Name /searxng
+
+    # see limiter.py
+    RequestHeader set X-Real-IP %{REMOTE_ADDR}s
+    RequestHeader append X-Forwarded-For %{REMOTE_ADDR}s
+
+</Location>
+
+# uWSGI serves the static files and in settings.yml we use::
+#
+#   ui:
+#     static_use_hash: true
+#
+# Alias /searxng/static/ /usr/local/searxng/searxng-src/searx/static/
+
+
+
+

Restart service:

+
+
sudo -H systemctl restart apache2
+sudo -H service uwsgi restart searxng
+
+
+
+
+
+

disable logs

+

For better privacy you can disable Apache logs. In the examples above activate +one of the lines and restart apache:

+
SetEnvIf Request_URI "/searxng" dontlog
+# CustomLog /dev/null combined env=dontlog
+
+
+

The CustomLog directive disables logs for the entire (virtual) server, use it +when the URL of the service does not have a path component (/searxng), so when +SearXNG is located at root (/).

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-docker.html b/admin/installation-docker.html new file mode 100644 index 000000000..ed98f8d5a --- /dev/null +++ b/admin/installation-docker.html @@ -0,0 +1,319 @@ + + + + + + + + Docker Container — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Docker Container

+
+ +

If you intend to create a public instance using Docker, use our well maintained +docker container

+ + +

The sources are hosted at searxng-docker and the container includes:

+ +

The default SearXNG setup +of this container:

+ +
+
+

Get Docker

+

If you plan to build and maintain a docker image by yourself, make sure you have +Docker installed. On Linux don’t +forget to add your user to the docker group (log out and log back in so that +your group membership is re-evaluated):

+
$ sudo usermod -a -G docker $USER
+
+
+
+
+

searxng/searxng

+ +

The docker image is based on git://Dockerfile and available from +searxng/searxng @dockerhub. Using the docker image is quite easy, for +instance you can pull the searxng/searxng @dockerhub image and deploy a local +instance using docker run:

+
$ mkdir my-instance
+$ cd my-instance
+$ export PORT=8080
+$ docker pull searxng/searxng
+$ docker run --rm \
+             -d -p ${PORT}:8080 \
+             -v "${PWD}/searxng:/etc/searxng" \
+             -e "BASE_URL=http://localhost:$PORT/" \
+             -e "INSTANCE_NAME=my-instance" \
+             searxng/searxng
+2f998.... # container's ID
+
+
+

Open your WEB browser and visit the URL:

+
$ xdg-open "http://localhost:$PORT"
+
+
+

Inside ${PWD}/searxng, you will find settings.yml and uwsgi.ini. You +can modify these files according to your needs and restart the Docker image.

+
$ docker container restart 2f998
+
+
+

Use command container ls to list running containers, add flag -a to list +exited containers also. With container stop a running container can be +stoped. To get rid of a container use container rm:

+
$ docker container ls
+CONTAINER ID   IMAGE             COMMAND                  CREATED         ...
+2f998d725993   searxng/searxng   "/sbin/tini -- /usr/…"   7 minutes ago   ...
+
+$ docker container stop 2f998
+$ docker container rm 2f998
+
+
+ +

If you won’t use docker anymore and want to get rid of all containers & images +use the following prune command:

+
$ docker stop $(docker ps -aq)       # stop all containers
+$ docker system prune                # make some housekeeping
+$ docker rmi -f $(docker images -q)  # drop all images
+
+
+
+

shell inside container

+ +

Like in many other distributions, Alpine’s /bin/sh is dash. Dash is meant to be +POSIX-compliant. +Compared to debian, in the Alpine image bash is not installed. The +git://dockerfiles/docker-entrypoint.sh script is checked against dash +(make tests.shell).

+

To open a shell inside the container:

+
$ docker exec -it 2f998 sh
+
+
+
+
+
+

Build the image

+

It’s also possible to build SearXNG from the embedded git://Dockerfile:

+
$ git clone https://github.com/searxng/searxng.git
+$ cd searxng
+$ make docker.build
+...
+Successfully built 49586c016434
+Successfully tagged searxng/searxng:latest
+Successfully tagged searxng/searxng:1.0.0-209-9c823800-dirty
+
+$ docker images
+REPOSITORY        TAG                        IMAGE ID       CREATED          SIZE
+searxng/searxng   1.0.0-209-9c823800-dirty   49586c016434   13 minutes ago   308MB
+searxng/searxng   latest                     49586c016434   13 minutes ago   308MB
+alpine            3.13                       6dbb9cc54074   3 weeks ago      5.61MB
+
+
+
+
+

Command line

+ +

In the git://Dockerfile the ENTRYPOINT is defined as +git://dockerfiles/docker-entrypoint.sh

+
docker run --rm -it searxng/searxng -h
+
+
+
Command line:
+  -h  Display this help
+  -d  Dry run to update the configuration files.
+  -f  Always update on the configuration files (existing files are renamed with
+      the .old suffix).  Without this option, the new configuration files are
+      copied with the .new suffix
+Environment variables:
+  INSTANCE_NAME settings.yml : general.instance_name
+  AUTOCOMPLETE  settings.yml : search.autocomplete
+  BASE_URL      settings.yml : server.base_url
+  MORTY_URL     settings.yml : result_proxy.url
+  MORTY_KEY     settings.yml : result_proxy.key
+  BIND_ADDRESS  uwsgi bind to the specified TCP socket using HTTP protocol.
+                Default value: 0.0.0.0:8080
+Volume:
+  /etc/searxng  the docker entry point copies settings.yml and uwsgi.ini in
+                this directory (see the -f command line option)"
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-nginx.html b/admin/installation-nginx.html new file mode 100644 index 000000000..a45e4dedb --- /dev/null +++ b/admin/installation-nginx.html @@ -0,0 +1,354 @@ + + + + + + + + NGINX — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

NGINX

+

This section explains how to set up a SearXNG instance using the HTTP server nginx. +If you have used the Installation Script and do not have any special preferences +you can install the SearXNG site using +searxng.sh:

+
$ sudo -H ./utils/searxng.sh install nginx
+
+
+

If you have special interests or problems with setting up nginx, the following +section might give you some guidance.

+ + +
+

The nginx HTTP server

+

If nginx is not installed, install it now.

+
+
sudo -H apt-get install nginx
+
+
+
+

Now at http://localhost you should see a Welcome to nginx! page, on Fedora you +see a Fedora Webserver - Test Page. The test page comes from the default +nginx server configuration. How this default site is configured, +depends on the linux distribution:

+
+
less /etc/nginx/nginx.conf
+
+
+

There is one line that includes site configurations from:

+
include /etc/nginx/sites-enabled/*;
+
+
+
+
+
+

NGINX’s SearXNG site

+

Now you have to create a configuration file (searxng.conf) for the SearXNG +site. If nginx is new to you, the nginx beginners guide is a good starting +point and the Getting Started wiki is always a good resource to keep in the +pocket.

+

Depending on what your SearXNG installation is listening on, you need a http or socket +communication to upstream.

+
+
location /searxng {
+
+    uwsgi_pass unix:///usr/local/searxng/run/socket;
+
+    include uwsgi_params;
+
+    uwsgi_param    HTTP_HOST             $host;
+    uwsgi_param    HTTP_CONNECTION       $http_connection;
+
+    # see flaskfix.py
+    uwsgi_param    HTTP_X_SCHEME         $scheme;
+    uwsgi_param    HTTP_X_SCRIPT_NAME    /searxng;
+
+    # see limiter.py
+    uwsgi_param    HTTP_X_REAL_IP        $remote_addr;
+    uwsgi_param    HTTP_X_FORWARDED_FOR  $proxy_add_x_forwarded_for;
+}
+
+# uWSGI serves the static files and in settings.yml we use::
+#
+#   ui:
+#     static_use_hash: true
+#
+# location /searxng/static/ {
+#     alias /usr/local/searxng/searxng-src/searx/static/;
+# }
+
+
+
+

The Installation Script installs the reference setup and a uWSGI setup that listens on a socket by default.

+
+

Create configuration at /etc/nginx/sites-available/ and place a +symlink to sites-enabled:

+
sudo -H ln -s /etc/nginx/sites-available/searxng.conf \
+              /etc/nginx/sites-enabled/searxng.conf
+
+
+
+

Restart services:

+
+
sudo -H systemctl restart nginx
+sudo -H service uwsgi restart searxng
+
+
+
+
+
+

Disable logs

+

For better privacy you can disable nginx logs in /etc/nginx/nginx.conf.

+
http {
+    # ...
+    access_log /dev/null;
+    error_log  /dev/null;
+    # ...
+}
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-scripts.html b/admin/installation-scripts.html new file mode 100644 index 000000000..b8bd655e3 --- /dev/null +++ b/admin/installation-scripts.html @@ -0,0 +1,194 @@ + + + + + + + + Installation Script — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Installation Script

+ +

The following will install a setup as shown in the reference architecture. First you need to get a clone of the repository. The clone is only needed for +the installation procedure and some maintenance tasks.

+ +

Jump to a folder that is readable by others and start to clone SearXNG, +alternatively you can create your own fork and clone from there.

+
$ cd ~/Downloads
+$ git clone https://github.com/searxng/searxng.git searxng
+$ cd searxng
+
+
+ +

To install a SearXNG reference setup +including a uWSGI setup as described in the +Step by step installation and in the uWSGI section type:

+
$ sudo -H ./utils/searxng.sh install all
+
+
+
+

Attention

+

For the installation procedure, use a sudoer login to run the scripts. If +you install from root, take into account that the scripts are creating a +searxng user. In the installation procedure this new created user does +need to have read access to the cloned SearXNG repository, which is not the case if you clone +it into a folder below /root!

+
+ +

When all services are installed and running fine, you can add SearXNG to your +HTTP server. We do not have any preferences regarding the HTTP server, you can use +whatever you prefer.

+

We use caddy in our docker image and we have +implemented installation procedures for:

+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-searxng.html b/admin/installation-searxng.html new file mode 100644 index 000000000..309e2eea9 --- /dev/null +++ b/admin/installation-searxng.html @@ -0,0 +1,599 @@ + + + + + + + + Step by step installation — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Step by step installation

+ +

In this section we show the setup of a SearXNG instance that will be installed +by the Installation Script.

+
+

Install packages

+
+
$ sudo -H apt-get install -y \
+    python3-dev python3-babel python3-venv \
+    uwsgi uwsgi-plugin-python3 \
+    git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev
+
+
+
+
+

Hint

+

This installs also the packages needed by uWSGI

+
+
+
+

Create user

+
+
$ sudo -H useradd --shell /bin/bash --system \
+    --home-dir "/usr/local/searxng" \
+    --comment 'Privacy-respecting metasearch engine' \
+    searxng
+
+$ sudo -H mkdir "/usr/local/searxng"
+$ sudo -H chown -R "searxng:searxng" "/usr/local/searxng"
+
+
+
+
+
+

Install SearXNG & dependencies

+

Start a interactive shell from new created user and clone SearXNG:

+
+
$ sudo -H -u searxng -i
+(searxng)$ git clone "https://github.com/searxng/searxng" \
+                   "/usr/local/searxng/searxng-src"
+
+
+
+

In the same shell create virtualenv:

+
+
(searxng)$ python3 -m venv "/usr/local/searxng/searx-pyenv"
+(searxng)$ echo ". /usr/local/searxng/searx-pyenv/bin/activate" \
+                   >>  "/usr/local/searxng/.profile"
+
+
+
+

To install SearXNG’s dependencies, exit the SearXNG bash session you opened above +and start a new one. Before installing, check if your virtualenv was sourced +from the login (~/.profile):

+
+
$ sudo -H -u searxng -i
+
+(searxng)$ command -v python && python --version
+/usr/local/searxng/searx-pyenv/bin/python
+Python 3.8.1
+
+# update pip's boilerplate ..
+pip install -U pip
+pip install -U setuptools
+pip install -U wheel
+pip install -U pyyaml
+
+# jump to SearXNG's working tree and install SearXNG into virtualenv
+(searxng)$ cd "/usr/local/searxng/searxng-src"
+(searxng)$ pip install -e .
+
+
+
+
+

Tip

+

Open a second terminal for the configuration tasks and leave the (searx)$ +terminal open for the tasks below.

+
+
+
+

Configuration

+ +

To create a initial /etc/searxng/settings.yml we recommend to start with a +copy of the file git://utils/templates/etc/searxng/settings.yml. This setup +use default settings from +git://searx/settings.yml and is shown in the tab “Use default settings” +below. This setup:

+ +

Modify the /etc/searxng/settings.yml to your needs:

+
+
# SearXNG settings
+
+use_default_settings: true
+
+general:
+  debug: false
+  instance_name: "SearXNG"
+
+search:
+  safe_search: 2
+  autocomplete: 'duckduckgo'
+
+server:
+  # Is overwritten by ${SEARXNG_SECRET}
+  secret_key: "ultrasecretkey"
+  limiter: true
+  image_proxy: true
+  # public URL of the instance, to ensure correct inbound links. Is overwritten
+  # by ${SEARXNG_URL}.
+  # base_url: http://example.com/location
+
+redis:
+  # URL to connect redis database. Is overwritten by ${SEARXNG_REDIS_URL}.
+  url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+ui:
+  static_use_hash: true
+
+# preferences:
+#   lock:
+#     - autocomplete
+#     - method
+
+enabled_plugins:
+  - 'Hash plugin'
+  - 'Search on category select'
+  - 'Self Informations'
+  - 'Tracker URL remover'
+  - 'Ahmia blacklist'
+  # - 'Hostname replace'  # see hostname_replace configuration below
+  # - 'Infinite scroll'
+  # - 'Open Access DOI rewrite'
+  # - 'Vim-like hotkeys'
+
+# plugins:
+#   - only_show_green_results
+
+
+

To see the entire file jump to git://utils/templates/etc/searxng/settings.yml

+
+

For a minimal setup you need to set server:secret_key.

+
+
$ sudo -H mkdir -p "/etc/searxng"
+$ sudo -H cp "/usr/local/searxng/searxng-src/utils/templates/etc/searxng/settings.yml" \
+             "/etc/searxng/settings.yml"
+
+
+
+
+
+

Check

+

To check your SearXNG setup, optional enable debugging and start the webapp. +SearXNG looks at the exported environment $SEARXNG_SETTINGS_PATH for a +configuration file.

+
+
# enable debug ..
+$ sudo -H sed -i -e "s/debug : False/debug : True/g" "/etc/searxng/settings.yml"
+
+# start webapp
+$ sudo -H -u searxng -i
+(searxng)$ cd /usr/local/searxng/searxng-src
+(searxng)$ export SEARXNG_SETTINGS_PATH="/etc/searxng/settings.yml"
+(searxng)$ python searx/webapp.py
+
+# disable debug
+$ sudo -H sed -i -e "s/debug : True/debug : False/g" "/etc/searxng/settings.yml"
+
+
+
+

Open WEB browser and visit http://127.0.0.1:8888 . If you are inside a +container or in a script, test with curl:

+
+
$ xdg-open http://127.0.0.1:8888
+
+
+
+

If everything works fine, hit [CTRL-C] to stop the webapp and disable the +debug option in settings.yml. You can now exit SearXNG user bash session (enter exit +command twice). At this point SearXNG is not demonized; uwsgi allows this.

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation-uwsgi.html b/admin/installation-uwsgi.html new file mode 100644 index 000000000..da3c5e05b --- /dev/null +++ b/admin/installation-uwsgi.html @@ -0,0 +1,630 @@ + + + + + + + + uWSGI — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

uWSGI

+ + +
+

Origin uWSGI

+

How uWSGI is implemented by distributors varies. The uWSGI project itself +recommends two methods:

+
    +

  1. systemd.unit template file as described here One service per app in systemd:

    2. +
+
+

There is one systemd unit template on the system installed and one uwsgi +ini file per uWSGI-app placed at dedicated locations. Take archlinux and a +searxng.ini as example:

+
systemd template unit: /usr/lib/systemd/system/uwsgi@.service
+        contains: [Service]
+                  ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/%I.ini
+
+SearXNG application:   /etc/uwsgi/searxng.ini
+        links to: /etc/uwsgi/apps-available/searxng.ini
+
+
+

The SearXNG app (template /etc/uwsgi/%I.ini) can be maintained as known +from common systemd units:

+
$ systemctl enable  uwsgi@searxng
+$ systemctl start   uwsgi@searxng
+$ systemctl restart uwsgi@searxng
+$ systemctl stop    uwsgi@searxng
+
+
+
+
    +

  1. The uWSGI Emperor which fits for maintaining a large range of uwsgi +apps and there is a Tyrant mode to secure multi-user hosting.

    2. +
+
+

The Emperor mode is a special uWSGI instance that will monitor specific +events. The Emperor mode (the service) is started by a (common, not template) +systemd unit.

+

The Emperor service will scan specific directories for uwsgi ini files +(also know as vassals). If a vassal is added, removed or the timestamp is +modified, a corresponding action takes place: a new uWSGI instance is started, +reload or stopped. Take Fedora and a searxng.ini as example:

+
to install & start SearXNG instance create --> /etc/uwsgi.d/searxng.ini
+to reload the instance edit timestamp      --> touch /etc/uwsgi.d/searxng.ini
+to stop instance remove ini                --> rm /etc/uwsgi.d/searxng.ini
+
+
+
+
+
+

Distributors

+

The uWSGI Emperor mode and systemd unit template is what the distributors +mostly offer their users, even if they differ in the way they implement both +modes and their defaults. Another point they might differ in is the packaging of +plugins (if so, compare Install packages) and what the default python +interpreter is (python2 vs. python3).

+

While archlinux does not start a uWSGI service by default, Fedora (RHEL) starts +a Emperor in Tyrant mode by default (you should have read Pitfalls of the Tyrant mode). Worth to know; debian (ubuntu) follow a complete different +approach, read see Debian’s uWSGI layout.

+
+

Debian’s uWSGI layout

+

Be aware, Debian’s uWSGI layout is quite different from the standard uWSGI +configuration. Your are familiar with Debian’s Apache layout? .. they do a +similar thing for the uWSGI infrastructure. The folders are:

+
/etc/uwsgi/apps-available/
+/etc/uwsgi/apps-enabled/
+
+
+

The uwsgi ini file is enabled by a symbolic link:

+
ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/
+
+
+

More details can be found in the uwsgi.README.Debian +(/usr/share/doc/uwsgi/README.Debian.gz). Some commands you should know on +Debian:

+
Commands recognized by init.d script
+====================================
+
+You can issue to init.d script following commands:
+  * start        | starts daemon
+  * stop         | stops daemon
+  * reload       | sends to daemon SIGHUP signal
+  * force-reload | sends to daemon SIGTERM signal
+  * restart      | issues 'stop', then 'start' commands
+  * status       | shows status of daemon instance (running/not running)
+
+'status' command must be issued with exactly one argument: '<confname>'.
+
+Controlling specific instances of uWSGI
+=======================================
+
+You could control specific instance(s) by issuing:
+
+    SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi <command> <confname> <confname>...
+
+where:
+  * <command> is one of 'start', 'stop' etc.
+  * <confname> is the name of configuration file (without extension)
+
+For example, this is how instance for /etc/uwsgi/apps-enabled/hello.xml is
+started:
+
+    SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi start hello
+
+
+
+
+
+

uWSGI maintenance

+
+
# init.d --> /usr/share/doc/uwsgi/README.Debian.gz
+# For uWSGI debian uses the LSB init process, this might be changed
+# one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067
+
+create     /etc/uwsgi/apps-available/searxng.ini
+enable:    sudo -H ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/
+start:     sudo -H service uwsgi start   searxng
+restart:   sudo -H service uwsgi restart searxng
+stop:      sudo -H service uwsgi stop    searxng
+disable:   sudo -H rm /etc/uwsgi/apps-enabled/searxng.ini
+
+
+
+
+
+

uWSGI setup

+

Create the configuration ini-file according to your distribution and restart the +uwsgi application. As shown below, the Installation Script installs by +default:

+
    +

  • a uWSGI setup that listens on a socket and

    • +

  • enables cache busting.

    • +
+
+
# -*- mode: conf; coding: utf-8  -*-
+[uwsgi]
+
+# uWSGI core
+# ----------
+#
+# https://uwsgi-docs.readthedocs.io/en/latest/Options.html#uwsgi-core
+
+# Who will run the code / Hint: in emperor-tyrant mode uid & gid setting will be
+# ignored [1].  Mode emperor-tyrant is the default on fedora (/etc/uwsgi.ini).
+#
+# [1] https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html#tyrant-mode-secure-multi-user-hosting
+#
+uid = searxng
+gid = searxng
+
+# set (python) default encoding UTF-8
+env = LANG=C.UTF-8
+env = LANGUAGE=C.UTF-8
+env = LC_ALL=C.UTF-8
+
+# chdir to specified directory before apps loading
+chdir = /usr/local/searxng/searxng-src/searx
+
+# SearXNG configuration (settings.yml)
+env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
+
+# disable logging for privacy
+disable-logging = true
+
+# The right granted on the created socket
+chmod-socket = 666
+
+# Plugin to use and interpreter config
+single-interpreter = true
+
+# enable master process
+master = true
+
+# load apps in each worker instead of the master
+lazy-apps = true
+
+# load uWSGI plugins
+plugin = python3,http
+
+# By default the Python plugin does not initialize the GIL.  This means your
+# app-generated threads will not run.  If you need threads, remember to enable
+# them with enable-threads.  Running uWSGI in multithreading mode (with the
+# threads options) will automatically enable threading support. This *strange*
+# default behaviour is for performance reasons.
+enable-threads = true
+
+
+# plugin: python
+# --------------
+#
+# https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-python
+
+# load a WSGI module
+module = searx.webapp
+
+# set PYTHONHOME/virtualenv
+virtualenv = /usr/local/searxng/searx-pyenv
+
+# add directory (or glob) to pythonpath
+pythonpath = /usr/local/searxng/searxng-src
+
+
+# speak to upstream
+# -----------------
+
+socket = /usr/local/searxng/run/socket
+buffer-size = 8192
+
+# uWSGI serves the static files and in settings.yml we use::
+#
+#   ui:
+#     static_use_hash: true
+#
+static-map = /static=/usr/local/searxng/searxng-src/searx/static
+# expires set to one year since there are hashes
+static-expires = /* 31557600
+static-gzip-all = True
+offload-threads = %k
+
+# Cache
+cache2 = name=searxngcache,items=2000,blocks=2000,blocksize=4096,bitmap=1
+
+
+
+
+
+

Pitfalls of the Tyrant mode

+

The implementation of the process owners and groups in the Tyrant mode is +somewhat unusual and requires special consideration. In Tyrant mode mode the +Emperor will run the vassal using the UID/GID of the vassal configuration file +(user and group of the app .ini file).

+

Without option emperor-tyrant-initgroups=true in /etc/uwsgi.ini the +process won’t get the additional groups, but this option is not available in +2.0.x branch (see #2099@uWSGI) the feature #752@uWSGI has been merged (on +Oct. 2014) to the master branch of uWSGI but had never been released; the last +major release is from Dec. 2013, since the there had been only bugfix releases +(see #2425uWSGI). To shorten up:

+
+

In Tyrant mode, there is no way to get additional groups, and the uWSGI +process misses additional permissions that may be needed.

+
+

For example on Fedora (RHEL): If you try to install a redis DB with socket +communication and you want to connect to it from the SearXNG uWSGI, you will see a +Permission denied in the log of your instance:

+
ERROR:searx.redisdb: [searxng (993)] can't connect redis DB ...
+ERROR:searx.redisdb:   Error 13 connecting to unix socket: /usr/local/searxng-redis/run/redis.sock. Permission denied.
+ERROR:searx.plugins.limiter: init limiter DB failed!!!
+
+
+

Even if your searxng user of the uWSGI process is added to additional groups +to give access to the socket from the redis DB:

+
$ groups searxng
+searxng : searxng searxng-redis
+
+
+

To see the effective groups of the uwsgi process, you have to look at the status +of the process, by example:

+
$ ps -aef | grep '/usr/sbin/uwsgi --ini searxng.ini'
+searxng       93      92  0 12:43 ?        00:00:00 /usr/sbin/uwsgi --ini searxng.ini
+searxng      186      93  0 12:44 ?        00:00:01 /usr/sbin/uwsgi --ini searxng.ini
+
+
+

Here you can see that the additional “Groups” of PID 186 are unset (missing gid +of searxng-redis):

+
$ cat /proc/186/task/186/status
+...
+Uid:      993     993     993     993
+Gid:      993     993     993     993
+FDSize:   128
+Groups:
+...
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/installation.html b/admin/installation.html new file mode 100644 index 000000000..947ebfad6 --- /dev/null +++ b/admin/installation.html @@ -0,0 +1,155 @@ + + + + + + + + Installation — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Installation

+

You’re spoilt for choice, choose your preferred method of installation.

+ +

The Step by step installation is an excellent illustration of how a SearXNG +instance is build up (see uWSGI Setup). If you do not have any +special preferences, it’s recommended to use the Docker Container or the +Installation Script.

+
+

Attention

+

SearXNG is growing rapidly, you should regularly read our Migrate and stay tuned! section. If you want to upgrade an existing instance or migrate +from searx to SearXNG, you should read this section first!

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/plugins.html b/admin/plugins.html new file mode 100644 index 000000000..720913258 --- /dev/null +++ b/admin/plugins.html @@ -0,0 +1,202 @@ + + + + + + + + Plugins builtin — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Plugins builtin

+ +

Configuration defaults (at built time):

+
+
DO:
+

Default on

+
+
+
+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 1 Plugins configured at built time (defaults)

Name

DO

Description

+

JS & CSS dependencies

+

Hash plugin

y

Converts strings to different hash digests.

Hostname replace

Rewrite result hostnames or remove results based on the hostname

Open Access DOI rewrite

Avoid paywalls by redirecting to open-access versions of publications when available

Search on category select

y

Perform search immediately if a category selected. Disable to select multiple categories. (JavaScript required)

Self Information

y

Displays your IP if the query is “ip” and your user agent if the query contains “user agent”.

Tor check plugin

This plugin checks if the address of the request is a Tor exit-node, and informs the user if it is; like check.torproject.org, but from SearXNG.

Tracker URL remover

y

Remove trackers arguments from the returned URL

Vim-like hotkeys

Navigate search results with Vim-like hotkeys (JavaScript required). Press “h” key on main or result page to get help.

+ + + +
+ + + + + +
+ + + + + + + \ No newline at end of file diff --git a/admin/settings/index.html b/admin/settings/index.html new file mode 100644 index 000000000..1195b9143 --- /dev/null +++ b/admin/settings/index.html @@ -0,0 +1,172 @@ + + + + + + + + Settings — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Settings

+ +
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings.html b/admin/settings/settings.html new file mode 100644 index 000000000..8c3fd28c3 --- /dev/null +++ b/admin/settings/settings.html @@ -0,0 +1,246 @@ + + + + + + + + settings.yml — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

settings.yml

+

This page describe the options possibilities of the git://searx/settings.yml +file.

+ + +
+

settings.yml location

+

The initial settings.yml we be load from these locations:

+
    +

  1. the full path specified in the SEARXNG_SETTINGS_PATH environment variable.

    2. +

  2. /etc/searxng/settings.yml

    3. +
+

If these files don’t exist (or are empty or can’t be read), SearXNG uses the +git://searx/settings.yml file. Read use_default_settings to +see how you can simplify your user defined settings.yml.

+
+
+

use_default_settings

+ +

The user defined settings.yml is loaded from the settings.yml location +and can relied on the default configuration git://searx/settings.yml using:

+
+

use_default_settings: true

+
+
+
server:

In the following example, the actual settings are the default settings defined +in git://searx/settings.yml with the exception of the secret_key and +the bind_address:

+
use_default_settings: true
+server:
+    secret_key: "ultrasecretkey"   # change this!
+    bind_address: "0.0.0.0"
+
+
+
+
engines:

With use_default_settings: true, each settings can be override in a +similar way, the engines section is merged according to the engine +name. In this example, SearXNG will load all the default engines, will +enable the bing engine and define a token for +the arch linux engine:

+
use_default_settings: true
+server:
+  secret_key: "ultrasecretkey"   # change this!
+engines:
+  - name: arch linux wiki
+    tokens: ['$ecretValue']
+  - name: bing
+    disabled: false
+
+
+
+
engines: / remove:

It is possible to remove some engines from the default settings. The following +example is similar to the above one, but SearXNG doesn’t load the the google +engine:

+
use_default_settings:
+  engines:
+    remove:
+      - google
+server:
+  secret_key: "ultrasecretkey"   # change this!
+engines:
+  - name: arch linux wiki
+    tokens: ['$ecretValue']
+
+
+
+
engines: / keep_only:

As an alternative, it is possible to specify the engines to keep. In the +following example, SearXNG has only two engines:

+
use_default_settings:
+  engines:
+    keep_only:
+      - google
+      - duckduckgo
+server:
+  secret_key: "ultrasecretkey"   # change this!
+engines:
+  - name: google
+    tokens: ['$ecretValue']
+  - name: duckduckgo
+    tokens: ['$ecretValue']
+
+
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_brand.html b/admin/settings/settings_brand.html new file mode 100644 index 000000000..0b95fcde8 --- /dev/null +++ b/admin/settings/settings_brand.html @@ -0,0 +1,161 @@ + + + + + + + + brand: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

brand:

+
brand:
+  issue_url: https://github.com/searxng/searxng/issues
+  docs_url: https://docs.searxng.org
+  public_instances: https://searx.space
+  wiki_url: https://github.com/searxng/searxng/wiki
+
+
+
+
issue_url :

If you host your own issue tracker change this URL.

+
+
docs_url :

If you host your own documentation change this URL.

+
+
public_instances :

If you host your own https://searx.space change this URL.

+
+
wiki_url :

Link to your wiki (or false)

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_categories_as_tabs.html b/admin/settings/settings_categories_as_tabs.html new file mode 100644 index 000000000..f7be24dfc --- /dev/null +++ b/admin/settings/settings_categories_as_tabs.html @@ -0,0 +1,166 @@ + + + + + + + + categories_as_tabs: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

categories_as_tabs:

+

A list of the categories that are displayed as tabs in the user interface. +Categories not listed here can still be searched with the Search syntax.

+
categories_as_tabs:
+  general:
+  images:
+  videos:
+  news:
+  map:
+  music:
+  it:
+  science:
+  files:
+  social media:
+
+
+

Engines are added to categories: (compare categories), the +categories listed in categories_as_tabs are shown as tabs in the UI. If +there are no active engines in a category, the tab is not displayed (e.g. if a +user disables all engines in a category).

+

On the preferences page (/preferences) – under engines – there is an +additional tab, called other. In this tab are all engines listed that are not +in one of the UI tabs (not included in categories_as_tabs).

+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_engine.html b/admin/settings/settings_engine.html new file mode 100644 index 000000000..be056235f --- /dev/null +++ b/admin/settings/settings_engine.html @@ -0,0 +1,344 @@ + + + + + + + + engine: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

engine:

+ +

In the code example below a full fledged example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional.

+
+

Hint

+

A few more options are possible, but they are pretty specific to some +engines (Engine Implementations).

+
+
- name: example engine
+  engine: example
+  shortcut: demo
+  base_url: 'https://{language}.example.com/'
+  send_accept_language_header: false
+  categories: general
+  timeout: 3.0
+  api_key: 'apikey'
+  disabled: false
+  language: en_US
+  tokens: [ 'my-secret-token' ]
+  weight: 1
+  display_error_messages: true
+  about:
+     website: https://example.com
+     wikidata_id: Q306656
+     official_api_documentation: https://example.com/api-doc
+     use_official_api: true
+     require_api_key: true
+     results: HTML
+
+  # overwrite values from section 'outgoing:'
+  enable_http2: false
+  retries: 1
+  max_connections: 100
+  max_keepalive_connections: 10
+  keepalive_expiry: 5.0
+  using_tor_proxy: false
+  proxies:
+    http:
+      - http://proxy1:8080
+      - http://proxy2:8080
+    https:
+      - http://proxy1:8080
+      - http://proxy2:8080
+      - socks5://user:password@proxy3:1080
+      - socks5h://user:password@proxy4:1080
+
+  # other network settings
+  enable_http: false
+  retry_on_http_error: true # or 403 or [404, 429]
+
+
+
+
name :

Name that will be used across SearXNG to define this engine. In settings, on +the result page…

+
+
engine :

Name of the python file used to handle requests and responses to and from this +search engine.

+
+
shortcut :

Code used to execute bang requests (in this case using !bi)

+
+
base_urloptional

Part of the URL that should be stable across every request. Can be useful to +use multiple sites using only one engine, or updating the site URL without +touching at the code.

+
+
send_accept_language_header :

Several engines that support languages (or regions) deal with the HTTP header +Accept-Language to build a response that fits to the locale. When this +option is activated, the language (locale) that is selected by the user is used +to build and send a Accept-Language header in the request to the origin +search engine.

+
+
+
+
categoriesoptional

Specifies to which categories the engine should be added. Engines can be +assigned to multiple categories.

+

Categories can be shown as tabs (categories_as_tabs:) in the +UI. A search in a tab (in the UI) will query all engines that are active in +this tab. In the preferences page (/preferences) – under engines – +users can select what engine should be active when querying in this tab.

+

Alternatively, !bang can be used to search all engines +in a category, regardless of whether they are active or not, or whether they +are in a tab of the UI or not. For example, !dictionaries can be used to +query all search engines in that category (group).

+
+
timeoutoptional

Timeout of the search with the current search engine. Overwrites +request_timeout from outgoing:. Be careful, it will +modify the global timeout of SearXNG.

+
+
api_keyoptional

In a few cases, using an API needs the use of a secret key. How to obtain them +is described in the file.

+
+
disabledoptional

To disable by default the engine, but not deleting it. It will allow the user +to manually activate it in the settings.

+
+
inactive: optional

Remove the engine from the settings (disabled & removed).

+
+
languageoptional

If you want to use another language for a specific engine, you can define it +by using the ISO code of language (and region), like fr, en-US, +de-DE.

+
+
tokensoptional

A list of secret tokens to make this engine private, more details see +Private Engines (tokens).

+
+
weightdefault 1

Weighting of the results of this engine.

+
+
display_error_messagesdefault true

When an engine returns an error, the message is displayed on the user interface.

+
+
networkoptional

Use the network configuration from another engine. +In addition, there are two default networks:

+
    +

  • ipv4 set local_addresses to 0.0.0.0 (use only IPv4 local addresses)

    • +

  • ipv6 set local_addresses to :: (use only IPv6 local addresses)

    • +
+
+
enable_httpoptional

Enable HTTP for this engine (by default only HTTPS is enabled).

+
+
retry_on_http_erroroptional

Retry request on some HTTP status code.

+

Example:

+
    +

  • true : on HTTP status code between 400 and 599.

    • +

  • 403 : on HTTP status code 403.

    • +

  • [403, 429]: on HTTP status code 403 and 429.

    • +
+
+
proxies :

Overwrites proxy settings from outgoing:.

+
+
using_tor_proxy :

Using tor proxy (true) or not (false) for this engine. The default is +taken from using_tor_proxy of the outgoing:.

+
+
+
+
max_keepalive_connection#s :
+
Pool limit configuration, overwrites value pool_maxsize from

outgoing: for this engine.

+
+
+
+
max_connections :

Pool limit configuration, overwrites value pool_connections from +outgoing: for this engine.

+
+
keepalive_expiry :

Pool limit configuration, overwrites value keepalive_expiry from +outgoing: for this engine.

+
+
+
+

Private Engines (tokens)

+

Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through Offline Engines. Or they would +rather share engines only with their trusted friends or colleagues.

+ +

To solve this issue the concept of private engines exists.

+

A new option was added to engines named tokens. It expects a list of strings. +If the user making a request presents one of the tokens of an engine, they can +access information about the engine and make search requests.

+

Example configuration to restrict access to the Arch Linux Wiki engine:

+
- name: arch linux wiki
+  engine: archlinux
+  shortcut: al
+  tokens: [ 'my-secret-token' ]
+
+
+

Unless a user has configured the right token, the engine is going to be hidden +from him/her. It is not going to be included in the list of engines on the +Preferences page and in the output of /config REST API call.

+

Tokens can be added to one’s configuration on the Preferences page under “Engine +tokens”. The input expects a comma separated list of strings.

+

The distribution of the tokens from the administrator to the users is not carved +in stone. As providing access to such engines implies that the admin knows and +trusts the user, we do not see necessary to come up with a strict process. +Instead, we would like to add guidelines to the documentation of the feature.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_general.html b/admin/settings/settings_general.html new file mode 100644 index 000000000..c5f253b97 --- /dev/null +++ b/admin/settings/settings_general.html @@ -0,0 +1,169 @@ + + + + + + + + general: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

general:

+
general:
+  debug: false
+  instance_name:  "SearXNG"
+  privacypolicy_url: false
+  donation_url: false
+  contact_url: false
+  enable_metrics: true
+
+
+
+
debug$SEARXNG_DEBUG

Allow a more detailed log if you run SearXNG directly. Display detailed error +messages in the browser too, so this must be deactivated in production.

+
+
donation_url :

Set value to true to use your own donation page written in the +searx/info/en/donate.md and use false to disable +the donation link altogether.

+
+
privacypolicy_url:

Link to privacy policy.

+
+
contact_url:

Contact mailto: address or WEB form.

+
+
enable_metrics:

Enabled by default. Record various anonymous metrics available at /stats, +/stats/errors and /preferences.

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_outgoing.html b/admin/settings/settings_outgoing.html new file mode 100644 index 000000000..2fb51fa4b --- /dev/null +++ b/admin/settings/settings_outgoing.html @@ -0,0 +1,234 @@ + + + + + + + + outgoing: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

outgoing:

+

Communication with search engines.

+
outgoing:
+  request_timeout: 2.0       # default timeout in seconds, can be override by engine
+  max_request_timeout: 10.0  # the maximum timeout in seconds
+  useragent_suffix: ""       # information like an email address to the administrator
+  pool_connections: 100      # Maximum number of allowable connections, or null
+                             # for no limits. The default is 100.
+  pool_maxsize: 10           # Number of allowable keep-alive connections, or null
+                             # to always allow. The default is 10.
+  enable_http2: true         # See https://www.python-httpx.org/http2/
+  # uncomment below section if you want to use a custom server certificate
+  # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
+  # and https://www.python-httpx.org/compatibility/#ssl-configuration
+  #  verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
+  #
+  # uncomment below section if you want to use a proxyq see: SOCKS proxies
+  #   https://2.python-requests.org/en/latest/user/advanced/#proxies
+  # are also supported: see
+  #   https://2.python-requests.org/en/latest/user/advanced/#socks
+  #
+  #  proxies:
+  #    all://:
+  #      - http://proxy1:8080
+  #      - http://proxy2:8080
+  #
+  #  using_tor_proxy: true
+  #
+  # Extra seconds to add in order to account for the time taken by the proxy
+  #
+  #  extra_proxy_timeout: 10.0
+  #
+
+
+
+
request_timeout :

Global timeout of the requests made to others engines in seconds. A bigger +timeout will allow to wait for answers from slow engines, but in consequence +will slow SearXNG reactivity (the result page may take the time specified in the +timeout to load). Can be override by timeout in the engine:.

+
+
useragent_suffix :

Suffix to the user-agent SearXNG uses to send requests to others engines. If an +engine wish to block you, a contact info here may be useful to avoid that.

+
+
+
+
pool_maxsize:

Number of allowable keep-alive connections, or null to always allow. The +default is 10. See max_keepalive_connections Pool limit configuration.

+
+
pool_connections :

Maximum number of allowable connections, or null # for no limits. The +default is 100. See max_connections Pool limit configuration.

+
+
keepalive_expiry :

Number of seconds to keep a connection in the pool. By default 5.0 seconds. +See keepalive_expiry Pool limit configuration.

+
+
+
+
proxies :

Define one or more proxies you wish to use, see httpx proxies. +If there are more than one proxy for one protocol (http, https), +requests to the engines are distributed in a round-robin fashion.

+
+
source_ips :

If you use multiple network interfaces, define from which IP the requests must +be made. Example:

+
    +

  • 0.0.0.0 any local IPv4 address.

    • +

  • :: any local IPv6 address.

    • +

  • 192.168.0.1

    • +

  • [ 192.168.0.1, 192.168.0.2 ] these two specific IP addresses

    • +

  • fe80::60a2:1691:e5a2:ee1f

    • +

  • fe80::60a2:1691:e5a2:ee1f/126 all IP addresses in this network.

    • +

  • [ 192.168.0.1, fe80::/126 ]

    • +
+
+
retries :

Number of retry in case of an HTTP error. On each retry, SearXNG uses an +different proxy and source ip.

+
+
enable_http2 :

Enable by default. Set to false to disable HTTP/2.

+
+
+
+
verify:$SSL_CERT_FILE, $SSL_CERT_DIR

Allow to specify a path to certificate. +see httpx verification defaults.

+

In addition to verify, SearXNG supports the $SSL_CERT_FILE (for a file) and +$SSL_CERT_DIR (for a directory) OpenSSL variables. +see httpx ssl configuration.

+
+
max_redirects :

30 by default. Maximum redirect before it is an error.

+
+
using_tor_proxy :

Using tor proxy (true) or not (false) for all engines. The default is +false and can be overwritten in the engine:

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_redis.html b/admin/settings/settings_redis.html new file mode 100644 index 000000000..d9680a09b --- /dev/null +++ b/admin/settings/settings_redis.html @@ -0,0 +1,179 @@ + + + + + + + + redis: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

redis:

+

A redis DB can be connected by an URL, in searx.redisdb you +will find a description to test your redis connection in SerXNG. When using +sockets, don’t forget to check the access rights on the socket:

+
ls -la /usr/local/searxng-redis/run/redis.sock
+srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
+
+
+

In this example read/write access is given to the searxng-redis group. To get +access rights to redis instance (the socket), your SearXNG (or even your +developer) account needs to be added to the searxng-redis group.

+
+
url$SEARXNG_REDIS_URL

URL to connect redis database, see Redis.from_url(url) & Redis DB:

+
redis://[[username]:[password]]@localhost:6379/0
+rediss://[[username]:[password]]@localhost:6379/0
+unix://[[username]:[password]]@/path/to/socket.sock?db=0
+
+
+
+
+
+

Redis Developer Notes

+

To set up a local redis instance, first set the socket path of the Redis DB +in your YAML setting:

+
redis:
+  url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+
+

Then use the following commands to install the redis instance (./manage redis.help):

+
$ ./manage redis.build
+$ sudo -H ./manage redis.install
+$ sudo -H ./manage redis.addgrp "${USER}"
+# don't forget to logout & login to get member of group
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_search.html b/admin/settings/settings_search.html new file mode 100644 index 000000000..356cb0aaf --- /dev/null +++ b/admin/settings/settings_search.html @@ -0,0 +1,232 @@ + + + + + + + + search: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ + + + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_server.html b/admin/settings/settings_server.html new file mode 100644 index 000000000..d04ddd7eb --- /dev/null +++ b/admin/settings/settings_server.html @@ -0,0 +1,186 @@ + + + + + + + + server: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

server:

+
server:
+    base_url: http://example.org/location  # change this!
+    port: 8888
+    bind_address: "127.0.0.1"
+    secret_key: "ultrasecretkey"           # change this!
+    limiter: false
+    image_proxy: false
+    default_http_headers:
+      X-Content-Type-Options : nosniff
+      X-XSS-Protection : 1; mode=block
+      X-Download-Options : noopen
+      X-Robots-Tag : noindex, nofollow
+      Referrer-Policy : no-referrer
+
+
+
+
base_url$SEARXNG_URL buildenv

The base URL where SearXNG is deployed. Used to create correct inbound links. +If you change the value, don’t forget to rebuild instance’s environment +(utils/brand.env)

+
+
port & bind_address: $SEARXNG_PORT & $SEARXNG_BIND_ADDRESS buildenv

Port number and bind address of the SearXNG web application if you run it +directly using python searx/webapp.py. Doesn’t apply to a SearXNG +services running behind a proxy and using socket communications. If you +change the value, don’t forget to rebuild instance’s environment +(utils/brand.env)

+
+
secret_key$SEARXNG_SECRET

Used for cryptography purpose.

+
+
+
+
limiter :

Rate limit the number of request on the instance, block some bots. The +Limiter requires a redis: database.

+
+
+
+
image_proxy :

Allow your instance of SearXNG of being able to proxy images. Uses memory space.

+
+
+
+
default_http_headers :

Set additional HTTP headers, see #755

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/settings/settings_ui.html b/admin/settings/settings_ui.html new file mode 100644 index 000000000..3d48b9e50 --- /dev/null +++ b/admin/settings/settings_ui.html @@ -0,0 +1,189 @@ + + + + + + + + ui: — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

ui:

+
ui:
+  static_use_hash: false
+  default_locale: ""
+  query_in_title: false
+  infinite_scroll: false
+  center_alignment: false
+  cache_url: https://web.archive.org/web/
+  default_theme: simple
+  theme_args:
+    simple_style: auto
+
+
+
+
static_use_hash :

Enables cache busting of static files.

+
+
default_locale :

SearXNG interface language. If blank, the locale is detected by using the +browser language. If it doesn’t work, or you are deploying a language +specific instance of searx, a locale can be defined using an ISO language +code, like fr, en, de.

+
+
query_in_title :

When true, the result page’s titles contains the query it decreases the +privacy, since the browser can records the page titles.

+
+
infinite_scroll:

When true, automatically loads the next page when scrolling to bottom of the current page.

+
+
center_alignmentdefault false

When enabled, the results are centered instead of being in the left (or RTL) +side of the screen. This setting only affects the desktop layout +(min-width: @tablet)

+
+
+
+
cache_urlhttps://web.archive.org/web/

URL prefix of the internet archive or cache, don’t forgett trailing slash (if +needed). The default is https://web.archive.org/web/ alternatives are:

+ +
+
default_theme :

Name of the theme you want to use by default on your SearXNG instance.

+
+
theme_args.simple_style:

Style of simple theme: auto, light, dark

+
+
results_on_new_tab:

Open result links in a new tab by default.

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/admin/update-searxng.html b/admin/update-searxng.html new file mode 100644 index 000000000..f2b16af0e --- /dev/null +++ b/admin/update-searxng.html @@ -0,0 +1,265 @@ + + + + + + + + SearXNG maintenance — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

SearXNG maintenance

+ + +
+

How to update

+

How to update depends on the Installation method. If you have used the +Installation Script, use the update command from the utils/searxng.sh +script.

+
sudo -H ./utils/searxng.sh instance update
+
+
+
+
+

How to inspect & debug

+

How to debug depends on the Installation method. If you have used the +Installation Script, use the inspect command from the utils/searxng.sh +script.

+
sudo -H ./utils/searxng.sh instance inspect
+
+
+
+
+

Migrate and stay tuned!

+ +

SearXNG is a rolling release; each commit to the master branch is a release. +SearXNG is growing rapidly, the services and opportunities are change every now +and then, to name just a few:

+
    +

  • Bot protection has been switched from filtron to SearXNG’s limiter, this requires a Redis database.

    • +

  • The image proxy morty is no longer needed, it has been replaced by the +image proxy from SearXNG.

    • +

  • To save bandwith cache busting has been implemented. +To get in use, the static-expires needs to be set in the uWSGI setup.

    • +
+

To stay tuned and get in use of the new features, instance maintainers have to +update the SearXNG code regularly (see How to update). As the above +examples show, this is not always enough, sometimes services have to be set up +or reconfigured and sometimes services that are no longer needed should be +uninstalled.

+
+

Hint

+

First of all: SearXNG is installed by the script utils/searxng.sh. If you +have old filtron, morty or searx setup you should consider complete +uninstall/reinstall.

+
+

Here you will find a list of changes that affect the infrastructure. Please +check to what extent it is necessary to update your installations:

+
+
PR 1595: [fix] uWSGI: increase buffer-size

Re-install uWSGI (utils/searxng.sh) or fix your uWSGI searxng.ini +file manually.

+
+
+
+

remove obsolete services

+

If your searx instance was installed “Step by step” or by the “Installation +scripts”, you need to undo the installation procedure completely. If you have +morty & filtron installed, it is recommended to uninstall these services also. +In case of scripts, to uninstall use the scripts from the origin you installed +searx from or try:

+
$ sudo -H ./utils/filtron.sh remove all
+$ sudo -H ./utils/morty.sh   remove all
+$ sudo -H ./utils/searx.sh   remove all
+
+
+
+

Hint

+

If you are migrate from searx take into account that the .config.sh is no +longer used.

+
+

If you upgrade from searx or from before PR 1332 has been merged and you +have filtron and/or morty installed, don’t forget to remove HTTP sites.

+

Apache:

+
$ sudo -H ./utils/filtron.sh apache remove
+$ sudo -H ./utils/morty.sh apache remove
+
+
+

nginx:

+
$ sudo -H ./utils/filtron.sh nginx remove
+$ sudo -H ./utils/morty.sh nginx remove
+
+
+
+
+

Check after Installation

+

Once you have done your installation, you can run a SearXNG check procedure, +to see if there are some left overs. In this example there exists a old +/etc/searx/settings.yml:

+
$ sudo -H ./utils/searxng.sh instance check
+
+SearXNG checks
+--------------
+ERROR: settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/
+INFO:  [OK] (old) account 'searx' does not exists
+INFO:  [OK] (old) account 'filtron' does not exists
+INFO:  [OK] (old) account 'morty' does not exists
+...
+INFO    searx.redisdb                 : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock'
+INFO    searx.redisdb                 : connected to Redis
+
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/contribution_guide.html b/dev/contribution_guide.html new file mode 100644 index 000000000..cf3bcc137 --- /dev/null +++ b/dev/contribution_guide.html @@ -0,0 +1,291 @@ + + + + + + + + How to contribute — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

How to contribute

+ +
+

Prime directives: Privacy, Hackability

+

SearXNG has two prime directives, privacy-by-design and hackability . The +hackability comes in three levels:

+
    +

  • support of search engines

    • +

  • plugins to alter search behaviour

    • +

  • hacking SearXNG itself

    • +
+

Note the lack of “world domination” among the directives. SearXNG has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +“privacy” deserves a separate chapter, as it’s quite uncommon unfortunately.

+
+

Privacy-by-design

+

SearXNG was born out of the need for a privacy-respecting search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities.

+

A few widely used features work differently or turned off by default or not +implemented at all as a consequence of privacy-by-design.

+

If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained.

+

Also, if you think that something works weird with searx, it’s might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. GET vs POST +requests in various browsers).

+

Remember the other prime directive of SearXNG is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it.

+
+

Happy hacking.

+
+
+
+
+

Code

+ +

In order to submit a patch, please follow the steps below:

+
    +

  • Follow coding conventions.

    +
      +

    • PEP8 standards apply, except the convention of line length

      • +

    • Maximum line length is 120 characters

      • +
    +
    • +

  • The cardinal rule for creating good commits is to ensure there is only one +logical change per commit / read Structural split of changes

    • +

  • Check if your code breaks existing tests. If so, update the tests or fix your +code.

    • +

  • If your code can be unit-tested, add unit tests.

    • +

  • Add yourself to the git://AUTHORS.rst file.

    • +

  • Choose meaningful commit messages, read Conventional Commits

    +
    <type>[optional scope]: <description>
+
+[optional body]
+
+[optional footer(s)]
+
    +
    +
    • +

  • Create a pull request.

    • +
+

For more help on getting started with SearXNG development, see Development Quickstart.

+
+
+

Translation

+

Translation currently takes place on weblate.

+
+
+

Documentation

+ +

The documentation is built using Sphinx. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +our Makefile & ./manage.

+

Here is an example which makes a complete rebuild:

+
$ make docs.clean docs.html
+...
+The HTML pages are in dist/docs.
+
+
+
+

live build

+ +

Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target docs.live builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed (make docs.clean docs.live).

+
$ make docs.live
+...
+The HTML pages are in dist/docs.
+... Serving on http://0.0.0.0:8000
+... Start watching changes
+
+
+

Live builds are implemented by sphinx-autobuild. Use environment +$(SPHINXOPTS) to pass arguments to the sphinx-autobuild command. Except +option --host (which is always set to 0.0.0.0) you can pass any +argument. E.g to find and use a free port, use:

+
$ SPHINXOPTS="--port 0" make docs.live
+...
+... Serving on http://0.0.0.0:50593
+...
+
+
+
+
+

deploy on github.io

+

To deploy documentation at github.io use Makefile target make docs.gh-pages, which builds the documentation and runs all the needed git add, +commit and push:

+
$ make docs.clean docs.gh-pages
+
+
+
+

Attention

+

If you are working in your own brand, don’t forgett to adjust your +brand:.

+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/demo/demo_offline.html b/dev/engines/demo/demo_offline.html new file mode 100644 index 000000000..e4df85652 --- /dev/null +++ b/dev/engines/demo/demo_offline.html @@ -0,0 +1,166 @@ + + + + + + + + Demo Offline Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Demo Offline Engine

+

Within this module we implement a demo offline engine. Do not look to +close to the implementation, its just a simple example. To get in use of this +demo engine add the following entry to your engines list in settings.yml:

+
- name: my offline engine
+  engine: demo_offline
+  shortcut: demo
+  disabled: false
+
+
+
+
+searx.engines.demo_offline.init(engine_settings=None)[source]
+

Initialization of the (offline) engine. The origin of this demo engine is a +simple json string which is loaded in this example while the engine is +initialized.

+
+ +
+
+searx.engines.demo_offline.search(query, request_params)[source]
+

Query (offline) engine and return results. Assemble the list of results from +your local engine. In this demo engine we ignore the ‘query’ term, usual +you would pass the ‘query’ term to your local engine to filter out the +results.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/demo/demo_online.html b/dev/engines/demo/demo_online.html new file mode 100644 index 000000000..01c187ad8 --- /dev/null +++ b/dev/engines/demo/demo_online.html @@ -0,0 +1,173 @@ + + + + + + + + Demo Online Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Demo Online Engine

+

Within this module we implement a demo online engine. Do not look to +close to the implementation, its just a simple example which queries The Art +Institute of Chicago

+

To get in use of this demo engine add the following entry to your engines +list in settings.yml:

+
- name: my online engine
+  engine: demo_online
+  shortcut: demo
+  disabled: false
+
+
+
+
+searx.engines.demo_online.init(engine_settings)[source]
+

Initialization of the (online) engine. If no initialization is needed, drop +this init function.

+
+ +
+
+searx.engines.demo_online.request(query, params)[source]
+

Build up the params for the online request. In this example we build a +URL to fetch images from artic.edu

+
+ +
+
+searx.engines.demo_online.response(resp)[source]
+

Parse out the result items from the response. In this example we parse the +response from api.artic.edu and filter out all +images.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/engine_overview.html b/dev/engines/engine_overview.html new file mode 100644 index 000000000..6f99e2ec4 --- /dev/null +++ b/dev/engines/engine_overview.html @@ -0,0 +1,822 @@ + + + + + + + + Engine Overview — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Engine Overview

+ + +

SearXNG is a metasearch-engine, so it uses different search engines to provide +better results.

+

Because there is no general search API which could be used for every search +engine, an adapter has to be built between SearXNG and the external search +engines. Adapters are stored under the folder git://searx/engines.

+
+

General Engine Configuration

+

It is required to tell SearXNG the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally +settings.yml). The arguments in the settings file override the ones in the +engine file.

+

It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following:

+
+

Engine File

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 2 Common options in the engine module

argument

type

information

categories

list

categories, in which the engine is working

paging

boolean

support multiple pages

time_range_support

boolean

support search time range

engine_type

str

    +

  • online [ref] by +default, other possibles values are:

    • +

  • offline [ref]

    • +

  • online_dictionary [ref]

    • +

  • online_currency [ref]

    • +

  • online_url_search [ref]

    • +
+
+
+
+

Engine settings.yml

+

For a more detailed description, see engine: in the settings.yml.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 3 Common options in the engine setup (settings.yml)

argument

type

information

name

string

name of search-engine

engine

string

name of searxng-engine (file name without .py)

enable_http

bool

enable HTTP (by default only HTTPS is enabled).

shortcut

string

shortcut of search-engine

timeout

string

specific timeout for search-engine

display_error_messages

boolean

display error messages on the web UI

proxies

dict

set proxies for a specific engine +(e.g. proxies : {http: socks5://proxy:port, +https: socks5://proxy:port})

+
+
+

Overrides

+

A few of the options have default values in the namespace of engine’s python +modul, but are often overwritten by the settings. If None is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +SearXNG will not start with that engine (global names with a leading underline can +be None).

+

Here is an very simple example of the global names in the namespace of engine’s +module:

+
# engine dependent config
+categories = ['general']
+paging = True
+_non_overwritten_global = 'foo'
+
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 4 The naming of overrides is arbitrary / recommended overrides are:

argument

type

information

base_url

string

base-url, can be overwritten to use same +engine on other URL

number_of_results

int

maximum number of results per request

language

string

ISO code of language and country like en_US

api_key

string

api-key if required by engine

+
+
+
+

Making a Request

+

To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query.

+
+

Passed Arguments (request)

+

These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 5 If the engine_type is online

argument

type

default-value, information

url

str

''

method

str

'GET'

headers

set

{}

data

set

{}

cookies

set

{}

verify

bool

True

headers.User-Agent

str

a random User-Agent

category

str

current category, like 'general'

safesearch

int

0, between 0 and 2 (normal, moderate, strict)

time_range

Optional[str]

None, can be day, week, month, year

pageno

int

current pagenumber

searxng_locale

str

SearXNG’s locale selected by user. Specific language code like +'en', 'en-US', or 'all' if unspecified.

+ + + + + + + + + + + + + + + + + + + + + + +
Table 6 If the engine_type is online_dictionary, + in addition to the online arguments:

argument

type

default-value, information

from_lang

str

specific language code like 'en_US'

to_lang

str

specific language code like 'en_US'

query

str

the text query without the languages

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 7 If the engine_type is online_currency, + in addition to the online arguments:

argument

type

default-value, information

amount

float

the amount to convert

from

str

ISO 4217 code

to

str

ISO 4217 code

from_name

str

currency name

to_name

str

currency name

+ + + + + + + + + + + + + + +
Table 8 If the engine_type is online_url_search, + in addition to the online arguments:

argument

type

default-value, information

search_url

dict

URLs from the search query:

+
{
+  'http': str,
+  'ftp': str,
+  'data:image': str
+}
+
+
+
+
+
+

Specify Request

+

The function def request(query, params): always returns the params variable, the +following parameters can be used to specify a search request:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

argument

type

information

url

str

requested url

method

str

HTTP request method

headers

set

HTTP header information

data

set

HTTP data information

cookies

set

HTTP cookies

verify

bool

Performing SSL-Validity check

allow_redirects

bool

Follow redirects

max_redirects

int

maximum redirects, hard limit

soft_max_redirects

int

maximum redirects, soft limit. Record an error but don’t stop the engine

raise_for_httperror

bool

True by default: raise an exception if the HTTP code of response is >= 300

+
+
+
+

Result Types (template)

+

Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as +default, the parameter template must be set to the desired +type.

+
+

default

+ + + + + + + + + + + + + + + + + + + + + +
Table 9 Parameter of the default media type:

result-parameter

information

url

string, url of the result

title

string, title of the result

content

string, general result-text

publishedDate

datetime.datetime, time of publish

+
+
+

images

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 10 Parameter of the images media type:

result-parameter

information

template

is set to images.html

url

string, url to the result site

title

string, title of the result (partly implemented)

content

(partly implemented)

publishedDate

datetime.datetime, +time of publish (partly implemented)

img_src

string, url to the result image

thumbnail_src

string, url to a small-preview image

+
+
+

videos

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 11 Parameter of the videos media type:

result-parameter

information

template

is set to videos.html

url

string, url of the result

title

string, title of the result

content

(not implemented yet)

publishedDate

datetime.datetime, time of publish

thumbnail

string, url to a small-preview image

+
+
+

torrent

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 12 Parameter of the torrent media type:

result-parameter

information

template

is set to torrent.html

url

string, url of the result

title

string, title of the result

content

string, general result-text

publishedDate

datetime.datetime, +time of publish (not implemented yet)

seed

int, number of seeder

leech

int, number of leecher

filesize

int, size of file in bytes

files

int, number of files

magnetlink

string, magnetlink of the result

torrentfile

string, torrentfile of the result

+
+
+

map

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 13 Parameter of the map media type:

result-parameter

information

template

is set to map.html

url

string, url of the result

title

string, title of the result

content

string, general result-text

publishedDate

datetime.datetime, time of publish

latitude

latitude of result (in decimal format)

longitude

longitude of result (in decimal format)

boundingbox

boundingbox of result (array of 4. values +[lat-min, lat-max, lon-min, lon-max])

geojson

geojson of result (https://geojson.org/)

osm.type

type of osm-object (if OSM-Result)

osm.id

id of osm-object (if OSM-Result)

address.name

name of object

address.road

street name of object

address.house_number

house number of object

address.locality

city, place of object

address.postcode

postcode of object

address.country

country of object

+
+
+

paper

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 14 Parameter of the paper media type / + see BibTeX field types and BibTeX format

result-parameter

Python type

information

template

str

is set to paper.html

title

str

title of the result

content

str

abstract

comments

str

free text display in italic below the content

tags

List[str]

free tag list

publishedDate

datetime

last publication date

type

str

short description of medium type, e.g. book, pdf or html …

authors

List[str]

list of authors of the work (authors with a “s”)

editor

str

list of editors of a book

publisher

str

name of the publisher

journal

str

name of the journal or magazine the article was +published in

volume

str

volume number

pages

str

page range where the article is

number

str

number of the report or the issue number for a journal article

doi

str

DOI number (like 10.1038/d41586-018-07848-2)

issn

List[str]

ISSN number like 1476-4687

isbn

List[str]

ISBN number like 9780201896831

pdf_url

str

URL to the full article, the PDF version

html_url

str

URL to full article, HTML version

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/enginelib.html b/dev/engines/enginelib.html new file mode 100644 index 000000000..a053233b9 --- /dev/null +++ b/dev/engines/enginelib.html @@ -0,0 +1,530 @@ + + + + + + + + Engine Library — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Engine Library

+ +

Implementations of the framework for the SearXNG engines.

+
+

Hint

+

The long term goal is to modularize all implementations of the engine +framework here in this Python package. ToDo:

+ +
+
+
+class searx.enginelib.Engine[source]
+

Class of engine instances build from YAML settings.

+

Further documentation see General Engine Configuration.

+
+

Hint

+

This class is currently never initialized and only used for type hinting.

+
+
+
+about: dict
+

Additional fileds describing the engine.

+
about:
+   website: https://example.com
+   wikidata_id: Q306656
+   official_api_documentation: https://example.com/api-doc
+   use_official_api: true
+   require_api_key: true
+   results: HTML
+
+
+
+ +
+
+categories: List[str]
+

Specifies to which categories the engine should be added.

+
+ +
+
+disabled: bool
+

To disable by default the engine, but not deleting it. It will allow the +user to manually activate it in the settings.

+
+ +
+
+display_error_messages: bool
+

Display error messages on the web UI.

+
+ +
+
+enable_http: bool
+

Enable HTTP (by default only HTTPS is enabled).

+
+ +
+
+engine: str
+

Name of the python file used to handle requests and responses to and from +this search engine (file name from git://searx/engines without +.py).

+
+ +
+
+engine_type: str
+

Type of the engine (Search processors)

+
+ +
+
+fetch_traits: Callable
+

Function to to fetch engine’s traits from origin.

+
+ +
+
+inactive: bool
+

Remove the engine from the settings (disabled & removed).

+
+ +
+
+language: str
+

For an engine, when there is language: ... in the YAML settings the engine +does support only this one language:

+
- name: google french
+  engine: google
+  language: fr
+
+
+
+ +
+
+language_support: bool
+

Engine supports languages (locales) search.

+
+ +
+
+name: str
+

Name that will be used across SearXNG to define this engine. In settings, on +the result page ..

+
+ +
+
+paging: bool
+

Engine supports multiple pages.

+
+ +
+
+proxies: dict
+

Set proxies for a specific engine (YAML):

+
proxies :
+  http:  socks5://proxy:port
+  https: socks5://proxy:port
+
+
+
+ +
+
+region: str
+

For an engine, when there is region: ... in the YAML settings the engine +does support only this one region:

+
.. code:: yaml
+
+
+
+
    +

  • name: google belgium +engine: google +region: fr-BE

    • +
+
+
+ +
+
+safesearch: bool
+

Engine supports SafeSearch

+
+ +
+
+send_accept_language_header: bool
+

When this option is activated, the language (locale) that is selected by +the user is used to build and send a Accept-Language header in the +request to the origin search engine.

+
+ +
+
+shortcut: str
+

Code used to execute bang requests (!foo)

+
+ +
+
+time_range_support: bool
+

Engine supports search time range.

+
+ +
+
+timeout: float
+

Specific timeout for search-engine.

+
+ +
+
+tokens: List[str]
+

A list of secret tokens to make this engine private, more details see +Private Engines (tokens).

+
+ +
+
+traits: EngineTraits
+

Traits of the engine.

+
+ +
+
+using_tor_proxy: bool
+

Using tor proxy (true) or not (false) for this engine.

+
+ +
+ +
+

Engine traits

+

Engine’s traits are fetched from the origin engines and stored in a JSON file +in the data folder. Most often traits are languages and region codes and +their mapping from SearXNG’s representation to the representation in the origin +search engine. For new traits new properties can be added to the class +EngineTraits.

+

To load traits from the persistence EngineTraitsMap.from_data can be +used.

+
+
+class searx.enginelib.traits.EngineTraits(regions: ~typing.Dict[str, str] = <factory>, languages: ~typing.Dict[str, str] = <factory>, all_locale: str | None = None, data_type: typing_extensions.Literal[traits_v1] = 'traits_v1', custom: ~typing.Dict[str, ~typing.Dict[str, ~typing.Dict] | ~typing.Iterable[str]] = <factory>)[source]
+

The class is intended to be instantiated for each engine.

+
+
+copy()[source]
+

Create a copy of the dataclass object.

+
+ +
+
+classmethod fetch_traits(engine: Engine) → Self | None[source]
+

Call a function fetch_traits(engine_traits) from engines namespace to fetch +and set properties from the origin engine in the object engine_traits. If +function does not exists, None is returned.

+
+ +
+
+get_language(searxng_locale: str, default=None)[source]
+

Return engine’s language string that best fits to SearXNG’s locale.

+
+
Parameters:
+
    +

  • searxng_locale – SearXNG’s internal representation of locale +selected by the user.

    • +

  • default – engine’s default language

    • +
+
+
+

The best fits rules are implemented in +searx.locales.get_engine_locale. Except for the special value all +which is determined from EngineTraits.all_locale.

+
+ +
+
+get_region(searxng_locale: str, default=None)[source]
+

Return engine’s region string that best fits to SearXNG’s locale.

+
+
Parameters:
+
    +

  • searxng_locale – SearXNG’s internal representation of locale +selected by the user.

    • +

  • default – engine’s default region

    • +
+
+
+

The best fits rules are implemented in +searx.locales.get_engine_locale. Except for the special value all +which is determined from EngineTraits.all_locale.

+
+ +
+
+is_locale_supported(searxng_locale: str) → bool[source]
+

A locale (SearXNG’s internal representation) is considered to be +supported by the engine if the region or the language is supported +by the engine.

+

For verification the functions EngineTraits.get_region() and +EngineTraits.get_language() are used.

+
+ +
+
+set_traits(engine: Engine)[source]
+

Set traits from self object in a Engine namespace.

+
+
Parameters:
+

engine – engine instance build by searx.engines.load_engine()

+
+
+
+ +
+
+all_locale: str | None = None
+

To which locale value SearXNG’s all language is mapped (shown a “Default +language”).

+
+ +
+
+custom: Dict[str, Dict[str, Dict] | Iterable[str]]
+

A place to store engine’s custom traits, not related to the SearXNG core.

+
+ +
+
+data_type: typing_extensions.Literal[traits_v1] = 'traits_v1'
+

Data type, default is ‘traits_v1’.

+
+ +
+
+languages: Dict[str, str]
+

Maps SearXNG’s internal representation of a language to the one of the engine.

+

SearXNG’s internal representation can be parsed by babel and the value is +send to the engine:

+
languages = {
+    'ca' : <engine's language name>,
+}
+
+for key, egnine_lang in languages.items():
+   searxng_lang = babel.Locale.parse(key)
+   ...
+
+
+
+ +
+
+regions: Dict[str, str]
+

Maps SearXNG’s internal representation of a region to the one of the engine.

+

SearXNG’s internal representation can be parsed by babel and the value is +send to the engine:

+
regions ={
+    'fr-BE' : <engine's region name>,
+}
+
+for key, egnine_region regions.items():
+   searxng_region = babel.Locale.parse(key, sep='-')
+   ...
+
+
+
+ +
+ +
+
+class searx.enginelib.traits.EngineTraitsEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]
+

Encodes EngineTraits to a serializable object, see +json.JSONEncoder.

+
+
+default(o)[source]
+

Return dictionary of a EngineTraits object.

+
+ +
+ +
+
+class searx.enginelib.traits.EngineTraitsMap[source]
+

A python dictionary to map EngineTraits by engine name.

+
+
+classmethod from_data() → Self[source]
+

Instantiate EngineTraitsMap object from ENGINE_TRAITS

+
+ +
+
+save_data()[source]
+

Store EngineTraitsMap in in file self.ENGINE_TRAITS_FILE

+
+ +
+
+set_traits(engine: Engine | types.ModuleType)[source]
+

Set traits in a Engine namespace.

+
+
Parameters:
+

engine – engine instance build by searx.engines.load_engine()

+
+
+
+ +
+
+ENGINE_TRAITS_FILE = PosixPath('/home/runner/work/searxng/searxng/searx/data/engine_traits.json')
+

File with persistence of the EngineTraitsMap.

+
+ +
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/engines.html b/dev/engines/engines.html new file mode 100644 index 000000000..aba8cffb3 --- /dev/null +++ b/dev/engines/engines.html @@ -0,0 +1,204 @@ + + + + + + + + SearXNG’s engines loader — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

SearXNG’s engines loader

+

Load and initialize the engines, see load_engines() and register +engine_shortcuts.

+

usage:

+
load_engines( settings['engines'] )
+
+
+
+
+searx.engines.is_missing_required_attributes(engine)[source]
+

An attribute is required when its name doesn’t start with _ (underline). +Required attributes must not be None.

+
+ +
+
+searx.engines.load_engine(engine_data: dict) → Engine | types.ModuleType | None[source]
+

Load engine from engine_data.

+
+
Parameters:
+

engine_data (dict) – Attributes from YAML settings:engines/<engine>

+
+
Returns:
+

initialized namespace of the <engine>.

+
+
+
    +

  1. create a namespace and load module of the <engine>

    2. +

  2. update namespace with the defaults from ENGINE_DEFAULT_ARGS

    3. +

  3. update namespace with values from engine_data

    4. +
+

If engine is active, return namespace of the engine, otherwise return +None.

+

This function also returns None if initialization of the namespace fails +for one of the following reasons:

+ +
+ +
+
+searx.engines.load_engines(engine_list)[source]
+

usage: engine_list = settings['engines']

+
+ +
+
+searx.engines.using_tor_proxy(engine: Engine | types.ModuleType)[source]
+

Return True if the engine configuration declares to use Tor.

+
+ +
+
+searx.engines.engine_shortcuts
+

Simple map of registered shortcuts to name of the engine (or None).

+
engine_shortcuts[engine.shortcut] = engine.name
+
+
+
+
+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/index.html b/dev/engines/index.html new file mode 100644 index 000000000..e69ea1ccd --- /dev/null +++ b/dev/engines/index.html @@ -0,0 +1,273 @@ + + + + + + + + Engine Implementations — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Engine Implementations

+ +
+

Framework Components

+ +
+
+

Engine Types

+

The engine_type of an engine +determines which search processor is used by +the engine.

+

In this section a list of the enignes that are documented is given, a complete +list of the engines can be found in the source under: git://searx/engines.

+
+

Online Engines

+ +
+ +
+
+ +
+
+
+

Offline Engines

+ +
+ +
+
+ +
+

Online Currency

+ +

no engine of this type is documented yet / comming soon

+
+
+

Online Dictionary

+ +

no engine of this type is documented yet / comming soon

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/mediawiki.html b/dev/engines/mediawiki.html new file mode 100644 index 000000000..7f740153e --- /dev/null +++ b/dev/engines/mediawiki.html @@ -0,0 +1,223 @@ + + + + + + + + MediaWiki Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

MediaWiki Engine

+ +

The MediaWiki engine is a generic engine to query Wikimedia wikis by +the MediaWiki Action API. For a query action all Wikimedia wikis have +endpoints that follow this pattern:

+
https://{base_url}/w/api.php?action=query&list=search&format=json
+
+
+
+

Note

+

In its actual state, this engine is implemented to parse JSON result +(format=json) from a search query (list=search). If you need other +action and list types ask SearXNG developers to extend the +implementation according to your needs.

+
+
+

Configuration

+

Request:

+ +
+
+

Implementations

+
+
+
+searx.engines.mediawiki.base_url: str = 'https://{language}.wikipedia.org/'
+

Base URL of the Wikimedia wiki.

+
+
{language}:

ISO 639-1 language code (en, de, fr ..) of the search language.

+
+
+
+ +
+
+searx.engines.mediawiki.search_type: str = 'nearmatch'
+

Which type of search to perform. One of the following values: nearmatch, +text or title.

+

See srwhat argument in list=search documentation.

+
+ +
+
+searx.engines.mediawiki.srenablerewrites: bool = True
+

Enable internal query rewriting (Type: boolean). Some search backends can +rewrite the query into another which is thought to provide better results, for +instance by correcting spelling errors.

+

See srenablerewrites argument in list=search documentation.

+
+ +
+
+searx.engines.mediawiki.srprop: str = 'sectiontitle|snippet|timestamp|categorysnippet'
+

Which properties to return.

+

See srprop argument in list=search documentation.

+
+ +
+
+searx.engines.mediawiki.srsort: str = 'relevance'
+

Set the sort order of returned results. One of the following values: +create_timestamp_asc, create_timestamp_desc, incoming_links_asc, +incoming_links_desc, just_match, last_edit_asc, last_edit_desc, +none, random, relevance, user_random.

+

See srenablerewrites argument in list=search documentation.

+
+ +
+
+searx.engines.mediawiki.timestamp_format = '%Y-%m-%dT%H:%M:%SZ'
+

The longhand version of MediaWiki time strings.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/offline/command-line-engines.html b/dev/engines/offline/command-line-engines.html new file mode 100644 index 000000000..6a6d9b727 --- /dev/null +++ b/dev/engines/offline/command-line-engines.html @@ -0,0 +1,230 @@ + + + + + + + + Command Line Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Command Line Engines

+ + + +

With command engines administrators can run engines to integrate arbitrary +shell commands.

+
+

Attention

+

When creating and enabling a command engine on a public instance, you +must be careful to avoid leaking private data.

+
+

The easiest solution is to limit the access by setting tokens as described +in section Private Engines (tokens). The engine base is flexible. Only your +imagination can limit the power of this engine (and maybe security concerns).

+
+

Configuration

+

The following options are available:

+
+
command:

A comma separated list of the elements of the command. A special token +{{QUERY}} tells where to put the search terms of the user. Example:

+
['ls', '-l', '-h', '{{QUERY}}']
+
+
+
+
delimiter:

A mapping containing a delimiter char and the titles of each element in +keys.

+
+
parse_regex:

A dict containing the regular expressions for each result key.

+
+
+

query_type:

+
+

The expected type of user search terms. Possible values: path and +enum.

+
+
path:

Checks if the user provided path is inside the working directory. If not, +the query is not executed.

+
+
enum:

Is a list of allowed search terms. If the user submits something which is +not included in the list, the query returns an error.

+
+
+
+
+
query_enum:

A list containing allowed search terms if query_type is set to enum.

+
+
working_dir:

The directory where the command has to be executed. Default: ./.

+
+
result_separator:

The character that separates results. Default: \n.

+
+
+
+
+

Example

+

The example engine below can be used to find files with a specific name in the +configured working directory:

+
- name: find
+  engine: command
+  command: ['find', '.', '-name', '{{QUERY}}']
+  query_type: path
+  shortcut: fnd
+  delimiter:
+      chars: ' '
+      keys: ['line']
+
+
+
+
+

Implementations

+
+
+
+searx.engines.command.check_parsing_options(engine_settings)[source]
+

Checks if delimiter based parsing or regex parsing is configured correctly

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/offline/nosql-engines.html b/dev/engines/offline/nosql-engines.html new file mode 100644 index 000000000..ad3bb2044 --- /dev/null +++ b/dev/engines/offline/nosql-engines.html @@ -0,0 +1,281 @@ + + + + + + + + NoSQL databases — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

NoSQL databases

+ + + +

The following NoSQL databases are supported:

+ +

All of the engines above are just commented out in the settings.yml, as you have to set various options and install +dependencies before using them.

+

By default, the engines use the key-value template for displaying results / +see simple +theme. If you are not satisfied with the original result layout, you can use +your own template, set result_template attribute to {template_name} and +place the templates at:

+
searx/templates/{theme_name}/result_templates/{template_name}
+
+
+

Furthermore, if you do not wish to expose these engines on a public instance, you +can still add them and limit the access by setting tokens as described in +section Private Engines (tokens).

+
+

Extra Dependencies

+

For using Redis Server or MongoDB you need to +install additional packages in Python’s Virtual Environment of your SearXNG +instance. To switch into the environment (Install SearXNG & dependencies) you can use +utils/searxng.sh:

+
$ sudo utils/searxng.sh instance cmd bash
+(searxng-pyenv)$ pip install ...
+
+
+
+
+

Configure the engines

+

NoSQL databases are used for storing arbitrary data without first defining +their structure.

+
+

Redis Server

+ +

Redis is an open source (BSD licensed), in-memory data structure (key value +based) store. Before configuring the redis_server engine, you must install +the dependency redis.

+
+

Configuration

+

Select a database to search in and set its index in the option db. You can +either look for exact matches or use partial keywords to find what you are +looking for by configuring exact_match_only.

+
+
+

Example

+

Below is an example configuration:

+
# Required dependency: redis
+
+- name: myredis
+  shortcut : rds
+  engine: redis_server
+  exact_match_only: false
+  host: '127.0.0.1'
+  port: 6379
+  enable_http: true
+  password: ''
+  db: 0
+
+
+
+
+

Implementations

+
+
+
+

MongoDB

+ +

MongoDB is a document based database program that handles JSON like data. +Before configuring the mongodb engine, you must install the dependency +pymongo.

+
+

Configuration

+

In order to query MongoDB, you have to select a database and a +collection. Furthermore, you have to select a key that is going to be +searched. MongoDB also supports the option exact_match_only, so configure +it as you wish.

+
+
+

Example

+

Below is an example configuration for using a MongoDB collection:

+
# MongoDB engine
+# Required dependency: pymongo
+
+- name: mymongo
+  engine: mongodb
+  shortcut: md
+  exact_match_only: false
+  host: '127.0.0.1'
+  port: 27017
+  enable_http: true
+  results_per_page: 20
+  database: 'business'
+  collection: 'reviews'  # name of the db collection
+  key: 'name'            # key in the collection to search for
+
+
+
+
+

Implementations

+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/offline/search-indexer-engines.html b/dev/engines/offline/search-indexer-engines.html new file mode 100644 index 000000000..7a3e92ecf --- /dev/null +++ b/dev/engines/offline/search-indexer-engines.html @@ -0,0 +1,270 @@ + + + + + + + + Local Search APIs — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Local Search APIs

+ + + +

Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now:

+ +

Each search engine is powerful, capable of full-text search. All of the engines +above are added to settings.yml just commented out, as you have to +base_url for all them.

+

Please note that if you are not using HTTPS to access these engines, you have to +enable HTTP requests by setting enable_http to True.

+

Furthermore, if you do not want to expose these engines on a public instance, +you can still add them and limit the access by setting tokens as described +in section Private Engines (tokens).

+
+

MeiliSearch

+ +

MeiliSearch is aimed at individuals and small companies. It is designed for +small-scale (less than 10 million documents) data collections. E.g. it is great +for storing web pages you have visited and searching in the contents later.

+

The engine supports faceted search, so you can search in a subset of documents +of the collection. Furthermore, you can search in MeiliSearch instances that +require authentication by setting auth_token.

+
+

Example

+

Here is a simple example to query a Meilisearch instance:

+
- name: meilisearch
+  engine: meilisearch
+  shortcut: mes
+  base_url: http://localhost:7700
+  index: my-index
+  enable_http: true
+
+
+
+
+
+

Elasticsearch

+ +

Elasticsearch supports numerous ways to query the data it is storing. At the +moment the engine supports the most popular search methods (query_type):

+
    +

  • match,

    • +

  • simple_query_string,

    • +

  • term and

    • +

  • terms.

    • +
+

If none of the methods fit your use case, you can select custom query type +and provide the JSON payload to submit to Elasticsearch in +custom_query_json.

+
+

Example

+

The following is an example configuration for an Elasticsearch instance with +authentication configured to read from my-index index.

+
- name: elasticsearch
+  shortcut: es
+  engine: elasticsearch
+  base_url: http://localhost:9200
+  username: elastic
+  password: changeme
+  index: my-index
+  query_type: match
+  # custom_query_json: '{ ... }'
+  enable_http: true
+
+
+
+
+
+

Solr

+ +

Solr is a popular search engine based on Lucene, just like Elasticsearch. But +instead of searching in indices, you can search in collections.

+
+

Example

+

This is an example configuration for searching in the collection +my-collection and get the results in ascending order.

+
- name: solr
+  engine: solr
+  shortcut: slr
+  base_url: http://localhost:8983
+  collection: my-collection
+  sort: asc
+  enable_http: true
+
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/offline/sql-engines.html b/dev/engines/offline/sql-engines.html new file mode 100644 index 000000000..1028bcc45 --- /dev/null +++ b/dev/engines/offline/sql-engines.html @@ -0,0 +1,323 @@ + + + + + + + + SQL Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

SQL Engines

+ + + +

With the SQL engines you can bind SQL databases into SearXNG. The following +Relational Database Management System (RDBMS) are supported:

+ +

All of the engines above are just commented out in the settings.yml, as you have to set the required attributes for the +engines, e.g. database: …

+
- name: ...
+  engine: {sqlite|postgresql|mysql_server}
+  database: ...
+  result_template: {template_name}
+  query_str: ...
+
+
+

By default, the engines use the key-value template for displaying results / +see simple +theme. If you are not satisfied with the original result layout, you can use +your own template, set result_template attribute to {template_name} and +place the templates at:

+
searx/templates/{theme_name}/result_templates/{template_name}
+
+
+

If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting tokens as described in section +Private Engines (tokens).

+
+

Extra Dependencies

+

For using PostgreSQL or MySQL you need to +install additional packages in Python’s Virtual Environment of your SearXNG +instance. To switch into the environment (Install SearXNG & dependencies) you can use +utils/searxng.sh:

+
$ sudo utils/searxng.sh instance cmd bash
+(searxng-pyenv)$ pip install ...
+
+
+
+
+

Configure the engines

+

The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in query_str. At the moment you can only bind at most one +parameter in your query. By setting the attribute limit you can define how +many results you want from the SQL server. Basically, it is the same as the +LIMIT keyword in SQL.

+

Please, do not include LIMIT or OFFSET in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option limit.

+
+

SQLite

+ +

SQLite is a small, fast and reliable SQL database engine. It does not require +any extra dependency.

+
+

Example

+

To demonstrate the power of database engines, here is a more complex example +which reads from a MediathekView (DE) movie database. For this example of the +SQlite engine download the database:

+ +

and unpack into searx/data/filmliste-v2.db. To search the database use e.g +Query to test: !mediathekview concert

+
- name: mediathekview
+  engine: sqlite
+  disabled: False
+  categories: general
+  result_template: default.html
+  database: searx/data/filmliste-v2.db
+  query_str:  >-
+    SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
+           COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
+           description AS content
+      FROM film
+     WHERE title LIKE :wildcard OR description LIKE :wildcard
+     ORDER BY duration DESC
+
+
+
+
+

Implementations

+
+
+
+searx.engines.sqlite.sqlite_cursor()[source]
+

Implements a Context Manager for a +sqlite3.Cursor.

+

Open database in read only mode: if the database doesn’t exist. The default +mode creates an empty file on the file system. See:

+ +
+ +
+
+

PostgreSQL

+ +

PostgreSQL is a powerful and robust open source database. Before configuring +the PostgreSQL engine, you must install the dependency psychopg2.

+
+

Example

+

Below is an example configuration:

+
- name: my_database
+  engine: postgresql
+  database: my_database
+  username: searxng
+  password: password
+  query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
+
+
+
+
+

Implementations

+
+
+
+

MySQL

+ +

MySQL is said to be the most popular open source database. Before enabling +MySQL engine, you must install the package mysql-connector-python.

+

The authentication plugin is configurable by setting auth_plugin in the +attributes. By default it is set to caching_sha2_password.

+
+

Example

+

This is an example configuration for querying a MySQL server:

+
- name: my_database
+  engine: mysql_server
+  database: my_database
+  username: searxng
+  password: password
+  limit: 5
+  query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
+
+
+
+
+

Implementations

+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/offline_concept.html b/dev/engines/offline_concept.html new file mode 100644 index 000000000..cbaeb1105 --- /dev/null +++ b/dev/engines/offline_concept.html @@ -0,0 +1,196 @@ + + + + + + + + Offline Concept — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Offline Concept

+ +

To extend the functionality of SearXNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate.

+

Offline engines can be configured, by adding those to the engines list of +settings.yml. An example skeleton for offline +engines can be found in Demo Offline Engine (demo_offline.py).

+
+

Programming Interface

+
+
init(engine_settings=None)

All offline engines can have their own init function to setup the engine before +accepting requests. The function gets the settings from settings.yml as a +parameter. This function can be omitted, if there is no need to setup anything +in advance.

+
+
search(query, params)

Each offline engine has a function named search. This function is +responsible to perform a search and return the results in a presentable +format. (Where presentable means presentable by the selected result +template.)

+

The return value is a list of results retrieved by the engine.

+
+
Engine representation in /config

If an engine is offline, the attribute offline is set to True.

+
+
+
+
+

Extra Dependencies

+

If an offline engine depends on an external tool, SearXNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance.

+

If there is a need to install additional packages in Python’s Virtual +Environment of your SearXNG instance you need to switch into the environment +(Install SearXNG & dependencies) first, for this you can use utils/searxng.sh:

+
$ sudo utils/searxng.sh instance cmd bash
+(searxng-pyenv)$ pip install ...
+
+
+
+
+

Private engines (Security)

+

To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the Private Engines (tokens). If a +query contains a valid token, then SearXNG performs the requested private +search. If not, requests from an offline engines return errors.

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/annas_archive.html b/dev/engines/online/annas_archive.html new file mode 100644 index 000000000..a1a0c3cfd --- /dev/null +++ b/dev/engines/online/annas_archive.html @@ -0,0 +1,217 @@ + + + + + + + + Anna’s Archive — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Anna’s Archive

+ +

Anna’s Archive is a free non-profit online shadow library metasearch +engine providing access to a variety of book resources (also via IPFS), created +by a team of anonymous archivists (AnnaArchivist).

+
+

Configuration

+

The engine has the following additional settings:

+ +

With this options a SearXNG maintainer is able to configure additional +engines for specific searches in Anna’s Archive. For example a engine to search +for newest articles and journals (PDF) / by shortcut !aaa <search-term>.

+
- name: annas articles
+  engine: annas_archive
+  shortcut: aaa
+  aa_content: 'journal_article'
+  aa_ext: 'pdf'
+  aa_sort: 'newest'
+
+
+
+
+

Implementations

+
+
+
+searx.engines.annas_archive.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and other search arguments from Anna’s search form.

+
+ +
+
+searx.engines.annas_archive.init(engine_settings=None)[source]
+

Check of engine’s settings.

+
+ +
+
+searx.engines.annas_archive.aa_content: str = ''
+

Anan’s search form field Content / possible values:

+
journal_article, book_any, book_fiction, book_unknown, book_nonfiction,
+book_comic, magazine, standards_document
+
+
+

To not filter use an empty string (default).

+
+ +
+
+searx.engines.annas_archive.aa_ext: str = ''
+

Filter Anna’s results by a file ending. Common filters for example are +pdf and epub.

+
+

Note

+

Anna’s Archive is a beta release: Filter results by file extension does not +really work on Anna’s Archive.

+
+
+ +
+
+searx.engines.annas_archive.aa_sort: str = ''
+

Sort Anna’s results, possible values:

+
newest, oldest, largest, smallest
+
+
+

To sort by most relevant use an empty string (default).

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/archlinux.html b/dev/engines/online/archlinux.html new file mode 100644 index 000000000..a98b704e4 --- /dev/null +++ b/dev/engines/online/archlinux.html @@ -0,0 +1,172 @@ + + + + + + + + Arch Linux — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Arch Linux

+ +
+

Arch Linux Wiki

+

This implementation does not use a official API: Mediawiki provides API, but +Arch Wiki blocks access to it.

+
+
+
+searx.engines.archlinux.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages from Archlinix-Wiki. The location of the Wiki address of a +language is mapped in a custom field (wiki_netloc). Depending +on the location, the title argument in the request is translated.

+
"custom": {
+  "wiki_netloc": {
+    "de": "wiki.archlinux.de",
+     # ...
+    "zh": "wiki.archlinuxcn.org"
+  }
+  "title": {
+    "de": "Spezial:Suche",
+     # ...
+    "zh": "Special:搜索"
+  },
+},
+
+
+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/bing.html b/dev/engines/online/bing.html new file mode 100644 index 000000000..c1cb1111d --- /dev/null +++ b/dev/engines/online/bing.html @@ -0,0 +1,325 @@ + + + + + + + + Bing Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Bing Engines

+ +
+

Bing WEB

+

This is the implementation of the Bing-WEB engine. Some of this +implementations are shared by other engines:

+ +

On the preference page Bing offers a lot of languages an regions (see section +‘Search results languages’ and ‘Country/region’). However, the abundant choice +does not correspond to reality, where Bing has a full-text indexer only for a +limited number of languages. By example: you can select a language like Māori +but you never get a result in this language.

+

What comes a bit closer to the truth are the search-APIs but they don`t seem +to be completely correct either (if you take a closer look you will find some +inaccuracies there too):

+ +
+
+searx.engines.bing.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Bing-Web.

+
+ +
+
+searx.engines.bing.request(query, params)[source]
+

Assemble a Bing-Web request.

+
+ +
+
+searx.engines.bing.base_url = 'https://www.bing.com/search'
+

Bing (Web) search URL

+
+ +
+
+searx.engines.bing.bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/reference/market-codes'
+

Bing (Web) search API description

+
+ +
+
+searx.engines.bing.send_accept_language_header = True
+

Bing tries to guess user’s language and territory from the HTTP +Accept-Language. Optional the user can select a search-language (can be +different to the UI language) and a region (market code).

+
+ +
+
+

Bing Images

+

Bing-Images: description see searx.engines.bing.

+
+
+searx.engines.bing_images.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Bing-News.

+
+ +
+
+searx.engines.bing_images.request(query, params)[source]
+

Assemble a Bing-Image request.

+
+ +
+
+searx.engines.bing_images.response(resp)[source]
+

Get response from Bing-Images

+
+ +
+
+searx.engines.bing_images.base_url = 'https://www.bing.com/images/async'
+

Bing (Images) search URL

+
+ +
+
+searx.engines.bing_images.bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-image-search/reference/market-codes'
+

Bing (Images) search API description

+
+ +
+
+

Bing Videos

+

Bing-Videos: description see searx.engines.bing.

+
+
+searx.engines.bing_videos.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Bing-Videos.

+
+ +
+
+searx.engines.bing_videos.request(query, params)[source]
+

Assemble a Bing-Video request.

+
+ +
+
+searx.engines.bing_videos.response(resp)[source]
+

Get response from Bing-Video

+
+ +
+
+searx.engines.bing_videos.base_url = 'https://www.bing.com/videos/asyncv2'
+

Bing (Videos) async search URL.

+
+ +
+
+searx.engines.bing_videos.bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-video-search/reference/market-codes'
+

Bing (Video) search API description

+
+ +
+
+

Bing News

+

Bing-News: description see searx.engines.bing.

+
+
+searx.engines.bing_news.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Bing-News.

+

The description of the +first table says “query parameter when calling the Video Search API.” +.. thats why I use the 4. table “News Category API markets” for the +xpath_market_codes.

+
+ +
+
+searx.engines.bing_news.request(query, params)[source]
+

Assemble a Bing-News request.

+
+ +
+
+searx.engines.bing_news.response(resp)[source]
+

Get response from Bing-Video

+
+ +
+
+searx.engines.bing_news.base_url = 'https://www.bing.com/news/infinitescrollajax'
+

Bing (News) search URL

+
+ +
+
+searx.engines.bing_news.bing_traits_url = 'https://learn.microsoft.com/en-us/bing/search-apis/bing-news-search/reference/market-codes'
+

Bing (News) search API description

+
+ +
+
+searx.engines.bing_news.mkt_alias = {'zh': 'en-WW', 'zh-CN': 'en-WW'}
+

Bing News has an official market code ‘zh-CN’ but we won’t get a result with +this market code. For ‘zh’ and ‘zh-CN’ we better use the Worldwide aggregate +market code (en-WW).

+
+ +
+
+searx.engines.bing_news.time_map = {'day': '4', 'month': '9', 'week': '8'}
+

A string ‘4’ means last hour. We use last hour for day here since the +difference of last day and last week in the result list is just marginally.

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/brave.html b/dev/engines/online/brave.html new file mode 100644 index 000000000..786929f62 --- /dev/null +++ b/dev/engines/online/brave.html @@ -0,0 +1,265 @@ + + + + + + + + Brave Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Brave Engines

+ +

Brave supports the categories listed in brave_category (General, +news, videos, images). The support of paging and time range is limited (see remarks).

+

Configured brave engines:

+
- name: brave
+  engine: brave
+  ...
+  brave_category: search
+  time_range_support: true
+  paging: true
+
+- name: brave.images
+  engine: brave
+  ...
+  brave_category: images
+
+- name: brave.videos
+  engine: brave
+  ...
+  brave_category: videos
+
+- name: brave.news
+  engine: brave
+  ...
+  brave_category: news
+
+
+
+

Brave regions

+

Brave uses two-digit tags for the regions like ca while SearXNG deals with +locales. To get a mapping, all officatl de-facto languages of the Brave +region are mapped to regions in SearXNG (see babel):

+
"regions": {
+  ..
+  "en-CA": "ca",
+  "fr-CA": "ca",
+  ..
+ }
+
+
+
+

Note

+

The language (aka region) support of Brave’s index is limited to very basic +languages. The search results for languages like Chinese or Arabic are of +low quality.

+
+
+
+

Brave languages

+

Brave’s language support is limited to the UI (menues, area local notations, +etc). Brave’s index only seems to support a locale, but it does not seem to +support any languages in its index. The choice of available languages is very +small (and its not clear to me where the differencee in UI is when switching +from en-us to en-ca or en-gb).

+

In the EngineTraits object the +UI languages are stored in a custom field named ui_lang:

+
"custom": {
+  "ui_lang": {
+    "ca": "ca",
+    "de-DE": "de-de",
+    "en-CA": "en-ca",
+    "en-GB": "en-gb",
+    "en-US": "en-us",
+    "es": "es",
+    "fr-CA": "fr-ca",
+    "fr-FR": "fr-fr",
+    "ja-JP": "ja-jp",
+    "pt-BR": "pt-br",
+    "sq-AL": "sq-al"
+  }
+},
+
+
+
+
+

Implementations

+
+
+
+searx.engines.brave.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Brave.

+
+ +
+
+searx.engines.brave.brave_category = 'search'
+

Brave supports common web-search, video search, image and video search.

+
    +

  • search: Common WEB search

    • +

  • videos: search for videos

    • +

  • images: search for images

    • +

  • news: search for news

    • +
+
+ +
+
+searx.engines.brave.brave_spellcheck = False
+

Brave supports some kind of spell checking. When activated, Brave tries to +fix typos, e.g. it searches for food when the user queries for fooh. In +the UI of Brave the user gets warned about this, since we can not warn the user +in SearXNG, the spellchecking is disabled by default.

+
+ +
+
+searx.engines.brave.paging = False
+

Brave only supports paging in brave_category search (UI +category All).

+
+ +
+
+searx.engines.brave.time_range_support = False
+

Brave only supports time-range in brave_category search (UI +category All).

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/bt4g.html b/dev/engines/online/bt4g.html new file mode 100644 index 000000000..a6d21d64b --- /dev/null +++ b/dev/engines/online/bt4g.html @@ -0,0 +1,194 @@ + + + + + + + + BT4G — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

BT4G

+ +

BT4G (bt4g.com) is not a tracker and doesn’t store any content and only +collects torrent metadata (such as file names and file sizes) and a magnet link +(torrent identifier).

+

This engine does not parse the HTML page because there is an API in XML (RSS). +The RSS feed provides fewer data like amount of seeders/leechers and the files +in the torrent file. It’s a tradeoff for a “stable” engine as the XML from RSS +content will change way less than the HTML page.

+
+

Configuration

+

The engine has the following additional settings:

+ +

With this options a SearXNG maintainer is able to configure additional +engines for specific torrent searches. For example a engine to search only for +Movies and sort the result list by the count of seeders.

+
- name: bt4g.movie
+  engine: bt4g
+  shortcut: bt4gv
+  categories: video
+  bt4g_order_by: seeders
+  bt4g_category: 'movie'
+
+
+
+
+

Implementations

+
+
+
+searx.engines.bt4g.bt4g_category = 'all'
+

BT$G offers categoies: all (default), audio, movie, doc, +app and `` other``.

+
+ +
+
+searx.engines.bt4g.bt4g_order_by = 'relevance'
+

Result list can be ordered by relevance (default), size, seeders +or time.

+
+

Hint

+

When time_range is activate, the results always orderd by time.

+
+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/dailymotion.html b/dev/engines/online/dailymotion.html new file mode 100644 index 000000000..b5957c2c5 --- /dev/null +++ b/dev/engines/online/dailymotion.html @@ -0,0 +1,204 @@ + + + + + + + + Dailymotion — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Dailymotion

+ +
+

Dailymotion (Videos)

+
+
+
+searx.engines.dailymotion.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch locales & languages from dailymotion.

+

Locales fetched from api/locales. +There are duplications in the locale codes returned from Dailymotion which +can be ignored:

+
en_EN --> en_GB, en_US
+ar_AA --> ar_EG, ar_AE, ar_SA
+
+
+

The language list api/languages +contains over 7000 languages codes (see PR1071). We use only those +language codes that are used in the locales.

+
+ +
+
+searx.engines.dailymotion.family_filter_map = {0: 'false', 1: 'true', 2: 'true'}
+

By default, the family filter is turned on. Setting this parameter to +false will stop filtering-out explicit content from searches and global +contexts (family_filter in Global API Parameters ).

+
+ +
+
+searx.engines.dailymotion.iframe_src = 'https://www.dailymotion.com/embed/video/{video_id}'
+

URL template to embed video in SearXNG’s result list.

+
+ +
+
+searx.engines.dailymotion.result_fields = ['allow_embed', 'description', 'title', 'created_time', 'duration', 'url', 'thumbnail_360_url', 'id']
+

Fields selection, by default, a few fields are returned. To request more +specific fields, the fields parameter is used with the list of fields +SearXNG needs in the response to build a video result list.

+
+ +
+
+searx.engines.dailymotion.safesearch_params = {0: {}, 1: {'is_created_for_kids': 'true'}, 2: {'is_created_for_kids': 'true'}}
+

True if this video is “Created for Kids” / intends to target an audience +under the age of 16 (is_created_for_kids in Video filters API )

+
+ +
+
+searx.engines.dailymotion.search_url = 'https://api.dailymotion.com/videos?'
+

URL to retrieve a list of videos.

+ +
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/duckduckgo.html b/dev/engines/online/duckduckgo.html new file mode 100644 index 000000000..9a7025666 --- /dev/null +++ b/dev/engines/online/duckduckgo.html @@ -0,0 +1,256 @@ + + + + + + + + DukcDukGo Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

DukcDukGo Engines

+ +
+

DuckDuckGo Lite

+
+
+
+searx.engines.duckduckgo.cache_vqd(query, value)[source]
+

Caches a vqd value from a query.

+

The vqd value depends on the query string and is needed for the follow up +pages or the images loaded by a XMLHttpRequest:

+
    +

  • DuckDuckGo Web: https://links.duckduckgo.com/d.js?q=…&vqd=…

    • +

  • DuckDuckGo Images: https://duckduckgo.com/i.js??q=…&vqd=…

    • +
+
+ +
+
+searx.engines.duckduckgo.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages & regions from DuckDuckGo.

+

SearXNG’s all locale maps DuckDuckGo’s “Alle regions” (wt-wt). +DuckDuckGo’s language “Browsers prefered language” (wt_WT) makes no +sense in a SearXNG request since SearXNG’s all will not add a +Accept-Language HTTP header. The value in engine_traits.all_locale +is wt-wt (the region).

+

Beside regions DuckDuckGo also defines its lanaguages by region codes. By +example these are the english languages in DuckDuckGo:

+
    +

  • en_US

    • +

  • en_AU

    • +

  • en_CA

    • +

  • en_GB

    • +
+

The function get_ddg_lang evaluates DuckDuckGo’s language from +SearXNG’s locale.

+
+ +
+
+searx.engines.duckduckgo.get_ddg_lang(eng_traits: EngineTraits, sxng_locale, default='en_US')[source]
+

Get DuckDuckGo’s language identifier from SearXNG’s locale.

+

DuckDuckGo defines its lanaguages by region codes (see +fetch_traits).

+

To get region and language of a DDG service use:

+

It might confuse, but the l value of the cookie is what SearXNG calls +the region:

+
# !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
+params['cookies']['ad'] = eng_lang
+params['cookies']['ah'] = eng_region
+params['cookies']['l'] = eng_region
+
+
+
+

Hint

+

DDG-lite does not offer a language +selection to the user, only a region can be selected by the user +(eng_region from the example above). DDG-lite stores the selected +region in a cookie:

+
params['cookies']['kl'] = eng_region  # 'ar-es'
+
+
+
+
+ +
+
+searx.engines.duckduckgo.get_vqd(query, headers)[source]
+

Returns the vqd that fits to the query. If there is no vqd cached +(cache_vqd) the query is sent to DDG to get a vqd value from the +response.

+
+ +
+
+searx.engines.duckduckgo.send_accept_language_header = True
+

DuckDuckGo-Lite tries to guess user’s prefered language from the HTTP +Accept-Language. Optional the user can select a region filter (but not a +language).

+
+ +
+

DuckDuckGo Images

+
+
+

DuckDuckGo Instant Answer API

+

The DDG-API is no longer documented but from +reverse engineering we can see that some services (e.g. instant answers) still +in use from the DDG search engine.

+

As far we can say the instant answers API does not support languages, or at +least we could not find out how language support should work. It seems that +most of the features are based on English terms.

+
+
+
+searx.engines.duckduckgo_definitions.area_to_str(area)[source]
+

parse {'unit': 'https://www.wikidata.org/entity/Q712226', 'amount': '+20.99'}

+
+ +
+
+searx.engines.duckduckgo_definitions.is_broken_text(text)[source]
+

duckduckgo may return something like <a href="xxxx">http://somewhere Related website<a/>

+

The href URL is broken, the “Related website” may contains some HTML.

+

The best solution seems to ignore these results.

+
+ +
+

DuckDuckGo Weather

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/google.html b/dev/engines/online/google.html new file mode 100644 index 000000000..d368e7267 --- /dev/null +++ b/dev/engines/online/google.html @@ -0,0 +1,413 @@ + + + + + + + + Google Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Google Engines

+ +
+

Google API

+

SearXNG’s implementation of the Google API is mainly done in +get_google_info.

+

For detailed description of the REST-full API see: Query Parameter +Definitions. The linked API documentation can sometimes be helpful during +reverse engineering. However, we cannot use it in the freely accessible WEB +services; not all parameters can be applied and some engines are more special +than other (e.g. Google News).

+
+
+

Google WEB

+

This is the implementation of the Google WEB engine. Some of this +implementations (manly the get_google_info) are shared by other +engines:

+ +
+
+searx.engines.google.fetch_traits(engine_traits: EngineTraits, add_domains: bool = True)[source]
+

Fetch languages from Google.

+
+ +
+
+searx.engines.google.get_google_info(params, eng_traits)[source]
+

Composing various (language) properties for the google engines (Google API).

+

This function is called by the various google engines (Google WEB, Google Images, Google News and +Google Videos).

+
+
Parameters:
+
    +

  • param (dict) – Request parameters of the engine. At least +a searxng_locale key should be in the dictionary.

    • +

  • eng_traits – Engine’s traits fetched from google preferences +(searx.enginelib.traits.EngineTraits)

    • +
+
+
Return type:
+

dict

+
+
Returns:
+

Py-Dictionary with the key/value pairs:

+
+
language:

The language code that is used by google (e.g. lang_en or +lang_zh-TW)

+
+
country:

The country code that is used by google (e.g. US or TW)

+
+
locale:

A instance of babel.core.Locale build from the +searxng_locale value.

+
+
subdomain:

Google subdomain google_domains that fits to the country +code.

+
+
params:

Py-Dictionary with additional request arguments (can be passed to +urllib.parse.urlencode()).

+
    +

  • hl parameter: specifies the interface language of user interface.

    • +

  • lr parameter: restricts search results to documents written in +a particular language.

    • +

  • cr parameter: restricts search results to documents +originating in a particular country.

    • +

  • ie parameter: sets the character encoding scheme that should +be used to interpret the query string (‘utf8’).

    • +

  • oe parameter: sets the character encoding scheme that should +be used to decode the XML result (‘utf8’).

    • +
+
+
headers:

Py-Dictionary with additional HTTP headers (can be passed to +request’s headers)

+
    +

  • Accept: '*/*

    • +
+
+
+

+
+
+
+ +
+
+searx.engines.google.request(query, params)[source]
+

Google search request

+
+ +
+
+searx.engines.google.response(resp)[source]
+

Get response from google’s search request

+
+ +
+
+searx.engines.google.UI_ASYNC = 'use_ac:true,_fmt:prog'
+

Format of the response from UI’s async request.

+
+ +
+
+

Google Autocomplete

+
+
+searx.autocomplete.google_complete(query, sxng_locale)[source]
+

Autocomplete from Google. Supports Google’s languages and subdomains +(searx.engines.google.get_google_info) by using the async REST +API:

+
https://{subdomain}/complete/search?{args}
+
+
+
+ +
+
+

Google Images

+

This is the implementation of the Google Images engine using the internal +Google API used by the Google Go Android app.

+

This internal API offer results in

+
    +

  • JSON (_fmt:json)

    • +

  • Protobuf (_fmt:pb)

    • +

  • Protobuf compressed? (_fmt:pc)

    • +

  • HTML (_fmt:html)

    • +

  • Protobuf encoded in JSON (_fmt:jspb).

    • +
+
+
+searx.engines.google_images.request(query, params)[source]
+

Google-Image search request

+
+ +
+
+searx.engines.google_images.response(resp)[source]
+

Get response from google’s search request

+
+ +
+
+

Google Videos

+

This is the implementation of the Google Videos engine.

+
+

Content-Security-Policy (CSP)

+

This engine needs to allow images from the data URLs (prefixed with the +data: scheme):

+
Header set Content-Security-Policy "img-src 'self' data: ;"
+
+
+
+
+
+searx.engines.google_videos.request(query, params)[source]
+

Google-Video search request

+
+ +
+
+searx.engines.google_videos.response(resp)[source]
+

Get response from google’s search request

+
+ +
+
+

Google News

+

This is the implementation of the Google News engine.

+

Google News has a different region handling compared to Google WEB.

+
    +

  • the ceid argument has to be set (ceid_list)

    • +

  • the hl argument has to be set correctly (and different to Google WEB)

    • +

  • the gl argument is mandatory

    • +
+

If one of this argument is not set correctly, the request is redirected to +CONSENT dialog:

+
https://consent.google.com/m?continue=
+
+
+

The google news API ignores some parameters from the common Google API:

+
    +

  • num : the number of search results is ignored / there is no paging all +results for a query term are in the first response.

    • +

  • save : is ignored / Google-News results are always SafeSearch

    • +
+
+
+searx.engines.google_news.request(query, params)[source]
+

Google-News search request

+
+ +
+
+searx.engines.google_news.response(resp)[source]
+

Get response from google’s search request

+
+ +
+
+searx.engines.google_news.ceid_list = ['AE:ar', 'AR:es-419', 'AT:de', 'AU:en', 'BD:bn', 'BE:fr', 'BE:nl', 'BG:bg', 'BR:pt-419', 'BW:en', 'CA:en', 'CA:fr', 'CH:de', 'CH:fr', 'CL:es-419', 'CN:zh-Hans', 'CO:es-419', 'CU:es-419', 'CZ:cs', 'DE:de', 'EG:ar', 'ES:es', 'ET:en', 'FR:fr', 'GB:en', 'GH:en', 'GR:el', 'HK:zh-Hant', 'HU:hu', 'ID:en', 'ID:id', 'IE:en', 'IL:en', 'IL:he', 'IN:bn', 'IN:en', 'IN:hi', 'IN:ml', 'IN:mr', 'IN:ta', 'IN:te', 'IT:it', 'JP:ja', 'KE:en', 'KR:ko', 'LB:ar', 'LT:lt', 'LV:en', 'LV:lv', 'MA:fr', 'MX:es-419', 'MY:en', 'NA:en', 'NG:en', 'NL:nl', 'NO:no', 'NZ:en', 'PE:es-419', 'PH:en', 'PK:en', 'PL:pl', 'PT:pt-150', 'RO:ro', 'RS:sr', 'RU:ru', 'SA:ar', 'SE:sv', 'SG:en', 'SI:sl', 'SK:sk', 'SN:fr', 'TH:th', 'TR:tr', 'TW:zh-Hant', 'TZ:en', 'UA:ru', 'UA:uk', 'UG:en', 'US:en', 'US:es-419', 'VE:es-419', 'VN:vi', 'ZA:en', 'ZW:en']
+

List of region/language combinations supported by Google News. Values of the +ceid argument of the Google News REST API.

+
+ +
+
+

Google Scholar

+

This is the implementation of the Google Scholar engine.

+

Compared to other Google services the Scholar engine has a simple GET REST-API +and there does not exists async API. Even though the API slightly vintage we +can make use of the Google API to assemble the arguments of the GET +request.

+
+
+searx.engines.google_scholar.detect_google_captcha(dom)[source]
+

In case of CAPTCHA Google Scholar open its own not a Robot dialog and is +not redirected to sorry.google.com.

+
+ +
+
+searx.engines.google_scholar.parse_gs_a(text: str | None)[source]
+

Parse the text written in green.

+

Possible formats: +* “{authors} - {journal}, {year} - {publisher}” +* “{authors} - {year} - {publisher}” +* “{authors} - {publisher}”

+
+ +
+
+searx.engines.google_scholar.request(query, params)[source]
+

Google-Scholar search request

+
+ +
+
+searx.engines.google_scholar.response(resp)[source]
+

Parse response from Google Scholar

+
+ +
+
+searx.engines.google_scholar.time_range_args(params)[source]
+

Returns a dictionary with a time range arguments based on +params['time_range'].

+

Google Scholar supports a detailed search by year. Searching by last +month or last week (as offered by SearXNG) is uncommon for scientific +publications and is not supported by Google Scholar.

+

To limit the result list when the users selects a range, all the SearXNG +ranges (day, week, month, year) are mapped to year. If no range +is set an empty dictionary of arguments is returned. Example; when +user selects a time range (current year minus one in 2022):

+
{ 'as_ylo' : 2021 }
+
+
+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/lemmy.html b/dev/engines/online/lemmy.html new file mode 100644 index 000000000..1a6b6ba89 --- /dev/null +++ b/dev/engines/online/lemmy.html @@ -0,0 +1,190 @@ + + + + + + + + Lemmy — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Lemmy

+ +

This engine uses the Lemmy API (https://lemmy.ml/api/v3/search), which is +documented at lemmy-js-client / Interface Search. Since Lemmy is +federated, results are from many different, independent lemmy instances, and not +only the official one.

+
+

Configuration

+

The engine has the following additional settings:

+ +

This implementation is used by different lemmy engines in the settings.yml:

+
- name: lemmy communities
+  lemmy_type: Communities
+  ...
+- name: lemmy users
+  lemmy_type: Users
+  ...
+- name: lemmy posts
+  lemmy_type: Posts
+  ...
+- name: lemmy comments
+  lemmy_type: Comments
+  ...
+
+
+
+
+

Implementations

+
+
+
+searx.engines.lemmy.base_url = 'https://lemmy.ml/'
+

By default, https://lemmy.ml is used for providing the results. If you want +to use a different lemmy instance, you can specify base_url.

+
+ +
+
+searx.engines.lemmy.lemmy_type = 'Communities'
+

Any of Communities, Users, Posts, Comments

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/peertube.html b/dev/engines/online/peertube.html new file mode 100644 index 000000000..bff730d59 --- /dev/null +++ b/dev/engines/online/peertube.html @@ -0,0 +1,190 @@ + + + + + + + + Peertube Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Peertube Engines

+ +
+

Peertube Video

+

Peertube and SepiaSearch do share +(more or less) the same REST API and the schema of the JSON result is identical.

+
+
+searx.engines.peertube.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages from peertube’s search-index source code.

+

See videoLanguages in commit 8ed5c729 - Refactor and redesign client

+
+ +
+
+searx.engines.peertube.request(query, params)[source]
+

Assemble request for the Peertube API

+
+ +
+
+searx.engines.peertube.video_response(resp)[source]
+

Parse video response from SepiaSearch and Peertube instances.

+
+ +
+
+searx.engines.peertube.base_url = 'https://peer.tube'
+

Base URL of the Peertube instance. A list of instances is available at:

+ +
+ +
+
+

SepiaSearch

+

SepiaSearch uses the same languages as Peertube and the response is identical to the response from the +peertube engines.

+
+
+searx.engines.sepiasearch.request(query, params)[source]
+

Assemble request for the SepiaSearch API

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/piped.html b/dev/engines/online/piped.html new file mode 100644 index 000000000..469d1c7e8 --- /dev/null +++ b/dev/engines/online/piped.html @@ -0,0 +1,208 @@ + + + + + + + + Piped — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Piped

+ +

An alternative privacy-friendly YouTube frontend which is efficient by +design. Piped’s architecture consists of 3 components:

+ +
+

Configuration

+

The backend_url and frontend_url has to be set in the engine +named piped and are used by all piped engines

+
- name: piped
+  engine: piped
+  piped_filter: videos
+  ...
+  frontend_url: https://..
+  backend_url:
+    - https://..
+    - https://..
+
+- name: piped.music
+  engine: piped
+  network: piped
+  shortcut: ppdm
+  piped_filter: music_songs
+  ...
+
+
+
+
+

Known Quirks

+

The implementation to support paging +is based on the nextpage method of Piped’s REST API / the frontend +API. This feature is next page driven and plays well with the +infinite_scroll setting in SearXNG but it does not really +fit into SearXNG’s UI to select a page by number.

+
+
+

Implementations

+
+
+
+searx.engines.piped.backend_url: list | str = 'https://pipedapi.kavin.rocks'
+

Piped-Backend: The core component behind Piped. The value is an URL or a +list of URLs. In the latter case instance will be selected randomly. For a +complete list of offical instances see Piped-Instances (JSON)

+
+ +
+
+searx.engines.piped.frontend_url: str = 'https://piped.video'
+

Piped-Frontend: URL to use as link and for embeds.

+
+ +
+
+searx.engines.piped.piped_filter = 'all'
+

Content filter music_songs or videos

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/recoll.html b/dev/engines/online/recoll.html new file mode 100644 index 000000000..716710b8e --- /dev/null +++ b/dev/engines/online/recoll.html @@ -0,0 +1,190 @@ + + + + + + + + Recoll Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Recoll Engine

+ + +

Recoll is a desktop full-text search tool based on Xapian. By itself Recoll +does not offer WEB or API access, this can be achieved using recoll-webui

+
+

Configuration

+

You must configure the following settings:

+
+
base_url:

Location where recoll-webui can be reached.

+
+
mount_prefix:

Location where the file hierarchy is mounted on your local filesystem.

+
+
dl_prefix:

Location where the file hierarchy as indexed by recoll can be reached.

+
+
search_dir:

Part of the indexed file hierarchy to be search, if empty the full domain is +searched.

+
+
+
+
+

Example

+

Scenario:

+
    +

  1. Recoll indexes a local filesystem mounted in /export/documents/reference,

    2. +

  2. the Recoll search interface can be reached at https://recoll.example.org/ and

    3. +

  3. the contents of this filesystem can be reached though https://download.example.org/reference

    4. +
+
base_url: https://recoll.example.org/
+mount_prefix: /export/documents
+dl_prefix: https://download.example.org
+search_dir: ''
+
+
+
+
+

Implementations

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/startpage.html b/dev/engines/online/startpage.html new file mode 100644 index 000000000..51f47be29 --- /dev/null +++ b/dev/engines/online/startpage.html @@ -0,0 +1,281 @@ + + + + + + + + Startpage Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Startpage Engines

+ +

Startpage’s language & region selectors are a mess ..

+
+

Startpage regions

+

In the list of regions there are tags we need to map to common region tags:

+
pt-BR_BR --> pt_BR
+zh-CN_CN --> zh_Hans_CN
+zh-TW_TW --> zh_Hant_TW
+zh-TW_HK --> zh_Hant_HK
+en-GB_GB --> en_GB
+
+
+

and there is at least one tag with a three letter language tag (ISO 639-2):

+
fil_PH --> fil_PH
+
+
+

The locale code no_NO from Startpage does not exists and is mapped to +nb-NO:

+
babel.core.UnknownLocaleError: unknown locale 'no_NO'
+
+
+

For reference see languages-subtag at iana; no is the macrolanguage [1] and +W3C recommends subtag over macrolanguage [2].

+ +
+
+

Startpage languages

+
+
send_accept_language_header:

The displayed name in Startpage’s settings page depend on the location of the +IP when Accept-Language HTTP header is unset. In fetch_traits +we use:

+
'Accept-Language': "en-US,en;q=0.5",
+..
+
+
+

to get uniform names independent from the IP).

+
+
+
+
+

Startpage categories

+

Startpage’s category (for Web-search, News, Videos, ..) is set by +startpage_categ in settings.yml:

+
- name: startpage
+  engine: startpage
+  startpage_categ: web
+  ...
+
+
+
+

Hint

+

The default category is web .. and other categories than web are not +yet implemented.

+
+
+
+
+searx.engines.startpage.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages and regions from Startpage.

+
+ +
+
+searx.engines.startpage.get_sc_code(searxng_locale, params)[source]
+

Get an actual sc argument from Startpage’s search form (HTML page).

+

Startpage puts a sc argument on every HTML search form. Without this argument Startpage considers the request +is from a bot. We do not know what is encoded in the value of the sc +argument, but it seems to be a kind of a time-stamp.

+

Startpage’s search form generates a new sc-code on each request. This +function scrap a new sc-code from Startpage’s home page every +sc_code_cache_sec seconds.

+
+ +
+
+searx.engines.startpage.request(query, params)[source]
+

Assemble a Startpage request.

+

To avoid CAPTCHA we need to send a well formed HTTP POST request with a +cookie. We need to form a request that is identical to the request build by +Startpage’s search form:

+
    +

  • in the cookie the region is selected

    • +

  • in the HTTP POST data the language is selected

    • +
+

Additionally the arguments form Startpage’s search form needs to be set in +HTML POST data / compare <input> elements: search_form_xpath.

+
+ +
+
+searx.engines.startpage.sc_code_cache_sec = 30
+

Time in seconds the sc-code is cached in memory get_sc_code.

+
+ +
+
+searx.engines.startpage.search_form_xpath = '//form[@id="search"]'
+

XPath of Startpage’s origin search form

+
+ +
+
+searx.engines.startpage.send_accept_language_header = True
+

Startpage tries to guess user’s language and territory from the HTTP +Accept-Language. Optional the user can select a search-language (can be +different to the UI language) and a region filter.

+
+ +
+
+searx.engines.startpage.startpage_categ = 'web'
+

Startpage’s category, visit Startpage categories.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/tagesschau.html b/dev/engines/online/tagesschau.html new file mode 100644 index 000000000..6ef472a51 --- /dev/null +++ b/dev/engines/online/tagesschau.html @@ -0,0 +1,146 @@ + + + + + + + + Tagesschau API — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Tagesschau API

+

ARD: Tagesschau API

+

The Tagesschau is a news program of the ARD. Via the Tagesschau API, current +news and media reports are available in JSON format. The Bundesstelle für Open +Data offers a OpenAPI portal at bundDEV where APIs are documented an can +be tested.

+

This SearXNG engine uses the /api2u/search API.

+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/torznab.html b/dev/engines/online/torznab.html new file mode 100644 index 000000000..37c2f0b63 --- /dev/null +++ b/dev/engines/online/torznab.html @@ -0,0 +1,212 @@ + + + + + + + + Torznab WebAPI — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Torznab WebAPI

+ +

Torznab is an API specification that provides a standardized way to query +torrent site for content. It is used by a number of torrent applications, +including Prowlarr and Jackett.

+

Using this engine together with Prowlarr or Jackett allows you to search +a huge number of torrent sites which are not directly supported.

+
+

Configuration

+

The engine has the following settings:

+
+
base_url:

Torznab endpoint URL.

+
+
api_key:

The API key to use for authentication.

+
+
torznab_categories:

The categories to use for searching. This is a list of category IDs. See +Prowlarr-categories or Jackett-categories for more information.

+
+
show_torrent_files:

Whether to show the torrent file in the search results. Be carful as using +this with Prowlarr or Jackett leaks the API key. This should be used only +if you are querying a Torznab endpoint without authentication or if the +instance is private. Be aware that private trackers may ban you if you share +the torrent file. Defaults to false.

+
+
show_magnet_links:

Whether to show the magnet link in the search results. Be aware that private +trackers may ban you if you share the magnet link. Defaults to true.

+
+
+
+
+

Implementations

+
+
+
+searx.engines.torznab.build_result(item: Element) → Dict[str, Any][source]
+

Build a result from a XML item.

+
+ +
+
+searx.engines.torznab.get_attribute(item: etree.Element, property_name: str) → str | None[source]
+

Get attribute from item.

+
+ +
+
+searx.engines.torznab.get_torznab_attribute(item: etree.Element, attribute_name: str) → str | None[source]
+

Get torznab special attribute from item.

+
+ +
+
+searx.engines.torznab.init(engine_settings=None)[source]
+

Initialize the engine.

+
+ +
+
+searx.engines.torznab.request(query: str, params: Dict[str, Any]) → Dict[str, Any][source]
+

Build the request params.

+
+ +
+
+searx.engines.torznab.response(resp: httpx.Response) → List[Dict[str, Any]][source]
+

Parse the XML response and return a list of results.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/wikipedia.html b/dev/engines/online/wikipedia.html new file mode 100644 index 000000000..577cf61c5 --- /dev/null +++ b/dev/engines/online/wikipedia.html @@ -0,0 +1,299 @@ + + + + + + + + Wikimedia — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Wikimedia

+ +
+

Wikipedia

+

This module implements the Wikipedia engine. Some of this implementations +are shared by other engines:

+ +

The list of supported languages is fetched from +the article linked by list_of_wikipedias.

+

Unlike traditional search engines, wikipedia does not support one Wikipedia for +all languages, but there is one Wikipedia for each supported language. Some of +these Wikipedias have a LanguageConverter enabled +(rest_v1_summary_url).

+

A LanguageConverter (LC) is a system based on language variants that +automatically converts the content of a page into a different variant. A variant +is mostly the same language in a different script.

+ +
+
PR-2554:

The Wikipedia link returned by the API is still the same in all cases +(https://zh.wikipedia.org/wiki/出租車) but if your browser’s +Accept-Language is set to any of zh, zh-CN, zh-TW, zh-HK +or .. Wikipedia’s LC automatically returns the desired script in their +web-page.

+ +
+
+

To support Wikipedia’s LanguageConverter, a SearXNG request to Wikipedia uses +get_wiki_params and wiki_lc_locale_variants' in the +:py:obj:`fetch_wikimedia_traits function.

+

To test in SearXNG, query for !wp 出租車 with each of the available Chinese +options:

+
    +

  • !wp 出租車 :zh should show 出租車

    • +

  • !wp 出租車 :zh-CN should show 出租车

    • +

  • !wp 出租車 :zh-TW should show 計程車

    • +

  • !wp 出租車 :zh-HK should show 的士

    • +

  • !wp 出租車 :zh-SG should show 德士

    • +
+
+
+searx.engines.wikipedia.fetch_wikimedia_traits(engine_traits: EngineTraits)[source]
+

Fetch languages from Wikipedia. Not all languages from the +list_of_wikipedias are supported by SearXNG locales, only those +known from searx.locales.LOCALE_NAMES or those with a minimal +editing depth.

+

The location of the Wikipedia address of a language is mapped in a +custom field +(wiki_netloc). Here is a reduced example:

+
traits.custom['wiki_netloc'] = {
+    "en": "en.wikipedia.org",
+    ..
+    "gsw": "als.wikipedia.org",
+    ..
+    "zh": "zh.wikipedia.org",
+    "zh-classical": "zh-classical.wikipedia.org"
+}
+
+
+
+ +
+
+searx.engines.wikipedia.get_wiki_params(sxng_locale, eng_traits)[source]
+

Returns the Wikipedia language tag and the netloc that fits to the +sxng_locale. To support LanguageConverter this function rates a locale +(region) higher than a language (compare wiki_lc_locale_variants).

+
+ +
+
+searx.engines.wikipedia.request(query, params)[source]
+

Assemble a request (wikipedia rest_v1 summary API).

+
+ +
+
+searx.engines.wikipedia.list_of_wikipedias = 'https://meta.wikimedia.org/wiki/List_of_Wikipedias'
+

List of all wikipedias

+
+ +
+
+searx.engines.wikipedia.rest_v1_summary_url = 'https://{wiki_netloc}/api/rest_v1/page/summary/{title}'
+
+
wikipedia rest_v1 summary API:

The summary response includes an extract of the first paragraph of the page in +plain text and HTML as well as the type of page. This is useful for page +previews (fka. Hovercards, aka. Popups) on the web and link previews in the +apps.

+
+
HTTP Accept-Language header (send_accept_language_header):

The desired language variant code for wikis where LanguageConverter is +enabled.

+
+
+
+ +
+
+searx.engines.wikipedia.send_accept_language_header = True
+

The HTTP Accept-Language header is needed for wikis where +LanguageConverter is enabled.

+
+ +
+
+searx.engines.wikipedia.wiki_lc_locale_variants = {'zh': ('zh-CN', 'zh-HK', 'zh-MO', 'zh-MY', 'zh-SG', 'zh-TW'), 'zh-classical': ('zh-classical',)}
+

Mapping rule of the LanguageConverter to map a language and its variants to +a Locale (used in the HTTP Accept-Language header). For example see LC +Chinese.

+
+ +
+
+searx.engines.wikipedia.wikipedia_article_depth = 'https://meta.wikimedia.org/wiki/Wikipedia_article_depth'
+

The editing depth of Wikipedia is one of several possible rough indicators +of the encyclopedia’s collaborative quality, showing how frequently its articles +are updated. The measurement of depth was introduced after some limitations of +the classic measurement of article count were realized.

+
+ +
+
+

Wikidata

+

This module implements the Wikidata engine. Some implementations are shared +from Wikipedia.

+
+
+searx.engines.wikidata.fetch_traits(engine_traits: EngineTraits)[source]
+

Uses languages evaluated from wikipedia.fetch_wikimedia_traits and removes

+
    +

  • traits.custom['wiki_netloc']: wikidata does not have net-locations for +the languages and the list of all

    • +

  • traits.custom['WIKIPEDIA_LANGUAGES']: not used in the wikipedia engine

    • +
+
+ +
+
+searx.engines.wikidata.get_thumbnail(img_src)[source]
+

Get Thumbnail image from wikimedia commons

+

Images from commons.wikimedia.org are (HTTP) redirected to +upload.wikimedia.org. The redirected URL can be calculated by this +function.

+ +
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/yacy.html b/dev/engines/online/yacy.html new file mode 100644 index 000000000..0009614e8 --- /dev/null +++ b/dev/engines/online/yacy.html @@ -0,0 +1,186 @@ + + + + + + + + Yacy — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Yacy

+ +

YaCy is a free distributed search engine, built on the principles of +peer-to-peer (P2P) networks.

+

API: Dev:APIyacysearch

+

Releases:

+ +
+

Configuration

+

The engine has the following (additional) settings:

+
- name: yacy
+  engine: yacy
+  shortcut: ya
+  base_url: http://localhost:8090
+  # Yacy search mode. 'global' or 'local'.
+  search_mode: 'global'
+  number_of_results: 5
+  http_digest_auth_user: ""
+  http_digest_auth_pass: ""
+
+
+
+
+

Implementations

+
+
+
+searx.engines.yacy.search_mode = 'global'
+

Yacy search mode global or local. By default, Yacy operates in global +mode.

+
+
global

Peer-to-Peer search

+
+
local

Privacy or Stealth mode, restricts the search to local yacy instance.

+
+
+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/yahoo.html b/dev/engines/online/yahoo.html new file mode 100644 index 000000000..889afdc0c --- /dev/null +++ b/dev/engines/online/yahoo.html @@ -0,0 +1,173 @@ + + + + + + + + Yahoo Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Yahoo Engine

+

Yahoo Search (Web)

+

Languages are supported by mapping the language to a domain. If domain is not +found in lang2domain URL <lang>.search.yahoo.com is used.

+
+
+searx.engines.yahoo.fetch_traits(engine_traits: EngineTraits)[source]
+

Fetch languages from yahoo

+
+ +
+
+searx.engines.yahoo.parse_url(url_string)[source]
+

remove yahoo-specific tracking-url

+
+ +
+
+searx.engines.yahoo.request(query, params)[source]
+

build request

+
+ +
+
+searx.engines.yahoo.response(resp)[source]
+

parse response

+
+ +
+
+searx.engines.yahoo.lang2domain = {'any': 'search.yahoo.com', 'bg': 'search.yahoo.com', 'cs': 'search.yahoo.com', 'da': 'search.yahoo.com', 'el': 'search.yahoo.com', 'en': 'search.yahoo.com', 'et': 'search.yahoo.com', 'he': 'search.yahoo.com', 'hr': 'search.yahoo.com', 'ja': 'search.yahoo.com', 'ko': 'search.yahoo.com', 'sk': 'search.yahoo.com', 'sl': 'search.yahoo.com', 'zh_chs': 'hk.search.yahoo.com', 'zh_cht': 'tw.search.yahoo.com'}
+

Map language to domain

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online/zlibrary.html b/dev/engines/online/zlibrary.html new file mode 100644 index 000000000..73499614a --- /dev/null +++ b/dev/engines/online/zlibrary.html @@ -0,0 +1,205 @@ + + + + + + + + Z-Library — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Z-Library

+ +

Z-Library (abbreviated as z-lib, formerly BookFinder) is a shadow library +project for file-sharing access to scholarly journal articles, academic texts +and general-interest books. It began as a mirror of Library Genesis, from which +most of its books originate.

+
+

Configuration

+

The engine has the following additional settings:

+ +

With this options a SearXNG maintainer is able to configure additional +engines for specific searches in Z-Library. For example a engine to search +only for EPUB from 2010 to 2020.

+
- name: z-library 2010s epub
+  engine: zlibrary
+  shortcut: zlib2010s
+  zlib_year_from: '2010'
+  zlib_year_to: '2020'
+  zlib_ext: 'EPUB'
+
+
+
+
+

Implementations

+
+
+
+searx.engines.zlibrary.fetch_traits(engine_traits: EngineTraits) → None[source]
+

Fetch languages and other search arguments from zlibrary’s search form.

+
+ +
+
+searx.engines.zlibrary.init(engine_settings=None) → None[source]
+

Check of engine’s settings.

+
+ +
+
+searx.engines.zlibrary.zlib_ext: str = ''
+

Filter z-library’s results by a file ending. Common filters for example are +PDF and EPUB.

+
+ +
+
+searx.engines.zlibrary.zlib_year_from: str = ''
+

Filter z-library’s results by year from. E.g ‘2010’.

+
+ +
+
+searx.engines.zlibrary.zlib_year_to: str = ''
+

Filter z-library’s results by year to. E.g. ‘2010’.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/online_url_search/tineye.html b/dev/engines/online_url_search/tineye.html new file mode 100644 index 000000000..8940849ba --- /dev/null +++ b/dev/engines/online_url_search/tineye.html @@ -0,0 +1,215 @@ + + + + + + + + Tineye — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Tineye

+

This engine implements Tineye - reverse image search

+

Using TinEye, you can search by image or perform what we call a reverse image +search. You can do that by uploading an image or searching by URL. You can also +simply drag and drop your images to start your search. TinEye constantly crawls +the web and adds images to its index. Today, the TinEye index is over 50.2 +billion images [tineye.com].

+
+

Hint

+

This SearXNG engine only supports ‘searching by URL’ and it does not use +the official API [api.tineye.com].

+
+
+
+searx.engines.tineye.parse_tineye_match(match_json)[source]
+

Takes parsed JSON from the API server and turns it into a dict +object.

+

Attributes (class Match)

+
    +

  • image_url, link to the result image.

    • +

  • domain, domain this result was found on.

    • +

  • score, a number (0 to 100) that indicates how closely the images match.

    • +

  • width, image width in pixels.

    • +

  • height, image height in pixels.

    • +

  • size, image area in pixels.

    • +

  • format, image format.

    • +

  • filesize, image size in bytes.

    • +

  • overlay, overlay URL.

    • +

  • tags, whether this match belongs to a collection or stock domain.

    • +

  • backlinks, a list of Backlink objects pointing to the original websites +and image URLs. List items are instances of dict, (Backlink):

    +
      +

    • url, the image URL to the image.

      • +

    • backlink, the original website URL.

      • +

    • crawl_date, the date the image was crawled.

      • +
    +
    • +
+
+ +
+
+searx.engines.tineye.request(query, params)[source]
+

Build TinEye HTTP request using search_urls of a engine_type.

+
+ +
+
+searx.engines.tineye.response(resp)[source]
+

Parse HTTP response from TinEye.

+
+ +
+
+searx.engines.tineye.DOWNLOAD_ERROR = 'The image could not be downloaded.'
+

TinEye error message

+
+ +
+
+searx.engines.tineye.FORMAT_NOT_SUPPORTED = 'Could not read that image url. This may be due to an unsupported file format. TinEye only supports images that are JPEG, PNG, GIF, BMP, TIFF or WebP.'
+

TinEye error message

+
+ +
+
+searx.engines.tineye.NO_SIGNATURE_ERROR = 'The image is too simple to find matches. TinEye requires a basic level of visual detail to successfully identify matches.'
+

TinEye error message

+
+ +
+
+searx.engines.tineye.engine_type = 'online_url_search'
+

searx.search.processors.online_url_search

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/engines/xpath.html b/dev/engines/xpath.html new file mode 100644 index 000000000..8286b20e7 --- /dev/null +++ b/dev/engines/xpath.html @@ -0,0 +1,389 @@ + + + + + + + + XPath Engine — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

XPath Engine

+ +

The XPath engine is a generic engine with which it is possible to configure +engines in the settings.

+
+

Configuration

+

Request:

+ +

Paging:

+ +

Time Range:

+ +

Safe-Search:

+ +

Response:

+ +

XPath selector:

+ +
+
+

Example

+

Here is a simple example of a XPath engine configured in the engine: section, further read Engine Overview.

+
- name : bitbucket
+  engine : xpath
+  paging : True
+  search_url : https://bitbucket.org/repo/all/{pageno}?name={query}
+  url_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]/@href
+  title_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]
+  content_xpath : //article[@class="repo-summary"]/p
+
+
+
+
+

Implementations

+
+
+
+searx.engines.xpath.request(query, params)[source]
+

Build request parameters (see Making a Request).

+
+ +
+
+searx.engines.xpath.response(resp)[source]
+

Scrap results from the response (see Result Types (template)).

+
+ +
+
+searx.engines.xpath.content_xpath = None
+

XPath selector of result’s content.

+
+ +
+
+searx.engines.xpath.cookies = {}
+

Some engines might offer different result based on cookies. +Possible use-case: To set safesearch cookie.

+
+ +
+
+searx.engines.xpath.first_page_num = 1
+

Number of the first page (usually 0 or 1).

+
+ +
+
+searx.engines.xpath.headers = {}
+

Some engines might offer different result based headers. Possible use-case: +To set header to moderate.

+
+ +
+
+searx.engines.xpath.lang_all = 'en'
+

Replacement {lang} in search_url if language all is +selected.

+
+ +
+
+searx.engines.xpath.no_result_for_http_status = []
+

Return empty result for these HTTP status codes instead of throwing an error.

+
no_result_for_http_status: []
+
+
+
+ +
+
+searx.engines.xpath.page_size = 1
+

Number of results on each page. Only needed if the site requires not a page +number, but an offset.

+
+ +
+
+searx.engines.xpath.paging = False
+

Engine supports paging [True or False].

+
+ +
+
+searx.engines.xpath.results_xpath = ''
+

XPath selector for the list of result items

+
+ +
+
+searx.engines.xpath.safe_search_map = {0: '&filter=none', 1: '&filter=moderate', 2: '&filter=strict'}
+

Maps safe-search value to {safe_search} in search_url.

+
safesearch: true
+safes_search_map:
+  0: '&filter=none'
+  1: '&filter=moderate'
+  2: '&filter=strict'
+
+
+
+ +
+
+searx.engines.xpath.safe_search_support = False
+

Engine supports safe-search.

+
+ +
+
+searx.engines.xpath.search_url = None
+

Search URL of the engine. Example:

+
https://example.org/?search={query}&page={pageno}{time_range}{safe_search}
+
+
+

Replacements are:

+
+
{query}:

Search terms from user.

+
+
{pageno}:

Page number if engine supports pagging paging

+
+
{lang}:

ISO 639-1 language code (en, de, fr ..)

+
+
{time_range}:

URL parameter if engine supports time +range. The value for the parameter is taken from +time_range_map.

+
+
{safe_search}:

Safe-search URL parameter if engine +supports safe-search. The {safe_search} +replacement is taken from the safes_search_map. Filter results:

+
0: none, 1: moderate, 2:strict
+
+
+

If not supported, the URL parameter is an empty string.

+
+
+
+ +
+
+searx.engines.xpath.soft_max_redirects = 0
+

Maximum redirects, soft limit. Record an error but don’t stop the engine

+
+ +
+
+searx.engines.xpath.suggestion_xpath = ''
+

XPath selector of result’s suggestion.

+
+ +
+
+searx.engines.xpath.thumbnail_xpath = False
+

XPath selector of result’s img_src.

+
+ +
+
+searx.engines.xpath.time_range_map = {'day': 24, 'month': 720, 'week': 168, 'year': 8760}
+

Maps time range value from user to {time_range_val} in +time_range_url.

+
time_range_map:
+  day: 1
+  week: 7
+  month: 30
+  year: 365
+
+
+
+ +
+
+searx.engines.xpath.time_range_support = False
+

Engine supports search time range.

+
+ +
+
+searx.engines.xpath.time_range_url = '&hours={time_range_val}'
+

Time range URL parameter in the in search_url. If no time range is +requested by the user, the URL parameter is an empty string. The +{time_range_val} replacement is taken from the time_range_map.

+
time_range_url : '&days={time_range_val}'
+
+
+
+ +
+
+searx.engines.xpath.title_xpath = None
+

XPath selector of result’s title.

+
+ +
+
+searx.engines.xpath.url_xpath = None
+

XPath selector of result’s url.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/index.html b/dev/index.html new file mode 100644 index 000000000..adc49b2e5 --- /dev/null +++ b/dev/index.html @@ -0,0 +1,211 @@ + + + + + + + + Developer documentation — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Developer documentation

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/lxcdev.html b/dev/lxcdev.html new file mode 100644 index 000000000..77929b8b3 --- /dev/null +++ b/dev/lxcdev.html @@ -0,0 +1,455 @@ + + + + + + + + Developing in Linux Containers — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Developing in Linux Containers

+

In this article we will show, how you can make use of Linux Containers (LXC) in +distributed and heterogeneous development cycles (TL;DR; jump to the +Summary).

+ + +
+

Motivation

+

Most often in our development cycle, we edit the sources and run some test +and/or builds by using make [ref] before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook.

+
+

The environment in which we run all our development processes matters!

+
+

The Makefile & ./manage and the Python environment (make install) encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer’s desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer’s desktop that would never +be set on a production system.

+
+

Linux Containers are isolate environments, we use them to not mix up all +the prerequisites from various projects on developer’s desktop.

+
+

The scripts from DevOps tooling box can divide in those to install and maintain +software

+ +

and the script

+ +

with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the:

+ +
+
+

Gentlemen, start your engines!

+

Before you can start with containers, you need to install and initiate LXD +once:

+
+
$ snap install lxd
+$ lxd init --auto
+
+
+
+

And you need to clone from origin or if you have your own fork, clone from your +fork:

+
+
$ cd ~/Downloads
+$ git clone https://github.com/searxng/searxng.git searxng
+$ cd searxng
+
+
+
+ +

The SearXNG suite config consists of several images, see export +LXC_SUITE=(... near by git://utils/lxc-searxng.env#L19. +For this blog post we exercise on a archlinux image. The container of this +image is named searxng-archlinux.

+

Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one:

+
+
$ sudo -H ./utils/lxc.sh remove searxng-archlinux
+$ sudo -H ./utils/lxc.sh build searxng-archlinux
+
+
+
+ +

To install the complete SearXNG suite and the HTTP +proxy NGINX into the archlinux container run:

+
+
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
+$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+$ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL
+...
+[searxng-archlinux]    SEARXNG_URL          : http://n.n.n.140/searxng
+
+
+
+ +

In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy.

+

In the example above the SearXNG instance in the container is wrapped to +http://n.n.n.140/searxng to the HOST system. Note, on your HOST system, the +IP of your searxng-archlinux container is different to this example. To +test the instance in the conatiner from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation

+
+
+

In containers, work as usual

+

Usually you open a root-bash using sudo -H bash. In case of LXC containers +open the root-bash in the container is done by the ./utils/lxc.sh cmd +searxng-archlinux command:

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash
+INFO:  [searxng-archlinux] bash
+[root@searxng-archlinux SearXNG]$
+
+
+
+

The prompt [root@searxng-archlinux ...] signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use:

+
+
$ ./utils/searxng.sh instance inspect
+...
+use [CTRL-C] to stop monitoring the log
+...
+
+
+
+

Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using CTRL-C, this also disables the “debug +option” in SearXNG’s settings file and restarts the SearXNG uwsgi application.

+

Another point we have to notice is that the service SearXNG +runs under dedicated system user account with the same name (compare +Create user). To get a login shell from these accounts, simply +call:

+
+
$ ./utils/searxng.sh instance cmd bash -l
+(searx-pyenv) [searxng@searxng-archlinux ~]$ pwd
+/usr/local/searxng
+
+
+
+

The prompt [searxng@searxng-archlinux] signals that you are logged in as system +user searxng in the searxng-archlinux container and the python virtualenv +(searxng-pyenv) environment is activated.

+
+
+

Wrap production into developer suite

+

In this section we will see how to change the “Fully functional SearXNG suite” +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the Step by step installation:

+
    +

  • SearXNG setup in: /etc/searxng/settings.yml

    • +

  • SearXNG user’s home: /usr/local/searxng

    • +

  • virtualenv in: /usr/local/searxng/searxng-pyenv

    • +

  • SearXNG software in: /usr/local/searxng/searxng-src

    • +
+

With the use of the utils/searxng.sh the SearXNG service was installed as +uWSGI application. To maintain this service, we can use +systemctl (compare uWSGI maintenance).

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng
+
+
+
+

With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container.

+

The uWSGI-App for the archlinux dsitros is configured in +git://utils/templates/etc/uwsgi/apps-archlinux/searxng.ini, from where at +least you should attend the settings of uid, chdir, env and +http:

+
env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
+http = 127.0.0.1:8888
+
+chdir = /usr/local/searxng/searxng-src/searx
+virtualenv = /usr/local/searxng/searxng-pyenv
+pythonpath = /usr/local/searxng/searxng-src
+
+
+

If you have read the Good to know you remember, that each container +shares the root folder of the repository and the command utils/lxc.sh cmd +handles relative path names transparent.

+

To wrap the SearXNG installation in the container into a developer one, we +simple have to create a smylink to the transparent reposetory from the +desktop. Now lets replace the repository at searxng-src in the container +with the working tree from outside of the container:

+
+
$ mv /usr/local/searxng/searxng-src  /usr/local/searxng/searxng-src.old
+$ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
+
+
+
+

Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container):

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
+
+
+
+

Remember: In containers, work as usual .. here are just some examples from my +daily usage:

+

To inspect the SearXNG instance (already described above):

+
+
$ ./utils/searx.sh inspect service
+
+
+
+

Run Makefile & ./manage, e.g. to test inside the container:

+
+
$ make test
+
+
+
+

To install all prerequisites needed for a Buildhosts:

+
+
$ ./utils/searxng.sh install buildhost
+
+
+
+

To build the docs on a buildhost Buildhosts:

+
+
$ make docs.html
+
+
+
+
+
+

Summary

+

We build up a fully functional SearXNG suite in a archlinux container:

+
$ sudo -H ./utils/lxc.sh build searxng-archlinux
+$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
+...
+Developer install? (wraps source from HOST into the running instance) [YES/no]
+
+
+

To wrap the suite into a developer one answer YES (or press Enter).

+
link SearXNG's sources to: /share/SearXNG
+=========================================
+
+mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup"
+ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src"
+ls -ld /usr/local/searxng/searxng-src
+  |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG
+
+
+

On code modification the instance has to be restarted (see uWSGI maintenance):

+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
+
+
+

To access HTTP from the desktop we installed nginx for the services inside the +container:

+
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+
+
+

To get information about the SearxNG suite in the archlinux container we can +use:

+
$ sudo -H ./utils/lxc.sh show suite searxng-archlinux
+[searxng-archlinux]  INFO:  (eth0) docs-live:  http:///n.n.n.140:8080/
+[searxng-archlinux]  INFO:  (eth0) IPv6:       http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744]
+[searxng-archlinux]  uWSGI:
+[searxng-archlinux]    SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket
+[searxng-archlinux]  environment /usr/local/searxng/searxng-src/utils/brand.env:
+[searxng-archlinux]    GIT_URL              : https://github.com/searxng/searxng
+[searxng-archlinux]    GIT_BRANCH           : master
+[searxng-archlinux]    SEARXNG_URL          : http:///n.n.n.140/searxng
+[searxng-archlinux]    SEARXNG_PORT         : 8888
+[searxng-archlinux]    SEARXNG_BIND_ADDRESS : 127.0.0.1
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/makefile.html b/dev/makefile.html new file mode 100644 index 000000000..574eb68b0 --- /dev/null +++ b/dev/makefile.html @@ -0,0 +1,625 @@ + + + + + + + + Makefile & ./manage — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + +
+
+
+
+ +
+

Makefile & ./manage

+

All relevant build and development tasks are implemented in the +./manage script and for CI or IDE integration a small +git://Makefile wrapper is available. If you are not familiar with +Makefiles, we recommend to read gnu-make introduction.

+ + +

The usage is simple, just type make {target-name} to build a target. +Calling the help target gives a first overview (make help):

+
+
INFO:  sourced NVM environment from /home/runner/.nvm
+nvm.: use nvm (without dot) to execute nvm commands directly
+  install   : install NVM locally at /home/runner/work/searxng/searxng/.nvm
+  clean     : remove NVM installation
+  status    : prompt some status informations about nvm & node
+  nodejs    : install Node.js latest LTS
+  cmd ...   : run command ... in NVM environment
+  bash      : start bash interpreter with NVM environment sourced
+buildenv:
+  rebuild ./utils/brand.env
+webapp.:
+  run       : run developer instance
+docs.:
+  html      : build HTML documentation
+  live      : autobuild HTML documentation while editing
+  gh-pages  : deploy on gh-pages branch
+  prebuild  : build reST include files (./build/docs/includes)
+  clean     : clean documentation build
+docker.:
+  build     : build docker image
+  push      : build and push docker image
+gecko.driver:
+  download & install geckodriver if not already installed (required for
+  robot_tests)
+redis:
+  build     : build redis binaries at /home/runner/work/searxng/searxng/dist/redis/6.2.6/amd64
+  install   : create user (searxng-redis) and install systemd service (searxng-redis)
+  help      : show more redis commands
+py.:
+  build     : Build python packages at ./dist
+  clean     : delete virtualenv and intermediate py files
+pyenv.:
+  install   : developer install of SearXNG into virtualenv
+  uninstall : uninstall developer installation
+  cmd ...   : run command ... in virtualenv
+  OK        : test if virtualenv is OK
+pypi.upload:
+  Upload python packages to PyPi (to test use pypi.upload.test)
+format.:
+  python    : format Python code source using black
+pygments.:
+  less      : build LESS files for pygments
+go.:
+  ls        : list golang binary archives (stable)
+  golang    : (re-) install golang binary in user's $HOME/local folder
+  install   : install go package in user's $HOME/go-apps folder
+  bash      : start bash interpreter with golang environment sourced
+node.:
+  env       : download & install SearXNG's npm dependencies locally
+  env.dev   : download & install developer and CI tools
+  clean     : drop locally npm installations
+weblate.:
+  push.translations: push translation changes from SearXNG to Weblate's counterpart
+  to.translations: Update 'translations' branch with last additions from Weblate.
+data.:
+  all       : update searx/sxng_locales.py and searx/data/*
+  traits    : update searx/data/engine_traits.json & searx/sxng_locales.py
+  useragents: update searx/data/useragents.json with the most recent versions of Firefox
+test.:
+  yamllint  : lint YAML files (YAMLLINT_FILES)
+  pylint    : lint PYLINT_FILES, searx/engines, searx & tests
+  pyright   : static type check of python sources
+  black     : check black code format
+  unit      : run unit tests
+  coverage  : run unit tests with coverage
+  robot     : run robot test
+  rst       : test .rst files incl. README.rst
+  clean     : clean intermediate test stuff
+themes.:
+  all       : build all themes
+  live      : to get live builds of CSS & JS use 'LIVE_THEME=simple make run'
+  simple.:
+    build   : build simple theme
+    test    : test simple theme
+static.build.:  [build] /static
+  commit    : build & commit /static folder
+  drop      : drop last commit if it was previously done by static.build.commit
+  restore   : git restore of the /static folder (after themes.all)
+environment ...
+  SEARXNG_REDIS_URL : 
+----
+run            - run developer instance
+install        - developer install of SearxNG into virtualenv
+uninstall      - uninstall developer installation
+clean          - clean up working tree
+search.checker - check search engines
+test           - run shell & CI tests
+test.shell     - test shell scripts
+ci.test        - run CI tests
+
+
+
+
+

Python environment (make install)

+ +

We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a make install to get a virtualenv with a +developer install of SearXNG (git://setup.py).

+
$ cd ~/searxng-clone
+$ make install
+PYENV     [virtualenv] installing ./requirements*.txt into local/py3
+...
+PYENV     OK
+PYENV     [install] pip install -e 'searx[test]'
+...
+Successfully installed argparse-1.4.0 searx
+BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
+BUILDENV  INFO:searx:Initialisation done
+BUILDENV  build utils/brand.env
+
+
+

If you release make install multiple times the installation will only +rebuild if the sha256 sum of the requirement files fails. With other words: +the check fails if you edit the requirements listed in +git://requirements-dev.txt and git://requirements.txt).

+
$ make install
+PYENV     OK
+PYENV     [virtualenv] requirements.sha256 failed
+          [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e...  ./requirements-dev.txt
+          [virtualenv] - 471efef6c73558e391c3adb35f4...  ./requirements.txt
+...
+PYENV     [virtualenv] installing ./requirements*.txt into local/py3
+...
+PYENV     OK
+PYENV     [install] pip install -e 'searx[test]'
+...
+Successfully installed argparse-1.4.0 searx
+BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
+BUILDENV  INFO:searx:Initialisation done
+BUILDENV  build utils/brand.env
+
+
+ +

If you think, something goes wrong with your ./local environment or you change +the git://setup.py file, you have to call make clean.

+
+
+

make buildenv

+

Rebuild instance’s environment with the modified settings from the +brand: and server: section of your +settings.yml.

+
+

What is the git://utils/brand.env needed for and why do you need to rebuild +it if necessary?

+

Short answer: installation and maintenance +scripts are running outside of instance’s runtime environment and need some +values defined in the runtime environment.

+
+

All the SearXNG setups are centralized in the settings.yml file. This +setup is available as long we are in a installed instance. E.g. the +installed instance on the server or the installed developer instance at +./local (the later one is created by a make install or +make run).

+

Tasks running outside of an installed instance, especially installation +and maintenance tasks running at (pre-) installation time +do not have access to the SearXNG setup (from a installed instance). Those +tasks need a build environment.

+

The make buildenv target will update the build environment in:

+ +

Tasks running outside of an installed instance, need the following settings +from the YAML configuration:

+ +

The GIT_URL and GIT_BRANCH in the origin:utils/brand.env file, are +readed from the git VCS and the branch that is checked out when make +buildenv command runs.

+

I would like to create my own brand, how should I proceed?

+

Create a remote branch (example.org), checkout the remote branch (on your +local developer desktop) and in the git://searx/settings.yml file in the +server: section set base_url. Run make buildenv and +create a commit for your brand.

+

On your server you clone the branch (example.org) into your HOME folder +~ from where you run the installation and +maintenance task.

+

To upgrade you brand, rebase on SearXNG’s master branch (on your local +developer desktop), force push it to your remote branch. Go to your server, do +a force pull and run sudo -H ./utils/searxng.sh instance update.

+
+
+

Node.js environment (make node.env)

+

Node.js version 16.13.0 or higher is required to build the themes. +If the requirement is not met, the build chain uses nvm (Node Version +Manager) to install latest LTS of Node.js locally: there is no need to +install nvm or npm on your system.

+

To install NVM and Node.js in once you can use make nvm.nodejs.

+
+

NVM make nvm.install nvm.status

+

Use make nvm.status to get the current status of your Node.js and nvm +setup.

+
+
$ LANG=C make nvm.install
+INFO:  install (update) NVM at ./searxng/.nvm
+INFO:  clone: https://github.com/nvm-sh/nvm.git
+  || Cloning into './searxng/.nvm'...
+INFO:  checkout v0.39.4
+  || HEAD is now at 8fbf8ab v0.39.4
+
+
+
+
+
+

make nvm.nodejs

+

Install latest Node.js LTS locally (uses nvm):

+
$ make nvm.nodejs
+INFO:  install (update) NVM at /share/searxng/.nvm
+INFO:  clone: https://github.com/nvm-sh/nvm.git
+...
+Downloading and installing node v16.13.0...
+...
+INFO:  Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node
+INFO:  Node.js is version v16.13.0
+INFO:  npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm
+INFO:  npm is version 8.1.0
+INFO:  NVM is installed at searxng/.nvm
+
+
+
+
+
+

make run

+

To get up a running a developer instance simply call make run. This enables +debug option in git://searx/settings.yml, starts a ./searx/webapp.py +instance and opens the URL in your favorite WEB browser (xdg-open):

+
$ make run
+
+
+

Changes to theme’s HTML templates (jinja2) are instant. Changes to the CSS & JS +sources of the theme need to be rebuild. You can do that by running:

+
$ make themes.all
+
+
+

Alternatively to themes.all you can run live builds of the theme you are +modify (make themes.*):

+
$ LIVE_THEME=simple make run
+
+
+
+
+

make format.python

+

Format Python sourcee code using Black code style. See $BLACK_OPTIONS +and $BLACK_TARGETS in git://Makefile.

+
+

Attention

+

We stuck at Black 22.12.0, please read comment in PR Bump black from 22.12.0 +to 23.1.0

+
+
+
+

make clean

+

Drops all intermediate files, all builds, but keep sources untouched. Before +calling make clean stop all processes using the Python environment (make install) or +Node.js environment (make node.env).

+
$ make clean
+CLEAN     pyenv
+PYENV     [virtualenv] drop local/py3
+CLEAN     docs -- build/docs dist/docs
+CLEAN     themes -- locally installed npm dependencies
+...
+CLEAN     test stuff
+CLEAN     common files
+
+
+
+
+

make docs

+

Target docs builds the documentation:

+
$ make docs
+HTML ./docs --> file://
+DOCS      build build/docs/includes
+...
+The HTML pages are in dist/docs.
+
+
+
+

make docs.clean docs.live

+

We describe the usage of the doc.* targets in the How to contribute / +Documentation section. If you want to edit the documentation +read our live build section. If you are working in your own brand, +adjust your brand:.

+
+
+

make docs.gh-pages

+

To deploy on github.io first adjust your brand:. For any +further read deploy on github.io.

+
+
+
+

make test

+

Runs a series of tests: make test.pylint, test.pep8, test.unit +and test.robot. You can run tests selective, e.g.:

+
$ make test.pep8 test.unit test.shell
+TEST      test.pep8 OK
+...
+TEST      test.unit OK
+...
+TEST      test.shell OK
+
+
+
+

make test.shell

+

Lint shell scripts / if you have changed some bash scripting run this test before +commit.

+
+
+

make test.pylint

+

Pylint is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile used in the SearXNG project is +found in project’s root folder git://.pylintrc.

+
+
+
+

make search.checker.{engine name}

+

To check all engines:

+
make search.checker
+
+
+

To check a engine with whitespace in the name like google news replace space +by underline:

+
make search.checker.google_news
+
+
+

To see HTTP requests and more use SEARXNG_DEBUG:

+
make SEARXNG_DEBUG=1 search.checker.google_news
+
+
+

To filter out HTTP redirects (3xx):

+
make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
+...
+Engine google news                   Checking
+https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
+https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
+--
+https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
+https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
+--
+
+
+
+
+

make themes.*

+ +

The git://Makefile targets make theme.* cover common tasks to build the +theme(s). The ./manage themes.* command line can be used to convenient run +common theme build tasks.

+
INFO:  sourced NVM environment from /home/runner/.nvm
+themes.:
+  all       : build all themes
+  live      : to get live builds of CSS & JS use 'LIVE_THEME=simple make run'
+  simple.:
+    build   : build simple theme
+    test    : test simple theme
+
+
+

To get live builds while modifying CSS & JS use (make run):

+
$ LIVE_THEME=simple make run
+
+
+
+
+

make static.build.*

+ +

The git://Makefile targets static.build.* cover common tasks to build (a +commit of) the static files. The ./manage static.build..* command line +can be used to convenient run common build tasks of the satic files.

+
INFO:  sourced NVM environment from /home/runner/.nvm
+static.build.:  [build] /static
+  commit    : build & commit /static folder
+  drop      : drop last commit if it was previously done by static.build.commit
+  restore   : git restore of the /static folder (after themes.all)
+
+
+
+
+

./manage redis.help

+

The ./manage redis.* command line can be used to convenient run common Redis +tasks (Redis Developer Notes).

+
INFO:  sourced NVM environment from /home/runner/.nvm
+redis.:
+  devpkg    : install essential packages to compile redis
+  build     : build redis binaries at /home/runner/work/searxng/searxng/dist/redis/6.2.6/amd64
+  install   : create user (searxng-redis) and install systemd service (searxng-redis)
+  remove    : delete user (searxng-redis) and remove service (searxng-redis)
+  shell     : start bash interpreter from user searxng-redis
+  src       : clone redis source code to <path> and checkput 6.2.6
+  useradd   : create user (searxng-redis) at /usr/local/searxng-redis
+  userdel   : delete user (searxng-redis)
+  addgrp    : add <user> to group (searxng-redis)
+  rmgrp     : remove <user> from group (searxng-redis)
+
+
+
+
+

./manage go.help

+

The ./manage go.* command line can be used to convenient run common go +(wiki) tasks.

+
INFO:  sourced NVM environment from /home/runner/.nvm
+go.:
+  ls        : list golang binary archives (stable)
+  golang    : (re-) install golang binary in user's $HOME/local folder
+  install   : install go package in user's $HOME/go-apps folder
+  bash      : start bash interpreter with golang environment sourced
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/plugins.html b/dev/plugins.html new file mode 100644 index 000000000..75c32e640 --- /dev/null +++ b/dev/plugins.html @@ -0,0 +1,254 @@ + + + + + + + + Plugins — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Plugins

+ +

Plugins can extend or replace functionality of various components of searx.

+
+

Example plugin

+
name = 'Example plugin'
+description = 'This plugin extends the suggestions with the word "example"'
+default_on = False  # disabled by default
+
+# attach callback to the post search hook
+#  request: flask request object
+#  ctx: the whole local context of the post search hook
+def post_search(request, search):
+    search.result_container.suggestions.add('example')
+    return True
+
+
+
+
+

External plugins

+

SearXNG supports external plugins / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from The Green Web Foundation [ref]:

+
$ sudo utils/searxng.sh instance cmd bash -c
+(searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins
+
+
+

In the settings.yml activate the plugins: section and add module +only_show_green_results from tgwf-searx-plugins.

+
plugins:
+  ...
+  - only_show_green_results
+  ...
+
+
+
+
+

Plugin entry points

+

Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. A plugin doesn’t need to implement all the hooks.

+
+ +

Runs BEFORE the search request.

+

search.result_container can be changed.

+

Return a boolean:

+
    +

  • True to continue the search

    • +

  • False to stop the search

    • +
+
+
Parameters:
+
+
+
Returns:
+

False to stop the search

+
+
Return type:
+

bool

+
+
+
+ +
+
+post_search(request, search) → None
+

Runs AFTER the search request.

+
+
Parameters:
+
+
+
+
+ +
+
+on_result(request, search, result) → bool
+

Runs for each result of each engine.

+

result can be changed.

+

If result[“url”] is defined, then result[“parsed_url”] = urlparse(result[‘url’])

+
+

Warning

+

result[“url”] can be changed, but result[“parsed_url”] must be updated too.

+
+

Return a boolean:

+
    +

  • True to keep the result

    • +

  • False to remove the result

    • +
+
+
Parameters:
+
+
+
Returns:
+

True to keep the result

+
+
Return type:
+

bool

+
+
+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/quickstart.html b/dev/quickstart.html new file mode 100644 index 000000000..d19148f39 --- /dev/null +++ b/dev/quickstart.html @@ -0,0 +1,200 @@ + + + + + + + + Development Quickstart — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Development Quickstart

+ +

SearXNG loves developers; Developers do not need to worry about tool chains, the +usual developer tasks can be comfortably executed via make.

+

Don’t hesitate, just clone SearXNG’s sources and start hacking right now ..

+
git clone https://github.com/searxng/searxng.git searxng
+
+
+

Here is how a minimal workflow looks like:

+
    +

  1. start hacking

    2. +

  2. run your code: make run

    3. +

  3. format & test your code: make format.python and make test

    4. +
+

If you think at some point something fails, go back to start. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in wild west we have some directives, please pay attention +to our “How to contribute” guideline.

+ +

If you implement themes, you will need to setup a Node.js environment: make node.env

+

Before you call make run (2.), you need to compile the modified styles and +JavaScript: make themes.all

+

Alternatively you can also compile selective the theme you have modified, +e.g. the simple theme.

+
make themes.simple
+
+
+
+

Tip

+

To get live builds while modifying CSS & JS use: LIVE_THEME=simple make run

+
+ +

If you finished your tests you can start to commit your changes. To separate +the modified source code from the build products first run:

+
make static.build.restore
+
+
+

This will restore the old build products and only your changes of the code +remain in the working tree which can now be added & commited. When all sources +are commited, you can commit the build products simply by:

+
make static.build.commit
+
+
+

Commiting the build products should be the last step, just before you send us +your PR. There is also a make target to rewind this last build commit:

+
make static.build.drop
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/reST.html b/dev/reST.html new file mode 100644 index 000000000..90b1a553b --- /dev/null +++ b/dev/reST.html @@ -0,0 +1,1694 @@ + + + + + + + + reST primer — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

reST primer

+ +

We at SearXNG are using reStructuredText (aka reST) markup for all kind of +documentation. With the builders from the Sphinx project a HTML output is +generated and deployed at docs.searxng.org. For build prerequisites read +Build docs.

+

The source files of SearXNG’s documentation are located at git://docs. +Sphinx assumes source files to be encoded in UTF-8 by default. Run make +docs.live to build HTML while editing.

+ + +

Sphinx and reST have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation [kernel doc].

+ +

reST is a plaintext markup language, its markup is mostly intuitive and +you will not need to learn much to produce well formed articles with. I use the +word mostly: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups).

+
+

Soft skills

+

Before going any deeper into the markup let’s face on some soft skills a +trained author brings with, to reach a well feedback from readers:

+
    +

  • Documentation is dedicated to an audience and answers questions from the +audience point of view.

    • +

  • Don’t detail things which are general knowledge from the audience point of +view.

    • +

  • Limit the subject, use cross links for any further reading.

    • +
+

To be more concrete what a point of view means. In the (git://docs) +folder we have three sections (and the blog folder), each dedicate to a +different group of audience.

+
+
User’s POV: git://docs/user

A typical user knows about search engines and might have heard about +meta crawlers and privacy.

+
+
Admin’s POV: git://docs/admin

A typical Admin knows about setting up services on a linux system, but he does +not know all the pros and cons of a SearXNG setup.

+
+
Developer’s POV: git://docs/dev

Depending on the readability of code, a typical developer is able to read and +understand source code. Describe what a item aims to do (e.g. a function). +If the chronological order matters, describe it. Name the out-of-limits +conditions and all the side effects a external developer will not know.

+
+
+
+
+

Basic inline markup

+ +

Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (\*pointer).

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + +
Table 15 basic inline markup

description

rendered

markup

one asterisk for emphasis

italics

*italics*

two asterisks for strong emphasis

boldface

**boldface**

backquotes for code samples and literals

foo()

``foo()``

quote asterisks or backquotes

*foo is a pointer

\*foo is a pointer

+
+
+

Basic article structure

+

The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections.

+
+

reST template

+

reST template for an simple article:

+
.. _doc refname:
+
+==============
+Document title
+==============
+
+Lorem ipsum dolor sit amet, consectetur adipisici elit ..  Further read
+:ref:`chapter refname`.
+
+.. _chapter refname:
+
+Chapter
+=======
+
+Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
+aliquid ex ea commodi consequat ...
+
+.. _section refname:
+
+Section
+-------
+
+lorem ..
+
+.. _subsection refname:
+
+Subsection
+~~~~~~~~~~
+
+lorem ..
+
+
+
+
+

Headings

+
    +

  1. title - with overline for document title:

    2. +
+
+
==============
+Document title
+==============
+
+
+
+
    +

  1. chapter - with anchor named anchor name:

    +
    .. _anchor name:
+
+Chapter
+=======
+
    +
    +
    2. +

  2. section

    +
    Section
+-------
+
    +
    +
    3. +

  3. subsection

    +
    Subsection
+~~~~~~~~~~
+
    +
    +
    4. +
+
+
+ +
+

Literal blocks

+

The simplest form of literal-blocks is a indented block introduced by +two colons (::). For highlighting use highlight or code-block directive. To include literals from external files use +literalinclude or kernel-include +directive (latter one expands environment variables in the path name).

+
+

::

+
::
+
+  Literal block
+
+Lorem ipsum dolor::
+
+  Literal block
+
+Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
+eirmod tempor invidunt ut labore ::
+
+  Literal block
+
+
+
+

Literal block

+
Literal block
+
+
+

Lorem ipsum dolor:

+
Literal block
+
+
+

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy +eirmod tempor invidunt ut labore

+
Literal block
+
+
+
+
+
+

code-block

+ +

The code-block directive is a variant of the code directive +with additional options. To learn more about code literals visit +Showing code examples.

+
The URL ``/stats`` handle is shown in :ref:`stats-handle`
+
+.. code-block:: Python
+   :caption: python code block
+   :name: stats-handle
+
+   @app.route('/stats', methods=['GET'])
+   def stats():
+       """Render engine statistics page."""
+       stats = get_engines_stats()
+       return render(
+           'stats.html'
+           , stats = stats )
+
+
+

+
+
+
+

Code block

+

The URL /stats handle is shown in python code block

+
+
Listing 1 python code block
+
@app.route('/stats', methods=['GET'])
+def stats():
+    """Render engine statistics page."""
+    stats = get_engines_stats()
+    return render(
+        'stats.html'
+        , stats = stats )
+
+
+
+
+
+
+
+

Unicode substitution

+

The unicode directive converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition.

+
.. |copy| unicode:: 0xA9 .. copyright sign
+.. |(TM)| unicode:: U+2122
+
+Trademark |(TM)| and copyright |copy| glyphs.
+
+
+
+

Unicode

+

Trademark ™ and copyright © glyphs.

+
+
+
+

Roles

+ +

A custom interpreted text role (ref) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way.

+

The general markup is one of:

+
:rolename:`ref-name`
+:rolename:`ref text <ref-name>`
+
+
+ + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 17 smart refs with sphinx.ext.extlinks and intersphinx

role

rendered example

markup

guilabel

Cancel

:guilabel:`&Cancel`

kbd

C-x C-f

:kbd:`C-x C-f`

menuselection

Open ‣ File

:menuselection:`Open --> File`

download

this file

:download:`this file <reST.rst>`

math

a^2 + b^2 = c^2

:math:`a^2 + b^2 = c^2`

ref

Simple SVG image.

:ref:`svg image example`

command

ls -la

:command:`ls -la`

emphasis

italic

:emphasis:`italic`

strong

bold

:strong:`bold`

literal

foo()

:literal:`foo()`

subscript

H2O

H\ :sub:`2`\ O

superscript

E = mc2

E = mc\ :sup:`2`

title-reference

Time

:title:`Time`

+
+
+

Figures & Images

+ +

SearXNG’s sphinx setup includes: Scalable figure and image handling. Scalable here means; +scalable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With Scalable figure and image handling the build process +continues and scales output quality in dependence of installed image processors.

+

If you want to add an image, you should use the kernel-figure (inheritance +of figure) and kernel-image (inheritance of image) +directives. E.g. to insert a figure with a scalable image format use SVG +(Simple SVG image.):

+
.. _svg image example:
+
+.. kernel-figure:: svg_image.svg
+   :alt: SVG image example
+
+   Simple SVG image
+
+ To refer the figure, a caption block is needed: :ref:`svg image example`.
+
+
+
+SVG image example
+

Fig. 4 Simple SVG image.

+
+
+

To refer the figure, a caption block is needed: Simple SVG image..

+
+

DOT files (aka Graphviz)

+

With kernel-figure & kernel-image reST support for DOT formatted files is +given.

+ +

A simple example is shown in DOT’s hello world example:

+
.. _dot file example:
+
+.. kernel-figure:: hello.dot
+   :alt: hello world
+
+   DOT's hello world example
+
+
+
+

hello.dot

+
+hello world
+

Fig. 5 DOT’s hello world example

+
+
+
+
+
+

kernel-render DOT

+

Embed render markups (or languages) like Graphviz’s DOT is provided by the +kernel-render directive. A simple example of embedded DOT is +shown in figure Embedded DOT (Graphviz) code:

+
.. _dot render example:
+
+.. kernel-render:: DOT
+   :alt: digraph
+   :caption: Embedded  DOT (Graphviz) code
+
+   digraph foo {
+     "bar" -> "baz";
+   }
+
+Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot
+render example`.
+
+
+

Please note build tools. If Graphviz is +installed, you will see an vector image. If not, the raw markup is inserted as +literal-block.

+
+

kernel-render DOT

+
+digraph
+

Fig. 6 Embedded DOT (Graphviz) code

+
+
+

Attribute caption is needed, if you want to refer the figure: Embedded DOT (Graphviz) code.

+
+
+
+

kernel-render SVG

+

A simple example of embedded SVG is shown in figure Embedded SVG markup:

+
.. _svg render example:
+
+.. kernel-render:: SVG
+   :caption: Embedded **SVG** markup
+   :alt: so-nw-arrow
+
+
+
+
<?xml version="1.0" encoding="UTF-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" version="1.1"
+     baseProfile="full" width="70px" height="40px"
+     viewBox="0 0 700 400"
+     >
+  <line x1="180" y1="370"
+        x2="500" y2="50"
+        stroke="black" stroke-width="15px"
+        />
+  <polygon points="585 0 525 25 585 50"
+           transform="rotate(135 525 25)"
+           />
+</svg>
+
+
+
+
+

kernel-render SVG

+
+so-nw-arrow
+

Fig. 7 Embedded SVG markup

+
+
+
+
+
+
+

List markups

+
+

Bullet list

+

List markup (ref) is simple:

+
- This is a bulleted list.
+
+  1. Nested lists are possible, but be aware that they must be separated from
+     the parent list items by blank line
+  2. Second item of nested list
+
+- It has two items, the second
+  item uses two lines.
+
+#. This is a numbered list.
+#. It has two items too.
+
+
+
+

bullet list

+
    +

  • This is a bulleted list.

    +
      +

    1. Nested lists are possible, but be aware that they must be separated from +the parent list items by blank line

      2. +

    2. Second item of nested list

      3. +
    +
    • +

  • It has two items, the second +item uses two lines.

    • +
+
    +

  1. This is a numbered list.

    2. +

  2. It has two items too.

    3. +
+
+
+
+

Horizontal list

+

The .. hlist:: transforms a bullet list into a more compact +list.

+
.. hlist::
+
+   - first list item
+   - second list item
+   - third list item
+   ...
+
+
+
+

hlist

+
    +

  • first list item

    • +

  • second list item

    • +

  • third list item

    • +

  • next list item

    • +
+
    +

  • next list item xxxx

    • +

  • next list item yyyy

    • +

  • next list item zzzz

    • +
+
+
+
+
+

Definition list

+ +

Each definition list (ref) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ‘ : ‘ (space, colon, space). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(this distinguishes definition lists from block quotes). Blank lines are +required before the first and after the last definition list item, but are +optional in-between.

+

Definition lists are created as follows:

+
term 1 (up to a line of text)
+    Definition 1.
+
+See the typo : this line is not a term!
+
+  And this is not term's definition.  **There is a blank line** in between
+  the line above and this paragraph.  That's why this paragraph is taken as
+  **block quote** (:duref:`ref <block-quotes>`) and not as term's definition!
+
+term 2
+    Definition 2, paragraph 1.
+
+    Definition 2, paragraph 2.
+
+term 3 : classifier
+    Definition 3.
+
+term 4 : classifier one : classifier two
+    Definition 4.
+
+
+
+

definition list

+
+
term 1 (up to a line of text)

Definition 1.

+
+
+

See the typo : this line is not a term!

+
+

And this is not term’s definition. There is a blank line in between +the line above and this paragraph. That’s why this paragraph is taken as +block quote (ref) and not as term’s definition!

+
+
+
term 2

Definition 2, paragraph 1.

+

Definition 2, paragraph 2.

+
+
term 3classifier

Definition 3.

+
+
+

term 4 : classifier one : classifier two

+
+
+
+

Quoted paragraphs

+

Quoted paragraphs (ref) are created by just indenting +them more than the surrounding paragraphs. Line blocks (ref) are a way of preserving line breaks:

+
normal paragraph ...
+lorem ipsum.
+
+   Quoted paragraph ...
+   lorem ipsum.
+
+| These lines are
+| broken exactly like in
+| the source file.
+
+
+
+

Quoted paragraph and line block

+

normal paragraph … +lorem ipsum.

+
+

Quoted paragraph … +lorem ipsum.

+
+
+
These lines are
+
broken exactly like in
+
the source file.
+
+
+
+
+

Field Lists

+ +

Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this:

+
:fieldname: Field content
+:foo:       first paragraph in field foo
+
+            second paragraph in field foo
+
+:bar:       Field content
+
+
+
+

Field List

+
+
fieldname:
+

Field content

+
+
foo:
+

first paragraph in field foo

+

second paragraph in field foo

+
+
bar:
+

Field content

+
+
+
+

They are commonly used in Python documentation:

+
def my_function(my_arg, my_other_arg):
+    """A function just for me.
+
+    :param my_arg: The first of my arguments.
+    :param my_other_arg: The second of my arguments.
+
+    :returns: A message (just for me, of course).
+    """
+
+
+
+
+

Further list blocks

+
    +

  • field lists (ref, with caveats noted in +Field Lists)

    • +

  • option lists (ref)

    • +

  • quoted literal blocks (ref)

    • +

  • doctest blocks (ref)

    • +
+
+
+
+

Admonitions

+ +
+

Generic admonition

+

The generic admonition needs a title:

+
.. admonition:: generic admonition title
+
+   lorem ipsum ..
+
+
+
+

generic admonition title

+

lorem ipsum ..

+
+
+
+

Specific admonitions

+

Specific admonitions: hint, note, tip attention, +caution, danger, error, , important, and +warning .

+
.. hint::
+
+   lorem ipsum ..
+
+.. note::
+
+   lorem ipsum ..
+
+.. warning::
+
+   lorem ipsum ..
+
+
+
+

Hint

+

lorem ipsum ..

+
+
+

Note

+

lorem ipsum ..

+
+
+

Tip

+

lorem ipsum ..

+
+
+

Attention

+

lorem ipsum ..

+
+
+

Caution

+

lorem ipsum ..

+
+
+

Danger

+

lorem ipsum ..

+
+
+

Important

+

lorem ipsum ..

+
+
+

Error

+

lorem ipsum ..

+
+
+

Warning

+

lorem ipsum ..

+
+
+
+
+

Tables

+ +

ASCII-art tables like Simple tables and Grid tables might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables.

+ +

Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers:

+ +
+

Simple tables

+

Simple tables allow colspan but not rowspan. If +your table need some metadata (e.g. a title) you need to add the .. table:: +directive (ref) in front and place the table in its body:

+
.. table:: foo gate truth table
+   :widths: grid
+   :align: left
+
+   ====== ====== ======
+       Inputs    Output
+   ------------- ------
+   A      B      A or B
+   ====== ====== ======
+   False
+   --------------------
+   True
+   --------------------
+   True   False  True
+          (foo)
+   ------ ------ ------
+   False  True
+          (foo)
+   ====== =============
+
+
+
+

Simple ASCII table

+ + +++++ + + + + + + + + + + + + + + + + + + + + + + +
Table 18 foo gate truth table

Inputs

Output

A

B

A or B

False

True

True

False +(foo)

True

False

True +(foo)

+
+
+
+

Grid tables

+

Grid tables allow colspan colspan and rowspan:

+
.. table:: grid table example
+   :widths: 1 1 5
+
+   +------------+------------+-----------+
+   | Header 1   | Header 2   | Header 3  |
+   +============+============+===========+
+   | body row 1 | column 2   | column 3  |
+   +------------+------------+-----------+
+   | body row 2 | Cells may span columns.|
+   +------------+------------+-----------+
+   | body row 3 | Cells may  | - Cells   |
+   +------------+ span rows. | - contain |
+   | body row 4 |            | - blocks. |
+   +------------+------------+-----------+
+
+
+
+

ASCII grid table

+ + +++++ + + + + + + + + + + + + + + + + + + + + + +
Table 19 grid table example

Header 1

Header 2

Header 3

body row 1

column 2

column 3

body row 2

Cells may span columns.

body row 3

Cells may +span rows.

    +

  • Cells

    • +

  • contain

    • +

  • blocks.

    • +
+

body row 4

+
+
+
+

flat-table

+

The flat-table is a further developed variant of the list tables. It is a double-stage list similar to the +list-table with some additional features:

+
+
column-span: cspan

with the role cspan a cell can be extended through additional columns

+
+
row-span: rspan

with the role rspan a cell can be extended through additional rows

+
+
auto-span:

spans rightmost cell of a table row over the missing cells on the right side +of that table-row. With Option :fill-cells: this behavior can changed +from auto span to auto fill, which automatically inserts (empty) cells +instead of spanning the last cell.

+
+
options:
+
header-rows:
+

[int] count of header rows

+
+
stub-columns:
+

[int] count of stub columns

+
+
widths:
+

[[int] [int] … ] widths of columns

+
+
fill-cells:
+

instead of auto-span missing cells, insert missing cells

+
+
+
+
roles:
+
cspan:
+

[int] additional columns (morecols)

+
+
rspan:
+

[int] additional rows (morerows)

+
+
+
+
+

The example below shows how to use this markup. The first level of the staged +list is the table-row. In the table-row there is only one markup allowed, +the list of the cells in this table-row. Exception are comments ( .. ) +and targets (e.g. a ref to row 2 of table’s body).

+
.. flat-table:: ``flat-table`` example
+   :header-rows: 2
+   :stub-columns: 1
+   :widths: 1 1 1 1 2
+
+   * - :rspan:`1` head / stub
+     - :cspan:`3` head 1.1-4
+
+   * - head 2.1
+     - head 2.2
+     - head 2.3
+     - head 2.4
+
+   * .. row body 1 / this is a comment
+
+     - row 1
+     - :rspan:`2` cell 1-3.1
+     - cell 1.2
+     - cell 1.3
+     - cell 1.4
+
+   * .. Comments and targets are allowed on *table-row* stage.
+     .. _`row body 2`:
+
+     - row 2
+     - cell 2.2
+     - :rspan:`1` :cspan:`1`
+       cell 2.3 with a span over
+
+       * col 3-4 &
+       * row 2-3
+
+   * - row 3
+     - cell 3.2
+
+   * - row 4
+     - cell 4.1
+     - cell 4.2
+     - cell 4.3
+     - cell 4.4
+
+   * - row 5
+     - cell 5.1 with automatic span to right end
+
+   * - row 6
+     - cell 6.1
+     - ..
+
+
+
+

List table

+ + +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 20 flat-table example

head / stub

head 1.1-4

head 2.1

head 2.2

head 2.3

head 2.4

row 1

cell 1-3.1

cell 1.2

cell 1.3

cell 1.4

row 2

+

cell 2.2

+

+cell 2.3 with a span over

+
    +

  • col 3-4 &

    • +

  • row 2-3

    • +
+

row 3

cell 3.2

row 4

cell 4.1

cell 4.2

cell 4.3

cell 4.4

row 5

cell 5.1 with automatic span to right end

row 6

cell 6.1

+
+
+
+

CSV table

+

CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation.

+
.. csv-table:: CSV table example
+   :header: .. , Column 1, Column 2
+   :widths: 2 5 5
+   :stub-columns: 1
+   :file: csv_table.txt
+
+
+

Content of file csv_table.txt:

+
stub col row 1, column, "loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
+eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam
+voluptua."
+stub col row 1, "At vero eos et accusam et justo duo dolores et ea rebum. Stet clita
+kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.", column
+stub col row 1, column, column
+
+
+
+

CSV table

+ + +++++ + + + + + + + + + + + + + + + + + + + + +
Table 21 CSV table example

Column 1

Column 2

stub col row 1

column

loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy +eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam +voluptua.

stub col row 1

At vero eos et accusam et justo duo dolores et ea rebum. Stet clita +kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

column

stub col row 1

column

column

+
+
+
+
+

Templating

+ +

Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja extension evaluates jinja templates in the Python environment (make install) (with SearXNG modules installed). We use this e.g. to build chapter: +Configured Engines. Below the jinja directive from the +git://docs/admin/engines.rst is shown:

+
==================
+Configured Engines
+==================
+
+.. sidebar:: Further reading ..
+
+   - :ref:`settings categories_as_tabs`
+   - :ref:`engines-dev`
+   - :ref:`settings engine`
+   - :ref:`general engine configuration`
+
+.. jinja:: searx
+
+   SearXNG supports {{engines | length}} search engines of which
+   {{enabled_engine_count}} are enabled by default.
+
+   Engines can be assigned to multiple :ref:`categories <engine categories>`.
+   The UI displays the tabs that are configured in :ref:`categories_as_tabs
+   <settings categories_as_tabs>`.  In addition to these UI categories (also
+   called *tabs*), engines can be queried by their name or the categories they
+   belong to, by using a :ref:`\!bing syntax <search-syntax>`.
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. jinja:: searx
+
+   {% for category, engines in categories_as_tabs.items() %}
+
+   tab ``!{{category.replace(' ', '_')}}``
+   ---------------------------------------
+
+   {% for group, group_bang, engines in engines | group_engines_in_tab %}
+
+   {% if loop.length > 1 %}
+   {% if group_bang %}group ``{{group_bang}}``{% else %}{{group}}{% endif %}
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+   {% endif %}
+
+   .. flat-table::
+      :header-rows: 2
+      :stub-columns: 1
+      :widths: 10 1 10 1 1 1 1 1 1 1
+
+      * - :cspan:`5` Engines configured by default (in :ref:`settings.yml <engine settings>`)
+        - :cspan:`3` :ref:`Supported features <engine file>`
+
+      * - Name
+        - !bang
+        - Module
+        - Disabled
+        - Timeout
+        - Weight
+        - Paging
+        - Locale
+        - Safe search
+        - Time range
+
+      {% for mod in engines %}
+
+      * - `{{mod.name}} <{{mod.about and mod.about.website}}>`_
+          {%- if mod.about and  mod.about.language %}
+          ({{mod.about.language | upper}})
+          {%- endif %}
+        - ``!{{mod.shortcut}}``
+        - {%- if 'searx.engines.' + mod.__name__ in documented_modules %}
+          :py:mod:`~searx.engines.{{mod.__name__}}`
+          {%- else %}
+          :origin:`{{mod.__name__}} <searx/engines/{{mod.__name__}}.py>`
+          {%- endif %}
+        - {{(mod.disabled and "y") or ""}}
+        - {{mod.timeout}}
+        - {{mod.weight or 1 }}
+        {% if mod.engine_type == 'online' %}
+        - {{(mod.paging and "y") or ""}}
+        - {{(mod.language_support and "y") or ""}}
+        - {{(mod.safesearch and "y") or ""}}
+        - {{(mod.time_range_support and "y") or ""}}
+        {% else %}
+        - :cspan:`3` not applicable ({{mod.engine_type}})
+        {% endif %}
+
+     {% endfor %}
+     {% endfor %}
+     {% endfor %}
+
+
+

The context for the template is selected in the line .. jinja:: searx. In +sphinx’s build configuration (git://docs/conf.py) the searx context +contains the engines and plugins.

+
import searx.search
+import searx.engines
+import searx.plugins
+searx.search.initialize()
+jinja_contexts = {
+   'searx': {
+      'engines': searx.engines.engines,
+      'plugins': searx.plugins.plugins
+   },
+}
+
+
+
+
+

Tabbed views

+

With sphinx-tabs extension we have tabbed views. To provide installation +instructions with one tab per distribution we use the group-tabs directive, +others are basic-tabs and code-tabs. Below a group-tab example from +Build docs is shown:

+
.. tabs::
+
+   .. group-tab:: Ubuntu / debian
+
+      .. code-block:: sh
+
+         $ sudo apt install shellcheck
+
+   .. group-tab:: Arch Linux
+
+      .. code-block:: sh
+
+         $ sudo pacman -S shellcheck
+
+   .. group-tab::  Fedora / RHEL
+
+      .. code-block:: sh
+
+         $ sudo dnf install ShellCheck
+
+
+
+
+

Math equations

+ +

The input language for mathematics is LaTeX markup using the CTAN: amsmath +package.

+

To embed LaTeX markup in reST documents, use role :math: for +inline and directive .. math:: for block markup.

+
In :math:numref:`schroedinger general` the time-dependent Schrödinger equation
+is shown.
+
+.. math::
+   :label: schroedinger general
+
+    \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle =
+          \hat{H} |\,\psi (t) \rangle.
+
+
+
+

LaTeX math equation

+

In (1) the time-dependent Schrödinger equation +is shown.

+
+

(1)\mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle.

+
+

The next example shows the difference of \tfrac (textstyle) and \dfrac +(displaystyle) used in a inline markup or another fraction.

+
``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}`
+``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}`
+
+
+
+

Line spacing

+

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy +eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam +voluptua. … +\tfrac inline example \tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z} +At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd +gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

+

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy +eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam +voluptua. … +\tfrac inline example \dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z} +At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd +gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/search_api.html b/dev/search_api.html new file mode 100644 index 000000000..56c793b03 --- /dev/null +++ b/dev/search_api.html @@ -0,0 +1,228 @@ + + + + + + + + Search API — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Search API

+

The search supports both GET and POST.

+

Furthermore, two endpoints / and /search are available for querying.

+

GET /

+

GET /search

+
+

Parameters

+ +
+
qrequired

The search query. This string is passed to external search services. Thus, +SearXNG supports syntax of each search service. For example, site:github.com +SearXNG is a valid query for Google. However, if simply the query above is +passed to any search engine which does not filter its results based on this +syntax, you might not get the results you wanted.

+

See more at Search syntax

+
+
categoriesoptional

Comma separated list, specifies the active search categories (see +Configured Engines)

+
+
enginesoptional

Comma separated list, specifies the active search engines (see +Configured Engines).

+
+
languagedefault from search:

Code of the language.

+
+
pagenodefault 1

Search page number.

+
+
time_rangeoptional

[ day, month, year ]

+

Time range of search for engines which support it. See if an engine supports +time range search in the preferences page of an instance.

+
+
formatoptional

[ json, csv, rss ]

+

Output format of results. Format needs to be activated in search:.

+
+
results_on_new_tabdefault 0

[ 0, 1 ]

+

Open search results on new tab.

+
+
image_proxydefault from server:

[ True, False ]

+

Proxy image results through SearXNG.

+
+
autocompletedefault from search:

[ google, dbpedia, duckduckgo, startpage, wikipedia, +swisscows, qwant ]

+

Service which completes words as you type.

+
+
safesearchdefault from search:

[ 0, 1, 2 ]

+

Filter search results of engines which support safe search. See if an engine +supports safe search in the preferences page of an instance.

+
+
themedefault simple

[ simple ]

+

Theme of instance.

+

Please note, available themes depend on an instance. It is possible that an +instance administrator deleted, created or renamed themes on their instance. +See the available options in the preferences page of the instance.

+
+
enabled_pluginsoptional

List of enabled plugins.

+
+
default:
+

Hash_plugin, Search_on_category_select, +Self_Information, Tracker_URL_remover, +Ahmia_blacklist

+
+
values:
+

Hash_plugin, Search_on_category_select, +Self_Information, Tracker_URL_remover, +Ahmia_blacklist,

+

Hostname_replace, Open_Access_DOI_rewrite, +Vim-like_hotkeys, Tor_check_plugin

+
+
+
+
disabled_plugins: optional

List of disabled plugins.

+
+
default:
+

Hostname_replace, Open_Access_DOI_rewrite, +Vim-like_hotkeys, Tor_check_plugin

+
+
values:
+

see values from enabled_plugins

+
+
+
+
enabled_enginesoptionalall engines

List of enabled engines.

+
+
disabled_enginesoptionalall engines

List of disabled engines.

+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/searxng_extra/index.html b/dev/searxng_extra/index.html new file mode 100644 index 000000000..6ef0a5e91 --- /dev/null +++ b/dev/searxng_extra/index.html @@ -0,0 +1,162 @@ + + + + + + + + Tooling box searxng_extra — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Tooling box searxng_extra

+

In the folder git://searxng_extra/ we maintain some tools useful for CI and +developers.

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/searxng_extra/standalone_searx.py.html b/dev/searxng_extra/standalone_searx.py.html new file mode 100644 index 000000000..c666f4135 --- /dev/null +++ b/dev/searxng_extra/standalone_searx.py.html @@ -0,0 +1,226 @@ + + + + + + + + searxng_extra/standalone_searx.py — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

searxng_extra/standalone_searx.py

+

Script to run SearXNG from terminal.

+
+

DON’T USE THIS SCRIPT!!

+
+
+

Danger

+

Be warned, using the standalone_searx.py won’t give you privacy!

+

On the contrary, this script behaves like a SearXNG server: your IP is +exposed and tracked by all active engines (google, bing, qwant, … ), with +every query!

+
+
+

Note

+

This is an old and grumpy hack / SearXNG is a Flask application with +client/server structure, which can’t be turned into a command line tool the +way it was done here.

+
+

Getting categories without initiate the engine will only return [‘general’]

+
>>> import searx.engines
+... list(searx.engines.categories.keys())
+['general']
+>>> import searx.search
+... searx.search.initialize()
+... list(searx.engines.categories.keys())
+['general', 'it', 'science', 'images', 'news', 'videos', 'music', 'files', 'social media', 'map']
+
+
+

Example to use this script:

+
$ python3 searxng_extra/standalone_searx.py rain
+
+
+
+
+searxng_extra.standalone_searx.get_search_query(args: Namespace, engine_categories: List[str] | None = None) → SearchQuery[source]
+

Get search results for the query

+
+ +
+
+searxng_extra.standalone_searx.json_serial(obj: Any) → Any[source]
+

JSON serializer for objects not serializable by default json code.

+
+
Raises:
+

TypeError – raised when obj is not serializable

+
+
+
+ +
+
+searxng_extra.standalone_searx.no_parsed_url(results: List[Dict[str, Any]]) → List[Dict[str, Any]][source]
+

Remove parsed url from dict.

+
+ +
+
+searxng_extra.standalone_searx.parse_argument(args: List[str] | None = None, category_choices: List[str] | None = None) → Namespace[source]
+

Parse command line.

+
+
Raises:
+

SystemExit – Query argument required on args

+
+
+

Examples:

+
>>> import importlib
+... # load module
+... spec = importlib.util.spec_from_file_location(
+...     'utils.standalone_searx', 'utils/standalone_searx.py')
+... sas = importlib.util.module_from_spec(spec)
+... spec.loader.exec_module(sas)
+... sas.parse_argument()
+usage: ptipython [-h] [--category [{general}]] [--lang [LANG]] [--pageno [PAGENO]] [--safesearch [{0,1,2}]] [--timerange [{day,week,month,year}]]
+                 query
+SystemExit: 2
+>>> sas.parse_argument(['rain'])
+Namespace(category='general', lang='all', pageno=1, query='rain', safesearch='0', timerange=None)
+
+
+
+ +
+
+searxng_extra.standalone_searx.to_dict(search_query: SearchQuery) → Dict[str, Any][source]
+

Get result from parsed arguments.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/searxng_extra/update.html b/dev/searxng_extra/update.html new file mode 100644 index 000000000..7b93d2fcf --- /dev/null +++ b/dev/searxng_extra/update.html @@ -0,0 +1,311 @@ + + + + + + + + searxng_extra/update/ — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

searxng_extra/update/

+

[source]

+

Scripts to update static data in git://searx/data/

+
+

update_ahmia_blacklist.py

+

[source]

+

This script saves Ahmia’s blacklist for onion sites.

+

Output file: git://searx/data/ahmia_blacklist.txt (CI Update data +…).

+
+
+

update_currencies.py

+

[source]

+

Fetch currencies from git://searx/engines/wikidata.py engine.

+

Output file: git://searx/data/currencies.json (CI Update data …).

+
+
+

update_engine_descriptions.py

+

[source]

+

Fetch website description from websites and from +git://searx/engines/wikidata.py engine.

+

Output file: git://searx/data/engine_descriptions.json.

+
+
+searxng_extra.update.update_engine_descriptions.get_output()[source]
+

From descriptions[engine][language] = [description, source] +To

+
    +

  • output[language][engine] = description_and_source

    • +
  • +
    description_and_source can be:
      +

    • [description, source]

      • +

    • description (if source = “wikipedia”)

      • +

    • [f”engine:lang”, “ref”] (reference to another existing description)

      • +
    +
    +
    +
    • +
+
+ +
+
+

update_external_bangs.py

+

[source]

+

Update git://searx/data/external_bangs.json using the duckduckgo bangs +(CI Update data …).

+

https://duckduckgo.com/newbang loads:

+ +

This script loads the javascript, then the bangs.

+

The javascript URL may change in the future ( for example +https://duckduckgo.com/bv2.js ), but most probably it will requires to update +RE_BANG_VERSION

+
+
+searxng_extra.update.update_external_bangs.merge_when_no_leaf(node)[source]
+

Minimize the number of nodes

+

A -> B -> C

+
    +

  • B is child of A

    • +

  • C is child of B

    • +
+

If there are no C equals to <LEAF_KEY>, then each C are merged +into A. For example (5 nodes):

+
d -> d -> g -> <LEAF_KEY> (ddg)
+  -> i -> g -> <LEAF_KEY> (dig)
+
+
+

becomes (3 noodes):

+
d -> dg -> <LEAF_KEY>
+  -> ig -> <LEAF_KEY>
+
+
+
+ +
+
+

update_firefox_version.py

+

[source]

+

Fetch firefox useragent signatures

+

Output file: git://searx/data/useragents.json (CI Update data …).

+
+
+

update_engine_traits.py

+

[source]

+

Update searx.enginelib.traits.EngineTraitsMap and git://searx/languages.py

+
+
searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE:

Persistence of engines traits, fetched from the engines.

+
+
git://searx/languages.py

Is generated from intersecting each engine’s supported traits.

+
+
+

The script git://searxng_extra/update/update_engine_traits.py is called in +the CI Update data …

+
+
+class searxng_extra.update.update_engine_traits.UnicodeEscape[source]
+

Escape unicode string in pprint.pformat

+
+ +
+
+searxng_extra.update.update_engine_traits.fetch_traits_map()[source]
+

Fetchs supported languages for each engine and writes json file with those.

+
+ +
+
+searxng_extra.update.update_engine_traits.filter_locales(traits_map: EngineTraitsMap)[source]
+

Filter language & region tags by a threshold.

+
+ +
+
+searxng_extra.update.update_engine_traits.get_unicode_flag(locale: Locale)[source]
+

Determine a unicode flag (emoji) that fits to the locale

+
+ +
+
+

update_osm_keys_tags.py

+

[source]

+

Fetch OSM keys and tags.

+

To get the i18n names, the scripts uses Wikidata Query Service instead of for +example OSM tags API (sidenote: the actual change log from +map.atownsend.org.uk might be useful to normalize OSM tags).

+

Output file: git://searx/data/osm_keys_tags (CI Update data …).

+
+
SPARQL_TAGS_REQUEST :

Wikidata SPARQL query that returns type-categories and types. The +returned tag is Tag:{category}={type} (see get_tags()). +Example:

+ +
+
SPARQL_KEYS_REQUEST :

Wikidata SPARQL query that returns keys. Example with “payment”:

+ +

rdfs:label get all the labels without language selection +(as opposed to SERVICE wikibase:label).

+
+
+
+
+

update_pygments.py

+

[source]

+

Update pygments style

+

Call this script after each upgrade of pygments

+
+
+

update_wikidata_units.py

+

[source]

+

Fetch units from git://searx/engines/wikidata.py engine.

+

Output file: git://searx/data/wikidata_units.json (CI Update data +…).

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/dev/translation.html b/dev/translation.html new file mode 100644 index 000000000..da83e7e5d --- /dev/null +++ b/dev/translation.html @@ -0,0 +1,193 @@ + + + + + + + + Translation — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Translation

+ +

Translation takes place on translate.codeberg.org.

+

Translations which has been added by translators on the translate.codeberg.org UI are +committed to Weblate’s counterpart of the SearXNG origin repository which is +located at https://translate.codeberg.org/git/searxng/searxng.

+

There is no need to clone this repository, SearXNG’s PR workflow to be in sync with Weblate take +care of the synchronization with the origin. To avoid merging commits from +the counterpart directly on the master branch of SearXNG origin, a pull +request (PR) is created by this workflow.

+

Weblate monitors the translations branch, not the master branch. This +branch is an orphan branch, decoupled from the master branch (we already know +orphan branches from the gh-pages). The translations branch contains +only the

+
    +

  • translation/messages.pot and the

    • +

  • translation/*/messages.po files, nothing else.

    • +
+
+../_images/translation.svg
+

Fig. 3 SearXNG’s PR workflow to be in sync with Weblate

+
+
+
+
Sync from origin to weblate: using make weblate.push.translations

For each commit on the master branch of SearXNG origin the GitHub job +babel / Update translations branch checks for updated translations.

+
+
Sync from weblate to origin: using make weblate.translations.commit

Every Friday, the GitHub workflow babel / create PR for additons from +weblate creates a PR with the +updated translation files:

+
    +

  • translation/messages.pot,

    • +

  • translation/*/messages.po and

    • +

  • translation/*/messages.mo

    • +
+
+
+
+

wlc

+

All weblate integration is done by GitHub workflows, but if you want to use wlc, +copy this content into wlc configuration in your HOME ~/.config/weblate

+
[keys]
+https://translate.codeberg.org/api/ = APIKEY
+
+
+

Replace APIKEY by your API key.

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/genindex.html b/genindex.html new file mode 100644 index 000000000..5c7873701 --- /dev/null +++ b/genindex.html @@ -0,0 +1,1815 @@ + + + + + + + + Index — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ + +

Index

+ +
+ A + | B + | C + | D + | E + | F + | G + | H + | I + | J + | L + | M + | N + | O + | P + | Q + | R + | S + | T + | U + | V + | W + | Z + +
+

A

+ + + +
+ +

B

+ + + +
+ +

C

+ + + +
+ +

D

+ + + +
+ +

E

+ + + +
+ +

F

+ + + +
+ +

G

+ + + +
+ +

H

+ + + +
+ +

I

+ + + +
+ +

J

+ + +
+ +

L

+ + + +
+ +

M

+ + +
+ +

N

+ + + +
+ +

O

+ + + +
+ +

P

+ + + +
+ +

Q

+ + + +
+ +

R

+ + + +
+ +

S

+ + + +
+ +

T

+ + + +
+ +

U

+ + + +
+ +

V

+ + +
+ +

W

+ + + +
+ +

Z

+ + + +
+ + + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/index.html b/index.html new file mode 100644 index 000000000..cafbfd646 --- /dev/null +++ b/index.html @@ -0,0 +1,221 @@ + + + + + + + + Welcome to SearXNG — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +
+

Welcome to SearXNG

+
+

Search without being tracked.

+
+

SearXNG is a free internet metasearch engine which aggregates results from more +than 70 search services. Users are neither tracked nor profiled. Additionally, +SearXNG can be used over Tor for online anonymity.

+

Get started with SearXNG by using one of the instances listed at searx.space. +If you don’t trust anyone, you can set up your own, see Installation.

+ + + +
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/objects.inv b/objects.inv new file mode 100644 index 000000000..495968cb4 Binary files /dev/null and b/objects.inv differ diff --git a/own-instance.html b/own-instance.html new file mode 100644 index 000000000..2e229b640 --- /dev/null +++ b/own-instance.html @@ -0,0 +1,201 @@ + + + + + + + + Why use a private instance? — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Why use a private instance?

+ + +

Public instances are open to everyone who has access to its URL. Usually, these +are operated by unknown parties (from the users’ point of view). Private +instances can be used by a select group of people. It is for example a SearXNG of +group of friends or a company which can be accessed through VPN. Also it can be +single user one which runs on the user’s laptop.

+

To gain more insight on how these instances work let’s dive into how SearXNG +protects its users.

+
+

How does SearXNG protect privacy?

+

SearXNG protects the privacy of its users in multiple ways regardless of the type +of the instance (private, public). Removal of private data from search requests +comes in three forms:

+
+
    +

  1. removal of private data from requests going to search services

    2. +

  2. not forwarding anything from a third party services through search services +(e.g. advertisement)

    3. +

  3. removal of private data from requests going to the result pages

    4. +
+
+

Removing private data means not sending cookies to external search engines and +generating a random browser profile for every request. Thus, it does not matter +if a public or private instance handles the request, because it is anonymized in +both cases. IP addresses will be the IP of the instance. But SearXNG can be +configured to use proxy or Tor. Result proxy is supported, too.

+

SearXNG does not serve ads or tracking content unlike most search services. So +private data is not forwarded to third parties who might monetize it. Besides +protecting users from search services, both referring page and search query are +hidden from visited result pages.

+
+

What are the consequences of using public instances?

+

If someone uses a public instance, they have to trust the administrator of that +instance. This means that the user of the public instance does not know whether +their requests are logged, aggregated and sent or sold to a third party.

+

Also, public instances without proper protection are more vulnerable to abusing +the search service, In this case the external service in exchange returns +CAPTCHAs or bans the IP of the instance. Thus, search requests return less +results.

+
+
+

I see. What about private instances?

+

If users run their own instances, everything is in their +control: the source code, logging settings and private data. Unknown instance +administrators do not have to be trusted.

+

Furthermore, as the default settings of their instance is editable, there is no +need to use cookies to tailor SearXNG to their needs. So preferences will not be +reset to defaults when clearing browser cookies. As settings are stored on +their computer, it will not be accessible to others as long as their computer is +not compromised.

+
+
+
+

Conclusion

+

Always use an instance which is operated by people you trust. The privacy +features of SearXNG are available to users no matter what kind of instance they +use.

+

If someone is on the go or just wants to try SearXNG for the first time public +instances are the best choices. Additionally, public instance are making a +world a better place, because those who cannot or do not want to run an +instance, have access to a privacy respecting search service.

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/py-modindex.html b/py-modindex.html new file mode 100644 index 000000000..c6595a39b --- /dev/null +++ b/py-modindex.html @@ -0,0 +1,542 @@ + + + + + + + + Python Module Index — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + + +
+
+
+
+ + +

Python Module Index

+ +
+ s +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
 
+ s
+ searx +
    + searx.babel_extract +
    + searx.botdetection +
    + searx.botdetection.http_accept +
    + searx.botdetection.http_accept_encoding +
    + searx.botdetection.http_accept_language +
    + searx.botdetection.http_connection +
    + searx.botdetection.http_user_agent +
    + searx.botdetection.ip_limit +
    + searx.botdetection.ip_lists +
    + searx.botdetection.limiter +
    + searx.botdetection.link_token +
    + searx.enginelib +
    + searx.enginelib.traits +
    + searx.engines +
    + searx.engines.annas_archive +
    + searx.engines.archlinux +
    + searx.engines.bing +
    + searx.engines.bing_images +
    + searx.engines.bing_news +
    + searx.engines.bing_videos +
    + searx.engines.brave +
    + searx.engines.bt4g +
    + searx.engines.command +
    + searx.engines.dailymotion +
    + searx.engines.demo_offline +
    + searx.engines.demo_online +
    + searx.engines.duckduckgo +
    + searx.engines.duckduckgo_definitions +
    + searx.engines.duckduckgo_images +
    + searx.engines.duckduckgo_weather +
    + searx.engines.elasticsearch +
    + searx.engines.google +
    + searx.engines.google_images +
    + searx.engines.google_news +
    + searx.engines.google_scholar +
    + searx.engines.google_videos +
    + searx.engines.lemmy +
    + searx.engines.mediawiki +
    + searx.engines.meilisearch +
    + searx.engines.mongodb +
    + searx.engines.mysql_server +
    + searx.engines.peertube +
    + searx.engines.piped +
    + searx.engines.postgresql +
    + searx.engines.recoll +
    + searx.engines.redis_server +
    + searx.engines.sepiasearch +
    + searx.engines.solr +
    + searx.engines.sqlite +
    + searx.engines.startpage +
    + searx.engines.tagesschau +
    + searx.engines.tineye +
    + searx.engines.torznab +
    + searx.engines.wikidata +
    + searx.engines.wikipedia +
    + searx.engines.xpath +
    + searx.engines.yacy +
    + searx.engines.yahoo +
    + searx.engines.zlibrary +
    + searx.exceptions +
    + searx.infopage +
    + searx.locales +
    + searx.plugins.tor_check +
    + searx.redisdb +
    + searx.redislib +
    + searx.search.processors.abstract +
    + searx.search.processors.offline +
    + searx.search.processors.online +
    + searx.search.processors.online_currency +
    + searx.search.processors.online_dictionary +
    + searx.search.processors.online_url_search +
    + searx.sxng_locales +
    + searx.utils +
+ searxng_extra +
    + searxng_extra.standalone_searx +
    + searxng_extra.update.update_ahmia_blacklist +
    + searxng_extra.update.update_currencies +
    + searxng_extra.update.update_engine_descriptions +
    + searxng_extra.update.update_engine_traits +
    + searxng_extra.update.update_external_bangs +
    + searxng_extra.update.update_firefox_version +
    + searxng_extra.update.update_osm_keys_tags +
    + searxng_extra.update.update_pygments +
    + searxng_extra.update.update_wikidata_units +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/search.html b/search.html new file mode 100644 index 000000000..0a93b2661 --- /dev/null +++ b/search.html @@ -0,0 +1,125 @@ + + + + + + + + Search — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + + + + + + +
+
+
+
+ +

Search

+ + + + +

+ Searching for multiple words only shows matches that contain + all words. +

+ + +
+ + + +
+ + + +
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/searchindex.js b/searchindex.js new file mode 100644 index 000000000..869af20e6 --- /dev/null +++ b/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["admin/answer-captcha", "admin/api", "admin/architecture", "admin/buildhosts", "admin/index", "admin/installation", "admin/installation-apache", "admin/installation-docker", "admin/installation-nginx", "admin/installation-scripts", "admin/installation-searxng", "admin/installation-uwsgi", "admin/plugins", "admin/settings/index", "admin/settings/settings", "admin/settings/settings_brand", "admin/settings/settings_categories_as_tabs", "admin/settings/settings_engine", "admin/settings/settings_general", "admin/settings/settings_outgoing", "admin/settings/settings_redis", "admin/settings/settings_search", "admin/settings/settings_server", "admin/settings/settings_ui", "admin/update-searxng", "dev/contribution_guide", "dev/engines/demo/demo_offline", "dev/engines/demo/demo_online", "dev/engines/engine_overview", "dev/engines/enginelib", "dev/engines/engines", "dev/engines/index", "dev/engines/mediawiki", "dev/engines/offline/command-line-engines", "dev/engines/offline/nosql-engines", "dev/engines/offline/search-indexer-engines", "dev/engines/offline/sql-engines", "dev/engines/offline_concept", "dev/engines/online/annas_archive", "dev/engines/online/archlinux", "dev/engines/online/bing", "dev/engines/online/brave", "dev/engines/online/bt4g", "dev/engines/online/dailymotion", "dev/engines/online/duckduckgo", "dev/engines/online/google", "dev/engines/online/lemmy", "dev/engines/online/peertube", "dev/engines/online/piped", "dev/engines/online/recoll", "dev/engines/online/startpage", "dev/engines/online/tagesschau", "dev/engines/online/torznab", "dev/engines/online/wikipedia", "dev/engines/online/yacy", "dev/engines/online/yahoo", "dev/engines/online/zlibrary", "dev/engines/online_url_search/tineye", "dev/engines/xpath", "dev/index", "dev/lxcdev", "dev/makefile", "dev/plugins", "dev/quickstart", "dev/reST", "dev/search_api", "dev/searxng_extra/index", "dev/searxng_extra/standalone_searx.py", "dev/searxng_extra/update", "dev/translation", "index", "own-instance", "src/index", "src/searx.babel_extract", "src/searx.botdetection", "src/searx.exceptions", "src/searx.infopage", "src/searx.locales", "src/searx.plugins.tor_check", "src/searx.redisdb", "src/searx.redislib", "src/searx.search", "src/searx.search.processors", "src/searx.utils", "user/about", "user/configured_engines", "user/index", "user/search-syntax", "utils/index", "utils/lxc.sh", "utils/searxng.sh"], "filenames": ["admin/answer-captcha.rst", "admin/api.rst", "admin/architecture.rst", "admin/buildhosts.rst", "admin/index.rst", "admin/installation.rst", "admin/installation-apache.rst", "admin/installation-docker.rst", "admin/installation-nginx.rst", "admin/installation-scripts.rst", "admin/installation-searxng.rst", "admin/installation-uwsgi.rst", "admin/plugins.rst", "admin/settings/index.rst", "admin/settings/settings.rst", "admin/settings/settings_brand.rst", "admin/settings/settings_categories_as_tabs.rst", "admin/settings/settings_engine.rst", "admin/settings/settings_general.rst", "admin/settings/settings_outgoing.rst", "admin/settings/settings_redis.rst", "admin/settings/settings_search.rst", "admin/settings/settings_server.rst", "admin/settings/settings_ui.rst", "admin/update-searxng.rst", "dev/contribution_guide.rst", "dev/engines/demo/demo_offline.rst", "dev/engines/demo/demo_online.rst", "dev/engines/engine_overview.rst", "dev/engines/enginelib.rst", "dev/engines/engines.rst", "dev/engines/index.rst", "dev/engines/mediawiki.rst", "dev/engines/offline/command-line-engines.rst", "dev/engines/offline/nosql-engines.rst", "dev/engines/offline/search-indexer-engines.rst", "dev/engines/offline/sql-engines.rst", "dev/engines/offline_concept.rst", "dev/engines/online/annas_archive.rst", "dev/engines/online/archlinux.rst", "dev/engines/online/bing.rst", "dev/engines/online/brave.rst", "dev/engines/online/bt4g.rst", "dev/engines/online/dailymotion.rst", "dev/engines/online/duckduckgo.rst", "dev/engines/online/google.rst", "dev/engines/online/lemmy.rst", "dev/engines/online/peertube.rst", "dev/engines/online/piped.rst", "dev/engines/online/recoll.rst", "dev/engines/online/startpage.rst", "dev/engines/online/tagesschau.rst", "dev/engines/online/torznab.rst", "dev/engines/online/wikipedia.rst", "dev/engines/online/yacy.rst", "dev/engines/online/yahoo.rst", "dev/engines/online/zlibrary.rst", "dev/engines/online_url_search/tineye.rst", "dev/engines/xpath.rst", "dev/index.rst", "dev/lxcdev.rst", "dev/makefile.rst", "dev/plugins.rst", "dev/quickstart.rst", "dev/reST.rst", "dev/search_api.rst", "dev/searxng_extra/index.rst", "dev/searxng_extra/standalone_searx.py.rst", "dev/searxng_extra/update.rst", "dev/translation.rst", "index.rst", "own-instance.rst", "src/index.rst", "src/searx.babel_extract.rst", "src/searx.botdetection.rst", "src/searx.exceptions.rst", "src/searx.infopage.rst", "src/searx.locales.rst", "src/searx.plugins.tor_check.rst", "src/searx.redisdb.rst", "src/searx.redislib.rst", "src/searx.search.rst", "src/searx.search.processors.rst", "src/searx.utils.rst", "user/about.rst", "user/configured_engines.rst", "user/index.rst", "user/search-syntax.rst", "utils/index.rst", "utils/lxc.sh.rst", "utils/searxng.sh.rst"], "titles": ["Answer CAPTCHA from server\u2019s IP", "Administration API", "Architecture", "Buildhosts", "Administrator documentation", "Installation", "Apache", "Docker Container", "NGINX", "Installation Script", "Step by step installation", "uWSGI", "Plugins builtin", "Settings", "settings.yml", "brand:", "categories_as_tabs:", "engine:", "general:", "outgoing:", "redis:", "search:", "server:", "ui:", "SearXNG maintenance", "How to contribute", "Demo Offline Engine", "Demo Online Engine", "Engine Overview", "Engine Library", "SearXNG\u2019s engines loader", "Engine Implementations", "MediaWiki Engine", "Command Line Engines", "NoSQL databases", "Local Search APIs", "SQL Engines", "Offline Concept", "Anna\u2019s Archive", "Arch Linux", "Bing Engines", "Brave Engines", "BT4G", "Dailymotion", "DukcDukGo Engines", "Google Engines", "Lemmy", "Peertube Engines", "Piped", "Recoll Engine", "Startpage Engines", "Tagesschau API", "Torznab WebAPI", "Wikimedia", "Yacy", "Yahoo Engine", "Z-Library", "Tineye", "XPath Engine", "Developer documentation", "Developing in Linux Containers", "Makefile & ./manage", "Plugins", "Development Quickstart", "reST primer", "Search API", "Tooling box searxng_extra", "searxng_extra/standalone_searx.py", "searxng_extra/update/", "Translation", "Welcome to SearXNG", "Why use a private instance?", "Source-Code", "Custom message extractor (i18n)", "Bot Detection", "SearXNG Exceptions", "Online /info", "Locales", "Tor check plugin", "Redis DB", "Redis Library", "Search", "Search processors", "Utility functions for the engines", "About SearXNG", "Configured Engines", "User information", "Search syntax", "DevOps tooling box", "utils/lxc.sh", "utils/searxng.sh"], "terms": {"With": [0, 6, 7, 14, 33, 36, 38, 42, 56, 60, 61, 64, 89], "tunnel": 0, "we": [0, 2, 3, 6, 8, 9, 10, 11, 14, 17, 26, 27, 40, 41, 43, 44, 45, 50, 57, 60, 61, 62, 63, 64, 66, 69, 72, 77, 84, 88, 89], "can": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 31, 32, 33, 34, 35, 36, 37, 40, 41, 42, 43, 44, 45, 46, 49, 50, 51, 53, 57, 60, 61, 62, 63, 64, 67, 68, 70, 71, 73, 74, 77, 79, 80, 83, 85, 86, 87, 89, 90], "send": [0, 11, 17, 19, 21, 29, 50, 63, 71, 74, 75, 84], "request": [0, 6, 10, 12, 17, 19, 21, 22, 25, 27, 29, 31, 32, 35, 37, 39, 40, 43, 44, 45, 47, 50, 52, 53, 55, 57, 58, 61, 62, 63, 64, 69, 71, 72, 75, 76, 78, 81, 82], "solv": [0, 17], "block": [0, 10, 11, 19, 22, 39, 59, 74, 75, 80, 89], "thi": [0, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 26, 27, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 48, 49, 50, 51, 52, 53, 56, 57, 60, 61, 62, 63, 64, 65, 67, 68, 69, 71, 72, 73, 74, 75, 77, 78, 79, 80, 84, 87, 89], "If": [0, 3, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 19, 22, 23, 24, 25, 27, 29, 30, 32, 33, 34, 35, 36, 37, 44, 45, 46, 55, 58, 60, 61, 62, 63, 64, 68, 70, 71, 74, 77, 80, 83, 84, 88, 89], "your": [0, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 22, 23, 24, 25, 26, 27, 32, 33, 34, 35, 36, 37, 49, 53, 57, 59, 61, 62, 63, 64, 67, 69, 70, 74, 84, 87, 89], "searxng": [0, 1, 2, 3, 4, 5, 9, 11, 12, 14, 15, 17, 18, 19, 20, 22, 23, 25, 28, 29, 31, 32, 34, 36, 37, 38, 41, 42, 43, 44, 45, 48, 51, 53, 56, 57, 59, 61, 62, 63, 64, 65, 67, 69, 72, 73, 74, 76, 78, 79, 80, 83, 85, 86, 87, 88], "instanc": [0, 1, 2, 5, 6, 7, 8, 10, 11, 17, 20, 22, 23, 24, 29, 32, 33, 34, 35, 36, 37, 45, 46, 47, 48, 52, 54, 57, 60, 61, 62, 65, 70, 74, 76, 77, 80, 84, 89, 90], "i": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 80, 82, 83, 85, 86, 89, 90], "host": [0, 3, 6, 7, 8, 10, 11, 15, 25, 34, 60, 70, 89], "exampl": [0, 1, 2, 6, 10, 11, 13, 14, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 37, 38, 40, 42, 44, 45, 53, 56, 59, 60, 61, 65, 67, 68, 71, 74, 77, 78, 83, 87, 88, 89], "org": [0, 1, 3, 10, 11, 12, 15, 19, 22, 23, 28, 32, 36, 39, 44, 47, 49, 53, 58, 61, 64, 68, 69, 74, 78, 83, 84, 89], "login": [0, 9, 10, 20, 60], "user": [0, 4, 7, 9, 11, 12, 14, 16, 17, 19, 20, 21, 25, 28, 29, 33, 40, 41, 44, 45, 46, 50, 58, 60, 61, 64, 70, 71, 74, 77, 78, 83, 84, 85, 87, 89, 90], "you": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 14, 15, 17, 18, 19, 20, 22, 23, 24, 25, 26, 32, 33, 34, 35, 36, 37, 40, 46, 49, 50, 52, 53, 57, 60, 61, 63, 64, 65, 67, 69, 70, 71, 84, 87, 88, 89, 90], "setup": [0, 3, 4, 5, 6, 7, 8, 9, 10, 17, 24, 37, 60, 61, 63, 64, 74, 79, 88], "proxi": [0, 2, 6, 7, 10, 17, 19, 22, 24, 28, 29, 48, 60, 65, 71, 74, 88, 89], "simpli": [0, 25, 57, 60, 61, 63, 65, 90], "sock": [0, 2, 10, 11, 19, 20, 24, 74, 79, 90], "127": [0, 6, 8, 10, 22, 34, 60, 90], "0": [0, 1, 2, 6, 7, 8, 10, 11, 14, 17, 19, 20, 21, 22, 24, 25, 28, 34, 43, 50, 57, 58, 60, 61, 64, 65, 67, 74, 79, 80, 81, 83, 85, 88, 89, 90], "1": [0, 1, 3, 6, 7, 8, 10, 11, 17, 19, 20, 21, 22, 32, 34, 43, 50, 58, 60, 61, 64, 65, 67, 74, 75, 77, 80, 81, 82, 83, 85, 89, 90], "8080": [0, 7, 10, 17, 19, 60, 89], "q": [0, 1, 7, 10, 23, 44, 50, 61, 65], "n": [0, 33, 60, 89], "d": [0, 6, 7, 8, 11, 44, 68, 83, 89], "The": [0, 1, 2, 3, 4, 5, 7, 9, 10, 11, 14, 17, 19, 21, 22, 23, 24, 26, 27, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 48, 50, 51, 52, 53, 54, 56, 57, 58, 61, 62, 64, 65, 68, 69, 71, 73, 74, 75, 77, 78, 80, 83, 84, 85, 88, 89, 90], "localhost": [0, 6, 7, 8, 20, 35, 54], "abov": [0, 6, 10, 14, 24, 25, 34, 35, 36, 44, 50, 60, 64, 65, 89], "test": [0, 3, 6, 7, 8, 10, 20, 25, 36, 51, 53, 59, 60, 63, 70, 74, 79, 83, 89, 90], "desktop": [0, 23, 49, 60, 61], "curl": [0, 10], "x": [0, 6, 8, 10, 11, 22, 64, 72, 89], "http": [0, 1, 2, 3, 4, 7, 9, 10, 11, 15, 17, 19, 21, 22, 23, 24, 25, 28, 29, 32, 35, 36, 37, 40, 43, 44, 45, 46, 47, 48, 49, 50, 53, 54, 57, 58, 60, 61, 62, 63, 64, 68, 69, 72, 78, 83, 89, 90], "ipecho": 0, "net": [0, 53, 54], "plain": [0, 53], "In": [0, 6, 7, 8, 9, 10, 11, 14, 16, 17, 19, 20, 24, 25, 26, 27, 28, 29, 31, 32, 34, 41, 45, 48, 50, 59, 62, 64, 66, 71, 74, 85, 87, 88, 89, 90], "set": [0, 2, 3, 4, 6, 7, 8, 10, 11, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 29, 30, 32, 33, 34, 35, 36, 37, 38, 42, 43, 45, 46, 48, 49, 50, 52, 53, 54, 56, 58, 60, 61, 62, 64, 65, 70, 71, 74, 75, 77, 78, 79, 80, 82, 83, 85, 86, 87, 88, 89, 90], "web": [0, 7, 10, 18, 21, 22, 23, 28, 29, 35, 41, 44, 49, 50, 53, 55, 57, 60, 61, 62, 74, 83, 89], "browser": [0, 7, 10, 18, 21, 23, 25, 44, 53, 60, 61, 71, 74, 83, 84, 87, 89], "open": [0, 6, 7, 10, 12, 23, 25, 34, 36, 45, 51, 60, 61, 64, 65, 70, 71, 84, 89], "network": [0, 10, 17, 19, 48, 54, 74, 89], "socks5": [0, 17, 28, 29], "see": [0, 3, 5, 6, 7, 8, 10, 11, 14, 17, 19, 20, 22, 24, 25, 29, 30, 32, 34, 36, 40, 41, 43, 44, 45, 47, 48, 50, 52, 53, 58, 60, 61, 62, 64, 65, 68, 70, 73, 74, 77, 80, 82, 83, 84, 87, 89], "screenshot": 0, "below": [0, 9, 10, 11, 17, 19, 25, 28, 33, 34, 36, 62, 64, 74, 83], "check": [0, 3, 4, 7, 12, 20, 25, 28, 33, 38, 41, 56, 61, 69, 70, 72, 74, 77, 83, 89, 90], "us": [0, 3, 5, 6, 7, 8, 9, 10, 11, 14, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 55, 57, 58, 60, 61, 63, 64, 66, 67, 68, 69, 70, 74, 76, 77, 79, 80, 82, 83, 85, 86, 87, 88, 89, 90], "now": [0, 6, 8, 10, 24, 35, 60, 61, 62, 63], "search": [0, 2, 4, 7, 10, 12, 13, 14, 16, 19, 23, 25, 26, 28, 29, 32, 33, 34, 36, 37, 38, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 62, 64, 67, 70, 71, 72, 74, 75, 77, 78, 83, 85, 86], "engin": [0, 1, 4, 10, 13, 14, 16, 19, 21, 25, 35, 38, 39, 42, 43, 46, 48, 51, 52, 53, 54, 56, 57, 59, 62, 64, 65, 67, 68, 70, 71, 72, 74, 75, 77, 81, 82, 86], "have": [0, 5, 6, 7, 8, 9, 10, 11, 17, 24, 25, 28, 32, 34, 35, 36, 37, 50, 53, 60, 61, 63, 64, 71, 74, 77, 84, 89], "issu": [0, 10, 11, 15, 17, 28, 60, 84, 89], "qwant": [0, 10, 21, 65, 67, 85], "com": [0, 2, 7, 9, 10, 15, 17, 23, 29, 31, 40, 42, 43, 44, 45, 53, 54, 55, 57, 60, 61, 62, 63, 64, 65, 68, 83, 89, 90], "firefox": [0, 3, 10, 61, 68, 84], "bind_address": [0, 7, 10, 14, 22, 61], "port": [0, 7, 10, 22, 25, 28, 29, 34, 61, 89], "specifi": [0, 7, 11, 14, 17, 19, 29, 45, 46, 65], "local": [0, 1, 2, 6, 7, 8, 10, 11, 17, 19, 20, 23, 24, 26, 28, 29, 31, 41, 43, 44, 45, 49, 50, 53, 54, 60, 61, 62, 64, 68, 70, 72, 74, 76, 79, 83, 85, 89, 90], "dynam": 0, "applic": [0, 11, 22, 52, 60, 64, 67, 75, 85], "level": [0, 25, 57, 64], "forward": [0, 6, 8, 71, 72, 89], "work": [0, 7, 10, 23, 25, 28, 29, 33, 38, 44, 59, 61, 63, 71, 74, 86, 89], "alloc": 0, "socket": [0, 6, 7, 8, 11, 20, 22, 60, 90], "listen": [0, 6, 8, 11, 89], "side": [0, 9, 23, 64, 68], "whenev": [0, 89], "connect": [0, 2, 8, 10, 11, 19, 20, 24, 36, 37, 60, 70, 74, 79], "made": [0, 19, 60, 68], "over": [0, 24, 43, 50, 57, 60, 64, 70, 76, 77, 89], "secur": [0, 10, 11, 33, 70], "channel": 0, "protocol": [0, 7, 19, 83], "determin": [0, 29, 31, 68, 74, 77], "where": [0, 11, 22, 28, 33, 36, 37, 40, 41, 49, 51, 53, 60, 61, 64, 78, 80, 84], "remot": [0, 61, 74, 89], "machin": 0, "act": 0, "do": [0, 3, 5, 6, 8, 9, 11, 12, 17, 25, 26, 27, 34, 35, 36, 47, 50, 57, 60, 61, 63, 64, 71, 77, 83, 86, 89], "execut": [0, 17, 29, 33, 61, 63, 73, 89], "command": [0, 3, 4, 6, 10, 11, 20, 24, 25, 31, 37, 60, 61, 64, 67, 69, 70, 80], "just": [0, 1, 24, 26, 27, 34, 35, 36, 40, 60, 61, 63, 64, 71, 84, 89], "config": [1, 10, 11, 17, 24, 28, 37, 60, 64, 69, 74, 88], "autocomplet": [1, 2, 7, 10, 21, 65], "categori": [1, 10, 12, 16, 17, 28, 29, 36, 40, 41, 42, 52, 64, 65, 67, 68, 81, 85, 86], "map": [1, 11, 16, 29, 30, 33, 39, 41, 44, 45, 50, 53, 55, 58, 64, 67, 68, 77, 83, 86, 87], "imag": [1, 3, 4, 9, 10, 16, 22, 24, 27, 41, 53, 57, 59, 60, 61, 65, 67, 86, 87, 89], "default_local": [1, 10, 23], "default_them": [1, 10, 23], "simpl": [1, 10, 23, 26, 27, 28, 30, 34, 35, 36, 45, 57, 58, 60, 61, 63, 65, 80, 89], "enabl": [1, 6, 7, 8, 10, 11, 14, 17, 18, 19, 23, 25, 28, 29, 32, 33, 35, 36, 53, 61, 64, 65, 74, 78, 85], "true": [1, 2, 6, 8, 11, 17, 18, 19, 23, 28, 29, 30, 32, 34, 35, 37, 40, 41, 43, 44, 45, 50, 52, 53, 58, 62, 64, 65, 74, 77, 79, 83], "name": [1, 6, 8, 10, 11, 12, 14, 17, 23, 24, 26, 27, 29, 30, 33, 34, 35, 36, 37, 38, 41, 42, 46, 48, 50, 54, 56, 58, 59, 60, 62, 68, 72, 75, 76, 77, 78, 80, 81, 83, 84, 85, 87, 89], "openstreetmap": [1, 68, 85], "shortcut": [1, 17, 26, 27, 28, 29, 30, 33, 34, 35, 38, 42, 48, 54, 56, 64], "osm": [1, 28, 68, 85], "arch": [1, 3, 6, 8, 10, 11, 14, 17, 31, 64, 85], "linux": [1, 3, 6, 7, 8, 10, 11, 14, 17, 31, 59, 64, 70, 85, 89], "wiki": [1, 7, 8, 10, 14, 15, 17, 32, 53, 61, 64, 68, 85, 89], "al": [1, 17, 41, 53, 83, 85], "googl": [1, 10, 14, 17, 21, 29, 31, 61, 65, 67, 84, 85], "goi": [1, 85], "fals": [1, 2, 10, 14, 15, 17, 18, 19, 22, 23, 26, 27, 29, 34, 36, 41, 43, 52, 58, 62, 64, 65, 74, 76, 77, 83], "bitbucket": [1, 58, 85], "bb": [1, 85], "instance_nam": [1, 2, 7, 10, 18], "searx": [1, 5, 6, 8, 10, 11, 14, 15, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 60, 61, 62, 64, 67, 68, 70, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 89, 90], "de": [1, 10, 11, 17, 21, 23, 32, 36, 39, 41, 45, 58, 77, 83, 85, 90], "deutsch": 1, "german": [1, 17], "en": [1, 10, 11, 17, 18, 19, 21, 23, 28, 32, 40, 41, 45, 50, 53, 55, 58, 61, 64, 76, 77, 83], "english": [1, 17, 21, 44, 77, 83], "eo": [1, 64, 83], "esperanto": 1, "plugin": [1, 3, 4, 10, 11, 25, 36, 59, 64, 65, 70, 72, 81], "rewrit": [1, 10, 12, 32], "vim": [1, 10, 12, 65], "like": [1, 3, 6, 7, 10, 12, 17, 19, 23, 25, 28, 34, 35, 36, 40, 41, 42, 44, 61, 63, 64, 67, 78, 83], "hotkei": [1, 10, 12], "safe_search": [1, 2, 10, 21, 58], "embed": [1, 7, 64], "websit": [1, 17, 29, 44, 57, 64, 68, 75, 84], "past": 1, "html": [1, 3, 6, 10, 11, 17, 21, 25, 28, 29, 36, 42, 44, 45, 50, 53, 60, 61, 64, 74, 76, 83], "site": [1, 4, 17, 24, 28, 52, 58, 64, 65, 68, 90], "url": [1, 2, 6, 7, 10, 12, 15, 17, 20, 22, 23, 25, 27, 28, 32, 36, 40, 43, 44, 45, 47, 48, 52, 53, 55, 57, 58, 60, 61, 62, 67, 68, 71, 72, 74, 79, 83, 89], "valu": [1, 7, 10, 17, 18, 21, 22, 28, 29, 30, 32, 33, 34, 36, 37, 38, 44, 45, 48, 50, 58, 61, 64, 65, 68, 74, 75, 76, 77, 80, 83, 90], "ar": [1, 3, 7, 9, 10, 11, 14, 16, 17, 19, 23, 24, 25, 29, 31, 33, 34, 35, 36, 37, 38, 40, 41, 43, 44, 45, 46, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 64, 65, 68, 69, 70, 72, 73, 74, 77, 80, 83, 85, 87, 89], "customiz": 1, "form": [1, 10, 18, 38, 50, 56, 64, 71], "method": [1, 5, 10, 11, 24, 28, 35, 48, 64, 72, 73, 76, 83], "post": [1, 10, 25, 46, 50, 60, 62, 65, 74, 85], "action": [1, 11, 32], "input": [1, 17, 50, 64], "type": [1, 9, 10, 22, 25, 29, 30, 32, 33, 35, 45, 50, 53, 58, 59, 61, 62, 65, 68, 71, 74, 75, 76, 82, 83, 89], "text": [1, 28, 32, 35, 40, 44, 45, 49, 53, 56, 64, 74, 83], "hidden": [1, 17, 71], "gener": [1, 2, 4, 6, 7, 10, 11, 13, 16, 17, 25, 29, 31, 32, 36, 41, 50, 56, 58, 67, 68, 71, 74, 77, 83, 84, 86, 87], "social": [1, 16, 67], "media": [1, 16, 51, 67], "languag": [1, 10, 11, 17, 21, 23, 28, 29, 32, 38, 39, 40, 43, 44, 45, 47, 53, 55, 56, 58, 61, 64, 65, 68, 74, 76, 77, 81, 83, 86], "lang": [1, 11, 55, 58, 61, 67, 68, 81, 83], "all": [1, 6, 7, 9, 10, 11, 14, 16, 17, 19, 21, 24, 25, 27, 28, 29, 32, 34, 35, 36, 37, 41, 42, 44, 45, 48, 53, 58, 60, 61, 62, 63, 64, 65, 67, 68, 69, 74, 76, 77, 78, 80, 81, 82, 83, 88, 89, 90], "date": [1, 28, 57], "filter": [1, 10, 21, 26, 27, 38, 43, 44, 48, 50, 56, 58, 61, 65, 68, 83, 87], "time_rang": [1, 28, 42, 45, 58, 65, 81], "month": [1, 28, 40, 45, 58, 65, 67], "revers": [2, 7, 44, 45, 57, 89], "apach": [2, 4, 9, 11, 24, 70, 74, 89, 90], "nginx": [2, 4, 9, 24, 60, 70, 74, 88, 89, 90], "step": [2, 4, 5, 9, 24, 25, 60, 63, 70, 90], "instal": [2, 3, 4, 6, 7, 8, 11, 20, 25, 34, 35, 36, 37, 59, 60, 62, 63, 64, 70, 88], "herein": 2, "find": [2, 7, 10, 17, 20, 24, 25, 33, 34, 35, 40, 44, 57, 77, 84, 87, 89], "some": [2, 6, 7, 8, 9, 10, 11, 14, 17, 21, 22, 24, 25, 32, 40, 41, 44, 45, 50, 53, 58, 60, 61, 63, 64, 66, 74, 84, 88, 89, 90], "hint": [2, 11, 29, 60, 64], "suggest": [2, 58, 62, 70], "about": [2, 17, 24, 25, 29, 41, 60, 61, 63, 70, 86, 89], "typic": [2, 61, 64], "infrastructur": [2, 11, 24], "start": [2, 6, 7, 8, 9, 10, 11, 25, 28, 30, 37, 57, 59, 61, 63, 70, 89], "refer": [2, 6, 8, 9, 40, 49, 50, 64, 68, 71, 81, 83], "public": [2, 7, 10, 12, 28, 33, 34, 35, 36, 45, 64, 84], "which": [2, 9, 10, 11, 17, 19, 25, 26, 27, 28, 29, 31, 32, 33, 36, 37, 43, 46, 48, 52, 56, 58, 60, 61, 63, 64, 65, 67, 68, 69, 70, 71, 74, 77, 80, 84, 85], "build": [2, 4, 5, 10, 17, 20, 27, 29, 43, 45, 50, 52, 55, 57, 58, 59, 60, 63, 68, 77, 89, 90], "up": [2, 5, 6, 7, 8, 10, 11, 17, 20, 24, 27, 44, 60, 61, 64, 70, 77], "maintain": [2, 7, 10, 11, 24, 38, 42, 56, 60, 64, 66, 70, 88], "script": [2, 4, 5, 6, 7, 8, 10, 11, 24, 50, 53, 60, 61, 67, 68, 70, 77, 80, 88, 89, 90], "from": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 17, 19, 21, 24, 25, 26, 27, 28, 29, 30, 32, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 50, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 71, 72, 73, 74, 76, 77, 78, 79, 80, 81, 83, 84, 87, 89, 90], "our": [2, 5, 7, 9, 25, 60, 61, 63, 64, 72, 88, 89], "devop": [2, 9, 24, 60, 70, 89], "tool": [2, 4, 9, 10, 24, 25, 37, 49, 59, 60, 61, 63, 64, 67, 70, 89], "box": [2, 9, 24, 59, 60, 62, 70, 89], "activ": [2, 6, 10, 16, 17, 25, 29, 30, 41, 42, 60, 62, 65, 67, 74, 87, 90], "server": [2, 4, 7, 9, 10, 13, 14, 19, 36, 57, 60, 61, 65, 67, 70, 74, 80, 84, 88, 89, 90], "limit": [2, 6, 7, 8, 10, 11, 17, 19, 22, 24, 28, 33, 34, 35, 36, 37, 40, 41, 45, 53, 58, 64, 72, 80], "image_proxi": [2, 10, 22, 65], "ui": [2, 4, 6, 8, 10, 11, 13, 16, 17, 28, 29, 40, 41, 45, 48, 50, 64, 69, 70, 77, 85], "static_use_hash": [2, 6, 8, 10, 11, 23], "etc": [2, 6, 7, 8, 10, 11, 14, 24, 25, 41, 60, 74, 81, 89, 90], "yml": [2, 4, 6, 7, 8, 10, 11, 13, 17, 24, 26, 27, 34, 35, 36, 37, 46, 50, 60, 61, 62, 64, 65, 78, 79, 83, 85, 90], "use_default_set": [2, 13], "debug": [2, 4, 9, 10, 18, 60, 61], "2": [2, 10, 11, 19, 21, 28, 43, 50, 57, 58, 61, 63, 64, 65, 67, 74, 80, 83, 85, 87], "duckduckgo": [2, 10, 14, 21, 65, 68, 85, 87], "overwritten": [2, 10, 19, 28], "searxng_secret": [2, 10, 22], "secret_kei": [2, 10, 14, 22, 80], "ultrasecretkei": [2, 10, 14, 22], "ensur": [2, 10, 25], "correct": [2, 10, 22, 32, 40, 74], "inbound": [2, 10, 22], "link": [2, 10, 11, 15, 18, 22, 23, 42, 44, 45, 48, 52, 53, 57, 58, 59, 60, 68, 74, 84], "searxng_url": [2, 10, 22, 60, 61, 89, 90], "base_url": [2, 7, 10, 17, 22, 28, 32, 35, 40, 46, 47, 49, 52, 54, 61, 83], "locat": [2, 6, 8, 10, 11, 13, 22, 39, 49, 50, 53, 64, 69, 76], "redi": [2, 4, 7, 10, 11, 13, 22, 24, 59, 60, 70, 72, 74, 90], "databas": [2, 10, 20, 22, 24, 31, 36, 37, 64, 74, 80], "searxng_redis_url": [2, 10, 20, 61], "unix": [2, 6, 8, 10, 11, 20, 74, 79], "usr": [2, 6, 7, 8, 10, 11, 20, 24, 60, 61, 74, 79, 90], "run": [2, 3, 6, 8, 9, 10, 11, 18, 20, 22, 24, 25, 33, 35, 59, 60, 62, 63, 64, 67, 74, 79, 84, 88, 90], "db": [2, 7, 10, 11, 20, 24, 34, 36, 60, 70, 72, 74, 80], "To": [3, 6, 7, 9, 10, 11, 17, 20, 24, 25, 26, 27, 28, 29, 34, 36, 37, 38, 41, 43, 44, 45, 50, 53, 58, 60, 61, 63, 64, 68, 69, 71, 74, 77, 84, 87, 89, 90], "get": [3, 4, 6, 8, 9, 10, 11, 12, 20, 24, 25, 26, 27, 28, 35, 37, 40, 41, 43, 44, 45, 50, 52, 53, 60, 61, 63, 64, 65, 67, 68, 70, 74, 75, 76, 77, 79, 80, 83, 84, 89, 90], "best": [3, 29, 44, 61, 64, 71, 77], "result": [3, 10, 12, 17, 19, 21, 23, 26, 27, 29, 31, 32, 33, 34, 35, 36, 37, 38, 40, 41, 42, 43, 44, 45, 46, 47, 52, 56, 57, 58, 62, 64, 65, 67, 70, 71, 74, 75, 77, 81, 83, 84], "": [3, 4, 5, 7, 10, 17, 22, 23, 24, 25, 28, 29, 31, 34, 36, 37, 40, 41, 42, 43, 44, 45, 47, 48, 50, 53, 56, 58, 59, 60, 61, 63, 64, 68, 69, 70, 71, 72, 73, 74, 78, 80, 83, 84, 89, 90], "recommend": [3, 5, 10, 11, 24, 25, 50, 61, 64], "addit": [3, 6, 11, 16, 17, 19, 22, 29, 34, 36, 37, 38, 42, 45, 46, 54, 56, 61, 64, 73, 74, 77, 85], "packag": [3, 4, 7, 11, 29, 34, 36, 37, 60, 61, 64, 73, 89, 90], "util": [3, 6, 8, 9, 10, 22, 24, 34, 36, 37, 60, 61, 62, 67, 70, 72, 88], "sh": [3, 6, 7, 8, 9, 24, 34, 36, 37, 60, 61, 62, 64, 70, 83, 88], "task": [3, 9, 10, 11, 60, 61, 63, 64, 74, 89], "onc": [3, 24, 60, 61, 80, 83, 89], "sudo": [3, 6, 7, 8, 9, 10, 11, 20, 24, 34, 36, 37, 60, 61, 62, 64, 88, 89, 90], "h": [3, 6, 7, 8, 9, 10, 11, 12, 20, 24, 32, 33, 60, 61, 64, 67, 88, 89, 90], "ubuntu": [3, 6, 8, 10, 11, 61, 64, 85, 89], "debian": [3, 7, 8, 10, 64], "fedora": [3, 6, 8, 10, 11, 64, 89], "rhel": [3, 6, 8, 10, 11, 64], "apt": [3, 6, 7, 8, 10, 64], "y": [3, 10, 12, 32, 64, 85], "python3": [3, 10, 11, 67], "dev": [3, 6, 8, 10, 54, 61, 64, 85], "babel": [3, 10, 29, 41, 45, 50, 69, 73, 77], "venv": [3, 10], "uwsgi": [3, 4, 5, 7, 8, 9, 10, 24, 60, 70, 90], "git": [3, 7, 9, 10, 14, 21, 25, 28, 29, 31, 49, 60, 61, 62, 63, 64, 66, 68, 69, 73, 88, 89, 90], "essenti": [3, 10, 61, 74], "libxslt": [3, 10], "zlib1g": [3, 10], "libffi": [3, 10], "libssl": [3, 10], "pacman": [3, 6, 8, 10, 64], "noconfirm": [3, 10], "python": [3, 10, 11, 17, 19, 22, 28, 29, 34, 36, 37, 59, 60, 63, 64, 73, 74, 77, 83, 90], "pip": [3, 10, 34, 36, 37, 61, 62], "lxml": [3, 10, 83], "base": [3, 7, 10, 12, 22, 28, 32, 33, 34, 35, 44, 45, 47, 48, 49, 53, 58, 60, 65, 74, 75, 82, 83, 84, 89], "devel": [3, 10], "libxml2": [3, 10], "dnf": [3, 6, 8, 10, 64], "openssl": [3, 10, 19], "docuemt": 3, "graphviz": 3, "imagemagick": [3, 64], "texliv": 3, "xetex": 3, "librsvg2": 3, "bin": [3, 7, 10, 11, 61, 85, 89], "latex": 3, "extra": [3, 6, 10, 19], "font": 3, "dejavu": 3, "latexmk": 3, "shellcheck": [3, 64, 89], "librsvg": 3, "core": [3, 11, 29, 45, 48, 50, 77], "latexextra": 3, "ttf": 3, "gd": [3, 83], "collect": [3, 34, 35, 42, 50, 57, 80], "fontsrecommend": 3, "san": 3, "serif": 3, "mono": 3, "dvisvgm": 3, "most": [3, 6, 17, 21, 29, 35, 36, 38, 44, 50, 56, 60, 61, 68, 71, 83, 90], "requir": [3, 6, 11, 12, 22, 24, 25, 28, 30, 34, 35, 36, 37, 57, 58, 61, 64, 65, 67, 68, 74, 75, 90], "py": [3, 6, 8, 10, 21, 22, 28, 29, 33, 34, 35, 36, 37, 45, 49, 53, 59, 61, 64, 66, 76, 77, 79, 90], "scratch": 3, "make": [3, 7, 10, 17, 25, 29, 31, 44, 45, 58, 59, 60, 63, 64, 69, 71, 74, 86, 89], "For": [3, 6, 8, 9, 10, 11, 17, 21, 25, 28, 29, 32, 34, 36, 38, 40, 42, 45, 48, 50, 53, 56, 60, 61, 64, 65, 68, 69, 72, 77, 84, 89], "better": [3, 6, 7, 8, 10, 28, 32, 40, 71, 84], "math": [3, 59], "process": [3, 7, 11, 17, 37, 60, 61], "onli": [3, 7, 9, 10, 11, 14, 17, 23, 25, 28, 29, 33, 36, 40, 41, 42, 43, 44, 46, 52, 53, 56, 57, 58, 61, 62, 63, 64, 67, 69, 74, 77, 80, 83, 84, 89], "pdf": [3, 28, 38, 56], "creation": [3, 64], "also": [3, 6, 7, 10, 11, 19, 24, 25, 30, 34, 37, 38, 44, 57, 60, 63, 64, 71, 74, 77, 83, 84, 85, 87], "equat": [3, 59], "when": [3, 6, 7, 9, 10, 12, 17, 20, 23, 29, 30, 33, 37, 40, 41, 42, 45, 50, 61, 62, 63, 64, 67, 71, 72, 75, 77, 82, 83, 84], "output": [3, 17, 61, 64, 65, 68, 89, 90], "abl": [3, 22, 25, 38, 42, 56, 64, 74, 87], "support": [3, 8, 10, 11, 17, 19, 25, 28, 29, 34, 35, 36, 41, 44, 45, 48, 52, 53, 55, 57, 58, 62, 64, 65, 68, 71, 76, 77, 82, 83, 84, 85, 87, 89], "without": [3, 7, 11, 17, 21, 28, 29, 34, 50, 52, 61, 67, 68, 70, 71, 76, 77, 84, 89], "cdn": 3, "render": [3, 76], "ext": 3, "imgmath": 3, "extens": [3, 11, 38, 64], "here": [3, 11, 16, 19, 24, 25, 28, 29, 35, 36, 40, 53, 58, 60, 61, 63, 64, 67, 89], "extract": [3, 53, 72, 73, 83], "conf": [3, 6, 8, 11, 64, 90], "file": [3, 6, 7, 8, 10, 11, 13, 14, 16, 17, 19, 23, 24, 25, 29, 33, 36, 38, 42, 49, 52, 56, 57, 60, 61, 67, 68, 69, 73, 76, 77, 86, 89], "html_math_render": 3, "imgmath_image_format": 3, "svg": 3, "imgmath_font_s": 3, "14": [3, 61, 83], "show": [3, 6, 10, 11, 24, 52, 53, 60, 61, 64, 87, 89], "warn": [3, 41, 61, 64, 67, 89], "dot": [3, 61], "found": [3, 11, 31, 37, 55, 57, 61, 83], "qualiti": [3, 41, 53, 61, 64, 70], "www": [3, 6, 10, 19, 36, 40, 43, 44, 64, 68, 83, 89], "cannot": [3, 45, 64, 71, 74, 84, 87], "displai": [3, 7, 10, 12, 16, 17, 18, 28, 29, 34, 36, 50, 64, 85], "imgmath_latex": 3, "static": [3, 6, 8, 10, 11, 23, 24, 59, 63, 68, 74], "analysi": 3, "brand": [4, 10, 13, 22, 25, 60, 61, 90], "outgo": [4, 10, 13, 17], "categories_as_tab": [4, 13, 17, 64, 85], "docker": [4, 5, 9, 60, 61, 70, 85], "contain": [4, 5, 10, 11, 12, 23, 30, 33, 37, 43, 44, 59, 64, 68, 69, 70, 74, 75, 81, 84, 88, 89], "line": [4, 6, 8, 25, 31, 37, 61, 67, 69, 89], "creat": [4, 6, 7, 8, 9, 11, 22, 29, 30, 33, 36, 38, 43, 60, 61, 64, 65, 69, 80, 89], "depend": [4, 6, 8, 12, 24, 28, 39, 44, 50, 60, 61, 64, 65, 74], "configur": [4, 6, 7, 8, 11, 14, 17, 19, 29, 30, 31, 35, 37, 41, 60, 61, 64, 65, 69, 70, 71, 73, 74, 83, 86], "origin": [4, 17, 24, 26, 29, 34, 36, 45, 50, 56, 57, 60, 61, 64, 69, 74], "distributor": 4, "mainten": [4, 9, 60, 61, 70, 90], "pitfal": 4, "tyrant": 4, "mode": [4, 6, 10, 22, 36, 54, 64, 89], "disabl": [4, 10, 11, 12, 14, 16, 17, 18, 19, 21, 26, 27, 29, 36, 41, 60, 62, 64, 65, 85, 89], "log": [4, 7, 11, 18, 60, 68, 71, 74, 84, 90], "how": [4, 5, 6, 7, 8, 9, 11, 14, 17, 36, 44, 53, 57, 59, 60, 61, 63, 64, 70, 86], "updat": [4, 7, 10, 17, 25, 30, 53, 59, 61, 62, 66, 69, 77, 90], "inspect": [4, 9, 60, 90], "migrat": [4, 5], "stai": [4, 5], "tune": [4, 5], "answer": [4, 19, 60, 61, 64, 70, 71], "captcha": [4, 10, 21, 45, 50, 70, 71, 74, 75], "ip": [4, 6, 8, 12, 19, 50, 60, 67, 70, 71, 74, 78, 89], "api": [4, 14, 17, 27, 28, 29, 31, 32, 39, 40, 42, 43, 46, 47, 48, 49, 52, 53, 54, 57, 59, 68, 69, 70, 74], "data": [4, 28, 29, 33, 34, 35, 36, 42, 45, 50, 51, 61, 64, 68, 71, 83, 84], "emb": [4, 43, 48, 64], "bar": [4, 64, 79, 83, 84], "architectur": [4, 9, 48, 70, 90], "builtin": [4, 62, 70], "buildhost": [4, 60, 61, 63, 70, 88, 90], "develop": [4, 10, 13, 25, 32, 61, 64, 66, 70, 83, 84, 88, 89], "doc": [4, 6, 10, 11, 15, 17, 29, 36, 42, 59, 60, 64, 83, 84, 89], "lint": [4, 61], "shell": [4, 10, 33, 60, 89], "re": [5, 7, 24, 61, 85], "spoilt": 5, "choic": [5, 40, 41, 64, 71, 83, 88], "choos": [5, 25, 63], "prefer": [5, 6, 8, 9, 10, 16, 17, 18, 40, 44, 45, 64, 65, 71, 76, 78, 87], "an": [5, 10, 14, 16, 17, 19, 20, 23, 25, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 40, 42, 43, 45, 48, 50, 51, 52, 53, 57, 58, 60, 61, 64, 65, 67, 69, 70, 71, 74, 75, 77, 81, 83, 84, 87], "excel": 5, "illustr": 5, "ani": [5, 6, 8, 9, 19, 25, 36, 41, 42, 46, 52, 53, 55, 61, 64, 65, 67, 76, 77, 83], "special": [5, 6, 8, 11, 28, 29, 33, 39, 45, 52, 86], "grow": [5, 24], "rapidli": [5, 24], "should": [5, 6, 8, 11, 17, 24, 25, 29, 44, 45, 50, 52, 53, 60, 61, 63, 64, 77], "regularli": [5, 24], "read": [5, 10, 20, 25, 57, 58], "section": [5, 6, 8, 9, 10, 14, 17, 19, 31, 33, 34, 35, 36, 40, 58, 60, 61, 62, 64, 78, 89], "want": [5, 7, 10, 11, 17, 19, 23, 25, 35, 36, 46, 61, 64, 65, 69, 71, 84, 88, 89], "upgrad": [5, 24, 61, 68], "exist": [5, 7, 10, 14, 17, 21, 24, 25, 29, 36, 45, 50, 60, 61, 68, 74, 80, 83, 89], "first": [5, 6, 20, 24, 34, 37, 40, 45, 53, 58, 60, 61, 63, 64, 71, 74, 77, 83, 89], "explain": [6, 8, 25], "did": [6, 74], "interest": [6, 7, 8, 56], "problem": [6, 8, 10, 84, 89], "follow": [6, 7, 8, 9, 10, 11, 14, 20, 25, 26, 27, 28, 30, 32, 33, 34, 35, 36, 38, 42, 44, 46, 49, 52, 54, 56, 61, 64, 74, 89], "might": [6, 7, 8, 11, 17, 25, 35, 44, 58, 60, 64, 65, 68, 71, 89], "give": [6, 8, 11, 61, 67, 84, 87], "guidanc": [6, 8], "apache2": [6, 90], "readm": [6, 11, 61], "direct": [6, 8, 59, 63, 64, 77], "new": [6, 7, 8, 9, 10, 11, 16, 17, 23, 24, 29, 36, 41, 50, 51, 61, 65, 67, 74, 80, 83, 86], "term": [6, 26, 29, 33, 35, 38, 44, 45, 58, 60, 64], "describ": [6, 9, 11, 14, 17, 29, 33, 34, 35, 36, 60, 61, 64, 83, 90], "document": [6, 15, 17, 29, 31, 32, 34, 35, 44, 45, 46, 49, 51, 61, 64, 70, 72, 76, 84, 89], "orient": 6, "There": [6, 8, 11, 25, 43, 63, 64, 69, 83, 89], "list": [6, 7, 16, 17, 21, 24, 26, 27, 28, 29, 31, 32, 33, 36, 37, 40, 41, 42, 43, 45, 47, 48, 50, 52, 53, 57, 58, 59, 61, 65, 67, 70, 74, 76, 77, 78, 81, 83, 84, 87], "keep": [6, 8, 10, 14, 19, 60, 61, 62], "pocket": [6, 8], "systemctl": [6, 8, 11, 60], "httpd": 6, "kind": [6, 41, 50, 64, 71], "welcom": [6, 8, 84], "page": [6, 8, 10, 12, 14, 16, 17, 18, 19, 23, 25, 28, 29, 35, 36, 40, 41, 42, 44, 45, 48, 50, 53, 58, 64, 65, 68, 69, 71, 76, 82, 84, 85, 87], "default": [6, 7, 8, 10, 11, 14, 17, 18, 19, 21, 23, 25, 29, 30, 33, 34, 36, 37, 38, 41, 42, 43, 44, 46, 50, 52, 54, 61, 62, 64, 65, 67, 71, 74, 75, 76, 77, 79, 80, 82, 83, 85, 86, 88, 89, 90], "distribut": [6, 7, 8, 11, 17, 19, 54, 60, 64, 83], "compar": [6, 7, 11, 16, 45, 50, 53, 60, 89], "less": [6, 8, 35, 42, 47, 61, 64, 71, 74], "000": 6, "documentroot": 6, "var": 6, "And": [6, 60, 64, 74], "index": [6, 34, 35, 40, 41, 47, 49, 57, 83], "srv": 6, "directori": [6, 7, 11, 19, 33, 76, 77, 85], "option": [6, 7, 10, 11, 14, 17, 22, 25, 29, 33, 34, 36, 38, 40, 42, 44, 50, 53, 56, 60, 61, 64, 65, 70, 71, 73, 83], "followsymlink": 6, "allowoverrid": 6, "none": [6, 10, 21, 26, 28, 29, 30, 32, 35, 37, 38, 45, 52, 56, 58, 61, 62, 64, 67, 74, 75, 76, 77, 80, 81, 82, 83], "grant": [6, 11], "mod_autoindex": 6, "loadmodul": 6, "autoindex_modul": 6, "so": [6, 7, 11, 18, 25, 28, 34, 35, 37, 60, 62, 64, 71, 83, 84], "includ": [6, 7, 8, 9, 16, 17, 33, 36, 52, 53, 60, 61, 64, 77, 89], "autoindex": 6, "allow": [6, 10, 17, 18, 19, 22, 29, 33, 45, 52, 64], "access": [6, 9, 10, 11, 12, 17, 20, 21, 33, 34, 35, 36, 37, 38, 39, 45, 49, 56, 60, 61, 64, 71, 74, 75], "On": [6, 7, 16, 19, 40, 60, 61, 67], "fresh": 6, "empti": [6, 14, 36, 38, 45, 49, 58, 64, 77, 83, 89], "shown": [6, 9, 10, 11, 16, 17, 21, 29, 60, 64, 78, 89], "Be": [6, 11, 17, 52, 67], "awar": [6, 11, 52, 64], "quit": [6, 7, 11, 25, 60], "differ": [6, 11, 12, 17, 19, 25, 28, 40, 45, 46, 50, 53, 58, 60, 64, 84, 87, 89], "standard": [6, 11, 25, 28, 52, 89], "detail": [6, 10, 11, 17, 18, 28, 29, 45, 57, 64], "look": [6, 10, 11, 26, 27, 34, 40, 61, 63], "share": [6, 11, 17, 40, 45, 47, 52, 53, 56, 60, 61, 84, 89], "gz": [6, 11], "know": [6, 11, 17, 50, 60, 64, 69, 71], "apache2ctl": 6, "control": [6, 11, 61, 71], "interfac": [6, 10, 16, 17, 19, 21, 23, 45, 46, 49], "a2enmod": 6, "a2dismod": 6, "switch": [6, 24, 25, 34, 36, 37, 41], "off": [6, 8, 10, 21, 25, 89], "a2enconf": 6, "a2disconf": 6, "a2ensit": 6, "a2dissit": 6, "load": [6, 10, 11, 14, 19, 23, 26, 29, 30, 44, 61, 67, 68, 75, 89], "un": [6, 10], "comment": [6, 10, 24, 28, 34, 35, 36, 37, 46, 61, 64, 85], "correspond": [6, 11, 40, 77], "except": [6, 14, 25, 28, 29, 64, 70, 72, 83], "ssl": [6, 10, 19, 28], "proxy_http": 6, "proxy_uwsgi": 6, "ssl_modul": 6, "mod_ssl": 6, "headers_modul": 6, "mod_head": 6, "proxy_modul": 6, "mod_proxi": 6, "proxy_http_modul": 6, "mod_proxy_http": 6, "proxy_uwsgi_modul": 6, "mod_proxy_uwsgi": 6, "save": [6, 7, 10, 24, 45, 68, 89], "avail": [6, 7, 8, 10, 11, 12, 18, 21, 33, 37, 41, 47, 51, 53, 61, 65, 71, 90], "folder": [6, 8, 9, 11, 24, 28, 29, 60, 61, 64, 66, 76, 88, 89], "add": [6, 7, 8, 9, 10, 11, 17, 19, 25, 26, 27, 34, 35, 36, 44, 57, 61, 62, 64, 72, 74, 81, 83, 84], "includeopt": 6, "two": [6, 7, 8, 11, 14, 17, 19, 25, 41, 64, 65], "one": [6, 8, 10, 11, 14, 16, 17, 19, 25, 29, 30, 36, 45, 46, 50, 53, 60, 61, 62, 64, 68, 70, 71, 73, 74, 77, 83, 89], "mkdir": [6, 7, 8, 10], "p": [6, 7, 8, 10, 11, 58, 83, 89], "place": [6, 8, 11, 25, 28, 29, 34, 36, 64, 69, 71], "symlink": [6, 8], "ln": [6, 8, 11, 60], "don": [6, 7, 10, 14, 20, 22, 23, 24, 25, 28, 40, 58, 63, 64, 67, 70, 89], "t": [6, 7, 10, 11, 14, 20, 22, 23, 24, 25, 28, 30, 36, 40, 42, 58, 62, 63, 64, 67, 70, 83, 84, 85, 89], "old": [6, 7, 24, 60, 63, 67], "mod_uwsgi": 6, "anymor": [6, 7], "incom": [6, 74], "need": [6, 7, 8, 9, 10, 11, 17, 20, 23, 24, 25, 27, 32, 34, 36, 37, 43, 44, 45, 50, 53, 58, 60, 61, 62, 63, 64, 65, 69, 70, 71, 74, 87, 89], "proxypreservehost": 6, "pass": [6, 25, 26, 45, 65, 74, 77, 83], "what": [6, 8, 11, 17, 24, 34, 40, 44, 50, 57, 60, 61, 64, 77, 80, 84, 89], "commun": [6, 8, 10, 11, 19, 22, 37, 46, 70, 84, 85], "upstream": [6, 8, 11], "own": [6, 7, 9, 10, 15, 18, 25, 34, 36, 37, 45, 60, 61, 70, 86], "code": [6, 10, 11, 17, 21, 23, 24, 28, 29, 32, 40, 43, 44, 45, 47, 50, 53, 58, 59, 60, 61, 63, 65, 67, 70, 71, 75, 80, 83, 84, 85, 89], "utf": [6, 11, 64, 89], "8": [6, 10, 11, 40, 61, 64, 89], "lib": [6, 11, 56, 85], "setenvif_modul": 6, "mod_setenvif": 6, "setenvif": 6, "request_uri": 6, "dontlog": 6, "customlog": 6, "null": [6, 8, 19], "combin": [6, 45, 80], "env": [6, 11, 22, 59, 60, 63, 89, 90], "order": [6, 10, 19, 25, 32, 34, 35, 36, 42, 64, 74], "deni": [6, 10, 11, 21, 75], "fd00": 6, "192": [6, 19, 74], "168": [6, 19, 58, 74], "16": [6, 10, 43, 50, 61, 74], "fe80": [6, 10, 19, 74], "10": [6, 10, 17, 19, 28, 35, 50, 64, 74, 85], "trail": [6, 10, 23, 83], "slash": [6, 10, 23, 83], "redirectmatch": 6, "308": 6, "proxypass": 6, "ud": 6, "flaskfix": [6, 8], "requesthead": [6, 74], "scheme": [6, 8, 45], "request_schem": 6, "real": [6, 8, 74], "remote_addr": [6, 8], "append": 6, "serv": [6, 8, 11, 25, 61, 71, 89], "alia": [6, 8], "src": [6, 8, 10, 11, 45, 60, 61, 90], "8888": [6, 8, 10, 22, 60, 90], "restart": [6, 7, 8, 11, 60], "servic": [6, 8, 9, 10, 11, 22, 44, 45, 60, 61, 64, 65, 68, 70, 71, 89, 90], "touch": [6, 8, 11, 17], "ini": [6, 7, 8, 11, 24, 60, 90], "privaci": [6, 7, 8, 10, 11, 18, 23, 48, 54, 59, 64, 67, 70, 74, 84, 87], "entir": [6, 10, 80], "virtual": [6, 34, 36, 37], "doe": [6, 9, 11, 17, 24, 28, 29, 36, 37, 38, 39, 40, 41, 42, 44, 45, 48, 49, 50, 53, 57, 60, 64, 65, 70, 80, 82, 83, 86, 89], "path": [6, 10, 14, 19, 20, 24, 33, 60, 61, 64, 83, 84, 89], "compon": [6, 48, 62, 74], "root": [6, 9, 60, 61, 89], "dockerhub": 7, "dockerfil": 7, "overview": [7, 17, 31, 58, 59, 61, 65, 68, 85], "cheat": 7, "sheet": 7, "alpin": 7, "dash": 7, "intend": [7, 29, 43], "well": [7, 48, 50, 53, 64, 70, 84], "rest": [7, 17, 43, 45, 47, 48, 59, 61, 70], "articl": [7, 28, 38, 53, 56, 58, 59, 60, 76, 80], "those": [7, 37, 43, 53, 60, 61, 68, 71], "who": [7, 11, 71], "sourc": [7, 10, 19, 26, 27, 29, 30, 31, 33, 34, 36, 38, 39, 40, 41, 43, 44, 45, 47, 50, 52, 53, 55, 56, 57, 58, 60, 61, 63, 64, 67, 68, 70, 71, 73, 74, 75, 76, 77, 80, 81, 82, 83, 84, 89], "caddi": [7, 9], "protect": [7, 10, 22, 24, 25, 70, 74, 87], "against": [7, 10], "bot": [7, 10, 22, 24, 50, 70, 72], "cach": [7, 10, 11, 23, 24, 44, 50, 76, 80, 83], "bust": [7, 10, 11, 23, 24], "bandwith": [7, 10, 24], "plan": 7, "yourself": [7, 25, 84], "sure": [7, 60], "forget": [7, 10, 20, 22, 24, 89], "group": [7, 11, 17, 20, 61, 64, 71], "out": [7, 25, 26, 27, 34, 35, 36, 43, 44, 61, 62, 64], "back": [7, 60, 63, 80], "membership": 7, "evalu": [7, 44, 53, 64, 74, 83, 89], "usermod": 7, "g": [7, 10, 16, 25, 28, 35, 36, 41, 42, 44, 45, 56, 60, 61, 63, 64, 68, 71, 74, 76, 77, 87, 89, 90], "rm": [7, 11, 83], "automat": [7, 10, 11, 23, 53, 64, 77], "clean": [7, 59], "exit": [7, 10, 12, 77, 78], "detach": 7, "v": [7, 10, 11, 25], "mount": [7, 49], "volum": [7, 28], "easi": [7, 60, 70], "pull": [7, 25, 61, 63, 64, 69], "deploi": [7, 22, 23, 61, 64], "my": [7, 17, 26, 27, 35, 45, 53, 60, 61, 64, 83, 86], "cd": [7, 9, 10, 60, 61], "export": [7, 10, 49, 60, 89], "pwd": [7, 60, 89], "e": [7, 10, 16, 25, 28, 35, 36, 41, 44, 45, 50, 56, 60, 61, 63, 64, 71, 74, 76, 77, 83, 87, 89, 90], "2f998": 7, "id": [7, 10, 28, 43, 45, 50, 52, 61, 83], "visit": [7, 10, 35, 50, 64, 71, 84], "xdg": [7, 10, 61], "modifi": [7, 10, 11, 17, 61, 63, 64, 87], "accord": [7, 11, 14, 32, 77], "l": [7, 20, 33, 44, 60, 61, 64, 89], "flag": [7, 68, 77], "stop": [7, 10, 11, 28, 43, 58, 60, 61, 62, 75, 80, 89], "stope": 7, "rid": [7, 61, 89], "2f998d725993": 7, "sbin": [7, 11], "tini": 7, "7": [7, 10, 58, 85], "minut": [7, 74], "ago": 7, "remov": [7, 10, 11, 12, 14, 17, 21, 29, 53, 55, 60, 61, 62, 67, 71, 80, 84, 89, 90], "item": [7, 11, 27, 28, 29, 52, 57, 58, 64, 72, 80], "won": [7, 11, 40, 67], "prune": 7, "aq": 7, "system": [7, 10, 11, 25, 36, 53, 60, 61, 64, 89], "housekeep": 7, "rmi": 7, "f": [7, 60, 64, 68, 83, 89], "drop": [7, 27, 57, 63, 80, 89], "A": [7, 16, 17, 19, 20, 24, 25, 28, 29, 33, 40, 45, 47, 53, 62, 64, 68, 73, 74, 76, 77, 78, 79, 80, 82, 83, 89], "tale": 7, "bash": [7, 10, 34, 36, 37, 60, 61, 62, 64, 89, 90], "mani": [7, 10, 21, 36, 46, 74, 75], "other": [7, 9, 16, 17, 19, 25, 28, 32, 38, 40, 42, 45, 50, 53, 56, 61, 64, 71, 74, 83, 84], "meant": [7, 21, 64], "posix": 7, "compliant": 7, "entrypoint": 7, "exec": 7, "It": [7, 14, 17, 25, 28, 29, 35, 36, 42, 44, 52, 56, 64, 65, 71, 78, 84, 89], "possibl": [7, 14, 17, 28, 33, 38, 45, 53, 58, 60, 64, 65], "clone": [7, 9, 10, 60, 61, 63, 69, 90], "github": [7, 9, 10, 15, 54, 60, 61, 62, 63, 64, 65, 69, 85, 89, 90], "successfulli": [7, 57, 61], "built": [7, 25, 28, 54], "49586c016434": 7, "tag": [7, 10, 22, 28, 41, 50, 53, 54, 57, 68, 77], "latest": [7, 10, 11, 19, 61], "209": 7, "9c823800": 7, "dirti": 7, "repositori": [7, 9, 60, 69, 89], "size": [7, 11, 24, 28, 42, 57, 83], "13": [7, 11, 61, 89], "308mb": 7, "3": [7, 10, 17, 36, 48, 61, 64, 68, 74, 80, 83, 85, 89], "6dbb9cc54074": 7, "week": [7, 28, 40, 45, 58, 67], "5": [7, 10, 17, 19, 21, 36, 50, 54, 64, 68, 74, 80, 83, 85], "61mb": 7, "interact": [7, 10, 88, 89], "defin": [7, 14, 17, 19, 23, 29, 34, 36, 44, 61, 62, 64, 89], "help": [7, 12, 20, 25, 45, 59, 70, 84, 88], "dry": 7, "alwai": [7, 8, 19, 24, 25, 28, 42, 45, 71, 89], "renam": [7, 65], "suffix": [7, 10, 19], "copi": [7, 10, 29, 64, 69, 89], "environ": [7, 10, 14, 22, 25, 34, 36, 37, 59, 60, 63, 70, 77, 89, 90], "variabl": [7, 14, 19, 28, 64], "morty_url": 7, "result_proxi": [7, 10], "morty_kei": 7, "kei": [7, 10, 12, 17, 28, 29, 33, 34, 36, 45, 52, 67, 68, 69, 74, 77, 80], "bind": [7, 22, 36], "tcp": 7, "entri": [7, 26, 27, 59, 64], "point": [7, 8, 10, 11, 57, 59, 60, 63, 64, 71], "beginn": 8, "guid": [8, 35, 64], "mainlin": 8, "webserv": 8, "come": [8, 17, 25, 40, 70, 71, 74, 84, 87, 89], "80": 8, "server_nam": 8, "good": [8, 60, 77], "resourc": [8, 35, 38], "uwsgi_pass": 8, "uwsgi_param": 8, "http_host": 8, "http_connect": 8, "http_x_scheme": 8, "http_x_script_nam": 8, "http_x_real_ip": 8, "http_x_forwarded_for": 8, "proxy_add_x_forwarded_for": 8, "proxy_pass": 8, "proxy_set_head": 8, "proxy_buff": 8, "proxy_request_buff": 8, "proxy_buffer_s": 8, "8k": 8, "app": [8, 11, 42, 45, 53, 60, 61, 64, 74, 76, 90], "access_log": 8, "error_log": 8, "avoid": [9, 12, 19, 33, 50, 69, 74], "unwant": 9, "effect": [9, 11, 25, 64], "befor": [9, 10, 11, 19, 24, 25, 34, 36, 37, 60, 61, 62, 63, 64, 71, 74, 89], "procedur": [9, 24, 89], "jump": [9, 10, 60, 61, 87], "readabl": 9, "altern": [9, 14, 17, 23, 35, 48, 61, 63, 64, 89], "fork": [9, 25, 60, 70, 83, 84], "download": [9, 10, 22, 36, 49, 54, 57, 60, 61, 64], "sudoer": 9, "take": [9, 11, 19, 24, 25, 40, 57, 69, 74, 80, 84, 89], "account": [9, 10, 19, 20, 24, 60], "case": [9, 10, 17, 19, 21, 24, 35, 45, 48, 50, 53, 58, 60, 71, 83, 90], "fine": [9, 10], "regard": [9, 84], "whatev": [9, 89], "implement": [9, 11, 17, 24, 25, 26, 27, 28, 29, 39, 40, 45, 50, 53, 57, 59, 61, 62, 63, 70, 73, 74, 77, 79, 80, 83, 89], "useradd": [10, 61], "home": [10, 29, 50, 60, 61, 69, 74], "dir": 10, "respect": [10, 25, 71], "metasearch": [10, 28, 38, 70, 84], "chown": 10, "r": [10, 45, 85, 89], "u": [10, 17, 21, 28, 40, 41, 45, 50, 60, 61, 63, 64, 70, 77, 84, 85], "same": [10, 28, 36, 47, 53, 60, 64, 83, 84, 89], "virtualenv": [10, 11, 60, 61, 90], "m": [10, 32, 45, 64, 83], "pyenv": [10, 11, 34, 36, 37, 60, 61, 62, 77, 90], "echo": [10, 89], "profil": [10, 61, 70, 71, 84], "session": [10, 74, 77], "wa": [10, 17, 24, 25, 28, 53, 57, 60, 61, 67, 79, 80, 83, 84, 85], "version": [10, 12, 28, 32, 61, 64, 68, 77], "boilerpl": 10, "setuptool": 10, "wheel": 10, "pyyaml": 10, "tree": [10, 60, 61, 63], "second": [10, 19, 21, 50, 64, 77, 80], "termin": [10, 60, 67], "leav": [10, 21, 89], "initi": [10, 11, 14, 17, 26, 27, 29, 30, 33, 34, 35, 36, 52, 60, 64, 67, 77, 79, 80, 89], "templat": [10, 11, 31, 34, 36, 37, 43, 58, 59, 60, 61, 62, 74], "tab": [10, 16, 17, 23, 59, 61, 65, 86, 89], "lock": [10, 83], "enabled_plugin": [10, 65, 78], "hash": [10, 11, 12, 74, 80, 87], "select": [10, 12, 17, 28, 29, 34, 35, 36, 37, 40, 43, 44, 45, 48, 50, 58, 61, 63, 64, 68, 71, 73, 77, 86, 89], "self": [10, 12, 29, 45, 70, 89, 90], "inform": [10, 12, 17, 19, 21, 25, 28, 52, 60, 61, 70, 76, 78, 81, 84, 89], "tracker": [10, 12, 15, 42, 52], "ahmia": [10, 68], "blacklist": [10, 68], "hostnam": [10, 12, 89], "replac": [10, 12, 24, 58, 60, 61, 62, 64, 69, 80], "hostname_replac": [10, 65], "infinit": [10, 80], "scroll": [10, 23], "doi": [10, 12, 28], "only_show_green_result": [10, 62], "searxng_debug": [10, 18, 61], "privacypolicy_url": [10, 18], "donat": [10, 18], "written": [10, 18, 45, 60], "info": [10, 18, 19, 60, 61, 70, 72, 89], "md": [10, 18, 34], "donation_url": [10, 18], "mailto": [10, 18], "contact": [10, 18, 19, 84], "contact_url": [10, 18], "record": [10, 18, 23, 28, 58, 64], "stat": [10, 18, 64, 84], "enable_metr": [10, 18], "new_issue_url": 10, "docs_url": [10, 15], "public_inst": [10, 15], "space": [10, 15, 22, 61, 70, 74], "wiki_url": [10, 15], "issue_url": [10, 15], "moder": [10, 21, 28, 58], "strict": [10, 17, 21, 28, 58], "backend": [10, 21, 32, 48], "dbpedia": [10, 21, 65], "yandex": 10, "seznam": [10, 85], "startpag": [10, 21, 31, 65, 85], "swisscow": [10, 21, 65], "wikipedia": [10, 21, 32, 65, 68, 85, 87], "blank": [10, 21, 23, 64], "turn": [10, 21, 25, 43, 57, 67, 80], "minimun": 10, "charact": [10, 25, 33, 45, 53, 64], "autocomplete_min": 10, "4": [10, 28, 40, 61, 64, 74, 80, 85], "detect": [10, 21, 23, 70, 72, 83], "default_lang": [10, 17, 21], "auto": [10, 23, 60, 64, 89], "IT": [10, 21, 45], "fr": [10, 17, 21, 23, 29, 32, 41, 45, 58, 77, 83, 85, 87], "BE": [10, 21, 29, 45, 77], "ban": [10, 21, 52, 71], "time": [10, 19, 21, 25, 28, 29, 32, 36, 41, 42, 45, 50, 58, 60, 61, 64, 65, 71, 74, 75, 80, 82, 83, 85, 89], "after": [10, 21, 53, 61, 62, 64, 68, 89], "error": [10, 11, 17, 18, 19, 21, 24, 28, 29, 32, 33, 37, 57, 58, 64, 74, 75, 83], "ban_time_on_fail": [10, 21], "max": [10, 21, 28, 80], "max_ban_time_on_fail": [10, 21], "120": [10, 21, 25], "suspended_tim": [10, 21, 75], "suspens": [10, 21], "402": [10, 21], "403": [10, 17, 21], "searxengineaccessdeni": [10, 21, 75], "86400": [10, 21, 75], "searxenginecaptcha": [10, 21, 75], "too": [10, 18, 21, 40, 57, 62, 64, 71, 75], "429": [10, 17, 21], "searxenginetoomanyrequest": [10, 21, 75], "3600": [10, 21, 74], "cloudflar": [10, 21], "cf_searxenginecaptcha": [10, 21], "1296000": [10, 21], "cf_searxengineaccessdeni": [10, 21], "recaptcha": 10, "recaptcha_searxenginecaptcha": [10, 21], "604800": [10, 21], "format": [10, 21, 32, 37, 45, 51, 57, 59, 63, 64, 65, 74], "lower": [10, 21, 80], "csv": [10, 21, 65], "json": [10, 21, 26, 29, 32, 34, 35, 45, 47, 48, 51, 57, 61, 65, 67, 68, 83], "rss": [10, 21, 42, 65], "chang": [10, 11, 14, 15, 19, 22, 24, 25, 42, 60, 61, 62, 63, 64, 68, 84], "rebuild": [10, 22, 25, 61], "buildenv": [10, 22, 59], "searxng_port": [10, 22, 60, 61, 90], "searxng_bind_address": [10, 22, 60, 61, 90], "rate": [10, 22, 53, 72], "number": [10, 19, 22, 28, 36, 40, 45, 48, 50, 52, 57, 58, 64, 65, 68, 74, 80, 82, 83, 89], "through": [10, 17, 64, 65, 71, 74], "http_protocol_vers": 10, "queri": [10, 12, 17, 21, 23, 26, 27, 28, 32, 33, 34, 35, 36, 37, 40, 41, 44, 45, 47, 50, 52, 53, 55, 57, 58, 64, 65, 67, 68, 71, 75, 78, 80, 81, 82, 83, 84, 85, 86], "more": [10, 11, 17, 18, 19, 25, 28, 29, 36, 43, 45, 47, 50, 52, 61, 64, 65, 70, 71, 74, 80, 83, 84, 87, 89], "thei": [10, 11, 17, 40, 64, 71, 74, 85], "histori": 10, "mai": [10, 11, 19, 44, 52, 57, 60, 64, 68, 74, 83, 84, 89], "caus": 10, "default_http_head": [10, 22], "content": [10, 22, 28, 35, 36, 38, 43, 49, 52, 53, 58, 69, 71, 76], "nosniff": [10, 22], "xss": [10, 22], "noopen": [10, 22], "robot": [10, 22, 45, 61], "noindex": [10, 22], "nofollow": [10, 22], "referr": [10, 22], "polici": [10, 18, 22, 89], "readthedoc": [10, 11], "io": [10, 11, 34, 61, 64, 83], "stabl": [10, 17, 42, 61], "client": [10, 46, 47, 60, 67, 69, 74, 79, 80], "from_url": [10, 20], "custom": [10, 19, 25, 29, 35, 39, 41, 53, 64, 69, 70, 72, 87], "didn": 10, "static_path": 10, "templates_path": 10, "query_in_titl": [10, 23], "titl": [10, 23, 28, 32, 33, 36, 39, 43, 53, 58, 76], "decreas": [10, 23], "sinc": [10, 11, 23, 40, 41, 44, 46, 64, 74, 89], "infinite_scrol": [10, 23, 48], "next": [10, 23, 48, 64], "bottom": [10, 23, 68], "current": [10, 17, 23, 25, 28, 29, 45, 51, 61, 74, 80], "theme": [10, 23, 34, 36, 59, 63, 65], "center": [10, 23], "center_align": [10, 23], "prefix": [10, 17, 23, 45, 74, 80, 87], "internet": [10, 23, 37, 60, 70, 84], "archiv": [10, 23, 31, 61, 74, 85], "forgett": [10, 23, 25], "cache_url": [10, 23], "webcach": [10, 23], "googleusercont": [10, 23], "results_on_new_tab": [10, 23, 65], "theme_arg": [10, 23], "style": [10, 23, 61, 63, 68, 83], "light": [10, 23], "dark": [10, 23], "simple_styl": [10, 23], "arbitrari": [10, 33, 34], "proxif": 10, "extern": [10, 28, 37, 59, 64, 65, 71, 86], "asciimoo": 10, "morti": [10, 24], "uncom": [10, 19], "base64": 10, "encod": [10, 11, 29, 45, 50, 64, 74], "binari": [10, 36, 61], "notat": [10, 41], "note": [10, 13, 25, 35, 60, 61, 65, 87], "commit": [10, 24, 47, 60, 61, 63, 69], "af77ec3": 10, "accept": [10, 17, 29, 37, 40, 44, 45, 50, 53, 74, 87, 89], "3000": 10, "string": [10, 12, 17, 26, 28, 29, 32, 38, 40, 44, 45, 58, 65, 68, 77, 80, 83, 87], "yaml": [10, 17, 20, 29, 30, 61], "your_morty_proxy_kei": 10, "button": 10, "each": [10, 11, 14, 19, 24, 28, 29, 33, 35, 37, 50, 53, 58, 60, 62, 64, 65, 68, 69, 80, 83, 89], "proxify_result": 10, "timeout": [10, 17, 19, 28, 29, 64, 85, 88], "overrid": [10, 14, 19], "request_timeout": [10, 17, 19], "maximum": [10, 19, 25, 28, 58, 74], "max_request_timeout": [10, 19], "searx_userag": [10, 72, 83], "could": [10, 11, 28, 44, 57, 64, 80], "email": [10, 19], "address": [10, 12, 17, 18, 19, 22, 28, 39, 53, 71, 78], "administr": [10, 17, 19, 33, 35, 37, 65, 70, 71, 88], "useragent_suffix": [10, 19], "concurr": 10, "establish": 10, "pool_connect": [10, 17, 19], "100": [10, 17, 19, 57, 70, 84, 85], "pool": [10, 17, 19], "aliv": [10, 19], "pool_maxs": [10, 17, 19], "20": [10, 34, 44, 61, 74, 83, 89], "httpx": [10, 19, 52], "http2": [10, 19], "enable_http2": [10, 17, 19], "certif": [10, 19], "advanc": [10, 19, 37], "verif": [10, 19, 29], "compat": [10, 19, 74], "verifi": [10, 19, 28], "mitmproxi": [10, 19], "ca": [10, 19, 29, 41, 45, 77, 83], "cert": [10, 19], "cer": [10, 19], "proxyq": [10, 19], "proxy1": [10, 17, 19], "proxy2": [10, 17, 19], "using_tor_proxi": [10, 17, 19, 29, 30, 31], "taken": [10, 17, 19, 58, 64, 74], "extra_proxy_timeout": [10, 19], "than": [10, 19, 35, 42, 45, 50, 53, 64, 70, 83, 84], "source_ip": [10, 19], "126": [10, 19], "plugin1": 10, "plugin2": 10, "deactiv": [10, 18], "noth": [10, 69], "tor": [10, 12, 17, 19, 29, 30, 70, 71, 72], "detriment": 10, "expect": [10, 17, 25, 33], "autodetect": 10, "minim": [10, 53, 61, 63, 64, 68], "cp": 10, "sed": [10, 64], "rand": 10, "hex": 10, "webapp": [10, 11, 22, 61], "searxng_settings_path": [10, 11, 14, 60], "insid": [10, 33, 60, 75, 89], "verbos": 10, "head": [10, 61], "insecur": 10, "try": [10, 11, 24, 50, 71], "tcp_nodelai": 10, "agent": [10, 12, 19, 28, 74, 83, 87], "68": 10, "mark": [10, 64], "bundl": 10, "multius": 10, "assum": [10, 64], "close": [10, 26, 27, 57, 74], "bodi": [10, 25, 64], "200": [10, 61], "ok": [10, 24, 61], "everyth": [10, 64, 71], "hit": 10, "ctrl": [10, 60], "c": [10, 11, 45, 55, 60, 61, 62, 64, 68, 83], "enter": [10, 60], "twice": 10, "At": [10, 35, 36, 45, 64, 89], "demon": 10, "systemd": [11, 61], "unit": [11, 25, 44, 61, 68, 77], "emperor": 11, "vari": 11, "project": [11, 56, 60, 61, 64, 70, 84], "itself": [11, 25, 49, 74], "One": [11, 32], "per": [11, 25, 28, 64, 68], "dedic": [11, 60, 64], "archlinux": [11, 17, 39, 85, 89], "execstart": 11, "known": [11, 53, 61, 74, 77, 84], "common": [11, 38, 41, 45, 50, 53, 56, 61, 70, 71], "fit": [11, 17, 29, 35, 44, 45, 48, 53, 62, 68, 74, 77], "larg": [11, 89, 90], "rang": [11, 28, 29, 41, 45, 58, 64, 65, 74, 80, 82, 85], "multi": 11, "monitor": [11, 60, 69], "specif": [11, 17, 19, 23, 28, 29, 33, 38, 42, 43, 50, 52, 55, 56], "event": 11, "scan": 11, "vassal": 11, "ad": [11, 16, 17, 20, 29, 35, 37, 44, 50, 63, 64, 69, 71, 73, 74, 77, 84], "timestamp": [11, 32], "reload": 11, "edit": [11, 25, 53, 60, 61, 64, 71], "mostli": [11, 53, 64], "offer": [11, 40, 42, 44, 45, 49, 51, 58, 84], "even": [11, 17, 20, 45, 60, 89], "wai": [11, 14, 25, 28, 35, 42, 52, 64, 67, 71, 74], "both": [11, 25, 65, 71, 74, 83], "anoth": [11, 17, 28, 32, 60, 64, 68, 89], "interpret": [11, 45, 61, 64], "python2": 11, "while": [11, 26, 41, 60, 61, 63, 64, 75, 80, 83, 84], "worth": 11, "complet": [11, 24, 25, 31, 40, 45, 48, 60, 61, 65, 89, 90], "approach": 11, "familiar": [11, 61], "similar": [11, 14, 36, 64, 77], "thing": [11, 64], "symbol": 11, "recogn": 11, "init": [11, 26, 27, 37, 38, 52, 56, 60, 77, 89], "daemon": 11, "sighup": 11, "signal": [11, 60], "forc": [11, 61], "sigterm": 11, "statu": [11, 17, 58, 63, 75, 77, 90], "must": [11, 18, 19, 25, 28, 30, 33, 34, 36, 49, 62, 64, 74, 83, 84], "exactli": [11, 64], "argument": [11, 12, 25, 32, 38, 39, 45, 50, 56, 64, 67, 74, 77, 80, 89], "confnam": 11, "systemctl_skip_redirect": 11, "hello": 11, "xml": [11, 42, 45, 52, 64, 83], "lsb": 11, "dai": [11, 28, 40, 45, 58, 65, 67, 75], "bug": [11, 61, 70], "cgi": 11, "bugreport": [11, 25], "833067": 11, "0pointer": 11, "blog": [11, 60, 64], "As": [11, 14, 17, 24, 44, 50, 71, 74, 89], "uid": [11, 60], "gid": 11, "ignor": [11, 26, 43, 44, 45, 74, 77], "lc_all": 11, "chdir": [11, 60], "right": [11, 17, 20, 62, 63, 64, 68, 77], "chmod": [11, 89], "666": 11, "singl": [11, 71, 89], "master": [11, 24, 25, 60, 61, 64, 69, 90], "worker": 11, "instead": [11, 17, 23, 35, 58, 64, 68, 80], "lazi": 11, "By": [11, 17, 19, 34, 36, 40, 43, 44, 46, 49, 54, 74, 75, 77, 83, 88, 89], "gil": 11, "mean": [11, 21, 37, 40, 60, 64, 71, 80], "thread": [11, 83], "rememb": [11, 25, 60], "them": [11, 17, 34, 35, 36, 60, 64], "multithread": 11, "strang": 11, "behaviour": [11, 25, 62], "perform": [11, 12, 28, 32, 37, 57, 74, 80, 87], "reason": [11, 30, 64, 80], "wsgi": 11, "modul": [11, 26, 27, 29, 30, 53, 62, 64, 67, 73, 75, 85], "pythonhom": 11, "glob": 11, "pythonpath": [11, 60], "speak": 11, "buffer": [11, 24], "8192": 11, "expir": [11, 24, 74, 80], "year": [11, 28, 45, 56, 58, 65, 67], "31557600": 11, "gzip": [11, 74], "offload": 11, "k": 11, "cache2": 11, "searxngcach": 11, "2000": [11, 64], "blocksiz": 11, "4096": 11, "bitmap": 11, "logger": 11, "owner": [11, 84, 89], "somewhat": 11, "unusu": 11, "consider": 11, "initgroup": 11, "branch": [11, 24, 25, 61, 69], "2099": 11, "featur": [11, 17, 24, 25, 44, 48, 64, 71, 85], "752": 11, "ha": [11, 14, 17, 24, 25, 28, 33, 37, 38, 40, 42, 45, 46, 48, 52, 54, 56, 60, 64, 69, 70, 71, 75, 76, 77, 80, 89], "been": [11, 24, 25, 60, 69, 70, 80, 89], "merg": [11, 14, 24, 60, 68, 69], "oct": 11, "2014": 11, "had": 11, "never": [11, 29, 40, 60, 84], "releas": [11, 24, 38, 54, 61, 89], "last": [11, 28, 40, 45, 61, 63, 64], "major": 11, "dec": 11, "2013": 11, "bugfix": 11, "2425uwsgi": 11, "shorten": 11, "miss": [11, 37, 64, 74, 75], "permiss": 11, "redisdb": [11, 20, 24, 79], "993": 11, "fail": [11, 30, 60, 61, 63], "aef": 11, "grep": [11, 60, 61, 89], "93": [11, 74], "92": 11, "12": [11, 61], "43": 11, "00": [11, 89], "186": 11, "44": 11, "01": 11, "pid": 11, "unset": [11, 21, 50, 74, 77], "cat": [11, 89], "proc": 11, "fdsize": 11, "128": 11, "descript": [12, 20, 25, 28, 36, 37, 40, 43, 45, 50, 62, 64, 68, 72, 77, 78, 83], "j": [12, 44, 46, 59, 63, 68], "css": [12, 61, 63, 74], "convert": [12, 28, 53, 64, 83, 87], "digest": [12, 87], "paywal": 12, "redirect": [12, 19, 28, 45, 53, 58, 61], "immedi": [12, 80], "multipl": [12, 17, 19, 28, 29, 53, 61, 64, 71, 85], "javascript": [12, 63, 68, 83], "node": [12, 59, 63, 68, 78, 83], "torproject": [12, 78], "return": [12, 17, 26, 28, 29, 30, 32, 33, 36, 37, 43, 44, 45, 52, 53, 58, 62, 64, 67, 68, 71, 74, 75, 76, 77, 80, 82, 83], "navig": [12, 84], "press": [12, 60], "main": 12, "privat": [13, 29, 33, 34, 35, 36, 52, 70, 74], "token": [13, 14, 29, 33, 34, 35, 36, 37, 74], "multilingu": 13, "full": [14, 17, 28, 35, 40, 45, 49, 64], "simplifi": [14, 53, 90], "reli": [14, 36], "actual": [14, 32, 50, 68], "bing": [14, 31, 64, 67, 85], "ecretvalu": 14, "doesn": [14, 22, 23, 30, 36, 42, 62, 83, 84], "keep_onli": 14, "still": [16, 34, 35, 36, 44, 53], "syntax": [16, 65, 70, 75, 83, 85, 86], "video": [16, 41, 42, 48, 50, 67, 86], "music": [16, 48, 67, 86], "scienc": [16, 67, 86], "under": [16, 17, 28, 31, 43, 60, 89], "call": [16, 17, 29, 40, 44, 45, 57, 60, 61, 63, 64, 68, 74, 80, 81, 83, 85, 89], "fledg": 17, "dummi": 17, "few": [17, 24, 25, 28, 43, 87], "pretti": 17, "demo": [17, 31, 37, 80], "send_accept_language_head": [17, 29, 40, 44, 50, 53], "api_kei": [17, 28, 52], "apikei": [17, 69], "en_u": [17, 28, 43, 44], "secret": [17, 29, 80], "weight": [17, 64, 85], "display_error_messag": [17, 28, 29], "wikidata_id": [17, 29], "q306656": [17, 29], "official_api_document": [17, 29], "use_official_api": [17, 29], "require_api_kei": [17, 29], "overwrit": 17, "retri": [17, 19], "max_connect": [17, 19], "max_keepalive_connect": [17, 19], "keepalive_expiri": [17, 19], "password": [17, 20, 34, 35, 36], "proxy3": 17, "1080": 17, "socks5h": 17, "proxy4": 17, "enable_http": [17, 28, 29, 34, 35], "retry_on_http_error": 17, "404": 17, "across": [17, 29], "handl": [17, 29, 34, 45, 60, 64, 71, 82, 83, 89], "respons": [17, 27, 28, 29, 37, 40, 43, 44, 45, 47, 52, 53, 55, 57, 58, 74], "bang": [17, 29, 64, 68, 85, 86], "bi": [17, 85], "part": [17, 49, 64, 89], "everi": [17, 24, 25, 28, 50, 60, 67, 69, 71, 72, 89], "sever": [17, 53, 60], "region": [17, 29, 40, 44, 45, 53, 68, 77, 83], "deal": [17, 41], "header": [17, 22, 28, 29, 44, 45, 50, 53, 58, 64, 72], "assign": [17, 28, 64, 74, 77, 85], "regardless": [17, 71], "whether": [17, 52, 57, 71, 74], "dictionari": [17, 29, 45, 72, 77, 80, 83], "care": [17, 33, 50, 64, 69, 84, 89], "global": [17, 19, 28, 43, 54, 77, 80], "obtain": 17, "delet": [17, 29, 61, 65, 80, 89], "manual": [17, 24, 29, 61, 64], "inact": [17, 29], "iso": [17, 23, 28, 32, 50, 58, 83], "messag": [17, 18, 25, 28, 29, 57, 63, 64, 69, 70, 72, 75], "ipv4": [17, 19, 74, 89], "local_address": 17, "ipv6": [17, 19, 60, 74, 89], "between": [17, 28, 53, 64, 77], "400": [17, 64], "599": 17, "themselv": [17, 35], "becaus": [17, 25, 28, 42, 71], "expos": [17, 34, 35, 36, 67], "offlin": [17, 28, 33, 64, 72], "Or": [17, 64], "would": [17, 26, 60, 61], "rather": 17, "trust": [17, 70, 71], "friend": [17, 71], "colleagu": 17, "sponsor": [17, 33, 34, 35, 36], "discoveri": [17, 33, 34, 35, 36], "fund": [17, 33, 34, 35, 36], "nlnet": [17, 33, 34, 35, 36], "foundat": [17, 33, 34, 35, 36, 62], "concept": [17, 31], "present": [17, 37], "restrict": [17, 45, 54], "unless": 17, "go": [17, 34, 37, 45, 59, 63, 64, 71, 74, 85], "him": 17, "her": 17, "comma": [17, 33, 65], "separ": [17, 25, 29, 33, 63, 64, 65], "carv": 17, "stone": 17, "provid": [17, 25, 28, 32, 33, 35, 38, 39, 42, 46, 52, 64, 68, 84], "impli": 17, "admin": [17, 37, 60, 64], "necessari": [17, 24, 61], "guidelin": [17, 63], "But": [17, 35, 62, 71, 89], "workaround": 17, "speaker": 17, "directli": [18, 22, 52, 61, 69, 87, 89], "product": [18, 59, 63], "altogeth": 18, "variou": [18, 25, 34, 45, 60, 62], "anonym": [18, 38, 70, 71, 84], "metric": 18, "bigger": 19, "wait": [19, 80], "slow": 19, "consequ": [19, 25], "reactiv": 19, "wish": [19, 34, 36], "round": [19, 25], "robin": 19, "fashion": 19, "60a2": 19, "1691": 19, "e5a2": 19, "ee1f": 19, "ssl_cert_fil": 19, "ssl_cert_dir": 19, "max_redirect": [19, 28], "30": [19, 50, 58, 85], "serxng": 20, "la": [20, 64, 83, 89], "srwxrwx": 20, "write": [20, 53, 68], "given": [20, 31, 64, 80, 83], "usernam": [20, 35, 36], "6379": [20, 34], "rediss": 20, "Then": 20, "manag": [20, 25, 36, 59, 60, 63, 70, 77, 90], "addgrp": [20, 61], "logout": 20, "member": [20, 74, 77], "otherwis": [21, 28, 30, 63, 84, 89], "paramet": [21, 29, 30, 36, 37, 40, 43, 45, 58, 59, 62, 76, 77, 80, 81, 82, 83], "appli": [22, 25, 45, 83, 89], "behind": [22, 48, 74, 84], "cryptographi": 22, "purpos": [22, 28], "being": [22, 23, 70], "memori": [22, 34, 50], "755": 22, "left": [23, 24, 64, 77], "rtl": 23, "screen": 23, "affect": [23, 24], "layout": [23, 34, 36], "min": [23, 28, 83], "width": [23, 57, 64], "tablet": 23, "todai": [23, 57], "pr": [24, 25, 53, 61, 63, 64, 69], "1332": 24, "456": 24, "roll": [24, 89], "opportun": 24, "filtron": 24, "longer": [24, 44, 61, 74, 89], "enough": 24, "sometim": [24, 45], "reconfigur": 24, "uninstal": [24, 61], "consid": [24, 29, 50], "reinstal": 24, "pleas": [24, 25, 35, 36, 61, 63, 64, 65, 87], "extent": 24, "1595": 24, "fix": [24, 25, 41], "increas": 24, "undo": 24, "done": [24, 45, 60, 61, 64, 67, 69, 74, 83, 89], "deprec": 24, "move": [24, 25, 29, 80], "three": [25, 50, 62, 64, 71], "alter": 25, "hack": [25, 63, 67], "lack": 25, "world": [25, 64, 71], "domin": 25, "among": [25, 71], "intent": [25, 74, 89], "wide": 25, "mass": 25, "adopt": 25, "corner": 25, "deserv": 25, "chapter": [25, 64, 90], "uncommon": [25, 45], "unfortun": 25, "born": 25, "extend": [25, 32, 37, 62, 64], "easili": 25, "maxim": 25, "its": [25, 26, 27, 30, 32, 34, 41, 44, 45, 53, 56, 57, 64, 65, 71, 83, 84, 89], "capabl": [25, 35], "reduc": [25, 53, 74, 83], "preserv": [25, 64], "aspect": [25, 60], "plenti": 25, "alreadi": [25, 60, 61, 69, 83], "think": [25, 61, 63, 64], "someth": [25, 33, 44, 61, 63], "weird": 25, "interfer": 25, "submit": [25, 33, 35], "vendor": 25, "misbehav": 25, "feedback": [25, 64], "reconsid": 25, "disrespect": 25, "concern": [25, 33, 84], "fanci": 25, "happi": [25, 63], "structur": [25, 34, 59, 67], "split": 25, "convent": 25, "practic": 25, "gitmoji": 25, "yet": [25, 28, 31, 50, 89], "semant": [25, 85], "patch": [25, 64, 77], "pep8": [25, 61], "length": [25, 64], "cardin": 25, "rule": [25, 29, 53, 77, 89], "logic": 25, "break": [25, 64], "author": [25, 28, 45, 64], "rst": [25, 61, 64], "meaning": [25, 63, 64], "scope": [25, 50], "footer": 25, "quickstart": [25, 59, 61, 70], "weblat": [25, 61, 69, 84], "gh": [25, 45, 69, 85], "sphinx": 25, "much": [25, 64], "easier": 25, "makefil": [25, 59, 60, 63, 70, 89], "dist": [25, 61], "assert": 25, "wysiwyg": 25, "target": [25, 43, 61, 63, 64], "favorit": [25, 61], "8000": 25, "watch": 25, "autobuild": [25, 61], "sphinxopt": 25, "free": [25, 28, 38, 54, 62, 70, 84, 85], "50593": 25, "push": [25, 61, 69], "adjust": [25, 61], "within": [26, 27, 64], "demo_offlin": [26, 37], "engine_set": [26, 27, 33, 37, 38, 52, 56], "request_param": 26, "assembl": [26, 40, 45, 47, 50, 53], "usual": [26, 58, 59, 63, 71, 89], "art": [27, 64], "institut": 27, "chicago": 27, "demo_onlin": 27, "function": [27, 28, 29, 30, 37, 44, 45, 50, 53, 62, 64, 70, 72, 73, 74, 77, 80, 89], "param": [27, 28, 37, 40, 44, 45, 47, 50, 52, 53, 55, 57, 58, 64, 82], "fetch": [27, 29, 38, 39, 40, 41, 43, 44, 45, 47, 50, 53, 55, 56, 68, 78, 90], "artic": [27, 85], "edu": 27, "resp": [27, 40, 45, 47, 52, 55, 57, 58], "pars": [27, 29, 32, 33, 42, 44, 45, 47, 52, 55, 57, 67, 75, 77, 78], "adapt": 28, "store": [28, 29, 34, 35, 41, 42, 44, 71, 74, 84, 85], "tell": [28, 33], "normal": [28, 64, 68, 83], "ones": [28, 35], "matter": [28, 60, 71], "howev": [28, 40, 45, 65], "boolean": [28, 32, 62, 64], "time_range_support": [28, 29, 41, 58, 64], "str": [28, 29, 32, 38, 45, 48, 52, 56, 67, 74, 75, 76, 77, 80, 81, 82, 83], "ref": [28, 60, 62, 68, 89], "bool": [28, 29, 32, 45, 62, 74, 77, 83], "dict": [28, 29, 30, 33, 45, 52, 57, 62, 67, 77, 81, 83], "namespac": [28, 29, 30, 67], "often": [28, 29, 60, 64, 88], "redefin": 28, "lead": 28, "underlin": [28, 30, 61], "veri": [28, 41, 89], "_non_overwritten_glob": 28, "foo": [28, 29, 79, 80], "number_of_result": [28, 54], "int": [28, 64, 75, 80, 81, 83], "countri": [28, 40, 45, 77], "These": [28, 64], "construct": [28, 64], "furthermor": [28, 34, 35, 65, 71], "cooki": [28, 44, 50, 58, 70, 71], "random": [28, 32, 71, 74, 83, 87], "safesearch": [28, 29, 45, 58, 64, 65, 67, 81], "pageno": [28, 58, 65, 67, 81], "pagenumb": 28, "searxng_local": [28, 29, 45, 50, 77], "unspecifi": 28, "from_lang": 28, "to_lang": 28, "amount": [28, 42, 44, 80], "float": [28, 29, 81, 83], "4217": 28, "from_nam": 28, "currenc": [28, 68, 72, 85], "to_nam": 28, "search_url": [28, 43, 57, 58, 83], "ftp": 28, "def": [28, 62, 64, 74, 76, 83], "valid": [28, 36, 37, 65, 74], "allow_redirect": 28, "hard": [28, 64], "soft_max_redirect": [28, 58], "soft": [28, 58, 59], "raise_for_httperror": 28, "rais": [28, 67, 75, 83], "300": 28, "desir": [28, 53], "publishedd": 28, "datetim": [28, 64], "publish": [28, 45], "partli": 28, "img_src": [28, 53, 58], "thumbnail_src": 28, "small": [28, 35, 36, 41, 61, 89, 90], "preview": [28, 53], "thumbnail": [28, 53], "seed": 28, "seeder": [28, 42], "leech": 28, "leecher": [28, 42], "files": [28, 57, 83], "byte": [28, 57, 83], "magnetlink": 28, "torrentfil": 28, "latitud": 28, "decim": 28, "longitud": 28, "boundingbox": 28, "arrai": 28, "lat": 28, "lon": 28, "geojson": 28, "object": [28, 29, 41, 57, 62, 64, 67, 77, 80, 83], "road": 28, "street": 28, "house_numb": 28, "hous": [28, 68], "citi": 28, "postcod": 28, "abstract": [28, 72], "ital": [28, 64], "short": [28, 61], "medium": 28, "book": [28, 38, 56], "editor": 28, "journal": [28, 38, 45, 56], "magazin": [28, 38], "report": [28, 51, 60, 70, 84], "1038": 28, "d41586": 28, "018": 28, "07848": 28, "issn": 28, "1476": 28, "4687": 28, "isbn": 28, "9780201896831": 28, "pdf_url": 28, "html_url": 28, "framework": 29, "long": [29, 61, 71], "goal": 29, "modular": 29, "todo": 29, "loader": [29, 31, 59, 67], "enginelib": [29, 45, 68], "class": [29, 57, 58, 68, 72, 74, 76, 81], "further": [29, 58, 83, 84], "engine_typ": [29, 31, 57, 64], "processor": [29, 31, 37, 57, 64, 70, 72], "fetch_trait": [29, 38, 39, 40, 41, 43, 44, 45, 47, 50, 53, 55, 56, 77], "callabl": 29, "french": [29, 77], "language_support": [29, 64], "belgium": 29, "enginetrait": [29, 38, 39, 40, 41, 43, 44, 45, 47, 50, 53, 55, 56], "represent": [29, 37, 77, 83], "properti": [29, 32, 45, 68, 76, 83], "persist": [29, 68], "enginetraitsmap": [29, 68], "from_data": 29, "factori": 29, "all_local": [29, 44], "data_typ": 29, "typing_extens": 29, "liter": [29, 59], "traits_v1": 29, "iter": [29, 76], "instanti": [29, 80], "dataclass": 29, "classmethod": 29, "engine_trait": [29, 38, 39, 40, 41, 43, 44, 45, 47, 50, 53, 55, 56, 61], "get_languag": 29, "intern": [29, 32, 45, 77, 83], "get_engine_local": [29, 72, 77, 83], "get_region": 29, "is_locale_support": 29, "set_trait": 29, "load_engin": [29, 30, 31], "relat": [29, 36, 44], "egnine_lang": 29, "searxng_lang": 29, "egnine_region": 29, "searxng_region": 29, "sep": [29, 85], "enginetraitsencod": 29, "skipkei": 29, "ensure_ascii": 29, "check_circular": 29, "allow_nan": 29, "sort_kei": 29, "indent": [29, 64, 89], "serializ": [29, 67], "jsonencod": 29, "o": [29, 64, 83, 90], "save_data": 29, "engine_traits_fil": [29, 68], "moduletyp": [29, 30], "posixpath": [29, 74], "runner": [29, 61, 74], "regist": 30, "engine_shortcut": [30, 31], "usag": [30, 60, 61, 64, 67, 76, 84, 89, 90], "is_missing_required_attribut": [30, 31], "attribut": [30, 34, 36, 37, 52, 57, 64], "_": [30, 64], "engine_data": [30, 81], "engine_default_arg": 30, "underscor": [30, 77], "lowercas": 30, "engine_list": 30, "declar": 30, "librari": [31, 36, 38, 59, 70, 72, 83, 85], "trait": [31, 45, 53, 61, 68], "enign": 31, "xpath": [31, 50, 75, 83, 85], "mediawiki": [31, 39, 85], "anna": [31, 85], "brave": [31, 85], "bt4g": [31, 85], "dailymot": [31, 85], "dukcdukgo": 31, "lemmi": [31, 85], "peertub": [31, 85], "pipe": [31, 85], "recol": 31, "tagesschau": [31, 85], "torznab": 31, "webapi": 31, "wikimedia": [31, 32], "yaci": 31, "yahoo": [31, 74, 85], "z": [31, 64, 85], "nosql": [31, 37], "sql": [31, 37], "online_url_search": [31, 57, 82, 85], "tiney": [31, 85], "online_curr": [31, 82, 85], "soon": [31, 74], "online_dictionari": [31, 82, 85], "endpoint": [32, 52, 65], "pattern": 32, "w": [32, 85], "php": 32, "state": [32, 77, 82, 89], "ask": 32, "search_typ": 32, "srenablerewrit": 32, "srsort": 32, "srprop": 32, "639": [32, 50, 58, 83], "nearmatch": 32, "srwhat": 32, "thought": 32, "spell": [32, 41], "sectiontitl": 32, "snippet": 32, "categorysnippet": 32, "relev": [32, 38, 42, 61], "sort": [32, 35, 38, 42, 80], "create_timestamp_asc": 32, "create_timestamp_desc": 32, "incoming_links_asc": 32, "incoming_links_desc": 32, "just_match": 32, "last_edit_asc": 32, "last_edit_desc": 32, "user_random": 32, "timestamp_format": 32, "dt": 32, "sz": 32, "longhand": 32, "integr": [33, 35, 61, 69, 70], "leak": [33, 52], "easiest": 33, "solut": [33, 44, 89], "flexibl": [33, 64], "imagin": 33, "power": [33, 35, 36, 64], "mayb": 33, "element": [33, 50, 52, 64, 83], "put": [33, 36, 50], "delimit": [33, 64, 77], "char": 33, "parse_regex": 33, "regular": [33, 74], "express": [33, 74, 75], "query_typ": [33, 35], "enum": 33, "query_enum": 33, "working_dir": 33, "result_separ": 33, "fnd": 33, "check_parsing_opt": 33, "regex": 33, "correctli": [33, 45, 74], "satisfi": [34, 36], "result_templ": [34, 36], "template_nam": [34, 36], "theme_nam": [34, 36], "cmd": [34, 36, 37, 60, 61, 62, 77, 88, 89, 90], "redis_serv": 34, "bsd": 34, "licens": [34, 89], "either": [34, 40, 83, 84], "exact": 34, "match": [34, 35, 57, 74, 77, 82], "partial": [34, 64, 72, 80], "keyword": [34, 36, 73, 78, 87], "exact_match_onli": 34, "myredi": 34, "rd": 34, "pymongo": 34, "program": [34, 51, 61], "mymongo": 34, "27017": 34, "results_per_pag": 34, "busi": 34, "review": 34, "comparison": 35, "aim": [35, 64, 72], "individu": 35, "compani": [35, 71], "design": [35, 48, 64], "scale": [35, 60, 64, 89], "million": 35, "great": 35, "later": [35, 61, 89], "facet": 35, "subset": [35, 83], "authent": [35, 36, 52], "auth_token": 35, "me": [35, 41, 64, 89], "7700": 35, "numer": [35, 64], "moment": [35, 36], "popular": [35, 36, 64], "simple_query_str": 35, "payload": 35, "custom_query_json": 35, "9200": 35, "elast": 35, "changem": 35, "lucen": 35, "indic": [35, 53, 57], "ascend": 35, "slr": 35, "8983": 35, "asc": 35, "rdbm": 36, "mysql_serv": 36, "query_str": 36, "basic": [36, 41, 57, 59, 84, 89], "offset": [36, 58], "dure": [36, 45], "fast": 36, "reliabl": 36, "demonstr": [36, 62], "complex": 36, "mediathekview": 36, "movi": [36, 42, 85], "filmlist": 36, "v2": 36, "bz2": 36, "unpack": 36, "concert": 36, "durat": [36, 43, 80], "unixepoch": 36, "AS": 36, "coalesc": 36, "nullif": 36, "url_video_hd": 36, "url_video_sd": 36, "url_video": 36, "film": 36, "wildcard": 36, "OR": 36, "BY": 36, "desc": 36, "sqlite_cursor": 36, "context": [36, 43, 62, 64, 76, 89], "sqlite3": 36, "cursor": 36, "uri": 36, "psycopg2": 36, "robust": 36, "psychopg2": 36, "my_databas": 36, "my_tabl": 36, "my_column": 36, "connector": 36, "said": 36, "auth_plugin": 36, "caching_sha2_password": 36, "introduc": [37, 53, 64, 89], "skeleton": 37, "omit": 37, "anyth": [37, 71, 84], "retriev": [37, 43], "publicli": 37, "non": 38, "profit": 38, "onlin": [38, 64, 70, 72], "shadow": [38, 56], "varieti": 38, "via": [38, 51, 63], "ipf": 38, "team": 38, "archivist": 38, "annaarchivist": 38, "aa_cont": 38, "aa_ext": 38, "aa_sort": 38, "newest": 38, "aaa": 38, "annas_arch": [38, 85], "journal_articl": 38, "anan": 38, "field": [38, 39, 41, 43, 50, 53], "book_ani": 38, "book_fict": 38, "book_unknown": 38, "book_nonfict": 38, "book_com": 38, "standards_docu": 38, "end": [38, 56, 63, 64, 89], "epub": [38, 56], "beta": 38, "realli": [38, 48], "oldest": 38, "largest": 38, "smallest": 38, "offici": [39, 40, 46, 57, 77], "archlinix": 39, "wiki_netloc": [39, 53], "translat": [39, 59, 61, 70, 77, 78, 84], "zh": [39, 40, 45, 50, 53, 77, 83], "archlinuxcn": 39, "spezial": 39, "Suche": 39, "\u641c\u7d22": 39, "lot": [40, 60, 80], "abund": 40, "realiti": 40, "m\u0101ori": 40, "bit": [40, 64, 89], "closer": 40, "truth": 40, "seem": [40, 41, 44, 50], "inaccuraci": 40, "bing_traits_url": 40, "bing_video": [40, 85], "bing_imag": [40, 85], "bing_new": [40, 85], "learn": [40, 64], "microsoft": [40, 84], "market": 40, "tri": [40, 41, 44, 50, 74, 77], "guess": [40, 44, 50], "territori": [40, 50, 77], "async": [40, 45], "asyncv2": 40, "tabl": [40, 59], "sai": [40, 44], "thats": 40, "why": [40, 61, 64, 70, 86], "xpath_market_cod": 40, "infinitescrollajax": 40, "mkt_alia": 40, "ww": [40, 74], "cn": [40, 45, 53, 77, 83], "worldwid": 40, "aggreg": [40, 70, 71, 84], "time_map": 40, "9": [40, 61, 85], "hour": [40, 58, 75], "margin": 40, "brave_categori": 41, "remark": 41, "digit": [41, 77, 84], "officatl": 41, "facto": [41, 77], "aka": [41, 53, 61, 76, 77], "chines": [41, 53], "arab": 41, "low": 41, "menu": 41, "area": [41, 44, 57], "clear": [41, 71], "difference": 41, "gb": [41, 45, 83], "ui_lang": 41, "ja": [41, 45, 55, 83, 85], "jp": [41, 45], "pt": [41, 45, 50, 77, 83], "br": [41, 45, 77, 83, 85], "sq": [41, 83], "brave_spellcheck": 41, "typo": [41, 64], "food": 41, "fooh": 41, "spellcheck": 41, "torrent": [42, 52], "metadata": [42, 64], "magnet": [42, 52], "identifi": [42, 44, 57, 89], "feed": 42, "fewer": 42, "tradeoff": 42, "bt4g_order_bi": 42, "bt4g_categori": 42, "count": [42, 53, 64, 74], "bt4gv": 42, "bt": [42, 85], "categoi": 42, "audio": 42, "orderd": 42, "duplic": [43, 84], "en_en": 43, "en_gb": [43, 44, 50], "ar_aa": 43, "ar_eg": 43, "ar_a": 43, "ar_sa": 43, "7000": 43, "pr1071": 43, "family_filter_map": 43, "famili": 43, "explicit": [43, 64], "family_filt": 43, "iframe_src": 43, "video_id": 43, "result_field": 43, "allow_emb": 43, "created_tim": 43, "thumbnail_360_url": 43, "safesearch_param": 43, "is_created_for_kid": 43, "kid": 43, "audienc": [43, 64], "ag": 43, "cache_vqd": 44, "vqd": 44, "xmlhttprequest": 44, "wt": 44, "wt_wt": 44, "sens": [44, 64], "besid": [44, 71], "lanaguag": 44, "en_au": 44, "en_ca": 44, "get_ddg_lang": 44, "eng_trait": [44, 45, 53], "sxng_local": [44, 45, 53, 61, 77, 83], "ddg": [44, 68, 85, 87], "confus": [44, 64], "ddi": [44, 85], "pari": [44, 87], "es_ar": 44, "ah": 44, "eng_lang": 44, "eng_region": 44, "kl": 44, "get_vqd": 44, "sent": [44, 71], "far": 44, "least": [44, 45, 50, 60, 74, 83], "duckduckgo_definit": [44, 85], "area_to_str": 44, "wikidata": [44, 68, 85], "entiti": 44, "q712226": 44, "99": 44, "is_broken_text": 44, "href": [44, 58, 74], "xxxx": [44, 64], "somewher": 44, "broken": [44, 64], "mainli": [45, 77], "get_google_info": 45, "definit": [45, 85], "freeli": 45, "manli": 45, "add_domain": 45, "compos": 45, "pair": 45, "lang_en": [45, 61], "lang_zh": 45, "tw": [45, 53, 55, 77, 85], "subdomain": 45, "google_domain": 45, "urllib": 45, "urlencod": 45, "hl": [45, 61], "lr": [45, 61, 85], "particular": 45, "cr": [45, 85], "ie": [45, 61, 83], "utf8": [45, 61], "oe": [45, 61], "decod": 45, "ui_async": 45, "use_ac": 45, "_fmt": 45, "prog": 45, "google_complet": 45, "arg": [45, 67, 83], "android": 45, "protobuf": 45, "pb": 45, "compress": [45, 83], "pc": 45, "jspb": 45, "google_imag": [45, 85], "img": 45, "google_video": [45, 85], "ceid": [45, 61], "ceid_list": 45, "gl": [45, 61, 83, 85], "mandatori": 45, "consent": 45, "dialog": 45, "continu": [45, 62, 64], "num": [45, 83], "google_new": [45, 61, 85], "ae": 45, "419": 45, "AT": 45, "au": 45, "bd": 45, "bn": [45, 83], "nl": [45, 77, 83], "bg": [45, 55, 83], "bw": 45, "ch": [45, 77], "cl": [45, 85], "han": [45, 77], "co": [45, 83], "cu": 45, "cz": [45, 85], "eg": 45, "et": [45, 55, 64, 83], "gr": 45, "el": [45, 55, 83], "hk": [45, 53, 55, 77], "hant": [45, 77], "hu": [45, 83], "il": 45, "he": [45, 55, 64, 77, 83], "IN": 45, "hi": [45, 83], "ml": [45, 46, 83], "mr": [45, 83], "ta": [45, 83], "te": [45, 83], "ke": 45, "kr": 45, "ko": [45, 55, 83, 85], "lb": [45, 83], "lt": [45, 61, 83, 89], "lv": [45, 83, 85], "ma": 45, "mx": 45, "na": 45, "ng": 45, "NO": [45, 50], "nz": 45, "pe": 45, "ph": [45, 85], "pk": 45, "pl": [45, 77, 83], "150": [45, 74, 89], "ro": [45, 83], "sr": [45, 83], "ru": [45, 83, 85], "sa": [45, 67, 83], "se": [45, 85], "sv": [45, 83], "sg": [45, 53], "si": [45, 77, 83], "sl": [45, 55, 83], "sk": [45, 55, 83], "sn": 45, "th": [45, 83], "tr": [45, 83], "tz": 45, "ua": 45, "uk": [45, 68, 83], "ug": [45, 83], "ve": 45, "vn": 45, "vi": [45, 83], "za": 45, "zw": 45, "though": [45, 49], "slightli": 45, "vintag": 45, "google_scholar": [45, 85], "detect_google_captcha": 45, "dom": 45, "sorri": 45, "parse_gs_a": 45, "green": [45, 62], "time_range_arg": 45, "scientif": 45, "minu": [45, 80], "2022": 45, "as_ylo": 45, "2021": [45, 70], "v3": 46, "feder": 46, "independ": [46, 50], "lemmy_typ": 46, "schema": [47, 74, 80], "ident": [47, 50, 89], "videolanguag": 47, "8ed5c729": 47, "refactor": 47, "redesign": 47, "video_respons": 47, "peer": [47, 54], "tube": 47, "joinpeertub": 47, "friendli": 48, "youtub": [48, 85], "frontend": 48, "effici": [48, 83], "consist": [48, 60], "backend_url": 48, "frontend_url": 48, "piped_filt": 48, "ppdm": [48, 85], "music_song": 48, "nextpag": 48, "driven": [48, 70, 84], "plai": [48, 85], "pipedapi": 48, "kavin": 48, "rock": 48, "latter": [48, 64], "randomli": [48, 74], "offic": [48, 77], "webui": 49, "xapian": 49, "achiev": 49, "reach": [49, 64, 80], "mount_prefix": 49, "hierarchi": 49, "filesystem": 49, "dl_prefix": 49, "search_dir": 49, "domain": [49, 55, 57, 64], "scenario": [49, 83], "selector": [50, 58], "mess": 50, "br_br": 50, "pt_br": [50, 77], "cn_cn": 50, "zh_hans_cn": [50, 76], "tw_tw": 50, "zh_hant_tw": 50, "tw_hk": 50, "zh_hant_hk": 50, "gb_gb": 50, "letter": 50, "fil_ph": 50, "no_no": 50, "nb": [50, 83], "unknownlocaleerror": 50, "unknown": [50, 71, 74], "subtag": 50, "iana": 50, "macrolanguag": 50, "w3c": 50, "registri": 50, "norwegian": 50, "bokm\u00e5l": 50, "2005": 50, "suppress": 50, "latn": 50, "primari": 50, "encompass": 50, "mention": 50, "w3": [50, 64], "uniform": 50, "startpage_categ": 50, "get_sc_cod": 50, "sc": [50, 83, 85], "stamp": 50, "scrap": [50, 58], "sc_code_cache_sec": 50, "addition": [50, 70, 71], "search_form_xpath": 50, "ard": 51, "bundesstel": 51, "f\u00fcr": 51, "openapi": 51, "portal": 51, "bunddev": 51, "api2u": 51, "prowlarr": 52, "jackett": 52, "togeth": 52, "huge": [52, 64], "torznab_categori": 52, "show_torrent_fil": 52, "car": 52, "show_magnet_link": 52, "build_result": 52, "get_attribut": 52, "etre": [52, 83], "property_nam": 52, "get_torznab_attribut": 52, "attribute_nam": 52, "list_of_wikipedia": 53, "unlik": [53, 71], "tradit": 53, "languageconvert": 53, "rest_v1_summary_url": 53, "lc": 53, "variant": [53, 64], "convers": 53, "2554": 53, "\u51fa\u79df\u8eca": 53, "reqbin": 53, "gesg2kvx": 53, "get_wiki_param": 53, "wiki_lc_locale_vari": 53, "obj": [53, 64, 67, 83], "fetch_wikimedia_trait": 53, "wp": [53, 85, 87], "\u51fa\u79df\u8f66": 53, "\u8a08\u7a0b\u8eca": 53, "\u7684\u58eb": 53, "\u5fb7\u58eb": 53, "Not": [53, 64, 82], "locale_nam": [53, 72, 77], "depth": [53, 64], "gsw": 53, "classic": 53, "netloc": 53, "higher": [53, 61], "rest_v1": 53, "summari": [53, 58, 59], "meta": [53, 64], "paragraph": 53, "fka": 53, "hovercard": 53, "popup": 53, "mo": [53, 69], "wikipedia_article_depth": 53, "rough": 53, "encyclopedia": 53, "collabor": 53, "frequent": 53, "measur": 53, "were": 53, "realiz": 53, "wikipedia_languag": 53, "get_thumbnail": 53, "upload": [53, 57, 61], "calcul": 53, "stackoverflow": [53, 85], "33691240": 53, "principl": [54, 64], "p2p": 54, "apiyacysearch": 54, "yacy_search_serv": 54, "ya": 54, "8090": 54, "search_mod": 54, "http_digest_auth_us": 54, "http_digest_auth_pass": 54, "oper": [54, 64, 71, 89], "stealth": 54, "lang2domain": 55, "parse_url": 55, "url_str": 55, "track": [55, 67, 70, 71, 84], "da": [55, 83, 85], "hr": [55, 83], "zh_ch": 55, "zh_cht": 55, "abbrevi": [56, 87], "formerli": 56, "bookfind": 56, "scholarli": 56, "academ": 56, "began": 56, "mirror": [56, 85], "genesi": [56, 85], "zlib_year_from": 56, "zlib_year_to": 56, "zlib_ext": 56, "2010": 56, "2020": 56, "zlibrari": [56, 85], "zlib2010": 56, "drag": 57, "constantli": 57, "crawl": 57, "50": [57, 64], "billion": 57, "parse_tineye_match": 57, "match_json": 57, "image_url": 57, "score": [57, 80], "pixel": 57, "height": [57, 64], "overlai": 57, "belong": [57, 64, 85], "stock": 57, "backlink": [57, 64], "crawl_dat": 57, "download_error": 57, "format_not_support": 57, "due": [57, 80], "unsupport": 57, "jpeg": 57, "png": 57, "gif": 57, "bmp": 57, "tiff": 57, "webp": 57, "no_signature_error": 57, "visual": 57, "lang_al": 58, "page_s": 58, "first_page_num": 58, "time_range_url": 58, "time_range_map": 58, "safe": [58, 64, 65, 85], "safe_search_support": 58, "safe_search_map": 58, "no_result_for_http_statu": 58, "results_xpath": 58, "url_xpath": 58, "title_xpath": 58, "content_xpath": 58, "thumbnail_xpath": 58, "suggestion_xpath": 58, "repo": 58, "throw": 58, "safes_search_map": 58, "pag": 58, "24": [58, 87], "720": 58, "8760": 58, "time_range_v": 58, "365": 58, "contribut": [59, 61, 63, 70, 84], "prime": 59, "hackabl": 59, "wlc": 59, "motiv": [59, 74], "gentlemen": 59, "wrap": [59, 89], "suit": [59, 88], "checker": 59, "primer": [59, 70], "skill": 59, "inlin": 59, "markup": [59, 76], "anchor": 59, "unicod": [59, 68, 77], "substitut": 59, "role": 59, "figur": 59, "admonit": 59, "view": [59, 71], "searxng_extra": [59, 70, 77], "standalone_searx": [59, 66], "lxc": [60, 70, 88], "heterogen": 60, "cycl": 60, "tl": [60, 83], "dr": 60, "experienc": 60, "reader": [60, 64], "seriou": 60, "perfect": 60, "overlook": 60, "encapsul": 60, "prerequisit": [60, 64], "preinstal": 60, "softwar": [60, 84, 85], "isol": 60, "mix": [60, 84], "divid": 60, "stack": [60, 89], "lxd": [60, 88], "snap": [60, 89], "exercis": 60, "lxc_suit": [60, 89], "l19": 60, "let": [60, 64, 71], "force_timeout": [60, 88, 89], "140": [60, 74, 89], "conatin": [60, 89], "outsid": [60, 61], "prompt": [60, 61, 88], "guest": 60, "notic": 60, "readi": 60, "ey": [60, 64], "dsitro": 60, "attend": 60, "rel": [60, 64, 74, 83, 89], "transpar": [60, 89], "smylink": 60, "reposetori": 60, "mv": 60, "daili": 60, "ye": 60, "backup": 60, "ld": 60, "lrwxrwxrwx": 60, "modif": 60, "eth0": [60, 89], "live": [60, 63, 64, 80, 89], "fd42": 60, "555b": 60, "2af9": 60, "e121": 60, "216": [60, 74], "3eff": 60, "fe5b": 60, "1744": 60, "searxng_uwsgi_socket": [60, 90], "git_url": [60, 61, 90], "git_branch": [60, 61, 90], "ci": [61, 66, 68, 70], "wrapper": 61, "gnu": 61, "introduct": 61, "deeper": [61, 64], "prebuild": 61, "gecko": 61, "driver": 61, "geckodriv": 61, "robot_test": 61, "6": [61, 64, 80, 83, 85], "amd64": 61, "intermedi": 61, "pypi": [61, 64, 85], "black": [61, 64], "pygment": [61, 64, 68], "golang": 61, "npm": [61, 85], "counterpart": [61, 69], "userag": [61, 68, 83], "recent": 61, "yamllint": 61, "yamllint_fil": 61, "pylint_fil": 61, "pyright": 61, "coverag": 61, "incl": 61, "stuff": [61, 64], "live_them": [61, 63], "previous": 61, "restor": [61, 63], "comfort": [61, 63, 64], "granular": 61, "py3": 61, "txt": [61, 64, 68], "argpars": 61, "initialis": 61, "sha256": 61, "sum": 61, "word": [61, 62, 64, 65], "6cea6eb6def9e14a18bf32f8a3": 61, "471efef6c73558e391c3adb35f4": 61, "goe": 61, "wrong": 61, "runtim": 61, "central": 61, "especi": [61, 64], "pre": 61, "public_url": 61, "vc": 61, "proce": 61, "checkout": 61, "rebas": 61, "met": 61, "chain": [61, 63, 87, 89], "ubu2004": [61, 89], "v0": 61, "39": 61, "8fbf8ab": 61, "04": [61, 87, 89], "v10": 61, "19": [61, 89], "v16": 61, "jinja2": 61, "instant": 61, "source": 61, "black_opt": 61, "black_target": 61, "stuck": 61, "22": [61, 89], "bump": 61, "23": 61, "untouch": 61, "seri": 61, "pylintrc": 61, "whitespac": 61, "3xx": 61, "a1": 61, "443": 61, "life": 61, "3aen": 61, "302": 61, "comput": [61, 71], "cover": 61, "conveni": [61, 80], "satic": 61, "devpkg": 61, "compil": [61, 63, 83], "checkput": 61, "userdel": 61, "rmgrp": 61, "default_on": 62, "attach": 62, "callback": 62, "hook": 62, "flask": [62, 64, 67, 76, 81], "ctx": 62, "whole": 62, "post_search": 62, "result_contain": [62, 81], "return42": [62, 64], "tgwf": 62, "feel": [62, 64], "pre_search": 62, "searchwithplugin": [62, 72, 81], "on_result": 62, "parsed_url": 62, "urlpars": 62, "love": 63, "worri": 63, "hesit": [63, 70], "workflow": [63, 69], "receiv": [63, 74], "wild": 63, "west": 63, "pai": 63, "attent": [63, 64], "nvm": 63, "finish": [63, 89], "remain": 63, "rewind": 63, "encourag": 64, "contributor": 64, "restructuredtext": 64, "builder": 64, "docutil": 64, "faq": 64, "doctre": 64, "cross": 64, "linuxdoc": 64, "jinja": [64, 76], "autodoc": 64, "ecosystem": 64, "therefor": 64, "spars": 64, "plaintext": 64, "intuit": 64, "produc": 64, "advantag": 64, "disadvantag": 64, "grumpi": [64, 67], "face": 64, "train": 64, "bring": [64, 77], "question": [64, 71, 84], "knowledg": 64, "subject": 64, "concret": 64, "pov": 64, "heard": 64, "crawler": 64, "pro": 64, "con": 64, "understand": [64, 77], "chronolog": 64, "condit": [64, 82, 83], "asterisk": 64, "backquot": 64, "appear": 64, "escap": [64, 68], "backslash": 64, "pointer": 64, "emphasi": 64, "strong": 64, "boldfac": 64, "sampl": 64, "adorn": 64, "subsect": 64, "_doc": 64, "refnam": 64, "lorem": [64, 87], "ipsum": [64, 87], "dolor": 64, "sit": 64, "amet": 64, "consectetur": 64, "adipisici": 64, "elit": 64, "_chapter": 64, "ut": 64, "enim": 64, "veniam": 64, "qui": 64, "nostrud": 64, "exercit": 64, "ullamco": 64, "labori": 64, "nisi": 64, "aliquid": 64, "ex": 64, "ea": 64, "commodi": 64, "consequat": 64, "_section": 64, "_subsect": 64, "overlin": 64, "_anchor": 64, "_rest": 64, "visist": 64, "_sphinx": 64, "raw": [64, 76], "__": 64, "referenc": 64, "becom": [64, 68], "rfc": 64, "822": 64, "pep": 64, "af2cae6": 64, "man": [64, 85], "intersphinx_map": 64, "palletsproject": 64, "inventori": 64, "inv": 64, "simplest": 64, "colon": 64, "literalinclud": 64, "expand": 64, "consetetur": 64, "sadipsc": 64, "elitr": 64, "diam": 64, "nonumi": 64, "eirmod": 64, "tempor": 64, "invidunt": 64, "labor": 64, "caption": 64, "rout": [64, 74, 76], "statist": [64, 84], "get_engines_stat": 64, "0xa9": 64, "copyright": 64, "sign": 64, "tm": 64, "2122": 64, "trademark": 64, "glyph": 64, "piec": 64, "signifi": 64, "enclos": 64, "rolenam": 64, "guilabel": 64, "ancel": 64, "cancel": 64, "kbd": 64, "menuselect": 64, "b": [64, 68, 79, 83], "bold": 64, "subscript": 64, "sub": 64, "superscript": 64, "mc": [64, 85], "sup": 64, "scalabl": 64, "absenc": 64, "annoi": 64, "inherit": [64, 81], "insert": [64, 89], "_svg": 64, "svg_imag": 64, "alt": 64, "_dot": 64, "digraph": 64, "baz": 64, "vector": 64, "nw": 64, "arrow": 64, "xmln": 64, "baseprofil": 64, "70px": 64, "40px": 64, "viewbox": 64, "700": 64, "x1": 64, "180": 64, "y1": 64, "370": 64, "x2": 64, "500": 64, "y2": 64, "stroke": 64, "15px": 64, "polygon": 64, "585": 64, "525": 64, "25": 64, "transform": 64, "rotat": 64, "135": 64, "parent": 64, "compact": 64, "third": [64, 71, 80, 84], "yyyi": 64, "zzzz": 64, "distinguish": [64, 77], "classifi": [64, 74], "phrase": 64, "That": 64, "duref": 64, "surround": 64, "fieldnam": 64, "commonli": 64, "my_funct": 64, "my_arg": 64, "my_other_arg": 64, "cours": 64, "caveat": 64, "doctest": 64, "catcher": 64, "top": 64, "kiss_": 64, "readability_": 64, "tip": 64, "caution": 64, "danger": 64, "import": [64, 67, 76, 77, 79], "ugli": 64, "row": 64, "column": 64, "cell": 64, "nightmar": 64, "big": [64, 80], "diff": 64, "widen": 64, "ascrib": 64, "anywai": 64, "helper": 64, "emac": 64, "colspan": 64, "rowspan": 64, "front": 64, "align": 64, "span": [64, 83], "doubl": 64, "stage": 64, "cspan": 64, "rspan": 64, "rightmost": 64, "fill": 64, "behavior": 64, "stub": 64, "morecol": 64, "morerow": 64, "col": 64, "outstand": 64, "csv_tabl": 64, "loremlorem": 64, "magna": 64, "aliquyam": 64, "erat": 64, "voluptua": 64, "vero": 64, "accusam": 64, "justo": 64, "duo": 64, "rebum": 64, "stet": 64, "clita": 64, "kasd": 64, "gubergren": 64, "sea": 64, "takimata": 64, "sanctu": 64, "est": 64, "suitabl": 64, "enabled_engine_count": 64, "group_bang": 64, "group_engines_in_tab": 64, "loop": [64, 80], "els": [64, 69, 89], "endif": 64, "mod": 64, "upper": 64, "__name__": 64, "documented_modul": 64, "endfor": 64, "jinja_context": 64, "instruct": [64, 84], "amsmath": 64, "mathemat": 64, "ctan": 64, "numref": 64, "schroeding": 64, "schr\u00f6dinger": 64, "label": [64, 68, 83], "mathrm": 64, "hbar": 64, "dfrac": 64, "psi": 64, "rangl": 64, "hat": 64, "tfrac": 64, "textstyl": 64, "displaystyl": 64, "fraction": 64, "thu": [65, 71, 74], "hash_plugin": 65, "search_on_category_select": 65, "self_inform": 65, "tracker_url_remov": 65, "ahmia_blacklist": [65, 68], "open_access_doi_rewrit": 65, "like_hotkei": 65, "tor_check_plugin": 65, "disabled_plugin": 65, "enabled_engin": 65, "disabled_engin": 65, "update_ahmia_blacklist": 66, "update_curr": 66, "update_engine_descript": 66, "update_external_bang": 66, "update_firefox_vers": 66, "update_engine_trait": [66, 77], "update_osm_keys_tag": 66, "update_pyg": 66, "update_wikidata_unit": 66, "get_search_queri": [66, 67], "json_seri": [66, 67], "no_parsed_url": [66, 67], "parse_argu": [66, 67], "to_dict": [66, 67], "contrari": 67, "behav": 67, "rain": 67, "engine_categori": [67, 82], "searchqueri": [67, 72, 81], "serial": 67, "typeerror": [67, 83], "category_choic": 67, "systemexit": 67, "importlib": 67, "spec": 67, "spec_from_file_loc": 67, "module_from_spec": 67, "exec_modul": 67, "ptipython": 67, "timerang": 67, "search_queri": [67, 81, 82], "onion": 68, "engine_descript": 68, "get_output": 68, "description_and_sourc": 68, "external_bang": [68, 81], "newbang": 68, "bv1": 68, "v260": 68, "futur": 68, "bv2": 68, "probabl": [68, 83], "re_bang_vers": 68, "merge_when_no_leaf": 68, "child": 68, "equal": 68, "leaf_kei": 68, "dig": 68, "nood": 68, "dg": 68, "ig": 68, "signatur": 68, "intersect": [68, 77], "unicodeescap": 68, "pprint": 68, "pformat": 68, "fetch_traits_map": 68, "filter_local": 68, "traits_map": 68, "threshold": [68, 83], "get_unicode_flag": 68, "emoji": [68, 77], "i18n": [68, 69, 70, 72], "sidenot": 68, "atownsend": 68, "osm_keys_tag": 68, "sparql_tags_request": 68, "sparql": 68, "get_tag": 68, "taginfo": 68, "3dhous": 68, "infobox": 68, "q3947": 68, "p1282": 68, "3abuild": 68, "3dbungalow": 68, "q850107": 68, "sparql_keys_request": 68, "payment": 68, "3apay": 68, "q1148747": 68, "confirm": 68, "cash": 68, "rdf": 68, "oppos": 68, "wikibas": 68, "wikidata_unit": 68, "extractor": [69, 70, 72], "pybabel": 69, "codeberg": [69, 85], "sync": 69, "synchron": 69, "orphan": 69, "decoupl": 69, "pot": 69, "po": 69, "job": [69, 88], "fridai": 69, "additon": 69, "70": 70, "neither": [70, 83], "nor": [70, 83], "anyon": 70, "encrypt": 70, "130": 70, "60": 70, "profession": 70, "assur": 70, "autom": 70, "join": [70, 83, 84], "expert": 70, "everyon": [70, 71, 84], "improv": [70, 84], "discuss": 70, "middl": 70, "conclus": 70, "parti": [71, 84], "peopl": [71, 84], "vpn": 71, "laptop": 71, "gain": 71, "insight": 71, "dive": 71, "advertis": 71, "monet": 71, "someon": 71, "sold": 71, "proper": 71, "vulner": 71, "abus": 71, "exchang": 71, "tailor": 71, "reset": [71, 74, 89, 90], "compromis": [71, 84], "limiter_cfg": [72, 74], "limiter_cfg_schema": [72, 74], "ip_list": 72, "block_ip": [72, 74], "pass_ip": [72, 74], "searxng_org": [72, 74], "probe": 72, "searxengineapiexcept": [72, 75], "searxengineaccessdeniedexcept": [72, 75], "searxenginecaptchaexcept": [72, 75], "searxengineexcept": [72, 75], "searxengineresponseexcept": [72, 75], "searxenginetoomanyrequestsexcept": [72, 75], "searxenginexpathexcept": [72, 75, 83], "searxexcept": [72, 75], "searxparameterexcept": [72, 75], "searxsettingsexcept": [72, 75], "searxxpathsyntaxexcept": [72, 75, 83], "infopag": [72, 76], "infopageset": [72, 76], "build_engine_local": [72, 77], "get_local": [72, 77], "get_locale_descr": [72, 77], "get_offical_local": [72, 77], "get_transl": [72, 77], "language_tag": [72, 77], "locales_initi": [72, 77], "match_local": [72, 77], "region_tag": [72, 77], "additional_transl": [72, 77], "locale_best_match": [72, 77], "rtl_local": [72, 77], "preference_sect": [72, 78], "query_exampl": [72, 78], "query_keyword": [72, 78], "old_redis_url_default_url": [72, 79], "drop_count": [72, 80], "incr_count": [72, 80], "incr_sliding_window": [72, 80], "lua_script_storag": [72, 80], "purge_by_prefix": [72, 80], "secret_hash": [72, 80], "engineref": [72, 81], "convert_str_to_int": [72, 83], "detect_languag": [72, 83], "dict_subset": [72, 83], "ecma_unescap": [72, 83], "eval_xpath": [72, 83], "eval_xpath_getindex": [72, 83], "eval_xpath_list": [72, 83], "extract_text": [72, 83], "extract_url": [72, 83], "gen_userag": [72, 83], "get_engine_from_set": [72, 83], "get_torrent_s": [72, 83], "get_xpath": [72, 83], "html_to_text": [72, 83], "int_or_zero": [72, 83], "is_valid_lang": [72, 83], "normalize_url": [72, 83], "to_str": [72, 83], "search_language_cod": [72, 83], "searxng_msg": 73, "msg": 73, "cfg": [73, 74], "babel_extract": 73, "yield": 73, "fileobj": 73, "comment_tag": 73, "botdetect": 74, "get_real_ip": 74, "fake": 74, "happen": 74, "werkzeug": 74, "proxyfix": 74, "x_for": 74, "inconsist": 74, "suspici": 74, "fact": 74, "toml": 74, "lokal": 74, "238": 74, "172": 74, "132": 74, "linkloc": 74, "184": 74, "34": 74, "257": 74, "invalid": 74, "real_ip": 74, "ipv4address": 74, "ipv6address": 74, "tupl": [74, 77, 83], "subnet": 74, "2603": 74, "c022": 74, "4900": 74, "56": 74, "passlist": 74, "organ": 74, "slide": [74, 80], "window": [74, 80], "investig": 74, "burst_max": 74, "burst_max_suspici": 74, "long_max": 74, "long_max_suspici": 74, "intercept": 74, "suspicious_ip_window": 74, "suspicious_ip_max": 74, "drope": 74, "api_max": 74, "api_wondow": 74, "sec": [74, 75, 80, 83], "15": [74, 89], "burst_window": 74, "burst": 74, "long_window": 74, "600": 74, "2592000": 74, "ping": 74, "client_token": 74, "mimetyp": 74, "stylesheet": 74, "get_token": 74, "url_for": 74, "get_ping_kei": 74, "ipv4network": 74, "ipv6network": 74, "token_live_tim": 74, "token_kei": 74, "is_suspici": 74, "renew": 74, "ping_live_tim": 74, "ping_kei": 74, "searxng_limit": 74, "livetim": 74, "AND": 74, "deflat": 74, "user_ag": 74, "cc": [74, 85], "uu": 74, "rr": 74, "ll": 74, "scrapi": 74, "splash": 74, "javafx": 74, "feedfetch": 74, "java": 74, "jakarta": 74, "okhttp": 74, "httpclient": 74, "jersei": 74, "libwww": 74, "perl": 74, "rubi": 74, "synhttpclient": 74, "universalfeedpars": 74, "googlebot": 74, "googleimageproxi": 74, "bingbot": 74, "baiduspid": 74, "yacybot": 74, "yandexmobilebot": 74, "yandexbot": 74, "slurp": 74, "mj12bot": 74, "ahrefsbot": 74, "org_bot": 74, "msnbot": 74, "seznambot": 74, "linkdexbot": 74, "netvib": 74, "smtbot": 74, "zgrab": 74, "jame": 74, "sogou": 74, "abonti": 74, "pixrai": 74, "spinn3r": 74, "semrushbot": 74, "exabot": 74, "zmeu": 74, "blexbot": 74, "bitlybot": 74, "mozilla": [74, 83], "farsid": 74, "petalbot": 74, "suspend_time_set": 75, "suspend": [75, 82], "imposs": 75, "3660": 75, "xpath_spec": [75, 83], "paramat": 75, "filenam": 75, "_info_pag": 76, "mistletoepag": 76, "pagenam": 76, "get_valu": 76, "get_pag": 76, "fname": 76, "get_ctx": 76, "conntext": 76, "markdown": 76, "commonmark": 76, "raw_cont": 76, "page_class": 76, "info_fold": 76, "parser": 76, "toc": 76, "i18n_origin": 76, "iter_pag": 76, "fallback_to_default": 76, "markdwon": 76, "locale_default": 76, "tag_list": 77, "engine_local": 77, "zh_han": [77, 83], "zh_hant": [77, 83], "model": [77, 83], "ca_e": 77, "fr_be": 77, "fr_ca": 77, "fr_ch": 77, "fr_fr": 77, "pl_pl": 77, "pt_pt": 77, "zh_tw": 77, "narrow": 77, "down": 77, "approxim": 77, "attempt": 77, "assumpt": 77, "optim": 77, "prioriti": 77, "terrirtori": 77, "locale_tag": 77, "fran\u00e7ai": 77, "portugu\u00ea": 77, "brasil": 77, "de_facto": 77, "get_official_languag": 77, "monkei": 77, "flask_babel": 77, "locale_tag_list": 77, "fallback": 77, "dv": [77, 83], "\u078b": 77, "\u0788": 77, "\u0780": 77, "dhivehi": 77, "oc": [77, 83], "occitan": 77, "pap": 77, "papiamento": 77, "szl": 77, "\u015bl\u014dnski": 77, "silesian": 77, "taiwan": 77, "hong": 77, "kong": 77, "fa": [77, 83], "ir": 77, "five": 77, "get_language_nam": 77, "get_territory_nam": 77, "english_nam": 77, "repres": 77, "globe": 77, "canada": 77, "belgiqu": 77, "tor_check": 78, "lua": 80, "inspir": [80, 84], "bullet": 80, "proof": 80, "redispi": 80, "redislib": 80, "counter": 80, "searxng_counter_": 80, "increment": 80, "64": 80, "incr": 80, "sleep": 80, "typedur": 80, "zadd": 80, "zremrangebyscor": 80, "refresh": 80, "zcount": 80, "until": 80, "register_script": 80, "searxng_": 80, "purg": 80, "zero": [80, 89], "del": 80, "anonymis": 80, "engineref_list": 81, "timeout_limit": 81, "resultcontain": 81, "ordered_plugin_list": 81, "engineprocessor": 82, "engine_nam": 82, "reqest": 82, "get_param": 82, "suspendedstatu": 82, "offlineprocessor": 82, "onlineprocessor": 82, "default_request_param": 82, "onlinecurrencyprocessor": 82, "parser_r": 82, "onlinedictionaryprocessor": 82, "onlineurlsearchprocessor": 82, "re_search_url": 82, "number_str": 83, "only_search_languag": 83, "whose": 83, "valueerror": 83, "fasttext": 83, "identif": 83, "zip": 83, "classif": 83, "bag": 83, "trick": 83, "af": 83, "am": 83, "arz": 83, "ast": 83, "av": 83, "az": 83, "azb": 83, "ba": 83, "bcl": 83, "bh": 83, "bo": 83, "bpy": 83, "bxr": 83, "cbk": 83, "ce": 83, "ceb": 83, "ckb": 83, "cv": [83, 85], "cy": 83, "diq": 83, "dsb": 83, "dty": 83, "eml": 83, "eu": 83, "fi": [83, 89], "frr": 83, "fy": 83, "ga": 83, "gn": 83, "gom": 83, "gu": 83, "gv": 83, "hif": 83, "hsb": 83, "ht": 83, "hy": 83, "ia": 83, "ilo": 83, "jbo": 83, "jv": 83, "ka": 83, "kk": 83, "km": 83, "kn": 83, "krc": 83, "ku": 83, "kv": 83, "kw": 83, "ky": 83, "lez": 83, "li": [83, 89], "lmo": 83, "lo": [83, 85], "lrc": 83, "mg": 83, "mhr": 83, "mk": 83, "mn": 83, "mrj": 83, "mt": 83, "mwl": 83, "myv": 83, "mzn": 83, "nah": 83, "nap": 83, "nd": 83, "ne": 83, "nn": 83, "pa": 83, "pam": 83, "pfl": 83, "pm": 83, "pnb": 83, "qu": 83, "rue": 83, "sah": 83, "scn": 83, "sco": 83, "sd": 83, "su": [83, 85], "sw": 83, "tg": 83, "tk": 83, "tt": [83, 85], "tyv": 83, "ur": 83, "uz": 83, "vec": 83, "vep": 83, "vl": 83, "vo": 83, "war": 83, "wuu": 83, "xal": 83, "xmf": 83, "yi": 83, "yo": 83, "yue": 83, "harmon": 83, "discrep": 83, "fastext": 83, "mutablemap": 83, "unescap": 83, "ecma": 83, "262": 83, "objets_globaux": 83, "u5409": 83, "\u5409": 83, "f3": 83, "\u00f3": 83, "elementbas": 83, "equival": 83, "xpath_str": 83, "xpathxslt": 83, "_notsetclass": 83, "ais": 83, "min_len": 83, "xpath_result": 83, "allow_non": 83, "concat": 83, "text_cont": 83, "union": 83, "htmlelement": 83, "fromstr": 83, "42": [83, 89], "parsererror": 83, "os_str": 83, "filesize_multipli": 83, "tb": 83, "tib": 83, "gib": 83, "5368709120": 83, "mib": 83, "3140000": 83, "worst": 83, "html_str": 83, "color": 83, "red": 83, "zz": 83, "ukrainian": 83, "espa\u00f1ol": 83, "spanish": 83, "absolut": 83, "frozenset": 83, "matrix": 84, "chat": 84, "person": 84, "consciou": 84, "believ": 84, "freedom": 84, "opensearch": 84, "edg": 84, "chrome": 84, "safari": 84, "chromium": 84, "encount": 84, "seek": 84, "platform": 84, "moreov": 84, "appreci": 84, "reclaim": 84, "freer": 84, "decentr": 84, "149": 85, "77": 85, "mojeek": 85, "mjk": 85, "qw": 85, "sp": 85, "wibi": 85, "wib": 85, "json_engin": 85, "yh": 85, "szn": 85, "goo": 85, "naver": 85, "nvr": 85, "wikibook": 85, "wb": 85, "wikiquot": 85, "wq": 85, "wikisourc": 85, "wikispeci": 85, "wsp": 85, "wikivers": 85, "wv": 85, "wikivoyag": 85, "wy": 85, "alexandria": 85, "alx": 85, "ai": 85, "crowdview": 85, "curli": 85, "currency_convert": 85, "ddd": 85, "dictzon": 85, "dc": 85, "lingva": 85, "tin": 85, "wd": 85, "wolframalpha": 85, "wolframalpha_noapi": 85, "yep": 85, "wikimini": 85, "wkmn": 85, "bii": 85, "brimg": 85, "duckduckgo_imag": 85, "qwi": 85, "1x": 85, "www1x": 85, "arc": 85, "deviantart": 85, "flickr": 85, "fl": 85, "flickr_noapi": 85, "frinkiac": 85, "frk": 85, "congress": 85, "loc": 85, "openvers": 85, "opv": 85, "unsplash": 85, "biv": 85, "brvid": 85, "gov": 85, "qwv": 85, "ccc": 85, "tv": 85, "c3tv": 85, "dm": 85, "gpm": 85, "google_plai": 85, "invidi": 85, "iv": 85, "ptb": 85, "ppd": 85, "rumbl": 85, "sepiasearch": 85, "vimeo": 85, "vm": 85, "yt": 85, "youtube_noapi": 85, "mediathekviewweb": 85, "mvw": 85, "ina": 85, "wikinew": 85, "wn": 85, "brnew": 85, "gon": 85, "qwn": 85, "yhn": 85, "yahoo_new": 85, "appl": 85, "apm": 85, "apple_map": 85, "photon": 85, "azlyr": 85, "geniu": 85, "gen": 85, "bandcamp": 85, "bc": 85, "deezer": 85, "dz": 85, "gpodder": 85, "gpod": 85, "mixcloud": 85, "soundcloud": 85, "hub": 85, "dh": 85, "docker_hub": 85, "hoogl": 85, "ho": 85, "metacpan": 85, "cpan": 85, "packagist": 85, "pack": 85, "pub": 85, "pd": 85, "rubygem": 85, "rbg": 85, "askubuntu": 85, "stackexchang": 85, "st": 85, "superus": 85, "cb": 85, "gitlab": 85, "sourcehut": 85, "srht": 85, "fsd": 85, "gentoo": 85, "ge": 85, "anaconda": 85, "conda": 85, "framalibr": 85, "frl": 85, "habrahabr": 85, "habr": 85, "lobst": 85, "mankier": 85, "searchcod": 85, "scc": 85, "searchcode_cod": 85, "arxiv": 85, "arx": 85, "crossref": 85, "scholar": 85, "pubm": 85, "semantic_scholar": 85, "openairedataset": 85, "oad": 85, "openairepubl": 85, "oap": 85, "pdbe": 85, "pdb": 85, "apk": 85, "apkm": 85, "apkmirror": 85, "ap": 85, "apple_app_stor": 85, "fdroid": 85, "fd": 85, "gpa": 85, "1337x": 85, "aa": 85, "btdigg": 85, "kickass": 85, "kc": 85, "lg": 85, "nyaa": 85, "nt": 85, "openrepo": 85, "piratebai": 85, "tpb": 85, "solidtorr": 85, "solid": 85, "tokyotoshokan": 85, "zlib": 85, "9gag": 85, "9g": 85, "lecom": 85, "leco": 85, "lepo": 85, "leu": 85, "reddit": 85, "twitter": 85, "social_media": 86, "wau": 87, "holland": 87, "inclus": 87, "wfr": 87, "uuid": 87, "averag": 87, "avg": 87, "123": 87, "548": 87, "md5": 87, "sha512": 87, "dispos": 88, "batch": 88, "snapcraft": 89, "cup": 89, "coffe": 89, "iptabl": 89, "fralef": 89, "conflict": 89, "reboot": 89, "7048": 89, "7851230": 89, "handi": 89, "ugo": 89, "ubu2204": 89, "fedora35": 89, "snapshot": 89, "upon": 89, "ever": 89, "47712402": 89, "rw": 89, "marku": 89, "2923": 89, "apr": 89, "52": 89, "11": 89, "inod": 89, "ouput": 89, "timezon": 89, "ubu2110": 89, "170": 89, "160": 89, "searxnggfedora35": 89, "200331": 89, "296": 89, "explanatori": [89, 90], "launch": 89, "storag": 89, "quot": 89, "prepar": 89, "nil": 89, "spdx": 89, "agpl": 89, "manipul": 89, "subshel": 89, "lxc_set_suite_env": 89, "lxc_suite_nam": 89, "linuxcontain": 89, "linuxcontainers_org_nam": 89, "lxc_host_prefix": 89, "eoss": 89, "april": 89, "2025": 89, "2027": 89, "eol": 89, "fedoraproject": 89, "35": 89, "releng": 89, "lxc_suite_install_info": 89, "eof": 89, "local_imag": 89, "lxc_suite_instal": 89, "lxc_repo_root": 89, "rst_titl": 89, "ask_yn": 89, "yn": 89, "link_src": 89, "lxc_suite_info": 89, "global_ip": 89, "info_msg": 89, "sc2034": 89, "sc2031": 89, "localtest": 90, "searxng_check": 90, "get_set": 90, "fv": 90, "az808": 90, "537": 90}, "objects": {"": [[62, 0, 1, "", "on_result"], [62, 0, 1, "", "post_search"], [62, 0, 1, "", "pre_search"]], "searx.autocomplete": [[45, 0, 1, "", "google_complete"]], "searx": [[73, 1, 0, "-", "babel_extract"], [74, 1, 0, "-", "botdetection"], [29, 1, 0, "-", "enginelib"], [30, 1, 0, "-", "engines"], [75, 1, 0, "-", "exceptions"], [76, 1, 0, "-", "infopage"], [77, 1, 0, "-", "locales"], [79, 1, 0, "-", "redisdb"], [80, 1, 0, "-", "redislib"], [77, 1, 0, "-", "sxng_locales"], [83, 1, 0, "-", "utils"]], "searx.babel_extract": [[73, 0, 1, "", "extract"]], "searx.botdetection": [[74, 0, 1, "", "get_real_ip"], [74, 1, 0, "-", "http_accept"], [74, 1, 0, "-", "http_accept_encoding"], [74, 1, 0, "-", "http_accept_language"], [74, 1, 0, "-", "http_connection"], [74, 1, 0, "-", "http_user_agent"], [74, 1, 0, "-", "ip_limit"], [74, 1, 0, "-", "ip_lists"], [74, 1, 0, "-", "limiter"], [74, 1, 0, "-", "link_token"]], "searx.botdetection.http_user_agent": [[74, 2, 1, "", "USER_AGENT"]], "searx.botdetection.ip_limit": [[74, 2, 1, "", "API_MAX"], [74, 2, 1, "", "API_WONDOW"], [74, 2, 1, "", "BURST_MAX"], [74, 2, 1, "", "BURST_MAX_SUSPICIOUS"], [74, 2, 1, "", "BURST_WINDOW"], [74, 2, 1, "", "LONG_MAX"], [74, 2, 1, "", "LONG_MAX_SUSPICIOUS"], [74, 2, 1, "", "LONG_WINDOW"], [74, 2, 1, "", "SUSPICIOUS_IP_MAX"], [74, 2, 1, "", "SUSPICIOUS_IP_WINDOW"]], "searx.botdetection.ip_lists": [[74, 2, 1, "", "SEARXNG_ORG"], [74, 0, 1, "", "block_ip"], [74, 0, 1, "", "pass_ip"]], "searx.botdetection.limiter": [[74, 2, 1, "", "LIMITER_CFG"], [74, 2, 1, "", "LIMITER_CFG_SCHEMA"]], "searx.botdetection.link_token": [[74, 2, 1, "", "PING_KEY"], [74, 2, 1, "", "PING_LIVE_TIME"], [74, 2, 1, "", "TOKEN_KEY"], [74, 2, 1, "", "TOKEN_LIVE_TIME"], [74, 0, 1, "", "get_ping_key"], [74, 0, 1, "", "get_token"], [74, 0, 1, "", "is_suspicious"], [74, 0, 1, "", "ping"]], "searx.enginelib": [[29, 3, 1, "", "Engine"], [29, 1, 0, "-", "traits"]], "searx.enginelib.Engine": [[29, 4, 1, "", "about"], [29, 4, 1, "", "categories"], [29, 4, 1, "", "disabled"], [29, 4, 1, "", "display_error_messages"], [29, 4, 1, "", "enable_http"], [29, 4, 1, "", "engine"], [29, 4, 1, "", "engine_type"], [29, 4, 1, "", "fetch_traits"], [29, 4, 1, "", "inactive"], [29, 4, 1, "", "language"], [29, 4, 1, "", "language_support"], [29, 4, 1, "", "name"], [29, 4, 1, "", "paging"], [29, 4, 1, "", "proxies"], [29, 4, 1, "", "region"], [29, 4, 1, "", "safesearch"], [29, 4, 1, "", "send_accept_language_header"], [29, 4, 1, "", "shortcut"], [29, 4, 1, "", "time_range_support"], [29, 4, 1, "", "timeout"], [29, 4, 1, "", "tokens"], [29, 4, 1, "", "traits"], [29, 4, 1, "", "using_tor_proxy"]], "searx.enginelib.traits": [[29, 3, 1, "", "EngineTraits"], [29, 3, 1, "", "EngineTraitsEncoder"], [29, 3, 1, "", "EngineTraitsMap"]], "searx.enginelib.traits.EngineTraits": [[29, 4, 1, "", "all_locale"], [29, 5, 1, "", "copy"], [29, 4, 1, "", "custom"], [29, 4, 1, "", "data_type"], [29, 5, 1, "", "fetch_traits"], [29, 5, 1, "", "get_language"], [29, 5, 1, "", "get_region"], [29, 5, 1, "", "is_locale_supported"], [29, 4, 1, "", "languages"], [29, 4, 1, "", "regions"], [29, 5, 1, "", "set_traits"]], "searx.enginelib.traits.EngineTraitsEncoder": [[29, 5, 1, "", "default"]], "searx.enginelib.traits.EngineTraitsMap": [[29, 4, 1, "", "ENGINE_TRAITS_FILE"], [29, 5, 1, "", "from_data"], [29, 5, 1, "", "save_data"], [29, 5, 1, "", "set_traits"]], "searx.engines": [[38, 1, 0, "-", "annas_archive"], [39, 1, 0, "-", "archlinux"], [40, 1, 0, "-", "bing"], [40, 1, 0, "-", "bing_images"], [40, 1, 0, "-", "bing_news"], [40, 1, 0, "-", "bing_videos"], [41, 1, 0, "-", "brave"], [42, 1, 0, "-", "bt4g"], [33, 1, 0, "-", "command"], [43, 1, 0, "-", "dailymotion"], [26, 1, 0, "-", "demo_offline"], [27, 1, 0, "-", "demo_online"], [44, 1, 0, "-", "duckduckgo"], [44, 1, 0, "-", "duckduckgo_definitions"], [44, 1, 0, "-", "duckduckgo_images"], [44, 1, 0, "-", "duckduckgo_weather"], [35, 1, 0, "-", "elasticsearch"], [30, 2, 1, "", "engine_shortcuts"], [45, 1, 0, "-", "google"], [45, 1, 0, "-", "google_images"], [45, 1, 0, "-", "google_news"], [45, 1, 0, "-", "google_scholar"], [45, 1, 0, "-", "google_videos"], [30, 0, 1, "", "is_missing_required_attributes"], [46, 1, 0, "-", "lemmy"], [30, 0, 1, "", "load_engine"], [30, 0, 1, "", "load_engines"], [32, 1, 0, "-", "mediawiki"], [35, 1, 0, "-", "meilisearch"], [34, 1, 0, "-", "mongodb"], [36, 1, 0, "-", "mysql_server"], [47, 1, 0, "-", "peertube"], [48, 1, 0, "-", "piped"], [36, 1, 0, "-", "postgresql"], [49, 1, 0, "-", "recoll"], [34, 1, 0, "-", "redis_server"], [47, 1, 0, "-", "sepiasearch"], [35, 1, 0, "-", "solr"], [36, 1, 0, "-", "sqlite"], [50, 1, 0, "-", "startpage"], [51, 1, 0, "-", "tagesschau"], [57, 1, 0, "-", "tineye"], [52, 1, 0, "-", "torznab"], [30, 0, 1, "", "using_tor_proxy"], [53, 1, 0, "-", "wikidata"], [53, 1, 0, "-", "wikipedia"], [58, 1, 0, "-", "xpath"], [54, 1, 0, "-", "yacy"], [55, 1, 0, "-", "yahoo"], [56, 1, 0, "-", "zlibrary"]], "searx.engines.annas_archive": [[38, 2, 1, "", "aa_content"], [38, 2, 1, "", "aa_ext"], [38, 2, 1, "", "aa_sort"], [38, 0, 1, "", "fetch_traits"], [38, 0, 1, "", "init"]], "searx.engines.archlinux": [[39, 0, 1, "", "fetch_traits"]], "searx.engines.bing": [[40, 2, 1, "", "base_url"], [40, 2, 1, "", "bing_traits_url"], [40, 0, 1, "", "fetch_traits"], [40, 0, 1, "", "request"], [40, 2, 1, "", "send_accept_language_header"]], "searx.engines.bing_images": [[40, 2, 1, "", "base_url"], [40, 2, 1, "", "bing_traits_url"], [40, 0, 1, "", "fetch_traits"], [40, 0, 1, "", "request"], [40, 0, 1, "", "response"]], "searx.engines.bing_news": [[40, 2, 1, "", "base_url"], [40, 2, 1, "", "bing_traits_url"], [40, 0, 1, "", "fetch_traits"], [40, 2, 1, "", "mkt_alias"], [40, 0, 1, "", "request"], [40, 0, 1, "", "response"], [40, 2, 1, "", "time_map"]], "searx.engines.bing_videos": [[40, 2, 1, "", "base_url"], [40, 2, 1, "", "bing_traits_url"], [40, 0, 1, "", "fetch_traits"], [40, 0, 1, "", "request"], [40, 0, 1, "", "response"]], "searx.engines.brave": [[41, 2, 1, "", "brave_category"], [41, 2, 1, "", "brave_spellcheck"], [41, 0, 1, "", "fetch_traits"], [41, 2, 1, "", "paging"], [41, 2, 1, "", "time_range_support"]], "searx.engines.bt4g": [[42, 2, 1, "", "bt4g_category"], [42, 2, 1, "", "bt4g_order_by"]], "searx.engines.command": [[33, 0, 1, "", "check_parsing_options"]], "searx.engines.dailymotion": [[43, 2, 1, "", "family_filter_map"], [43, 0, 1, "", "fetch_traits"], [43, 2, 1, "", "iframe_src"], [43, 2, 1, "", "result_fields"], [43, 2, 1, "", "safesearch_params"], [43, 2, 1, "", "search_url"]], "searx.engines.demo_offline": [[26, 0, 1, "", "init"], [26, 0, 1, "", "search"]], "searx.engines.demo_online": [[27, 0, 1, "", "init"], [27, 0, 1, "", "request"], [27, 0, 1, "", "response"]], "searx.engines.duckduckgo": [[44, 0, 1, "", "cache_vqd"], [44, 0, 1, "", "fetch_traits"], [44, 0, 1, "", "get_ddg_lang"], [44, 0, 1, "", "get_vqd"], [44, 2, 1, "", "send_accept_language_header"]], "searx.engines.duckduckgo_definitions": [[44, 0, 1, "", "area_to_str"], [44, 0, 1, "", "is_broken_text"]], "searx.engines.google": [[45, 2, 1, "", "UI_ASYNC"], [45, 0, 1, "", "fetch_traits"], [45, 0, 1, "", "get_google_info"], [45, 0, 1, "", "request"], [45, 0, 1, "", "response"]], "searx.engines.google_images": [[45, 0, 1, "", "request"], [45, 0, 1, "", "response"]], "searx.engines.google_news": [[45, 2, 1, "", "ceid_list"], [45, 0, 1, "", "request"], [45, 0, 1, "", "response"]], "searx.engines.google_scholar": [[45, 0, 1, "", "detect_google_captcha"], [45, 0, 1, "", "parse_gs_a"], [45, 0, 1, "", "request"], [45, 0, 1, "", "response"], [45, 0, 1, "", "time_range_args"]], "searx.engines.google_videos": [[45, 0, 1, "", "request"], [45, 0, 1, "", "response"]], "searx.engines.lemmy": [[46, 2, 1, "", "base_url"], [46, 2, 1, "", "lemmy_type"]], "searx.engines.mediawiki": [[32, 2, 1, "", "base_url"], [32, 2, 1, "", "search_type"], [32, 2, 1, "", "srenablerewrites"], [32, 2, 1, "", "srprop"], [32, 2, 1, "", "srsort"], [32, 2, 1, "", "timestamp_format"]], "searx.engines.peertube": [[47, 2, 1, "", "base_url"], [47, 0, 1, "", "fetch_traits"], [47, 0, 1, "", "request"], [47, 0, 1, "", "video_response"]], "searx.engines.piped": [[48, 2, 1, "", "backend_url"], [48, 2, 1, "", "frontend_url"], [48, 2, 1, "", "piped_filter"]], "searx.engines.sepiasearch": [[47, 0, 1, "", "request"]], "searx.engines.sqlite": [[36, 0, 1, "", "sqlite_cursor"]], "searx.engines.startpage": [[50, 0, 1, "", "fetch_traits"], [50, 0, 1, "", "get_sc_code"], [50, 0, 1, "", "request"], [50, 2, 1, "", "sc_code_cache_sec"], [50, 2, 1, "", "search_form_xpath"], [50, 2, 1, "", "send_accept_language_header"], [50, 2, 1, "", "startpage_categ"]], "searx.engines.tineye": [[57, 2, 1, "", "DOWNLOAD_ERROR"], [57, 2, 1, "", "FORMAT_NOT_SUPPORTED"], [57, 2, 1, "", "NO_SIGNATURE_ERROR"], [57, 2, 1, "", "engine_type"], [57, 0, 1, "", "parse_tineye_match"], [57, 0, 1, "", "request"], [57, 0, 1, "", "response"]], "searx.engines.torznab": [[52, 0, 1, "", "build_result"], [52, 0, 1, "", "get_attribute"], [52, 0, 1, "", "get_torznab_attribute"], [52, 0, 1, "", "init"], [52, 0, 1, "", "request"], [52, 0, 1, "", "response"]], "searx.engines.wikidata": [[53, 0, 1, "", "fetch_traits"], [53, 0, 1, "", "get_thumbnail"]], "searx.engines.wikipedia": [[53, 0, 1, "", "fetch_wikimedia_traits"], [53, 0, 1, "", "get_wiki_params"], [53, 2, 1, "", "list_of_wikipedias"], [53, 0, 1, "", "request"], [53, 2, 1, "", "rest_v1_summary_url"], [53, 2, 1, "", "send_accept_language_header"], [53, 2, 1, "", "wiki_lc_locale_variants"], [53, 2, 1, "", "wikipedia_article_depth"]], "searx.engines.xpath": [[58, 2, 1, "", "content_xpath"], [58, 2, 1, "", "cookies"], [58, 2, 1, "", "first_page_num"], [58, 2, 1, "", "headers"], [58, 2, 1, "", "lang_all"], [58, 2, 1, "", "no_result_for_http_status"], [58, 2, 1, "", "page_size"], [58, 2, 1, "", "paging"], [58, 0, 1, "", "request"], [58, 0, 1, "", "response"], [58, 2, 1, "", "results_xpath"], [58, 2, 1, "", "safe_search_map"], [58, 2, 1, "", "safe_search_support"], [58, 2, 1, "", "search_url"], [58, 2, 1, "", "soft_max_redirects"], [58, 2, 1, "", "suggestion_xpath"], [58, 2, 1, "", "thumbnail_xpath"], [58, 2, 1, "", "time_range_map"], [58, 2, 1, "", "time_range_support"], [58, 2, 1, "", "time_range_url"], [58, 2, 1, "", "title_xpath"], [58, 2, 1, "", "url_xpath"]], "searx.engines.yacy": [[54, 2, 1, "", "search_mode"]], "searx.engines.yahoo": [[55, 0, 1, "", "fetch_traits"], [55, 2, 1, "", "lang2domain"], [55, 0, 1, "", "parse_url"], [55, 0, 1, "", "request"], [55, 0, 1, "", "response"]], "searx.engines.zlibrary": [[56, 0, 1, "", "fetch_traits"], [56, 0, 1, "", "init"], [56, 2, 1, "", "zlib_ext"], [56, 2, 1, "", "zlib_year_from"], [56, 2, 1, "", "zlib_year_to"]], "searx.exceptions": [[75, 6, 1, "", "SearxEngineAPIException"], [75, 6, 1, "", "SearxEngineAccessDeniedException"], [75, 6, 1, "", "SearxEngineCaptchaException"], [75, 6, 1, "", "SearxEngineException"], [75, 6, 1, "", "SearxEngineResponseException"], [75, 6, 1, "", "SearxEngineTooManyRequestsException"], [75, 6, 1, "", "SearxEngineXPathException"], [75, 6, 1, "", "SearxException"], [75, 6, 1, "", "SearxParameterException"], [75, 6, 1, "", "SearxSettingsException"], [75, 6, 1, "", "SearxXPathSyntaxException"]], "searx.exceptions.SearxEngineAccessDeniedException": [[75, 4, 1, "", "SUSPEND_TIME_SETTING"]], "searx.exceptions.SearxEngineCaptchaException": [[75, 4, 1, "", "SUSPEND_TIME_SETTING"]], "searx.exceptions.SearxEngineTooManyRequestsException": [[75, 4, 1, "", "SUSPEND_TIME_SETTING"]], "searx.infopage": [[76, 3, 1, "", "InfoPage"], [76, 3, 1, "", "InfoPageSet"]], "searx.infopage.InfoPage": [[76, 7, 1, "", "content"], [76, 5, 1, "", "get_ctx"], [76, 7, 1, "", "html"], [76, 7, 1, "", "raw_content"], [76, 7, 1, "", "title"]], "searx.infopage.InfoPageSet": [[76, 4, 1, "", "folder"], [76, 5, 1, "", "get_page"], [76, 5, 1, "", "iter_pages"], [76, 4, 1, "", "locale_default"], [76, 4, 1, "", "locales"], [76, 4, 1, "", "toc"]], "searx.locales": [[77, 2, 1, "", "ADDITIONAL_TRANSLATIONS"], [77, 2, 1, "", "LOCALE_BEST_MATCH"], [77, 2, 1, "", "LOCALE_NAMES"], [77, 2, 1, "", "RTL_LOCALES"], [77, 0, 1, "", "build_engine_locales"], [77, 0, 1, "", "get_engine_locale"], [77, 0, 1, "", "get_locale"], [77, 0, 1, "", "get_locale_descr"], [77, 0, 1, "", "get_offical_locales"], [77, 0, 1, "", "get_translations"], [77, 0, 1, "", "language_tag"], [77, 0, 1, "", "locales_initialize"], [77, 0, 1, "", "match_locale"], [77, 0, 1, "", "region_tag"]], "searx.plugins": [[78, 1, 0, "-", "tor_check"]], "searx.plugins.tor_check": [[78, 2, 1, "", "description"], [78, 2, 1, "", "name"], [78, 2, 1, "", "preference_section"], [78, 2, 1, "", "query_examples"], [78, 2, 1, "", "query_keywords"]], "searx.redisdb": [[79, 2, 1, "", "OLD_REDIS_URL_DEFAULT_URL"]], "searx.redislib": [[80, 2, 1, "", "LUA_SCRIPT_STORAGE"], [80, 0, 1, "", "drop_counter"], [80, 0, 1, "", "incr_counter"], [80, 0, 1, "", "incr_sliding_window"], [80, 0, 1, "", "lua_script_storage"], [80, 0, 1, "", "purge_by_prefix"], [80, 0, 1, "", "secret_hash"]], "searx.search": [[81, 3, 1, "", "EngineRef"], [81, 3, 1, "", "Search"], [81, 3, 1, "", "SearchQuery"], [81, 3, 1, "", "SearchWithPlugins"]], "searx.search.Search": [[81, 4, 1, "", "result_container"], [81, 5, 1, "", "search"], [81, 4, 1, "", "search_query"]], "searx.search.SearchWithPlugins": [[81, 4, 1, "", "ordered_plugin_list"], [81, 4, 1, "", "request"], [81, 4, 1, "", "result_container"], [81, 5, 1, "", "search"], [81, 4, 1, "", "search_query"]], "searx.search.processors": [[82, 1, 0, "-", "abstract"], [82, 1, 0, "-", "offline"], [82, 1, 0, "-", "online"], [82, 1, 0, "-", "online_currency"], [82, 1, 0, "-", "online_dictionary"], [82, 1, 0, "-", "online_url_search"]], "searx.search.processors.abstract": [[82, 3, 1, "", "EngineProcessor"], [82, 3, 1, "", "SuspendedStatus"]], "searx.search.processors.abstract.EngineProcessor": [[82, 5, 1, "", "get_params"]], "searx.search.processors.offline": [[82, 3, 1, "", "OfflineProcessor"]], "searx.search.processors.online": [[82, 3, 1, "", "OnlineProcessor"], [82, 0, 1, "", "default_request_params"]], "searx.search.processors.online.OnlineProcessor": [[82, 5, 1, "", "get_params"]], "searx.search.processors.online_currency": [[82, 3, 1, "", "OnlineCurrencyProcessor"]], "searx.search.processors.online_currency.OnlineCurrencyProcessor": [[82, 5, 1, "", "get_params"]], "searx.search.processors.online_dictionary": [[82, 3, 1, "", "OnlineDictionaryProcessor"]], "searx.search.processors.online_dictionary.OnlineDictionaryProcessor": [[82, 5, 1, "", "get_params"]], "searx.search.processors.online_url_search": [[82, 3, 1, "", "OnlineUrlSearchProcessor"]], "searx.search.processors.online_url_search.OnlineUrlSearchProcessor": [[82, 5, 1, "", "get_params"]], "searx.sxng_locales": [[77, 2, 1, "", "sxng_locales"]], "searx.utils": [[83, 2, 1, "", "SEARCH_LANGUAGE_CODES"], [83, 0, 1, "", "convert_str_to_int"], [83, 0, 1, "", "detect_language"], [83, 0, 1, "", "dict_subset"], [83, 0, 1, "", "ecma_unescape"], [83, 0, 1, "", "eval_xpath"], [83, 0, 1, "", "eval_xpath_getindex"], [83, 0, 1, "", "eval_xpath_list"], [83, 0, 1, "", "extract_text"], [83, 0, 1, "", "extract_url"], [83, 0, 1, "", "gen_useragent"], [83, 0, 1, "", "get_engine_from_settings"], [83, 0, 1, "", "get_torrent_size"], [83, 0, 1, "", "get_xpath"], [83, 0, 1, "", "html_to_text"], [83, 0, 1, "", "int_or_zero"], [83, 0, 1, "", "is_valid_lang"], [83, 0, 1, "", "normalize_url"], [83, 0, 1, "", "searx_useragent"], [83, 0, 1, "", "to_string"]], "searxng_extra": [[67, 1, 0, "-", "standalone_searx"]], "searxng_extra.standalone_searx": [[67, 0, 1, "", "get_search_query"], [67, 0, 1, "", "json_serial"], [67, 0, 1, "", "no_parsed_url"], [67, 0, 1, "", "parse_argument"], [67, 0, 1, "", "to_dict"]], "searxng_extra.update": [[68, 1, 0, "-", "update_ahmia_blacklist"], [68, 1, 0, "-", "update_currencies"], [68, 1, 0, "-", "update_engine_descriptions"], [68, 1, 0, "-", "update_engine_traits"], [68, 1, 0, "-", "update_external_bangs"], [68, 1, 0, "-", "update_firefox_version"], [68, 1, 0, "-", "update_osm_keys_tags"], [68, 1, 0, "-", "update_pygments"], [68, 1, 0, "-", "update_wikidata_units"]], "searxng_extra.update.update_engine_descriptions": [[68, 0, 1, "", "get_output"]], "searxng_extra.update.update_engine_traits": [[68, 3, 1, "", "UnicodeEscape"], [68, 0, 1, "", "fetch_traits_map"], [68, 0, 1, "", "filter_locales"], [68, 0, 1, "", "get_unicode_flag"]], "searxng_extra.update.update_external_bangs": [[68, 0, 1, "", "merge_when_no_leaf"]]}, "objtypes": {"0": "py:function", "1": "py:module", "2": "py:data", "3": "py:class", "4": "py:attribute", "5": "py:method", "6": "py:exception", "7": "py:property"}, "objnames": {"0": ["py", "function", "Python function"], "1": ["py", "module", "Python module"], "2": ["py", "data", "Python data"], "3": ["py", "class", "Python class"], "4": ["py", "attribute", "Python attribute"], "5": ["py", "method", "Python method"], "6": ["py", "exception", "Python exception"], "7": ["py", "property", "Python property"]}, "titleterms": {"answer": [0, 44], "captcha": 0, "from": 0, "server": [0, 6, 8, 22, 34], "": [0, 6, 8, 11, 30, 38, 77], "ip": 0, "ssh": 0, "manual": 0, "administr": [1, 4], "api": [1, 35, 44, 45, 51, 65], "get": [1, 7], "configur": [1, 10, 12, 28, 32, 33, 34, 36, 38, 42, 46, 48, 49, 52, 54, 56, 58, 85], "data": 1, "sampl": 1, "respons": 1, "emb": 1, "search": [1, 17, 21, 31, 35, 61, 65, 81, 82, 84, 87], "bar": 1, "architectur": 2, "further": [2, 6, 8, 9, 11, 12, 13, 14, 17, 24, 28, 34, 35, 36, 60, 61, 62, 63, 64, 65, 85, 89, 90], "read": [2, 6, 8, 9, 11, 12, 13, 14, 17, 24, 28, 34, 35, 36, 60, 61, 62, 63, 64, 65, 85, 89, 90], "uwsgi": [2, 6, 11], "setup": [2, 11, 28, 89], "buildhost": [3, 89], "build": [3, 7, 25, 61, 64], "develop": [3, 20, 59, 60, 63], "tool": [3, 66, 88], "doc": [3, 25, 61], "sphinx": [3, 64], "need": 3, "lint": 3, "shell": [3, 7, 61], "script": [3, 9], "document": [4, 25, 59], "instal": [5, 9, 10, 24, 61, 89, 90], "apach": 6, "The": [6, 8, 25, 28, 60], "http": [6, 8, 74], "debian": [6, 11], "layout": [6, 11], "modul": [6, 28], "site": [6, 8], "searxng": [6, 7, 8, 10, 24, 30, 60, 70, 71, 75, 77, 84, 89, 90], "header": [6, 74], "disabl": [6, 8], "log": [6, 8], "docker": [7, 89], "contain": [7, 60], "info": [7, 17, 24, 31, 33, 34, 35, 36, 49, 74, 76], "hint": [7, 89], "run": [7, 61, 71, 89], "warn": 7, "insid": 7, "bashism": 7, "imag": [7, 28, 40, 44, 45, 64, 85], "command": [7, 33, 88, 89, 90], "line": [7, 33, 64], "nginx": 8, "updat": [9, 24, 68], "o": 9, "first": 9, "step": 10, "packag": [10, 85], "creat": [10, 25], "user": [10, 86], "depend": [10, 34, 36, 37], "use_default_set": [10, 14], "true": [10, 14], "check": [10, 24, 78], "origin": [11, 70], "distributor": 11, "mainten": [11, 24], "pitfal": 11, "tyrant": 11, "mode": 11, "plugin": [12, 62, 78], "builtin": 12, "built": 12, "time": 12, "default": [12, 28, 84], "set": [13, 14, 28, 84], "yml": [14, 28], "locat": 14, "brand": 15, "categories_as_tab": 16, "engin": [17, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36, 37, 40, 41, 44, 45, 47, 49, 50, 55, 58, 60, 61, 83, 84, 85, 87], "privat": [17, 37, 71], "token": 17, "exampl": [17, 33, 34, 35, 36, 49, 58, 62, 64], "multilingu": 17, "gener": [18, 28, 64, 85], "outgo": 19, "redi": [20, 34, 61, 79, 80], "note": [20, 64], "ui": 23, "how": [24, 25, 71, 84], "inspect": 24, "debug": 24, "migrat": 24, "stai": 24, "tune": 24, "remov": 24, "obsolet": 24, "servic": 24, "after": 24, "contribut": 25, "prime": 25, "direct": 25, "privaci": [25, 71], "hackabl": 25, "design": 25, "code": [25, 64, 72, 77], "good": [25, 89], "commit": 25, "translat": [25, 69], "rest": [25, 64], "sourc": [25, 72], "live": [25, 61], "clean": [25, 61, 89], "deploi": 25, "github": 25, "io": 25, "demo": [26, 27], "offlin": [26, 31, 37, 82], "onlin": [27, 28, 31, 76, 82], "overview": 28, "file": [28, 64, 85], "common": [28, 88], "option": 28, "overrid": 28, "name": [28, 61, 64], "i": [28, 71, 84], "arbitrari": 28, "recommend": 28, "ar": [28, 71], "make": [28, 61, 84], "request": [28, 74], "pass": 28, "argument": 28, "If": 28, "engine_typ": 28, "online_dictionari": 28, "addit": 28, "online_curr": 28, "online_url_search": 28, "specifi": 28, "result": 28, "type": [28, 31], "templat": [28, 64], "paramet": [28, 65], "media": 28, "video": [28, 40, 43, 45, 47, 85], "torrent": 28, "map": [28, 85], "paper": 28, "see": [28, 71], "bibtex": 28, "field": [28, 64], "format": [28, 61], "librari": [29, 56, 80], "trait": 29, "loader": 30, "implement": [31, 32, 33, 34, 36, 38, 41, 42, 46, 48, 49, 52, 54, 56, 58], "framework": 31, "compon": 31, "url": [31, 64, 82], "currenc": [31, 82], "dictionari": [31, 82], "mediawiki": 32, "nosql": 34, "databas": 34, "extra": [34, 36, 37], "mongodb": 34, "local": [35, 77], "meilisearch": 35, "elasticsearch": 35, "solr": 35, "sql": 36, "sqlite": 36, "postgresql": 36, "mysql": 36, "concept": 37, "program": 37, "interfac": 37, "secur": [37, 45], "anna": 38, "archiv": 38, "arch": 39, "linux": [39, 60], "wiki": 39, "bing": 40, "web": [40, 45, 85], "new": [40, 45, 85], "brave": 41, "content": [41, 42, 45, 46, 48, 54, 56, 64], "region": [41, 50], "languag": [41, 50, 87], "bt4g": 42, "dailymot": 43, "dukcdukgo": 44, "duckduckgo": 44, "lite": 44, "instant": 44, "weather": 44, "googl": 45, "autocomplet": 45, "polici": 45, "csp": 45, "scholar": 45, "lemmi": 46, "peertub": 47, "sepiasearch": 47, "pipe": 48, "known": 48, "quirk": 48, "recol": 49, "startpag": 50, "categori": [50, 87], "tagesschau": 51, "torznab": 52, "webapi": 52, "wikimedia": [53, 85], "wikipedia": 53, "wikidata": 53, "yaci": 54, "yahoo": 55, "z": 56, "tiney": 57, "xpath": 58, "audienc": 60, "motiv": 60, "gentlemen": 60, "start": 60, "your": 60, "archlinux": 60, "fulli": 60, "function": [60, 83], "suit": [60, 89], "In": 60, "work": [60, 84], "usual": 60, "wrap": 60, "product": 60, "summari": 60, "makefil": 61, "manag": 61, "environ": [61, 64, 88], "python": 61, "activ": 61, "drop": 61, "buildenv": 61, "node": 61, "j": 61, "env": 61, "nvm": 61, "statu": 61, "nodej": 61, "gh": 61, "page": 61, "test": 61, "pylint": 61, "checker": 61, "theme": 61, "static": 61, "help": [61, 89, 90], "go": 61, "extern": [62, 87], "entri": 62, "point": 62, "quickstart": 63, "primer": 64, "kiss": 64, "readabl": 64, "matter": 64, "soft": 64, "skill": 64, "basic": 64, "inlin": 64, "markup": 64, "articl": 64, "structur": 64, "head": 64, "anchor": 64, "link": 64, "ref": 64, "role": 64, "ordinari": 64, "hyperlink": 64, "smart": 64, "ext": 64, "extlink": 64, "intersphinx": 64, "liter": 64, "block": 64, "syntax": [64, 87], "highlight": 64, "unicod": 64, "substitut": 64, "figur": 64, "process": 64, "dot": 64, "aka": 64, "graphviz": 64, "hello": 64, "kernel": 64, "render": 64, "svg": 64, "list": 64, "bullet": 64, "horizont": 64, "hlist": 64, "definit": 64, "quot": 64, "paragraph": 64, "bibliograph": 64, "admonit": 64, "sidebar": 64, "titl": 64, "specif": 64, "tabl": 64, "nest": 64, "simpl": 64, "ascii": 64, "foo": 64, "gate": 64, "truth": 64, "grid": 64, "flat": 64, "csv": 64, "tab": [64, 85], "view": 64, "math": 64, "equat": 64, "about": [64, 71, 84], "latex": 64, "space": 64, "box": [66, 88], "searxng_extra": [66, 67, 68], "standalone_searx": 67, "py": [67, 68], "update_ahmia_blacklist": 68, "update_curr": 68, "update_engine_descript": 68, "update_external_bang": 68, "update_firefox_vers": 68, "update_engine_trait": 68, "update_osm_keys_tag": 68, "update_pyg": 68, "update_wikidata_unit": 68, "wlc": 69, "welcom": 70, "featur": 70, "part": 70, "why": [71, 84], "us": [71, 84], "instanc": 71, "worth": 71, "my": [71, 84], "own": [71, 84], "doe": [71, 84], "protect": 71, "what": 71, "consequ": 71, "public": 71, "conclus": 71, "custom": 73, "messag": 73, "extractor": 73, "i18n": 73, "bot": 74, "detect": 74, "x": 74, "forward": 74, "For": 74, "flask": 74, "remote_addr": 74, "limit": 74, "method": 74, "ip_list": 74, "rate": 74, "ip_limit": 74, "link_token": 74, "probe": 74, "http_accept": 74, "http_accept_encod": 74, "http_accept_languag": 74, "http_connect": 74, "http_user_ag": 74, "except": 75, "tor": 78, "db": 79, "processor": 82, "abstract": 82, "class": 82, "util": [83, 89, 90], "do": 84, "can": 84, "group": 85, "without": 85, "subgroup": 85, "music": 85, "lyric": 85, "q": 85, "repo": 85, "software_wiki": 85, "scienc": 85, "scientific_publ": 85, "app": 85, "social_media": 85, "inform": 86, "select": 87, "bang": 87, "special": 87, "queri": 87, "devop": 88, "lxc": 89, "sh": [89, 90], "lxd": 89, "internet": 89, "connect": 89, "know": 89, "up": 89, "config": 89}, "envversion": {"sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1, "sphinx.ext.intersphinx": 1, "sphinx": 58}, "alltitles": {"Answer CAPTCHA from server\u2019s IP": [[0, "answer-captcha-from-server-s-ip"]], "ssh manual:": [[0, null]], "Administration API": [[1, "administration-api"]], "Get configuration data": [[1, "get-configuration-data"]], "Sample response": [[1, "sample-response"]], "Embed search bar": [[1, "embed-search-bar"]], "Architecture": [[2, "architecture"]], "Further reading": [[2, null], [64, null], [64, null]], "uWSGI Setup": [[2, "uwsgi-setup"]], "Buildhosts": [[3, "buildhosts"]], "Build and Development tools": [[3, "build-and-development-tools"]], "Build docs": [[3, "build-docs"]], "Sphinx build needs": [[3, null]], "Lint shell scripts": [[3, "lint-shell-scripts"]], "Administrator documentation": [[4, "administrator-documentation"]], "Installation": [[5, "installation"]], "Apache": [[6, "apache"]], "further read": [[6, null], [9, null], [9, null], [9, null], [24, null], [34, null], [35, null], [36, null], [60, null], [61, null], [61, null], [63, null], [63, null], [63, null], [89, null], [89, null]], "The Apache HTTP server": [[6, "the-apache-http-server"]], "Debian\u2019s Apache layout": [[6, "debian-s-apache-layout"]], "Apache modules": [[6, "apache-modules"]], "Apache sites": [[6, "apache-sites"]], "Apache\u2019s SearXNG site": [[6, "apache-s-searxng-site"]], "uWSGI": [[6, null], [11, "uwsgi"]], "HTTP headers": [[6, null]], "disable logs": [[6, "disable-logs"]], "Docker Container": [[7, "docker-container"]], "info": [[7, null], [17, null], [24, null], [31, null], [31, null], [31, null], [31, null], [31, null], [33, null], [33, null], [34, null], [34, null], [34, null], [35, null], [35, null], [35, null], [35, null], [36, null], [36, null], [36, null], [36, null], [49, null], [74, null]], "hint": [[7, null], [89, null]], "Get Docker": [[7, "get-docker"]], "searxng/searxng": [[7, "searxng-searxng"]], "docker run": [[7, null], [7, null]], "Warning": [[7, null]], "shell inside container": [[7, "shell-inside-container"]], "Bashism": [[7, null]], "Build the image": [[7, "build-the-image"]], "Command line": [[7, "command-line"]], "NGINX": [[8, "nginx"]], "further reading": [[8, null], [11, null], [89, null], [90, null]], "The nginx HTTP server": [[8, "the-nginx-http-server"]], "NGINX\u2019s SearXNG site": [[8, "nginx-s-searxng-site"]], "Disable logs": [[8, "disable-logs"]], "Installation Script": [[9, "installation-script"]], "Update the OS first!": [[9, null]], "Step by step installation": [[10, "step-by-step-installation"]], "Install packages": [[10, "install-packages"]], "Create user": [[10, "create-user"]], "Install SearXNG & dependencies": [[10, "install-searxng-dependencies"]], "Configuration": [[10, "configuration"], [32, "configuration"], [33, "configuration"], [34, "configuration"], [34, "id3"], [38, "configuration"], [42, "configuration"], [46, "configuration"], [48, "configuration"], [49, "configuration"], [52, "configuration"], [54, "configuration"], [56, "configuration"], [58, "configuration"]], "use_default_settings: True": [[10, null]], "Check": [[10, "check"]], "Origin uWSGI": [[11, "origin-uwsgi"]], "Distributors": [[11, "distributors"]], "Debian\u2019s uWSGI layout": [[11, "debian-s-uwsgi-layout"]], "uWSGI maintenance": [[11, "uwsgi-maintenance"]], "uWSGI setup": [[11, "uwsgi-setup"]], "Pitfalls of the Tyrant mode": [[11, "pitfalls-of-the-tyrant-mode"]], "Plugins builtin": [[12, "plugins-builtin"]], "Further reading ..": [[12, null], [13, null], [14, null], [17, null], [28, null], [62, null], [65, null], [85, null]], "Plugins configured at built time (defaults)": [[12, "id1"]], "Settings": [[13, "settings"]], "settings.yml": [[14, "settings-yml"]], "settings.yml location": [[14, "settings-yml-location"]], "use_default_settings": [[14, "use-default-settings"]], "use_default_settings: true": [[14, null]], "brand:": [[15, "brand"]], "categories_as_tabs:": [[16, "categories-as-tabs"]], "engine:": [[17, "engine"]], "Private Engines (tokens)": [[17, "private-engines-tokens"]], "Example: Multilingual Search": [[17, "example-multilingual-search"]], "general:": [[18, "general"]], "outgoing:": [[19, "outgoing"]], "redis:": [[20, "redis"]], "Redis Developer Notes": [[20, "redis-developer-notes"]], "search:": [[21, "search"]], "server:": [[22, "server"]], "ui:": [[23, "ui"]], "SearXNG maintenance": [[24, "searxng-maintenance"]], "How to update": [[24, "how-to-update"]], "How to inspect & debug": [[24, "how-to-inspect-debug"]], "Migrate and stay tuned!": [[24, "migrate-and-stay-tuned"]], "remove obsolete services": [[24, "remove-obsolete-services"]], "Check after Installation": [[24, "check-after-installation"]], "How to contribute": [[25, "how-to-contribute"]], "Prime directives: Privacy, Hackability": [[25, "prime-directives-privacy-hackability"]], "Privacy-by-design": [[25, "privacy-by-design"]], "Code": [[25, "code"]], "Create good commits!": [[25, null]], "Translation": [[25, "translation"], [69, "translation"]], "Documentation": [[25, "documentation"]], "The reST sources": [[25, null]], "live build": [[25, "live-build"]], "docs.clean": [[25, null]], "deploy on github.io": [[25, "deploy-on-github-io"]], "Demo Offline Engine": [[26, "demo-offline-engine"]], "Demo Online Engine": [[27, "demo-online-engine"]], "Engine Overview": [[28, "engine-overview"]], "General Engine Configuration": [[28, "general-engine-configuration"]], "Engine File": [[28, "engine-file"]], "Common options in the engine module": [[28, "id3"]], "Engine settings.yml": [[28, "engine-settings-yml"]], "Common options in the engine setup (settings.yml)": [[28, "id4"]], "Overrides": [[28, "overrides"]], "The naming of overrides is arbitrary / recommended overrides are:": [[28, "id5"]], "Making a Request": [[28, "making-a-request"]], "Passed Arguments (request)": [[28, "passed-arguments-request"]], "If the engine_type is online": [[28, "id6"]], "If the engine_type is online_dictionary,\n in addition to the online arguments:": [[28, "id7"]], "If the engine_type is online_currency,\n in addition to the online arguments:": [[28, "id8"]], "If the engine_type is online_url_search,\n in addition to the online arguments:": [[28, "id9"]], "Specify Request": [[28, "specify-request"]], "Result Types (template)": [[28, "result-types-template"]], "default": [[28, "default"]], "Parameter of the default media type:": [[28, "id10"]], "images": [[28, "images"]], "Parameter of the images media type:": [[28, "id11"]], "videos": [[28, "videos"]], "Parameter of the videos media type:": [[28, "id12"]], "torrent": [[28, "torrent"]], "Parameter of the torrent media type:": [[28, "id13"]], "map": [[28, "map"]], "Parameter of the map media type:": [[28, "id14"]], "paper": [[28, "paper"]], "Parameter of the paper media type /\n see BibTeX field types and BibTeX format": [[28, "id15"]], "Engine Library": [[29, "engine-library"]], "Engine traits": [[29, "module-searx.enginelib.traits"]], "SearXNG\u2019s engines loader": [[30, "module-searx.engines"]], "Engine Implementations": [[31, "engine-implementations"]], "Framework Components": [[31, null]], "Engine Types": [[31, "engine-types"]], "Online Engines": [[31, "online-engines"]], "Offline Engines": [[31, "offline-engines"]], "Online URL Search": [[31, "online-url-search"]], "Online Currency": [[31, "online-currency"]], "Online Dictionary": [[31, "online-dictionary"]], "MediaWiki Engine": [[32, "mediawiki-engine"]], "Implementations": [[32, "implementations"], [33, "implementations"], [34, "implementations"], [34, "id5"], [36, "implementations"], [36, "id5"], [36, "id8"], [38, "implementations"], [41, "implementations"], [42, "implementations"], [46, "implementations"], [48, "implementations"], [49, "implementations"], [52, "implementations"], [54, "implementations"], [56, "implementations"], [58, "implementations"]], "Command Line Engines": [[33, "command-line-engines"]], "Example": [[33, "example"], [34, "example"], [34, "id4"], [35, "example"], [35, "id3"], [35, "id5"], [36, "example"], [36, "id4"], [36, "id7"], [49, "example"], [58, "example"]], "NoSQL databases": [[34, "nosql-databases"]], "Extra Dependencies": [[34, "extra-dependencies"], [36, "extra-dependencies"], [37, "extra-dependencies"]], "Configure the engines": [[34, "configure-the-engines"], [36, "configure-the-engines"]], "Redis Server": [[34, "redis-server"]], "MongoDB": [[34, "engine-mongodb"]], "Local Search APIs": [[35, "local-search-apis"]], "MeiliSearch": [[35, "module-searx.engines.meilisearch"]], "Elasticsearch": [[35, "module-searx.engines.elasticsearch"]], "Solr": [[35, "module-searx.engines.solr"]], "SQL Engines": [[36, "sql-engines"]], "SQLite": [[36, "engine-sqlite"]], "PostgreSQL": [[36, "engine-postgresql"]], "MySQL": [[36, "engine-mysql-server"]], "Offline Concept": [[37, "offline-concept"]], "offline engines": [[37, null]], "Programming Interface": [[37, "programming-interface"]], "Private engines (Security)": [[37, "private-engines-security"]], "Anna\u2019s Archive": [[38, "anna-s-archive"]], "Arch Linux": [[39, "arch-linux"]], "Arch Linux Wiki": [[39, "arch-linux-wiki"]], "Bing Engines": [[40, "bing-engines"]], "Bing WEB": [[40, "module-searx.engines.bing"]], "Bing Images": [[40, "module-searx.engines.bing_images"]], "Bing Videos": [[40, "module-searx.engines.bing_videos"]], "Bing News": [[40, "module-searx.engines.bing_news"]], "Brave Engines": [[41, "brave-engines"]], "Contents": [[41, "contents"], [42, "contents"], [46, "contents"], [48, "contents"], [54, "contents"], [56, "contents"]], "Brave regions": [[41, "brave-regions"]], "Brave languages": [[41, "brave-languages"]], "BT4G": [[42, "bt4g"]], "Dailymotion": [[43, "dailymotion"]], "Dailymotion (Videos)": [[43, "dailymotion-videos"]], "DukcDukGo Engines": [[44, "dukcdukgo-engines"]], "DuckDuckGo Lite": [[44, "duckduckgo-lite"]], "DuckDuckGo Images": [[44, "duckduckgo-images"]], "DuckDuckGo Instant Answer API": [[44, "duckduckgo-instant-answer-api"]], "DuckDuckGo Weather": [[44, "duckduckgo-weather"]], "Google Engines": [[45, "google-engines"]], "Google API": [[45, "google-api"]], "Google WEB": [[45, "module-searx.engines.google"]], "Google Autocomplete": [[45, "google-autocomplete"]], "Google Images": [[45, "module-searx.engines.google_images"]], "Google Videos": [[45, "module-searx.engines.google_videos"]], "Content-Security-Policy (CSP)": [[45, null]], "Google News": [[45, "module-searx.engines.google_news"]], "Google Scholar": [[45, "module-searx.engines.google_scholar"]], "Lemmy": [[46, "lemmy"]], "Peertube Engines": [[47, "peertube-engines"]], "Peertube Video": [[47, "module-searx.engines.peertube"]], "SepiaSearch": [[47, "module-searx.engines.sepiasearch"]], "Piped": [[48, "piped"]], "Known Quirks": [[48, "known-quirks"]], "Recoll Engine": [[49, "recoll-engine"]], "Startpage Engines": [[50, "startpage-engines"]], "Startpage regions": [[50, "startpage-regions"]], "Startpage languages": [[50, "startpage-languages"]], "Startpage categories": [[50, "startpage-categories"]], "Tagesschau API": [[51, "tagesschau-api"]], "Torznab WebAPI": [[52, "torznab-webapi"]], "Wikimedia": [[53, "wikimedia"]], "Wikipedia": [[53, "module-searx.engines.wikipedia"]], "Wikidata": [[53, "module-searx.engines.wikidata"]], "Yacy": [[54, "yacy"]], "Yahoo Engine": [[55, "yahoo-engine"]], "Z-Library": [[56, "z-library"]], "Tineye": [[57, "tineye"]], "XPath Engine": [[58, "xpath-engine"]], "Developer documentation": [[59, "developer-documentation"]], "Developing in Linux Containers": [[60, "developing-in-linux-containers"]], "Audience": [[60, null]], "Motivation": [[60, "motivation"]], "Gentlemen, start your engines!": [[60, "gentlemen-start-your-engines"]], "The searxng-archlinux container": [[60, null]], "Fully functional SearXNG suite": [[60, null]], "In containers, work as usual": [[60, "in-containers-work-as-usual"]], "Wrap production into developer suite": [[60, "wrap-production-into-developer-suite"]], "Summary": [[60, "summary"]], "Makefile & ./manage": [[61, "makefile-manage"]], "build environment": [[61, null]], "Python environment (make install)": [[61, "python-environment-make-install"]], "activate environment": [[61, null]], "drop environment": [[61, null]], "make buildenv": [[61, "make-buildenv"]], "Node.js environment (make node.env)": [[61, "node-js-environment-make-node-env"]], "NVM make nvm.install nvm.status": [[61, "nvm-make-nvm-install-nvm-status"]], "make nvm.nodejs": [[61, "make-nvm-nodejs"]], "make run": [[61, "make-run"]], "make format.python": [[61, "make-format-python"]], "make clean": [[61, "make-clean"]], "make docs": [[61, "make-docs"]], "make docs.clean docs.live": [[61, "make-docs-clean-docs-live"]], "make docs.gh-pages": [[61, "make-docs-gh-pages"]], "make test": [[61, "make-test"]], "make test.shell": [[61, "make-test-shell"]], "make test.pylint": [[61, "make-test-pylint"]], "make search.checker.{engine name}": [[61, "make-search-checker-engine-name"]], "make themes.*": [[61, "make-themes"]], "make static.build.*": [[61, "make-static-build"]], "./manage redis.help": [[61, "manage-redis-help"]], "./manage go.help": [[61, "manage-go-help"]], "Plugins": [[62, "plugins"]], "Example plugin": [[62, "example-plugin"]], "External plugins": [[62, "external-plugins"]], "Plugin entry points": [[62, "plugin-entry-points"]], "Development Quickstart": [[63, "development-quickstart"]], "reST primer": [[64, "rest-primer"]], "KISS and readability": [[64, null]], "Content matters": [[64, null]], "Soft skills": [[64, "soft-skills"]], "Basic inline markup": [[64, "basic-inline-markup"]], "Inline markup": [[64, null]], "basic inline markup": [[64, "id4"]], "Basic article structure": [[64, "basic-article-structure"]], "reST template": [[64, "rest-template"]], "Headings": [[64, "headings"]], "Anchors & Links": [[64, "anchors-links"]], "Anchors": [[64, "anchors"]], ":ref: role": [[64, null]], "Link ordinary URL": [[64, "link-ordinary-url"]], "Named hyperlink": [[64, null]], "Smart refs": [[64, "smart-refs"]], "smart refs with sphinx.ext.extlinks and intersphinx": [[64, "id5"], [64, "id6"]], "Literal blocks": [[64, "literal-blocks"]], "::": [[64, "rest-literal"]], "Literal block": [[64, null]], "code-block": [[64, "code-block"]], "Syntax highlighting": [[64, null]], "Code block": [[64, null]], "Unicode substitution": [[64, "unicode-substitution"]], "Unicode": [[64, null]], "Roles": [[64, "roles"]], "Figures & Images": [[64, "figures-images"]], "Image processing": [[64, null]], "DOT files (aka Graphviz)": [[64, "dot-files-aka-graphviz"]], "hello.dot": [[64, null]], "kernel-render DOT": [[64, "kernel-render-dot"], [64, null]], "kernel-render SVG": [[64, "kernel-render-svg"], [64, null]], "List markups": [[64, "list-markups"]], "Bullet list": [[64, "bullet-list"]], "bullet list": [[64, null]], "Horizontal list": [[64, "horizontal-list"]], "hlist": [[64, null]], "Definition list": [[64, "definition-list"]], "Note ..": [[64, null]], "definition list": [[64, null]], "Quoted paragraphs": [[64, "quoted-paragraphs"]], "Quoted paragraph and line block": [[64, null]], "Field Lists": [[64, "field-lists"]], "bibliographic fields": [[64, null]], "Field List": [[64, null]], "Further list blocks": [[64, "further-list-blocks"]], "Admonitions": [[64, "admonitions"]], "Sidebar": [[64, "sidebar"]], "Generic admonition": [[64, "generic-admonition"]], "generic admonition title": [[64, null]], "Specific admonitions": [[64, "specific-admonitions"]], "Tables": [[64, "tables"]], "Nested tables": [[64, null]], "List tables": [[64, null]], "Simple tables": [[64, "simple-tables"]], "Simple ASCII table": [[64, null]], "foo gate truth table": [[64, "id11"]], "Grid tables": [[64, "grid-tables"]], "ASCII grid table": [[64, null]], "grid table example": [[64, "id12"]], "flat-table": [[64, "flat-table"]], "List table": [[64, null]], "flat-table example": [[64, "id13"]], "CSV table": [[64, "csv-table"], [64, null]], "CSV table example": [[64, "id14"]], "Templating": [[64, "templating"]], "Build environment": [[64, null]], "Tabbed views": [[64, "tabbed-views"]], "Math equations": [[64, "math-equations"]], "About LaTeX": [[64, null]], "LaTeX math equation": [[64, null]], "Line spacing": [[64, null]], "Search API": [[65, "search-api"]], "Parameters": [[65, "parameters"]], "Tooling box searxng_extra": [[66, "tooling-box-searxng-extra"]], "searxng_extra/standalone_searx.py": [[67, "module-searxng_extra.standalone_searx"]], "searxng_extra/update/": [[68, "searxng-extra-update"]], "update_ahmia_blacklist.py": [[68, "update-ahmia-blacklist-py"]], "update_currencies.py": [[68, "update-currencies-py"]], "update_engine_descriptions.py": [[68, "update-engine-descriptions-py"]], "update_external_bangs.py": [[68, "update-external-bangs-py"]], "update_firefox_version.py": [[68, "update-firefox-version-py"]], "update_engine_traits.py": [[68, "update-engine-traits-py"]], "update_osm_keys_tags.py": [[68, "update-osm-keys-tags-py"]], "update_pygments.py": [[68, "update-pygments-py"]], "update_wikidata_units.py": [[68, "update-wikidata-units-py"]], "translated": [[69, null]], "wlc": [[69, "id2"]], "Welcome to SearXNG": [[70, "welcome-to-searxng"]], "features": [[70, null]], "be a part": [[70, null]], "the origin": [[70, null]], "Why use a private instance?": [[71, "why-use-a-private-instance"]], "Is it worth to run my own instance?": [[71, null]], "How does SearXNG protect privacy?": [[71, "how-does-searxng-protect-privacy"]], "What are the consequences of using public instances?": [[71, "what-are-the-consequences-of-using-public-instances"]], "I see. What about private instances?": [[71, "i-see-what-about-private-instances"]], "Conclusion": [[71, "conclusion"]], "Source-Code": [[72, "source-code"]], "Custom message extractor (i18n)": [[73, "module-searx.babel_extract"]], "Bot Detection": [[74, "bot-detection"]], "X-Forwarded-For": [[74, "x-forwarded-for"]], "flask.Request.remote_addr": [[74, null]], "Limiter": [[74, "limiter"]], "Method ip_lists": [[74, "method-ip-lists"]], "Rate limit": [[74, "module-searx.botdetection.ip_limit"]], "Method ip_limit": [[74, "method-ip-limit"]], "Method link_token": [[74, "method-link-token"]], "Probe HTTP headers": [[74, "module-searx.botdetection.http_accept"]], "Method http_accept": [[74, "method-http-accept"]], "Method http_accept_encoding": [[74, "method-http-accept-encoding"]], "Method http_accept_language": [[74, "method-http-accept-language"]], "Method http_connection": [[74, "method-http-connection"]], "Method http_user_agent": [[74, "method-http-user-agent"]], "SearXNG Exceptions": [[75, "module-searx.exceptions"]], "Online /info": [[76, "module-searx.infopage"]], "Locales": [[77, "locales"]], "SearXNG\u2019s locale codes": [[77, "module-searx.sxng_locales"]], "Tor check plugin": [[78, "tor-check-plugin"]], "Redis DB": [[79, "redis-db"]], "Redis Library": [[80, "module-searx.redislib"]], "Search": [[81, "search"]], "Search processors": [[82, "search-processors"]], "Abstract processor class": [[82, "module-searx.search.processors.abstract"]], "Offline processor": [[82, "module-searx.search.processors.offline"]], "Online processor": [[82, "module-searx.search.processors.online"]], "Online currency processor": [[82, "module-searx.search.processors.online_currency"]], "Online dictionary processor": [[82, "module-searx.search.processors.online_dictionary"]], "Online URL search processor": [[82, "module-searx.search.processors.online_url_search"]], "Utility functions for the engines": [[83, "module-searx.utils"]], "About SearXNG": [[84, "about-searxng"]], "Why use it?": [[84, "why-use-it"]], "How do I set it as the default search engine?": [[84, "how-do-i-set-it-as-the-default-search-engine"]], "How does it work?": [[84, "how-does-it-work"]], "How can I make it my own?": [[84, "how-can-i-make-it-my-own"]], "User information": [[86, "user-information"]], "Search syntax": [[87, "search-syntax"]], "! select engine and category": [[87, "select-engine-and-category"]], ": select language": [[87, "select-language"]], "!! external bangs": [[87, "external-bangs"]], "Special Queries": [[87, "special-queries"]], "DevOps tooling box": [[88, "devops-tooling-box"]], "Common command environments": [[88, "common-command-environments"]], "utils/lxc.sh": [[89, "utils-lxc-sh"]], "Install LXD": [[89, "install-lxd"]], "Internet Connectivity & Docker": [[89, "internet-connectivity-docker"]], "SearXNG LXC suite": [[89, "searxng-lxc-suite"]], "Running commands": [[89, "running-commands"]], "Good to know": [[89, "good-to-know"]], "Install suite": [[89, "install-suite"]], "Clean up": [[89, "clean-up"]], "Setup SearXNG buildhost": [[89, "setup-searxng-buildhost"]], "Command Help": [[89, "command-help"], [90, "command-help"]], "SearXNG suite config": [[89, "searxng-suite-config"]], "utils/searxng.sh": [[90, "utils-searxng-sh"]], "Install": [[90, "install"]], "Configured Engines": [[85, "configured-engines"]], "tab !general": [[85, "tab-general"]], "group !web": [[85, "group-web"], [85, "id2"], [85, "id4"]], "group !wikimedia": [[85, "group-wikimedia"], [85, "id6"], [85, "id13"]], "without further subgrouping": [[85, "without-further-subgrouping"], [85, "id3"], [85, "id5"], [85, "id7"], [85, "id9"], [85, "id12"], [85, "id15"], [85, "id16"]], "tab !images": [[85, "tab-images"]], "tab !videos": [[85, "tab-videos"]], "tab !news": [[85, "tab-news"]], "tab !map": [[85, "tab-map"]], "tab !music": [[85, "tab-music"]], "group !lyrics": [[85, "group-lyrics"]], "tab !it": [[85, "tab-it"]], "group !packages": [[85, "group-packages"]], "group !q&a": [[85, "group-q-a"]], "group !repos": [[85, "group-repos"]], "group !software_wikis": [[85, "group-software-wikis"]], "tab !science": [[85, "tab-science"]], "group !scientific_publications": [[85, "group-scientific-publications"]], "tab !files": [[85, "tab-files"]], "group !apps": [[85, "group-apps"]], "tab !social_media": [[85, "tab-social-media"]]}, "indexentries": {"init() (in module searx.engines.demo_offline)": [[26, "searx.engines.demo_offline.init"]], "module": [[26, "module-searx.engines.demo_offline"], [27, "module-searx.engines.demo_online"], [29, "module-searx.enginelib"], [29, "module-searx.enginelib.traits"], [30, "module-searx.engines"], [32, "module-searx.engines.mediawiki"], [33, "module-searx.engines.command"], [34, "module-searx.engines.mongodb"], [34, "module-searx.engines.redis_server"], [35, "module-searx.engines.elasticsearch"], [35, "module-searx.engines.meilisearch"], [35, "module-searx.engines.solr"], [36, "module-searx.engines.mysql_server"], [36, "module-searx.engines.postgresql"], [36, "module-searx.engines.sqlite"], [38, "module-searx.engines.annas_archive"], [39, "module-searx.engines.archlinux"], [40, "module-searx.engines.bing"], [40, "module-searx.engines.bing_images"], [40, "module-searx.engines.bing_news"], [40, "module-searx.engines.bing_videos"], [41, "module-searx.engines.brave"], [42, "module-searx.engines.bt4g"], [43, "module-searx.engines.dailymotion"], [44, "module-searx.engines.duckduckgo"], [44, "module-searx.engines.duckduckgo_definitions"], [44, "module-searx.engines.duckduckgo_images"], [44, "module-searx.engines.duckduckgo_weather"], [45, "module-searx.engines.google"], [45, "module-searx.engines.google_images"], [45, "module-searx.engines.google_news"], [45, "module-searx.engines.google_scholar"], [45, "module-searx.engines.google_videos"], [46, "module-searx.engines.lemmy"], [47, "module-searx.engines.peertube"], [47, "module-searx.engines.sepiasearch"], [48, "module-searx.engines.piped"], [49, "module-searx.engines.recoll"], [50, "module-searx.engines.startpage"], [51, "module-searx.engines.tagesschau"], [52, "module-searx.engines.torznab"], [53, "module-searx.engines.wikidata"], [53, "module-searx.engines.wikipedia"], [54, "module-searx.engines.yacy"], [55, "module-searx.engines.yahoo"], [56, "module-searx.engines.zlibrary"], [57, "module-searx.engines.tineye"], [58, "module-searx.engines.xpath"], [67, "module-searxng_extra.standalone_searx"], [68, "module-searxng_extra.update.update_ahmia_blacklist"], [68, "module-searxng_extra.update.update_currencies"], [68, "module-searxng_extra.update.update_engine_descriptions"], [68, "module-searxng_extra.update.update_engine_traits"], [68, "module-searxng_extra.update.update_external_bangs"], [68, "module-searxng_extra.update.update_firefox_version"], [68, "module-searxng_extra.update.update_osm_keys_tags"], [68, "module-searxng_extra.update.update_pygments"], [68, "module-searxng_extra.update.update_wikidata_units"], [73, "module-searx.babel_extract"], [74, "module-searx.botdetection"], [74, "module-searx.botdetection.http_accept"], [74, "module-searx.botdetection.http_accept_encoding"], [74, "module-searx.botdetection.http_accept_language"], [74, "module-searx.botdetection.http_connection"], [74, "module-searx.botdetection.http_user_agent"], [74, "module-searx.botdetection.ip_limit"], [74, "module-searx.botdetection.ip_lists"], [74, "module-searx.botdetection.limiter"], [74, "module-searx.botdetection.link_token"], [75, "module-searx.exceptions"], [76, "module-searx.infopage"], [77, "module-searx.locales"], [77, "module-searx.sxng_locales"], [78, "module-searx.plugins.tor_check"], [79, "module-searx.redisdb"], [80, "module-searx.redislib"], [82, "module-searx.search.processors.abstract"], [82, "module-searx.search.processors.offline"], [82, "module-searx.search.processors.online"], [82, "module-searx.search.processors.online_currency"], [82, "module-searx.search.processors.online_dictionary"], [82, "module-searx.search.processors.online_url_search"], [83, "module-searx.utils"]], "search() (in module searx.engines.demo_offline)": [[26, "searx.engines.demo_offline.search"]], "searx.engines.demo_offline": [[26, "module-searx.engines.demo_offline"]], "init() (in module searx.engines.demo_online)": [[27, "searx.engines.demo_online.init"]], "request() (in module searx.engines.demo_online)": [[27, "searx.engines.demo_online.request"]], "response() (in module searx.engines.demo_online)": [[27, "searx.engines.demo_online.response"]], "searx.engines.demo_online": [[27, "module-searx.engines.demo_online"]], "engine_traits_file (searx.enginelib.traits.enginetraitsmap attribute)": [[29, "searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE"]], "engine (class in searx.enginelib)": [[29, "searx.enginelib.Engine"]], "enginetraits (class in searx.enginelib.traits)": [[29, "searx.enginelib.traits.EngineTraits"]], "enginetraitsencoder (class in searx.enginelib.traits)": [[29, "searx.enginelib.traits.EngineTraitsEncoder"]], "enginetraitsmap (class in searx.enginelib.traits)": [[29, "searx.enginelib.traits.EngineTraitsMap"]], "about (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.about"]], "all_locale (searx.enginelib.traits.enginetraits attribute)": [[29, "searx.enginelib.traits.EngineTraits.all_locale"]], "categories (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.categories"]], "copy() (searx.enginelib.traits.enginetraits method)": [[29, "searx.enginelib.traits.EngineTraits.copy"]], "custom (searx.enginelib.traits.enginetraits attribute)": [[29, "searx.enginelib.traits.EngineTraits.custom"]], "data_type (searx.enginelib.traits.enginetraits attribute)": [[29, "searx.enginelib.traits.EngineTraits.data_type"]], "default() (searx.enginelib.traits.enginetraitsencoder method)": [[29, "searx.enginelib.traits.EngineTraitsEncoder.default"]], "disabled (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.disabled"]], "display_error_messages (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.display_error_messages"]], "enable_http (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.enable_http"]], "engine (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.engine"]], "engine_type (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.engine_type"]], "fetch_traits (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.fetch_traits"]], "fetch_traits() (searx.enginelib.traits.enginetraits class method)": [[29, "searx.enginelib.traits.EngineTraits.fetch_traits"]], "from_data() (searx.enginelib.traits.enginetraitsmap class method)": [[29, "searx.enginelib.traits.EngineTraitsMap.from_data"]], "get_language() (searx.enginelib.traits.enginetraits method)": [[29, "searx.enginelib.traits.EngineTraits.get_language"]], "get_region() (searx.enginelib.traits.enginetraits method)": [[29, "searx.enginelib.traits.EngineTraits.get_region"]], "inactive (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.inactive"]], "is_locale_supported() (searx.enginelib.traits.enginetraits method)": [[29, "searx.enginelib.traits.EngineTraits.is_locale_supported"]], "language (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.language"]], "language_support (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.language_support"]], "languages (searx.enginelib.traits.enginetraits attribute)": [[29, "searx.enginelib.traits.EngineTraits.languages"]], "name (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.name"]], "paging (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.paging"]], "proxies (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.proxies"]], "region (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.region"]], "regions (searx.enginelib.traits.enginetraits attribute)": [[29, "searx.enginelib.traits.EngineTraits.regions"]], "safesearch (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.safesearch"]], "save_data() (searx.enginelib.traits.enginetraitsmap method)": [[29, "searx.enginelib.traits.EngineTraitsMap.save_data"]], "searx.enginelib": [[29, "module-searx.enginelib"]], "searx.enginelib.traits": [[29, "module-searx.enginelib.traits"]], "send_accept_language_header (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.send_accept_language_header"]], "set_traits() (searx.enginelib.traits.enginetraits method)": [[29, "searx.enginelib.traits.EngineTraits.set_traits"]], "set_traits() (searx.enginelib.traits.enginetraitsmap method)": [[29, "searx.enginelib.traits.EngineTraitsMap.set_traits"]], "shortcut (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.shortcut"]], "time_range_support (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.time_range_support"]], "timeout (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.timeout"]], "tokens (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.tokens"]], "traits (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.traits"]], "using_tor_proxy (searx.enginelib.engine attribute)": [[29, "searx.enginelib.Engine.using_tor_proxy"]], "engine_shortcuts (in module searx.engines)": [[30, "searx.engines.engine_shortcuts"]], "is_missing_required_attributes() (in module searx.engines)": [[30, "searx.engines.is_missing_required_attributes"]], "load_engine() (in module searx.engines)": [[30, "searx.engines.load_engine"]], "load_engines() (in module searx.engines)": [[30, "searx.engines.load_engines"]], "searx.engines": [[30, "module-searx.engines"]], "using_tor_proxy() (in module searx.engines)": [[30, "searx.engines.using_tor_proxy"]], "base_url (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.base_url"]], "search_type (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.search_type"]], "searx.engines.mediawiki": [[32, "module-searx.engines.mediawiki"]], "srenablerewrites (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.srenablerewrites"]], "srprop (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.srprop"]], "srsort (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.srsort"]], "timestamp_format (in module searx.engines.mediawiki)": [[32, "searx.engines.mediawiki.timestamp_format"]], "check_parsing_options() (in module searx.engines.command)": [[33, "searx.engines.command.check_parsing_options"]], "searx.engines.command": [[33, "module-searx.engines.command"]], "searx.engines.mongodb": [[34, "module-searx.engines.mongodb"]], "searx.engines.redis_server": [[34, "module-searx.engines.redis_server"]], "searx.engines.elasticsearch": [[35, "module-searx.engines.elasticsearch"]], "searx.engines.meilisearch": [[35, "module-searx.engines.meilisearch"]], "searx.engines.solr": [[35, "module-searx.engines.solr"]], "searx.engines.mysql_server": [[36, "module-searx.engines.mysql_server"]], "searx.engines.postgresql": [[36, "module-searx.engines.postgresql"]], "searx.engines.sqlite": [[36, "module-searx.engines.sqlite"]], "sqlite_cursor() (in module searx.engines.sqlite)": [[36, "searx.engines.sqlite.sqlite_cursor"]], "aa_content (in module searx.engines.annas_archive)": [[38, "searx.engines.annas_archive.aa_content"]], "aa_ext (in module searx.engines.annas_archive)": [[38, "searx.engines.annas_archive.aa_ext"]], "aa_sort (in module searx.engines.annas_archive)": [[38, "searx.engines.annas_archive.aa_sort"]], "fetch_traits() (in module searx.engines.annas_archive)": [[38, "searx.engines.annas_archive.fetch_traits"]], "init() (in module searx.engines.annas_archive)": [[38, "searx.engines.annas_archive.init"]], "searx.engines.annas_archive": [[38, "module-searx.engines.annas_archive"]], "fetch_traits() (in module searx.engines.archlinux)": [[39, "searx.engines.archlinux.fetch_traits"]], "searx.engines.archlinux": [[39, "module-searx.engines.archlinux"]], "base_url (in module searx.engines.bing)": [[40, "searx.engines.bing.base_url"]], "base_url (in module searx.engines.bing_images)": [[40, "searx.engines.bing_images.base_url"]], "base_url (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.base_url"]], "base_url (in module searx.engines.bing_videos)": [[40, "searx.engines.bing_videos.base_url"]], "bing_traits_url (in module searx.engines.bing)": [[40, "searx.engines.bing.bing_traits_url"]], "bing_traits_url (in module searx.engines.bing_images)": [[40, "searx.engines.bing_images.bing_traits_url"]], "bing_traits_url (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.bing_traits_url"]], "bing_traits_url (in module searx.engines.bing_videos)": [[40, "searx.engines.bing_videos.bing_traits_url"]], "fetch_traits() (in module searx.engines.bing)": [[40, "searx.engines.bing.fetch_traits"]], "fetch_traits() (in module searx.engines.bing_images)": [[40, "searx.engines.bing_images.fetch_traits"]], "fetch_traits() (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.fetch_traits"]], "fetch_traits() (in module searx.engines.bing_videos)": [[40, "searx.engines.bing_videos.fetch_traits"]], "mkt_alias (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.mkt_alias"]], "request() (in module searx.engines.bing)": [[40, "searx.engines.bing.request"]], "request() (in module searx.engines.bing_images)": [[40, "searx.engines.bing_images.request"]], "request() (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.request"]], "request() (in module searx.engines.bing_videos)": [[40, "searx.engines.bing_videos.request"]], "response() (in module searx.engines.bing_images)": [[40, "searx.engines.bing_images.response"]], "response() (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.response"]], "response() (in module searx.engines.bing_videos)": [[40, "searx.engines.bing_videos.response"]], "searx.engines.bing": [[40, "module-searx.engines.bing"]], "searx.engines.bing_images": [[40, "module-searx.engines.bing_images"]], "searx.engines.bing_news": [[40, "module-searx.engines.bing_news"]], "searx.engines.bing_videos": [[40, "module-searx.engines.bing_videos"]], "send_accept_language_header (in module searx.engines.bing)": [[40, "searx.engines.bing.send_accept_language_header"]], "time_map (in module searx.engines.bing_news)": [[40, "searx.engines.bing_news.time_map"]], "brave_category (in module searx.engines.brave)": [[41, "searx.engines.brave.brave_category"]], "brave_spellcheck (in module searx.engines.brave)": [[41, "searx.engines.brave.brave_spellcheck"]], "fetch_traits() (in module searx.engines.brave)": [[41, "searx.engines.brave.fetch_traits"]], "paging (in module searx.engines.brave)": [[41, "searx.engines.brave.paging"]], "searx.engines.brave": [[41, "module-searx.engines.brave"]], "time_range_support (in module searx.engines.brave)": [[41, "searx.engines.brave.time_range_support"]], "bt4g_category (in module searx.engines.bt4g)": [[42, "searx.engines.bt4g.bt4g_category"]], "bt4g_order_by (in module searx.engines.bt4g)": [[42, "searx.engines.bt4g.bt4g_order_by"]], "searx.engines.bt4g": [[42, "module-searx.engines.bt4g"]], "family_filter_map (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.family_filter_map"]], "fetch_traits() (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.fetch_traits"]], "iframe_src (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.iframe_src"]], "result_fields (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.result_fields"]], "safesearch_params (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.safesearch_params"]], "search_url (in module searx.engines.dailymotion)": [[43, "searx.engines.dailymotion.search_url"]], "searx.engines.dailymotion": [[43, "module-searx.engines.dailymotion"]], "area_to_str() (in module searx.engines.duckduckgo_definitions)": [[44, "searx.engines.duckduckgo_definitions.area_to_str"]], "cache_vqd() (in module searx.engines.duckduckgo)": [[44, "searx.engines.duckduckgo.cache_vqd"]], "fetch_traits() (in module searx.engines.duckduckgo)": [[44, "searx.engines.duckduckgo.fetch_traits"]], "get_ddg_lang() (in module searx.engines.duckduckgo)": [[44, "searx.engines.duckduckgo.get_ddg_lang"]], "get_vqd() (in module searx.engines.duckduckgo)": [[44, "searx.engines.duckduckgo.get_vqd"]], "is_broken_text() (in module searx.engines.duckduckgo_definitions)": [[44, "searx.engines.duckduckgo_definitions.is_broken_text"]], "searx.engines.duckduckgo": [[44, "module-searx.engines.duckduckgo"]], "searx.engines.duckduckgo_definitions": [[44, "module-searx.engines.duckduckgo_definitions"]], "searx.engines.duckduckgo_images": [[44, "module-searx.engines.duckduckgo_images"]], "searx.engines.duckduckgo_weather": [[44, "module-searx.engines.duckduckgo_weather"]], "send_accept_language_header (in module searx.engines.duckduckgo)": [[44, "searx.engines.duckduckgo.send_accept_language_header"]], "ui_async (in module searx.engines.google)": [[45, "searx.engines.google.UI_ASYNC"]], "ceid_list (in module searx.engines.google_news)": [[45, "searx.engines.google_news.ceid_list"]], "detect_google_captcha() (in module searx.engines.google_scholar)": [[45, "searx.engines.google_scholar.detect_google_captcha"]], "fetch_traits() (in module searx.engines.google)": [[45, "searx.engines.google.fetch_traits"]], "get_google_info() (in module searx.engines.google)": [[45, "searx.engines.google.get_google_info"]], "google_complete() (in module searx.autocomplete)": [[45, "searx.autocomplete.google_complete"]], "parse_gs_a() (in module searx.engines.google_scholar)": [[45, "searx.engines.google_scholar.parse_gs_a"]], "request() (in module searx.engines.google)": [[45, "searx.engines.google.request"]], "request() (in module searx.engines.google_images)": [[45, "searx.engines.google_images.request"]], "request() (in module searx.engines.google_news)": [[45, "searx.engines.google_news.request"]], "request() (in module searx.engines.google_scholar)": [[45, "searx.engines.google_scholar.request"]], "request() (in module searx.engines.google_videos)": [[45, "searx.engines.google_videos.request"]], "response() (in module searx.engines.google)": [[45, "searx.engines.google.response"]], "response() (in module searx.engines.google_images)": [[45, "searx.engines.google_images.response"]], "response() (in module searx.engines.google_news)": [[45, "searx.engines.google_news.response"]], "response() (in module searx.engines.google_scholar)": [[45, "searx.engines.google_scholar.response"]], "response() (in module searx.engines.google_videos)": [[45, "searx.engines.google_videos.response"]], "searx.engines.google": [[45, "module-searx.engines.google"]], "searx.engines.google_images": [[45, "module-searx.engines.google_images"]], "searx.engines.google_news": [[45, "module-searx.engines.google_news"]], "searx.engines.google_scholar": [[45, "module-searx.engines.google_scholar"]], "searx.engines.google_videos": [[45, "module-searx.engines.google_videos"]], "time_range_args() (in module searx.engines.google_scholar)": [[45, "searx.engines.google_scholar.time_range_args"]], "base_url (in module searx.engines.lemmy)": [[46, "searx.engines.lemmy.base_url"]], "lemmy_type (in module searx.engines.lemmy)": [[46, "searx.engines.lemmy.lemmy_type"]], "searx.engines.lemmy": [[46, "module-searx.engines.lemmy"]], "base_url (in module searx.engines.peertube)": [[47, "searx.engines.peertube.base_url"]], "fetch_traits() (in module searx.engines.peertube)": [[47, "searx.engines.peertube.fetch_traits"]], "request() (in module searx.engines.peertube)": [[47, "searx.engines.peertube.request"]], "request() (in module searx.engines.sepiasearch)": [[47, "searx.engines.sepiasearch.request"]], "searx.engines.peertube": [[47, "module-searx.engines.peertube"]], "searx.engines.sepiasearch": [[47, "module-searx.engines.sepiasearch"]], "video_response() (in module searx.engines.peertube)": [[47, "searx.engines.peertube.video_response"]], "backend_url (in module searx.engines.piped)": [[48, "searx.engines.piped.backend_url"]], "frontend_url (in module searx.engines.piped)": [[48, "searx.engines.piped.frontend_url"]], "piped_filter (in module searx.engines.piped)": [[48, "searx.engines.piped.piped_filter"]], "searx.engines.piped": [[48, "module-searx.engines.piped"]], "searx.engines.recoll": [[49, "module-searx.engines.recoll"]], "fetch_traits() (in module searx.engines.startpage)": [[50, "searx.engines.startpage.fetch_traits"]], "get_sc_code() (in module searx.engines.startpage)": [[50, "searx.engines.startpage.get_sc_code"]], "request() (in module searx.engines.startpage)": [[50, "searx.engines.startpage.request"]], "sc_code_cache_sec (in module searx.engines.startpage)": [[50, "searx.engines.startpage.sc_code_cache_sec"]], "search_form_xpath (in module searx.engines.startpage)": [[50, "searx.engines.startpage.search_form_xpath"]], "searx.engines.startpage": [[50, "module-searx.engines.startpage"]], "send_accept_language_header (in module searx.engines.startpage)": [[50, "searx.engines.startpage.send_accept_language_header"]], "startpage_categ (in module searx.engines.startpage)": [[50, "searx.engines.startpage.startpage_categ"]], "searx.engines.tagesschau": [[51, "module-searx.engines.tagesschau"]], "build_result() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.build_result"]], "get_attribute() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.get_attribute"]], "get_torznab_attribute() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.get_torznab_attribute"]], "init() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.init"]], "request() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.request"]], "response() (in module searx.engines.torznab)": [[52, "searx.engines.torznab.response"]], "searx.engines.torznab": [[52, "module-searx.engines.torznab"]], "fetch_traits() (in module searx.engines.wikidata)": [[53, "searx.engines.wikidata.fetch_traits"]], "fetch_wikimedia_traits() (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.fetch_wikimedia_traits"]], "get_thumbnail() (in module searx.engines.wikidata)": [[53, "searx.engines.wikidata.get_thumbnail"]], "get_wiki_params() (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.get_wiki_params"]], "list_of_wikipedias (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.list_of_wikipedias"]], "request() (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.request"]], "rest_v1_summary_url (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.rest_v1_summary_url"]], "searx.engines.wikidata": [[53, "module-searx.engines.wikidata"]], "searx.engines.wikipedia": [[53, "module-searx.engines.wikipedia"]], "send_accept_language_header (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.send_accept_language_header"]], "wiki_lc_locale_variants (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.wiki_lc_locale_variants"]], "wikipedia_article_depth (in module searx.engines.wikipedia)": [[53, "searx.engines.wikipedia.wikipedia_article_depth"]], "search_mode (in module searx.engines.yacy)": [[54, "searx.engines.yacy.search_mode"]], "searx.engines.yacy": [[54, "module-searx.engines.yacy"]], "fetch_traits() (in module searx.engines.yahoo)": [[55, "searx.engines.yahoo.fetch_traits"]], "lang2domain (in module searx.engines.yahoo)": [[55, "searx.engines.yahoo.lang2domain"]], "parse_url() (in module searx.engines.yahoo)": [[55, "searx.engines.yahoo.parse_url"]], "request() (in module searx.engines.yahoo)": [[55, "searx.engines.yahoo.request"]], "response() (in module searx.engines.yahoo)": [[55, "searx.engines.yahoo.response"]], "searx.engines.yahoo": [[55, "module-searx.engines.yahoo"]], "fetch_traits() (in module searx.engines.zlibrary)": [[56, "searx.engines.zlibrary.fetch_traits"]], "init() (in module searx.engines.zlibrary)": [[56, "searx.engines.zlibrary.init"]], "searx.engines.zlibrary": [[56, "module-searx.engines.zlibrary"]], "zlib_ext (in module searx.engines.zlibrary)": [[56, "searx.engines.zlibrary.zlib_ext"]], "zlib_year_from (in module searx.engines.zlibrary)": [[56, "searx.engines.zlibrary.zlib_year_from"]], "zlib_year_to (in module searx.engines.zlibrary)": [[56, "searx.engines.zlibrary.zlib_year_to"]], "download_error (in module searx.engines.tineye)": [[57, "searx.engines.tineye.DOWNLOAD_ERROR"]], "format_not_supported (in module searx.engines.tineye)": [[57, "searx.engines.tineye.FORMAT_NOT_SUPPORTED"]], "no_signature_error (in module searx.engines.tineye)": [[57, "searx.engines.tineye.NO_SIGNATURE_ERROR"]], "engine_type (in module searx.engines.tineye)": [[57, "searx.engines.tineye.engine_type"]], "parse_tineye_match() (in module searx.engines.tineye)": [[57, "searx.engines.tineye.parse_tineye_match"]], "request() (in module searx.engines.tineye)": [[57, "searx.engines.tineye.request"]], "response() (in module searx.engines.tineye)": [[57, "searx.engines.tineye.response"]], "searx.engines.tineye": [[57, "module-searx.engines.tineye"]], "content_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.content_xpath"]], "cookies (in module searx.engines.xpath)": [[58, "searx.engines.xpath.cookies"]], "first_page_num (in module searx.engines.xpath)": [[58, "searx.engines.xpath.first_page_num"]], "headers (in module searx.engines.xpath)": [[58, "searx.engines.xpath.headers"]], "lang_all (in module searx.engines.xpath)": [[58, "searx.engines.xpath.lang_all"]], "no_result_for_http_status (in module searx.engines.xpath)": [[58, "searx.engines.xpath.no_result_for_http_status"]], "page_size (in module searx.engines.xpath)": [[58, "searx.engines.xpath.page_size"]], "paging (in module searx.engines.xpath)": [[58, "searx.engines.xpath.paging"]], "request() (in module searx.engines.xpath)": [[58, "searx.engines.xpath.request"]], "response() (in module searx.engines.xpath)": [[58, "searx.engines.xpath.response"]], "results_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.results_xpath"]], "safe_search_map (in module searx.engines.xpath)": [[58, "searx.engines.xpath.safe_search_map"]], "safe_search_support (in module searx.engines.xpath)": [[58, "searx.engines.xpath.safe_search_support"]], "search_url (in module searx.engines.xpath)": [[58, "searx.engines.xpath.search_url"]], "searx.engines.xpath": [[58, "module-searx.engines.xpath"]], "soft_max_redirects (in module searx.engines.xpath)": [[58, "searx.engines.xpath.soft_max_redirects"]], "suggestion_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.suggestion_xpath"]], "thumbnail_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.thumbnail_xpath"]], "time_range_map (in module searx.engines.xpath)": [[58, "searx.engines.xpath.time_range_map"]], "time_range_support (in module searx.engines.xpath)": [[58, "searx.engines.xpath.time_range_support"]], "time_range_url (in module searx.engines.xpath)": [[58, "searx.engines.xpath.time_range_url"]], "title_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.title_xpath"]], "url_xpath (in module searx.engines.xpath)": [[58, "searx.engines.xpath.url_xpath"]], "built-in function": [[62, "on_result"], [62, "post_search"], [62, "pre_search"]], "on_result()": [[62, "on_result"]], "post_search()": [[62, "post_search"]], "pre_search()": [[62, "pre_search"]], "pep 8": [[64, "index-1"]], "python enhancement proposals": [[64, "index-1"]], "rfc": [[64, "index-0"]], "rfc 822": [[64, "index-0"]], "get_search_query() (in module searxng_extra.standalone_searx)": [[67, "searxng_extra.standalone_searx.get_search_query"]], "json_serial() (in module searxng_extra.standalone_searx)": [[67, "searxng_extra.standalone_searx.json_serial"]], "no_parsed_url() (in module searxng_extra.standalone_searx)": [[67, "searxng_extra.standalone_searx.no_parsed_url"]], "parse_argument() (in module searxng_extra.standalone_searx)": [[67, "searxng_extra.standalone_searx.parse_argument"]], "searxng_extra.standalone_searx": [[67, "module-searxng_extra.standalone_searx"]], "to_dict() (in module searxng_extra.standalone_searx)": [[67, "searxng_extra.standalone_searx.to_dict"]], "unicodeescape (class in searxng_extra.update.update_engine_traits)": [[68, "searxng_extra.update.update_engine_traits.UnicodeEscape"]], "fetch_traits_map() (in module searxng_extra.update.update_engine_traits)": [[68, "searxng_extra.update.update_engine_traits.fetch_traits_map"]], "filter_locales() (in module searxng_extra.update.update_engine_traits)": [[68, "searxng_extra.update.update_engine_traits.filter_locales"]], "get_output() (in module searxng_extra.update.update_engine_descriptions)": [[68, "searxng_extra.update.update_engine_descriptions.get_output"]], "get_unicode_flag() (in module searxng_extra.update.update_engine_traits)": [[68, "searxng_extra.update.update_engine_traits.get_unicode_flag"]], "merge_when_no_leaf() (in module searxng_extra.update.update_external_bangs)": [[68, "searxng_extra.update.update_external_bangs.merge_when_no_leaf"]], "searxng_extra.update.update_ahmia_blacklist": [[68, "module-searxng_extra.update.update_ahmia_blacklist"]], "searxng_extra.update.update_currencies": [[68, "module-searxng_extra.update.update_currencies"]], "searxng_extra.update.update_engine_descriptions": [[68, "module-searxng_extra.update.update_engine_descriptions"]], "searxng_extra.update.update_engine_traits": [[68, "module-searxng_extra.update.update_engine_traits"]], "searxng_extra.update.update_external_bangs": [[68, "module-searxng_extra.update.update_external_bangs"]], "searxng_extra.update.update_firefox_version": [[68, "module-searxng_extra.update.update_firefox_version"]], "searxng_extra.update.update_osm_keys_tags": [[68, "module-searxng_extra.update.update_osm_keys_tags"]], "searxng_extra.update.update_pygments": [[68, "module-searxng_extra.update.update_pygments"]], "searxng_extra.update.update_wikidata_units": [[68, "module-searxng_extra.update.update_wikidata_units"]], "extract() (in module searx.babel_extract)": [[73, "searx.babel_extract.extract"]], "searx.babel_extract": [[73, "module-searx.babel_extract"]], "api_max (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.API_MAX"]], "api_wondow (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.API_WONDOW"]], "burst_max (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.BURST_MAX"]], "burst_max_suspicious (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.BURST_MAX_SUSPICIOUS"]], "burst_window (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.BURST_WINDOW"]], "limiter_cfg (in module searx.botdetection.limiter)": [[74, "searx.botdetection.limiter.LIMITER_CFG"]], "limiter_cfg_schema (in module searx.botdetection.limiter)": [[74, "searx.botdetection.limiter.LIMITER_CFG_SCHEMA"]], "long_max (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.LONG_MAX"]], "long_max_suspicious (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.LONG_MAX_SUSPICIOUS"]], "long_window (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.LONG_WINDOW"]], "ping_key (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.PING_KEY"]], "ping_live_time (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.PING_LIVE_TIME"]], "searxng_org (in module searx.botdetection.ip_lists)": [[74, "searx.botdetection.ip_lists.SEARXNG_ORG"]], "suspicious_ip_max (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.SUSPICIOUS_IP_MAX"]], "suspicious_ip_window (in module searx.botdetection.ip_limit)": [[74, "searx.botdetection.ip_limit.SUSPICIOUS_IP_WINDOW"]], "token_key (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.TOKEN_KEY"]], "token_live_time (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.TOKEN_LIVE_TIME"]], "user_agent (in module searx.botdetection.http_user_agent)": [[74, "searx.botdetection.http_user_agent.USER_AGENT"]], "block_ip() (in module searx.botdetection.ip_lists)": [[74, "searx.botdetection.ip_lists.block_ip"]], "get_ping_key() (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.get_ping_key"]], "get_real_ip() (in module searx.botdetection)": [[74, "searx.botdetection.get_real_ip"]], "get_token() (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.get_token"]], "is_suspicious() (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.is_suspicious"]], "pass_ip() (in module searx.botdetection.ip_lists)": [[74, "searx.botdetection.ip_lists.pass_ip"]], "ping() (in module searx.botdetection.link_token)": [[74, "searx.botdetection.link_token.ping"]], "searx.botdetection": [[74, "module-searx.botdetection"]], "searx.botdetection.http_accept": [[74, "module-searx.botdetection.http_accept"]], "searx.botdetection.http_accept_encoding": [[74, "module-searx.botdetection.http_accept_encoding"]], "searx.botdetection.http_accept_language": [[74, "module-searx.botdetection.http_accept_language"]], "searx.botdetection.http_connection": [[74, "module-searx.botdetection.http_connection"]], "searx.botdetection.http_user_agent": [[74, "module-searx.botdetection.http_user_agent"]], "searx.botdetection.ip_limit": [[74, "module-searx.botdetection.ip_limit"]], "searx.botdetection.ip_lists": [[74, "module-searx.botdetection.ip_lists"]], "searx.botdetection.limiter": [[74, "module-searx.botdetection.limiter"]], "searx.botdetection.link_token": [[74, "module-searx.botdetection.link_token"]], "suspend_time_setting (searx.exceptions.searxengineaccessdeniedexception attribute)": [[75, "searx.exceptions.SearxEngineAccessDeniedException.SUSPEND_TIME_SETTING"]], "suspend_time_setting (searx.exceptions.searxenginecaptchaexception attribute)": [[75, "searx.exceptions.SearxEngineCaptchaException.SUSPEND_TIME_SETTING"]], "suspend_time_setting (searx.exceptions.searxenginetoomanyrequestsexception attribute)": [[75, "searx.exceptions.SearxEngineTooManyRequestsException.SUSPEND_TIME_SETTING"]], "searxengineapiexception": [[75, "searx.exceptions.SearxEngineAPIException"]], "searxengineaccessdeniedexception": [[75, "searx.exceptions.SearxEngineAccessDeniedException"]], "searxenginecaptchaexception": [[75, "searx.exceptions.SearxEngineCaptchaException"]], "searxengineexception": [[75, "searx.exceptions.SearxEngineException"]], "searxengineresponseexception": [[75, "searx.exceptions.SearxEngineResponseException"]], "searxenginetoomanyrequestsexception": [[75, "searx.exceptions.SearxEngineTooManyRequestsException"]], "searxenginexpathexception": [[75, "searx.exceptions.SearxEngineXPathException"]], "searxexception": [[75, "searx.exceptions.SearxException"]], "searxparameterexception": [[75, "searx.exceptions.SearxParameterException"]], "searxsettingsexception": [[75, "searx.exceptions.SearxSettingsException"]], "searxxpathsyntaxexception": [[75, "searx.exceptions.SearxXPathSyntaxException"]], "searx.exceptions": [[75, "module-searx.exceptions"]], "infopage (class in searx.infopage)": [[76, "searx.infopage.InfoPage"]], "infopageset (class in searx.infopage)": [[76, "searx.infopage.InfoPageSet"]], "content (searx.infopage.infopage property)": [[76, "searx.infopage.InfoPage.content"]], "folder (searx.infopage.infopageset attribute)": [[76, "searx.infopage.InfoPageSet.folder"]], "get_ctx() (searx.infopage.infopage method)": [[76, "searx.infopage.InfoPage.get_ctx"]], "get_page() (searx.infopage.infopageset method)": [[76, "searx.infopage.InfoPageSet.get_page"]], "html (searx.infopage.infopage property)": [[76, "searx.infopage.InfoPage.html"]], "iter_pages() (searx.infopage.infopageset method)": [[76, "searx.infopage.InfoPageSet.iter_pages"]], "locale_default (searx.infopage.infopageset attribute)": [[76, "searx.infopage.InfoPageSet.locale_default"]], "locales (searx.infopage.infopageset attribute)": [[76, "searx.infopage.InfoPageSet.locales"]], "raw_content (searx.infopage.infopage property)": [[76, "searx.infopage.InfoPage.raw_content"]], "searx.infopage": [[76, "module-searx.infopage"]], "title (searx.infopage.infopage property)": [[76, "searx.infopage.InfoPage.title"]], "toc (searx.infopage.infopageset attribute)": [[76, "searx.infopage.InfoPageSet.toc"]], "additional_translations (in module searx.locales)": [[77, "searx.locales.ADDITIONAL_TRANSLATIONS"]], "locale_best_match (in module searx.locales)": [[77, "searx.locales.LOCALE_BEST_MATCH"]], "locale_names (in module searx.locales)": [[77, "searx.locales.LOCALE_NAMES"]], "rtl_locales (in module searx.locales)": [[77, "searx.locales.RTL_LOCALES"]], "build_engine_locales() (in module searx.locales)": [[77, "searx.locales.build_engine_locales"]], "get_engine_locale() (in module searx.locales)": [[77, "searx.locales.get_engine_locale"]], "get_locale() (in module searx.locales)": [[77, "searx.locales.get_locale"]], "get_locale_descr() (in module searx.locales)": [[77, "searx.locales.get_locale_descr"]], "get_offical_locales() (in module searx.locales)": [[77, "searx.locales.get_offical_locales"]], "get_translations() (in module searx.locales)": [[77, "searx.locales.get_translations"]], "language_tag() (in module searx.locales)": [[77, "searx.locales.language_tag"]], "locales_initialize() (in module searx.locales)": [[77, "searx.locales.locales_initialize"]], "match_locale() (in module searx.locales)": [[77, "searx.locales.match_locale"]], "region_tag() (in module searx.locales)": [[77, "searx.locales.region_tag"]], "searx.locales": [[77, "module-searx.locales"]], "searx.sxng_locales": [[77, "module-searx.sxng_locales"]], "sxng_locales (in module searx.sxng_locales)": [[77, "searx.sxng_locales.sxng_locales"]], "description (in module searx.plugins.tor_check)": [[78, "searx.plugins.tor_check.description"]], "name (in module searx.plugins.tor_check)": [[78, "searx.plugins.tor_check.name"]], "preference_section (in module searx.plugins.tor_check)": [[78, "searx.plugins.tor_check.preference_section"]], "query_examples (in module searx.plugins.tor_check)": [[78, "searx.plugins.tor_check.query_examples"]], "query_keywords (in module searx.plugins.tor_check)": [[78, "searx.plugins.tor_check.query_keywords"]], "searx.plugins.tor_check": [[78, "module-searx.plugins.tor_check"]], "old_redis_url_default_url (in module searx.redisdb)": [[79, "searx.redisdb.OLD_REDIS_URL_DEFAULT_URL"]], "searx.redisdb": [[79, "module-searx.redisdb"]], "lua_script_storage (in module searx.redislib)": [[80, "searx.redislib.LUA_SCRIPT_STORAGE"]], "drop_counter() (in module searx.redislib)": [[80, "searx.redislib.drop_counter"]], "incr_counter() (in module searx.redislib)": [[80, "searx.redislib.incr_counter"]], "incr_sliding_window() (in module searx.redislib)": [[80, "searx.redislib.incr_sliding_window"]], "lua_script_storage() (in module searx.redislib)": [[80, "searx.redislib.lua_script_storage"]], "purge_by_prefix() (in module searx.redislib)": [[80, "searx.redislib.purge_by_prefix"]], "searx.redislib": [[80, "module-searx.redislib"]], "secret_hash() (in module searx.redislib)": [[80, "searx.redislib.secret_hash"]], "engineref (class in searx.search)": [[81, "searx.search.EngineRef"]], "search (class in searx.search)": [[81, "searx.search.Search"]], "searchquery (class in searx.search)": [[81, "searx.search.SearchQuery"]], "searchwithplugins (class in searx.search)": [[81, "searx.search.SearchWithPlugins"]], "ordered_plugin_list (searx.search.searchwithplugins attribute)": [[81, "searx.search.SearchWithPlugins.ordered_plugin_list"]], "request (searx.search.searchwithplugins attribute)": [[81, "searx.search.SearchWithPlugins.request"]], "result_container (searx.search.search attribute)": [[81, "searx.search.Search.result_container"]], "result_container (searx.search.searchwithplugins attribute)": [[81, "searx.search.SearchWithPlugins.result_container"]], "search() (searx.search.search method)": [[81, "searx.search.Search.search"]], "search() (searx.search.searchwithplugins method)": [[81, "searx.search.SearchWithPlugins.search"]], "search_query (searx.search.search attribute)": [[81, "searx.search.Search.search_query"]], "search_query (searx.search.searchwithplugins attribute)": [[81, "searx.search.SearchWithPlugins.search_query"]], "engineprocessor (class in searx.search.processors.abstract)": [[82, "searx.search.processors.abstract.EngineProcessor"]], "offlineprocessor (class in searx.search.processors.offline)": [[82, "searx.search.processors.offline.OfflineProcessor"]], "onlinecurrencyprocessor (class in searx.search.processors.online_currency)": [[82, "searx.search.processors.online_currency.OnlineCurrencyProcessor"]], "onlinedictionaryprocessor (class in searx.search.processors.online_dictionary)": [[82, "searx.search.processors.online_dictionary.OnlineDictionaryProcessor"]], "onlineprocessor (class in searx.search.processors.online)": [[82, "searx.search.processors.online.OnlineProcessor"]], "onlineurlsearchprocessor (class in searx.search.processors.online_url_search)": [[82, "searx.search.processors.online_url_search.OnlineUrlSearchProcessor"]], "suspendedstatus (class in searx.search.processors.abstract)": [[82, "searx.search.processors.abstract.SuspendedStatus"]], "default_request_params() (in module searx.search.processors.online)": [[82, "searx.search.processors.online.default_request_params"]], "get_params() (searx.search.processors.abstract.engineprocessor method)": [[82, "searx.search.processors.abstract.EngineProcessor.get_params"]], "get_params() (searx.search.processors.online.onlineprocessor method)": [[82, "searx.search.processors.online.OnlineProcessor.get_params"]], "get_params() (searx.search.processors.online_currency.onlinecurrencyprocessor method)": [[82, "searx.search.processors.online_currency.OnlineCurrencyProcessor.get_params"]], "get_params() (searx.search.processors.online_dictionary.onlinedictionaryprocessor method)": [[82, "searx.search.processors.online_dictionary.OnlineDictionaryProcessor.get_params"]], "get_params() (searx.search.processors.online_url_search.onlineurlsearchprocessor method)": [[82, "searx.search.processors.online_url_search.OnlineUrlSearchProcessor.get_params"]], "searx.search.processors.abstract": [[82, "module-searx.search.processors.abstract"]], "searx.search.processors.offline": [[82, "module-searx.search.processors.offline"]], "searx.search.processors.online": [[82, "module-searx.search.processors.online"]], "searx.search.processors.online_currency": [[82, "module-searx.search.processors.online_currency"]], "searx.search.processors.online_dictionary": [[82, "module-searx.search.processors.online_dictionary"]], "searx.search.processors.online_url_search": [[82, "module-searx.search.processors.online_url_search"]], "search_language_codes (in module searx.utils)": [[83, "searx.utils.SEARCH_LANGUAGE_CODES"]], "convert_str_to_int() (in module searx.utils)": [[83, "searx.utils.convert_str_to_int"]], "detect_language() (in module searx.utils)": [[83, "searx.utils.detect_language"]], "dict_subset() (in module searx.utils)": [[83, "searx.utils.dict_subset"]], "ecma_unescape() (in module searx.utils)": [[83, "searx.utils.ecma_unescape"]], "eval_xpath() (in module searx.utils)": [[83, "searx.utils.eval_xpath"]], "eval_xpath_getindex() (in module searx.utils)": [[83, "searx.utils.eval_xpath_getindex"]], "eval_xpath_list() (in module searx.utils)": [[83, "searx.utils.eval_xpath_list"]], "extract_text() (in module searx.utils)": [[83, "searx.utils.extract_text"]], "extract_url() (in module searx.utils)": [[83, "searx.utils.extract_url"]], "gen_useragent() (in module searx.utils)": [[83, "searx.utils.gen_useragent"]], "get_engine_from_settings() (in module searx.utils)": [[83, "searx.utils.get_engine_from_settings"]], "get_torrent_size() (in module searx.utils)": [[83, "searx.utils.get_torrent_size"]], "get_xpath() (in module searx.utils)": [[83, "searx.utils.get_xpath"]], "html_to_text() (in module searx.utils)": [[83, "searx.utils.html_to_text"]], "int_or_zero() (in module searx.utils)": [[83, "searx.utils.int_or_zero"]], "is_valid_lang() (in module searx.utils)": [[83, "searx.utils.is_valid_lang"]], "normalize_url() (in module searx.utils)": [[83, "searx.utils.normalize_url"]], "searx.utils": [[83, "module-searx.utils"]], "searx_useragent() (in module searx.utils)": [[83, "searx.utils.searx_useragent"]], "to_string() (in module searx.utils)": [[83, "searx.utils.to_string"]]}}) \ No newline at end of file diff --git a/src/index.html b/src/index.html new file mode 100644 index 000000000..269bd201b --- /dev/null +++ b/src/index.html @@ -0,0 +1,254 @@ + + + + + + + + Source-Code — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Source-Code

+

This is a partial documentation of our source code. We are not aiming to document +every item from the source code, but we will add documentation when requested.

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.babel_extract.html b/src/searx.babel_extract.html new file mode 100644 index 000000000..0a87805d2 --- /dev/null +++ b/src/searx.babel_extract.html @@ -0,0 +1,161 @@ + + + + + + + + Custom message extractor (i18n) — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Custom message extractor (i18n)

+

This module implements the searxng_msg extractor to +extract messages from:

+ +

The searxng.msg files are selected by Babel, see Babel’s configuration in +git://babel.cfg:

+
searxng_msg = searx.babel_extract.extract
+...
+[searxng_msg: **/searxng.msg]
+
+
+

A searxng.msg file is a python file that is executed by the +extract function. Additional searxng.msg files can be added by:

+
    +

  1. Adding a searxng.msg file in one of the SearXNG python packages and

    2. +

  2. implement a method in extract that yields messages from this file.

    3. +
+
+
+searx.babel_extract.extract(fileobj, keywords, comment_tags, options)[source]
+

Extract messages from searxng.msg files by a custom extractor.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.botdetection.html b/src/searx.botdetection.html new file mode 100644 index 000000000..2fc563e6f --- /dev/null +++ b/src/searx.botdetection.html @@ -0,0 +1,485 @@ + + + + + + + + Bot Detection — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Bot Detection

+ +
+

X-Forwarded-For

+
+

Attention

+

A correct setup of the HTTP request headers X-Forwarded-For and +X-Real-IP is essential to be able to assign a request to an IP correctly:

+ +
+
+
+searx.botdetection.get_real_ip(request: Request) → str[source]
+

Returns real IP of the request. Since not all proxies set all the HTTP +headers and incoming headers can be faked it may happen that the IP cannot +be determined correctly.

+ +

This function tries to get the remote IP in the order listed below, +additional some tests are done and if inconsistencies or errors are +detected, they are logged.

+

The remote IP of the request is taken from (first match):

+ +
+ +
+
+

Limiter

+ +

Bot protection / IP rate limitation. The intention of rate limitation is to +limit suspicious requests from an IP. The motivation behind this is the fact +that SearXNG passes through requests from bots and is thus classified as a bot +itself. As a result, the SearXNG engine then receives a CAPTCHA or is blocked +by the search engine (the origin) in some other way.

+

To avoid blocking, the requests from bots to SearXNG must also be blocked, this +is the task of the limiter. To perform this task, the limiter uses the methods +from the searx.botdetection.

+

To enable the limiter activate:

+
server:
+  ...
+  limiter: true  # rate limit the number of request on the instance, block some bots
+
+
+

and set the redis-url connection. Check the value, it depends on your redis DB +(see redis:), by example:

+
redis:
+  url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+
+
+
+
+searx.botdetection.limiter.LIMITER_CFG = PosixPath('/etc/searxng/limiter.toml')
+

Lokal Limiter configuration.

+
+ +
+
+searx.botdetection.limiter.LIMITER_CFG_SCHEMA = PosixPath('/home/runner/work/searxng/searxng/searx/botdetection/limiter.toml')
+

Base configuration (schema) of the botdetection.

+
+ +
+

Method ip_lists

+

The ip_lists method implements IP block- and +pass-lists.

+
[botdetection.ip_lists]
+
+pass_ip = [
+ '140.238.172.132', # IPv4 of check.searx.space
+ '192.168.0.0/16',  # IPv4 private network
+ 'fe80::/10'        # IPv6 linklocal
+]
+block_ip = [
+   '93.184.216.34', # IPv4 of example.org
+   '257.1.1.1',     # invalid IP --> will be ignored, logged in ERROR class
+]
+
+
+
+
+
+searx.botdetection.ip_lists.block_ip(real_ip: IPv4Address | IPv6Address, cfg: config.Config) → Tuple[bool, str][source]
+

Checks if the IP on the subnet is in one of the members of the +botdetection.ip_lists.block_ip list.

+
+ +
+
+searx.botdetection.ip_lists.pass_ip(real_ip: IPv4Address | IPv6Address, cfg: config.Config) → Tuple[bool, str][source]
+

Checks if the IP on the subnet is in one of the members of the +botdetection.ip_lists.pass_ip list.

+
+ +
+
+searx.botdetection.ip_lists.SEARXNG_ORG = ['140.238.172.132', '2603:c022:0:4900::/56']
+

Passlist of IPs from the SearXNG organization, e.g. check.searx.space.

+
+ +
+

Rate limit

+
+

Method ip_limit

+

The ip_limit method counts request from an IP in sliding windows. If +there are to many requests in a sliding window, the request is evaluated as a +bot request. This method requires a redis DB and needs a HTTP X-Forwarded-For +header. To take privacy only the hash value of an IP is stored in the redis DB +and at least for a maximum of 10 minutes.

+

The link_token method can be used to investigate whether a request is +suspicious. To activate the link_token method in the +ip_limit method add the following to your +/etc/searxng/limiter.toml:

+
[botdetection.ip_limit]
+link_token = true
+
+
+

If the link_token method is activated and a request is suspicious +the request rates are reduced:

+ +

To intercept bots that get their IPs from a range of IPs, there is a +SUSPICIOUS_IP_WINDOW. In this window the suspicious IPs are stored +for a longer time. IPs stored in this sliding window have a maximum of +SUSPICIOUS_IP_MAX accesses before they are blocked. As soon as the IP +makes a request that is not suspicious, the sliding window for this IP is +droped.

+
+
+
+searx.botdetection.ip_limit.API_MAX = 4
+

Maximum requests from one IP in the API_WONDOW

+
+ +
+
+searx.botdetection.ip_limit.API_WONDOW = 3600
+

Time (sec) before sliding window for API requests (format != html) expires.

+
+ +
+
+searx.botdetection.ip_limit.BURST_MAX = 15
+

Maximum requests from one IP in the BURST_WINDOW

+
+ +
+
+searx.botdetection.ip_limit.BURST_MAX_SUSPICIOUS = 2
+

Maximum of suspicious requests from one IP in the BURST_WINDOW

+
+ +
+
+searx.botdetection.ip_limit.BURST_WINDOW = 20
+

Time (sec) before sliding window for burst requests expires.

+
+ +
+
+searx.botdetection.ip_limit.LONG_MAX = 150
+

Maximum requests from one IP in the LONG_WINDOW

+
+ +
+
+searx.botdetection.ip_limit.LONG_MAX_SUSPICIOUS = 10
+

Maximum suspicious requests from one IP in the LONG_WINDOW

+
+ +
+
+searx.botdetection.ip_limit.LONG_WINDOW = 600
+

Time (sec) before the longer sliding window expires.

+
+ +
+
+searx.botdetection.ip_limit.SUSPICIOUS_IP_MAX = 3
+

Maximum requests from one suspicious IP in the SUSPICIOUS_IP_WINDOW.

+
+ +
+
+searx.botdetection.ip_limit.SUSPICIOUS_IP_WINDOW = 2592000
+

Time (sec) before sliding window for one suspicious IP expires.

+
+ + +
+ +

Generates a hashed key that fits (more or less) to a WEB-browser +session in a network.

+
+ +
+ +

Returns current token. If there is no currently active token a new token +is generated randomly and stored in the redis DB.

+ +
+ +
+ +

Checks whether a valid ping is exists for this (client) network, if not +this request is rated as suspicious. If a valid ping exists and argument +renew is True the expire time of this ping is reset to +PING_LIVE_TIME.

+
+ +
+ +

This function is called by a request to URL /client<token>.css. If +token is valid a PING_KEY for the client is stored in the DB. +The expire time of this ping-key is PING_LIVE_TIME.

+
+ +
+ +

Prefix of all ping-keys generated by get_ping_key

+
+ +
+ +

Livetime (sec) of the ping-key from a client (request)

+
+ +
+ +

Key for which the current token is stored in the DB

+
+ +
+ +

Livetime (sec) of limiter’s CSS token.

+
+ +
+
+

Probe HTTP headers

+
+

Method http_accept

+

The http_accept method evaluates a request as the request of a bot if the +Accept header ..

+
    +

  • did not contain text/html

    • +
+
+
+

Method http_accept_encoding

+

The http_accept_encoding method evaluates a request as the request of a +bot if the Accept-Encoding header ..

+
    +

  • did not contain gzip AND deflate (if both values are missed)

    • +

  • did not contain text/html

    • +
+
+
+

Method http_accept_language

+

The http_accept_language method evaluates a request as the request of a bot +if the Accept-Language header is unset.

+
+
+

Method http_connection

+

The http_connection method evaluates a request as the request of a bot if +the Connection header is set to close.

+
+
+

Method http_user_agent

+

The http_user_agent method evaluates a request as the request of a bot if +the User-Agent header is unset or matches the regular expression +USER_AGENT.

+
+
+
+searx.botdetection.http_user_agent.USER_AGENT = '(unknown|[Cc][Uu][Rr][Ll]|[wW]get|Scrapy|splash|JavaFX|FeedFetcher|python-requests|Go-http-client|Java|Jakarta|okhttp|HttpClient|Jersey|Python|libwww-perl|Ruby|SynHttpClient|UniversalFeedParser|Googlebot|GoogleImageProxy|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT|Sogou|Abonti|Pixray|Spinn3r|SemrushBot|Exabot|ZmEu|BLEXBot|bitlybot|Mozilla/5\\.0\\ \\(compatible;\\ Farside/0\\.1\\.0;\\ \\+https://farside\\.link\\)|.*PetalBot.*)'
+

Regular expression that matches to User-Agent from known bots

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.exceptions.html b/src/searx.exceptions.html new file mode 100644 index 000000000..9611daa44 --- /dev/null +++ b/src/searx.exceptions.html @@ -0,0 +1,226 @@ + + + + + + + + SearXNG Exceptions — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

SearXNG Exceptions

+

Exception types raised by SearXNG modules.

+
+
+exception searx.exceptions.SearxEngineAPIException[source]
+

The website has returned an application error

+
+ +
+
+exception searx.exceptions.SearxEngineAccessDeniedException(suspended_time: int | None = None, message: str = 'Access denied')[source]
+

The website is blocking the access

+
+
+SUSPEND_TIME_SETTING = 'search.suspended_times.SearxEngineAccessDenied'
+

This settings contains the default suspended time (default 86400 sec / 1 +day).

+
+ +
+ +
+
+exception searx.exceptions.SearxEngineCaptchaException(suspended_time=None, message='CAPTCHA')[source]
+

The website has returned a CAPTCHA.

+
+
+SUSPEND_TIME_SETTING = 'search.suspended_times.SearxEngineCaptcha'
+

This settings contains the default suspended time (default 86400 sec / 1 +day).

+
+ +
+ +
+
+exception searx.exceptions.SearxEngineException[source]
+

Error inside an engine

+
+ +
+
+exception searx.exceptions.SearxEngineResponseException[source]
+

Impossible to parse the result of an engine

+
+ +
+
+exception searx.exceptions.SearxEngineTooManyRequestsException(suspended_time=None, message='Too many request')[source]
+

The website has returned a Too Many Request status code

+

By default, searx stops sending requests to this engine for 1 hour.

+
+
+SUSPEND_TIME_SETTING = 'search.suspended_times.SearxEngineTooManyRequests'
+

This settings contains the default suspended time (default 3660 sec / 1 +hour).

+
+ +
+ +
+
+exception searx.exceptions.SearxEngineXPathException(xpath_spec, message)[source]
+

Error while getting the result of an XPath expression

+
+ +
+
+exception searx.exceptions.SearxException[source]
+

Base SearXNG exception.

+
+ +
+
+exception searx.exceptions.SearxParameterException(name, value)[source]
+

Raised when query miss a required paramater

+
+ +
+
+exception searx.exceptions.SearxSettingsException(message: str | Exception, filename: str | None)[source]
+

Error while loading the settings

+
+ +
+
+exception searx.exceptions.SearxXPathSyntaxException(xpath_spec, message)[source]
+

Syntax error in a XPATH

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.infopage.html b/src/searx.infopage.html new file mode 100644 index 000000000..55960a7d2 --- /dev/null +++ b/src/searx.infopage.html @@ -0,0 +1,245 @@ + + + + + + + + Online /info — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Online /info

+

Render SearXNG instance documentation.

+

Usage in a Flask app route:

+
from searx import infopage
+
+_INFO_PAGES = infopage.InfoPageSet(infopage.MistletoePage)
+
+@app.route('/info/<pagename>', methods=['GET'])
+def info(pagename):
+
+    locale = request.preferences.get_value('locale')
+    page = _INFO_PAGES.get_page(pagename, locale)
+
+
+
+
+class searx.infopage.InfoPage(fname)[source]
+

A page of the online documentation.

+
+
+get_ctx()[source]
+

Jinja context to render InfoPage.content

+
+ +
+
+property content
+

Content of the page (rendered in a Jinja conntext)

+
+ +
+
+property html
+

Render Markdown (CommonMark) to HTML by using markdown-it-py.

+
+ +
+
+property raw_content
+

Raw content of the page (without any jinja rendering)

+
+ +
+
+property title
+

Title of the content (without any markup)

+
+ +
+ +
+
+class searx.infopage.InfoPageSet(page_class: Type[InfoPage] | None = None, info_folder: str | None = None)[source]
+

Cached rendering of the online documentation a SearXNG instance has.

+
+
Parameters:
+
    +

  • page_class (InfoPage) – render online documentation by InfoPage parser.

    • +

  • info_folder (str) – information directory

    • +
+
+
+
+
+get_page(pagename: str, locale: str | None = None)[source]
+

Return pagename instance of InfoPage

+
+
Parameters:
+
    +

  • pagename (str) – name of the page, a value from InfoPageSet.toc

    • +

  • locale (str) – language of the page, e.g. en, zh_Hans_CN +(default: InfoPageSet.i18n_origin)

    • +
+
+
+
+ +
+
+iter_pages(locale: str | None = None, fallback_to_default=False)[source]
+

Iterate over all pages of the TOC

+
+ +
+
+folder: str
+

location of the Markdwon files

+
+ +
+
+locale_default: str
+

default language

+
+ +
+
+locales: List[str]
+

list of supported languages (aka locales)

+
+ +
+
+toc: List[str]
+

list of articles in the online documentation

+
+ +
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.locales.html b/src/searx.locales.html new file mode 100644 index 000000000..7f9d105bf --- /dev/null +++ b/src/searx.locales.html @@ -0,0 +1,388 @@ + + + + + + + + Locales — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Locales

+ +

Initialize LOCALE_NAMES, RTL_LOCALES.

+
+
+searx.locales.build_engine_locales(tag_list: List[str])[source]
+

From a list of locale tags a dictionary is build that can be passed by +argument engine_locales to get_engine_locale. This function +is mainly used by match_locale and is similar to what the +fetch_traits(..) function of engines do.

+

If there are territory codes in the tag_list that have a script code +additional keys are added to the returned dictionary.

+
>>> import locales
+>>> engine_locales = locales.build_engine_locales(['en', 'en-US', 'zh', 'zh-CN', 'zh-TW'])
+>>> engine_locales
+{
+    'en': 'en', 'en-US': 'en-US',
+    'zh': 'zh', 'zh-CN': 'zh-CN', 'zh_Hans': 'zh-CN',
+    'zh-TW': 'zh-TW', 'zh_Hant': 'zh-TW'
+}
+>>> get_engine_locale('zh-Hans', engine_locales)
+'zh-CN'
+
+
+

This function is a good example to understand the language/region model +of SearXNG:

+
+

SearXNG only distinguishes between search languages and search +regions, by adding the script-tags, languages with script-tags can +be assigned to the regions that SearXNG supports.

+
+
+ +
+
+searx.locales.get_engine_locale(searxng_locale, engine_locales, default=None)[source]
+

Return engine’s language (aka locale) string that best fits to argument +searxng_locale.

+

Argument engine_locales is a python dict that maps SearXNG locales to +corresponding engine locales:

+
<engine>: {
+    # SearXNG string : engine-string
+    'ca-ES'          : 'ca_ES',
+    'fr-BE'          : 'fr_BE',
+    'fr-CA'          : 'fr_CA',
+    'fr-CH'          : 'fr_CH',
+    'fr'             : 'fr_FR',
+    ...
+    'pl-PL'          : 'pl_PL',
+    'pt-PT'          : 'pt_PT'
+    ..
+    'zh'             : 'zh'
+    'zh_Hans'        : 'zh'
+    'zh_Hant'        : 'zh_TW'
+}
+
+
+
+

Hint

+

The SearXNG locale string has to be known by babel!

+
+

If there is no direct 1:1 mapping, this functions tries to narrow down +engine’s language (locale). If no value can be determined by these +approximation attempts the default value is returned.

+

Assumptions:

+
    +

  1. When user select a language the results should be optimized according to +the selected language.

    2. +

  2. When user select a language and a territory the results should be +optimized with first priority on terrirtory and second on language.

    3. +
+

First approximation rule (by territory):

+
+

When the user selects a locale with terrirtory (and a language), the +territory has priority over the language. If any of the offical languages +in the terrirtory is supported by the engine (engine_locales) it will +be used.

+
+

Second approximation rule (by language):

+
+

If “First approximation rule” brings no result or the user selects only a +language without a terrirtory. Check in which territories the language +has an offical status and if one of these territories is supported by the +engine.

+
+
+ +
+
+searx.locales.get_locale(locale_tag: str) → Locale | None[source]
+

Returns a babel.Locale object parsed from argument +locale_tag

+
+ +
+
+searx.locales.get_locale_descr(locale, locale_name)[source]
+

Get locale name e.g. ‘Français - fr’ or ‘Português (Brasil) - pt-BR’

+
+
Parameters:
+
    +

  • locale – instance of Locale

    • +

  • locale_name – name e.g. ‘fr’ or ‘pt_BR’ (delimiter is underscore)

    • +
+
+
+
+ +
+
+searx.locales.get_offical_locales(territory: str, languages=None, regional: bool = False, de_facto: bool = True) → Set[Locale][source]
+

Returns a list of babel.Locale with languages from +babel.languages.get_official_languages.

+
+
Parameters:
+
    +

  • territory – The territory (country or region) code.

    • +

  • languages – A list of language codes the languages from +babel.languages.get_official_languages should be in +(intersection). If this argument is None, all official languages in +this territory are used.

    • +

  • regional – If the regional flag is set, then languages which are +regionally official are also returned.

    • +

  • de_facto – If the de_facto flag is set to False, then languages +which are “de facto” official are not returned.

    • +
+
+
+
+ +
+
+searx.locales.get_translations()[source]
+

Monkey patch of flask_babel.get_translations

+
+ +
+
+searx.locales.language_tag(locale: Locale) → str[source]
+

Returns SearXNG’s language tag from the locale and if exits, the tag +includes the script name (e.g. en, zh_Hant).

+
+ +
+
+searx.locales.locales_initialize(directory=None)[source]
+

Initialize locales environment of the SearXNG session.

+ +
+ +
+
+searx.locales.match_locale(searxng_locale: str, locale_tag_list: List[str], fallback: str | None = None) → str | None[source]
+

Return tag from locale_tag_list that best fits to searxng_locale.

+
+
Parameters:
+
    +

  • searxng_locale (str) – SearXNG’s internal representation of locale (de, +de-DE, fr-BE, zh, zh-CN, zh-TW ..).

    • +

  • locale_tag_list (list) – The list of locale tags to select from

    • +

  • fallback (str) – fallback locale tag (if unset –> None)

    • +
+
+
+

The rules to find a match are implemented in get_engine_locale, +the engine_locales is build up by build_engine_locales.

+
+

Hint

+

The SearXNG locale string and the members of locale_tag_list has to +be known by babel! The ADDITIONAL_TRANSLATIONS are used in the +UI and are not known by babel –> will be ignored.

+
+
+ +
+
+searx.locales.region_tag(locale: Locale) → str[source]
+

Returns SearXNG’s region tag from the locale (e.g. zh-TW , en-US).

+
+ +
+
+searx.locales.ADDITIONAL_TRANSLATIONS = {'dv': 'ދިވެހި (Dhivehi)', 'oc': 'Occitan', 'pap': 'Papiamento', 'szl': 'Ślōnski (Silesian)'}
+

Additional languages SearXNG has translations for but not supported by +python-babel (see locales_initialize).

+
+ +
+
+searx.locales.LOCALE_BEST_MATCH = {'dv': 'si', 'nl-BE': 'nl', 'oc': 'fr-FR', 'pap': 'pt-BR', 'szl': 'pl', 'zh-HK': 'zh-Hant-TW'}
+

Map a locale we do not have a translations for to a locale we have a +translation for. By example: use Taiwan version of the translation for Hong +Kong.

+
+ +
+
+searx.locales.LOCALE_NAMES
+

Mapping of locales and their description. Locales e.g. ‘fr’ or ‘pt-BR’ (see +locales_initialize).

+
+
+
+ +
+
+searx.locales.RTL_LOCALES: Set[str] = {'ar', 'fa-IR', 'he'}
+

List of Right-To-Left locales e.g. ‘he’ or ‘fa-IR’ (see +locales_initialize).

+
+ +
+

SearXNG’s locale codes

+

List of SearXNG’s locale codes.

+

This file is generated automatically by:

+
./manage pyenv.cmd searxng_extra/update/update_engine_traits.py
+
+
+
+
+searx.sxng_locales.sxng_locales
+

A list of five-digit tuples:

+
    +

  1. SearXNG’s internal locale tag (a language or region tag)

    2. +

  2. Name of the language (babel.core.Locale.get_language_name)

    3. +

  3. For region tags the name of the region (babel.core.Locale.get_territory_name). +Empty string for language tags.

    4. +

  4. English language name (from babel.core.Locale.english_name)

    5. +

  5. Unicode flag (emoji) that fits to SearXNG’s internal region tag. Languages +are represented by a globe (🌐)

    6. +
+
('en',    'English', '',              'English', '🌐'),
+('en-CA', 'English', 'Canada',        'English', '🇨🇦'),
+('en-US', 'English', 'United States', 'English', '🇺🇸'),
+..
+('fr',    'Français', '',             'French',  '🌐'),
+('fr-BE', 'Français', 'Belgique',     'French',  '🇧🇪'),
+('fr-CA', 'Français', 'Canada',       'French',  '🇨🇦'),
+
+
+
+
+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.plugins.tor_check.html b/src/searx.plugins.tor_check.html new file mode 100644 index 000000000..1f8b8f4c6 --- /dev/null +++ b/src/searx.plugins.tor_check.html @@ -0,0 +1,177 @@ + + + + + + + + Tor check plugin — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Tor check plugin

+

A plugin to check if the ip address of the request is a Tor exit-node if the +user searches for tor-check. It fetches the tor exit node list from +https://check.torproject.org/exit-addresses and parses all the IPs into a list, +then checks if the user’s IP address is in it.

+

Enable in settings.yml:

+
enabled_plugins:
+  ..
+  - 'Tor check plugin'
+
+
+
+
+searx.plugins.tor_check.description = 'This plugin checks if the address of the request is a Tor exit-node, and informs the user if it is; like check.torproject.org, but from SearXNG.'
+

Translated description of the plugin.

+
+ +
+
+searx.plugins.tor_check.name = 'Tor check plugin'
+

Translated name of the plugin

+
+ +
+
+searx.plugins.tor_check.preference_section = 'query'
+

The preference section where the plugin is shown.

+
+ +
+
+searx.plugins.tor_check.query_examples = ''
+

Query examples shown in the preferences.

+
+ +
+
+searx.plugins.tor_check.query_keywords = ['tor-check']
+

Query keywords shown in the preferences.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.redisdb.html b/src/searx.redisdb.html new file mode 100644 index 000000000..dffbeac20 --- /dev/null +++ b/src/searx.redisdb.html @@ -0,0 +1,157 @@ + + + + + + + + Redis DB — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Redis DB

+

Implementation of the redis client (redis-py).

+

This implementation uses the redis: setup from settings.yml. +A redis DB connect can be tested by:

+
>>> from searx import redisdb
+>>> redisdb.initialize()
+True
+>>> db = redisdb.client()
+>>> db.set("foo", "bar")
+True
+>>> db.get("foo")
+b'bar'
+>>>
+
+
+
+
+searx.redisdb.OLD_REDIS_URL_DEFAULT_URL = 'unix:///usr/local/searxng-redis/run/redis.sock?db=0'
+

This was the default Redis URL in settings.yml.

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.redislib.html b/src/searx.redislib.html new file mode 100644 index 000000000..203262384 --- /dev/null +++ b/src/searx.redislib.html @@ -0,0 +1,283 @@ + + + + + + + + Redis Library — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Redis Library

+

A collection of convenient functions and redis/lua scripts.

+

This code was partial inspired by the Bullet-Proofing Lua Scripts in RedisPy +article.

+
+
+searx.redislib.drop_counter(client, name)[source]
+

Drop counter with redis key SearXNG_counter_<name>

+

The replacement <name> is a secret hash of the value from argument +name (see incr_counter() and incr_sliding_window()).

+
+ +
+
+searx.redislib.incr_counter(client, name: str, limit: int = 0, expire: int = 0)[source]
+

Increment a counter and return the new value.

+

If counter with redis key SearXNG_counter_<name> does not exists it is +created with initial value 1 returned. The replacement <name> is a +secret hash of the value from argument name (see +secret_hash()).

+

The implementation of the redis counter is the lua script from string +INCR_COUNTER.

+
+
Parameters:
+
    +

  • name (str) – name of the counter

    • +

  • expire (int / see EXPIRE) – live-time of the counter in seconds (default None means +infinite).

    • +

  • limit (int / limit is 2^64 see INCR) – limit where the counter stops to increment (default None)

    • +
+
+
Returns:
+

value of the incremented counter

+
+
+

A simple demo of a counter with expire time and limit:

+
>>> for i in range(6):
+...   i, incr_counter(client, "foo", 3, 5) # max 3, duration 5 sec
+...   time.sleep(1) # from the third call on max has been reached
+...
+(0, 1)
+(1, 2)
+(2, 3)
+(3, 3)
+(4, 3)
+(5, 1)
+
+
+
+ +
+
+searx.redislib.incr_sliding_window(client, name: str, duration: int)[source]
+

Increment a sliding-window counter and return the new value.

+

If counter with redis key SearXNG_counter_<name> does not exists it is +created with initial value 1 returned. The replacement <name> is a +secret hash of the value from argument name (see +secret_hash()).

+
+
Parameters:
+
    +

  • name (str) – name of the counter

    • +

  • duration – live-time of the sliding window in seconds

    • +
+
+
Typeduration:
+

int

+
+
Returns:
+

value of the incremented counter

+
+
+

The implementation of the redis counter is the lua script from string +INCR_SLIDING_WINDOW. The lua script uses sorted sets in Redis +to implement a sliding window for the redis key SearXNG_counter_<name> +(ZADD). The current TIME is used to score the items in the sorted set and +the time window is moved by removing items with a score lower current time +minus duration time (ZREMRANGEBYSCORE).

+

The EXPIRE time (the duration of the sliding window) is refreshed on each +call (incrementation) and if there is no call in this duration, the sorted +set expires from the redis DB.

+

The return value is the amount of items in the sorted set (ZCOUNT), what +means the number of calls in the sliding window.

+

A simple demo of the sliding window:

+
>>> for i in range(5):
+...   incr_sliding_window(client, "foo", 3) # duration 3 sec
+...   time.sleep(1) # from the third call (second) on the window is moved
+...
+1
+2
+3
+3
+3
+>>> time.sleep(3)  # wait until expire
+>>> incr_sliding_window(client, "foo", 3)
+1
+
+
+
+ +
+
+searx.redislib.lua_script_storage(client, script)[source]
+

Returns a redis Script instance.

+

Due to performance reason the Script object is instantiated only once +for a client (client.register_script(..)) and is cached in +LUA_SCRIPT_STORAGE.

+
+ +
+
+searx.redislib.purge_by_prefix(client, prefix: str = 'SearXNG_')[source]
+

Purge all keys with prefix from database.

+

Queries all keys in the database by the given prefix and set expire time to +zero. The default prefix will drop all keys which has been set by SearXNG +(drops SearXNG schema entirely from database).

+

The implementation is the lua script from string PURGE_BY_PREFIX. +The lua script uses EXPIRE instead of DEL: if there are a lot keys to +delete and/or their values are big, DEL could take more time and blocks +the command loop while EXPIRE turns back immediate.

+
+
Parameters:
+

prefix – prefix of the key to delete (default: SearXNG_)

+
+
+
+ +
+
+searx.redislib.secret_hash(name: str)[source]
+

Creates a hash of the name.

+

Combines argument name with the secret_key from server:. This function can be used to get a more anonymised name of a Redis +KEY.

+
+
Parameters:
+

name (str) – the name to create a secret hash for

+
+
+
+ +
+
+searx.redislib.LUA_SCRIPT_STORAGE = {}
+

A global dictionary to cache client’s Script objects, used by +lua_script_storage

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.search.html b/src/searx.search.html new file mode 100644 index 000000000..d16aa4928 --- /dev/null +++ b/src/searx.search.html @@ -0,0 +1,201 @@ + + + + + + + + Search — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ + + + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.search.processors.html b/src/searx.search.processors.html new file mode 100644 index 000000000..06cfdcead --- /dev/null +++ b/src/searx.search.processors.html @@ -0,0 +1,259 @@ + + + + + + + + Search processors — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Search processors

+ +
+

Abstract processor class

+

Abstract base classes for engine request processores.

+
+
+class searx.search.processors.abstract.EngineProcessor(engine, engine_name: str)[source]
+

Base classes used for all types of reqest processores.

+
+
+get_params(search_query, engine_category)[source]
+

Returns a set of (see request params) or +None if request is not supported.

+

Not supported conditions (None is returned):

+
    +

  • A page-number > 1 when engine does not support paging.

    • +

  • A time range when the engine does not support time range.

    • +
+
+ +
+ +
+
+class searx.search.processors.abstract.SuspendedStatus[source]
+

Class to handle suspend state.

+
+ +
+
+

Offline processor

+

Processores for engine-type: offline

+
+
+class searx.search.processors.offline.OfflineProcessor(engine, engine_name: str)[source]
+

Processor class used by offline engines

+
+ +
+
+

Online processor

+

Processores for engine-type: online

+
+
+class searx.search.processors.online.OnlineProcessor(engine, engine_name: str)[source]
+

Processor class for online engines.

+
+
+get_params(search_query, engine_category)[source]
+

Returns a set of request params or None +if request is not supported.

+
+ +
+ +
+
+searx.search.processors.online.default_request_params()[source]
+

Default request parameters for online engines.

+
+ +
+
+

Online currency processor

+

Processores for engine-type: online_currency

+
+
+class searx.search.processors.online_currency.OnlineCurrencyProcessor(engine, engine_name: str)[source]
+

Processor class used by online_currency engines.

+
+
+get_params(search_query, engine_category)[source]
+

Returns a set of request params +or None if search query does not match to parser_re.

+
+ +
+ +
+
+

Online dictionary processor

+

Processores for engine-type: online_dictionary

+
+
+class searx.search.processors.online_dictionary.OnlineDictionaryProcessor(engine, engine_name: str)[source]
+

Processor class used by online_dictionary engines.

+
+
+get_params(search_query, engine_category)[source]
+

Returns a set of request params or +None if search query does not match to parser_re.

+
+ +
+ +
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/src/searx.utils.html b/src/searx.utils.html new file mode 100644 index 000000000..b25b030df --- /dev/null +++ b/src/searx.utils.html @@ -0,0 +1,531 @@ + + + + + + + + Utility functions for the engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + +
+
+
+
+ +
+

Utility functions for the engines

+

Utility functions for the engines

+
+
+searx.utils.convert_str_to_int(number_str: str) → int[source]
+

Convert number_str to int or 0 if number_str is not a number.

+
+ +
+
+searx.utils.detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) → str | None[source]
+

Detect the language of the text parameter.

+
+
Parameters:
+
    +

  • text (str) – The string whose language is to be detected.

    • +

  • threshold (float) – Threshold filters the returned labels by a threshold +on probability. A choice of 0.3 will return labels with at least 0.3 +probability.

    • +

  • only_search_languages (bool) – If True, returns only supported +SearXNG search languages. see searx.languages

    • +
+
+
Return type:
+

str, None

+
+
Returns:
+

The detected language code or None. See below.

+
+
Raises:
+

ValueError – If text is not a string.

+
+
+

The language detection is done by using a fork of the fastText library +(python fasttext). fastText distributes the language identification +model, for reference:

+ +

The language identification model support the language codes +(ISO-639-3):

+
af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs
+bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es
+et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia
+id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li
+lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah
+nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru
+rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl
+tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh
+
+
+

By using only_search_languages=True the language identification model +is harmonized with the SearXNG’s language (locale) model. General +conditions of SearXNG’s locale model are:

+
    +

  1. SearXNG’s locale of a query is passed to the +searx.locales.get_engine_locale to get a language and/or region +code that is used by an engine.

    2. +

  2. Most of SearXNG’s engines do not support all the languages from language +identification model and there is also a discrepancy in the ISO-639-3 +(fastext) and ISO-639-2 (SearXNG)handling. Further more, in SearXNG the +locales like zh-TH (zh-CN) are mapped to zh_Hant +(zh_Hans) while the language identification model reduce both to +zh.

    3. +
+
+ +
+
+searx.utils.dict_subset(dictionary: MutableMapping, properties: Set[str]) → Dict[source]
+

Extract a subset of a dict

+
+
Examples:
>>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
+{'A': 'a', 'C': 'c'}
+>>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
+{'A': 'a'}
+
+
+
+
+
+ +
+
+searx.utils.ecma_unescape(string: str) → str[source]
+

Python implementation of the unescape javascript function

+

https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string +https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape

+
+
Examples:
>>> ecma_unescape('%u5409')
+'吉'
+>>> ecma_unescape('%20')
+' '
+>>> ecma_unescape('%F3')
+'ó'
+
+
+
+
+
+ +
+
+searx.utils.eval_xpath(element: ElementBase, xpath_spec: str | XPath)[source]
+

Equivalent of element.xpath(xpath_str) but compile xpath_str once for all. +See https://lxml.de/xpathxslt.html#xpath-return-values

+
+
Args:
    +

  • element (ElementBase): [description]

    • +

  • xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath

    • +
+
+
Returns:
    +

  • result (bool, float, list, str): Results.

    • +
+
+
Raises:
    +

  • TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath

    • +

  • SearxXPathSyntaxException: Raise when there is a syntax error in the XPath

    • +

  • SearxEngineXPathException: Raise when the XPath can’t be evaluated.

    • +
+
+
+
+ +
+
+searx.utils.eval_xpath_getindex(elements: ~lxml.etree.ElementBase, xpath_spec: str | ~lxml.etree.XPath, index: int, default=<searx.utils._NotSetClass object>)[source]
+

Call eval_xpath_list then get one element using the index parameter. +If the index does not exist, either aise an exception is default is not set, +other return the default value (can be None).

+
+
Args:
    +

  • elements (ElementBase): lxml element to apply the xpath.

    • +

  • xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath.

    • +

  • index (int): index to get

    • +

  • default (Object, optional): Defaults if index doesn’t exist.

    • +
+
+
Raises:
    +

  • TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath

    • +

  • SearxXPathSyntaxException: Raise when there is a syntax error in the XPath

    • +

  • SearxEngineXPathException: if the index is not found. Also see eval_xpath.

    • +
+
+
Returns:
    +

  • result (bool, float, list, str): Results.

    • +
+
+
+
+ +
+
+searx.utils.eval_xpath_list(element: ElementBase, xpath_spec: str | XPath, min_len: int | None = None)[source]
+

Same as eval_xpath, check if the result is a list

+
+
Args:
    +

  • element (ElementBase): [description]

    • +

  • xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath

    • +

  • min_len (int, optional): [description]. Defaults to None.

    • +
+
+
Raises:
    +

  • TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath

    • +

  • SearxXPathSyntaxException: Raise when there is a syntax error in the XPath

    • +

  • SearxEngineXPathException: raise if the result is not a list

    • +
+
+
Returns:
    +

  • result (bool, float, list, str): Results.

    • +
+
+
+
+ +
+
+searx.utils.extract_text(xpath_results, allow_none: bool = False) → str | None[source]
+

Extract text from a lxml result

+
    +

  • if xpath_results is list, extract the text from each result and concat the list

    • +

  • if xpath_results is a xml element, extract all the text node from it +( text_content() method from lxml )

    • +

  • if xpath_results is a string element, then it’s already done

    • +
+
+ +
+
+searx.utils.extract_url(xpath_results, base_url) → str[source]
+

Extract and normalize URL from lxml Element

+
+
Args:
    +

  • xpath_results (Union[List[html.HtmlElement], html.HtmlElement]): lxml Element(s)

    • +

  • base_url (str): Base URL

    • +
+
+
Example:
>>> def f(s, search_url):
+>>>    return searx.utils.extract_url(html.fromstring(s), search_url)
+>>> f('<span id="42">https://example.com</span>', 'http://example.com/')
+'https://example.com/'
+>>> f('https://example.com', 'http://example.com/')
+'https://example.com/'
+>>> f('//example.com', 'http://example.com/')
+'http://example.com/'
+>>> f('//example.com', 'https://example.com/')
+'https://example.com/'
+>>> f('/path?a=1', 'https://example.com')
+'https://example.com/path?a=1'
+>>> f('', 'https://example.com')
+raise lxml.etree.ParserError
+>>> searx.utils.extract_url([], 'https://example.com')
+raise ValueError
+
+
+
+
Raises:
    +

  • ValueError

    • +

  • lxml.etree.ParserError

    • +
+
+
Returns:
    +

  • str: normalized URL

    • +
+
+
+
+ +
+
+searx.utils.gen_useragent(os_string: str | None = None) → str[source]
+

Return a random browser User Agent

+

See searx/data/useragents.json

+
+ +
+
+searx.utils.get_engine_from_settings(name: str) → Dict[source]
+

Return engine configuration from settings.yml of a given engine name

+
+ +
+
+searx.utils.get_torrent_size(filesize: str, filesize_multiplier: str) → int | None[source]
+
+
Args:
    +

  • filesize (str): size

    • +

  • filesize_multiplier (str): TB, GB, …. TiB, GiB…

    • +
+
+
Returns:
    +

  • int: number of bytes

    • +
+
+
Example:
>>> get_torrent_size('5', 'GB')
+5368709120
+>>> get_torrent_size('3.14', 'MiB')
+3140000
+
+
+
+
+
+ +
+
+searx.utils.get_xpath(xpath_spec: str | XPath) → XPath[source]
+

Return cached compiled XPath

+

There is no thread lock. +Worst case scenario, xpath_str is compiled more than one time.

+
+
Args:
    +

  • xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath

    • +
+
+
Returns:
    +

  • result (bool, float, list, str): Results.

    • +
+
+
Raises:
    +

  • TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath

    • +

  • SearxXPathSyntaxException: Raise when there is a syntax error in the XPath

    • +
+
+
+
+ +
+
+searx.utils.html_to_text(html_str: str) → str[source]
+

Extract text from a HTML string

+
+
Args:
    +

  • html_str (str): string HTML

    • +
+
+
Returns:
    +

  • str: extracted text

    • +
+
+
Examples:
>>> html_to_text('Example <span id="42">#2</span>')
+'Example #2'
+
+
+
>>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
+'Example'
+
+
+
+
+
+ +
+
+searx.utils.int_or_zero(num: List[str] | str) → int[source]
+

Convert num to int or 0. num can be either a str or a list. +If num is a list, the first element is converted to int (or return 0 if the list is empty). +If num is a str, see convert_str_to_int

+
+ +
+
+searx.utils.is_valid_lang(lang) → Tuple[bool, str, str] | None[source]
+

Return language code and name if lang describe a language.

+
+
Examples:
>>> is_valid_lang('zz')
+None
+>>> is_valid_lang('uk')
+(True, 'uk', 'ukrainian')
+>>> is_valid_lang(b'uk')
+(True, 'uk', 'ukrainian')
+>>> is_valid_lang('en')
+(True, 'en', 'english')
+>>> searx.utils.is_valid_lang('Español')
+(True, 'es', 'spanish')
+>>> searx.utils.is_valid_lang('Spanish')
+(True, 'es', 'spanish')
+
+
+
+
+
+ +
+
+searx.utils.normalize_url(url: str, base_url: str) → str[source]
+

Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path

+
+
Args:
    +

  • url (str): Relative URL

    • +

  • base_url (str): Base URL, it must be an absolute URL.

    • +
+
+
Example:
>>> normalize_url('https://example.com', 'http://example.com/')
+'https://example.com/'
+>>> normalize_url('//example.com', 'http://example.com/')
+'http://example.com/'
+>>> normalize_url('//example.com', 'https://example.com/')
+'https://example.com/'
+>>> normalize_url('/path?a=1', 'https://example.com')
+'https://example.com/path?a=1'
+>>> normalize_url('', 'https://example.com')
+'https://example.com/'
+>>> normalize_url('/test', '/path')
+raise ValueError
+
+
+
+
Raises:
    +

  • lxml.etree.ParserError

    • +
+
+
Returns:
    +

  • str: normalized URL

    • +
+
+
+
+ +
+
+searx.utils.searx_useragent() → str[source]
+

Return the searx User Agent

+
+ +
+
+searx.utils.to_string(obj: Any) → str[source]
+

Convert obj to its string representation.

+
+ +
+
+searx.utils.SEARCH_LANGUAGE_CODES = frozenset({'ar', 'bg', 'ca', 'cs', 'da', 'de', 'el', 'en', 'es', 'et', 'fa', 'fi', 'fr', 'he', 'hi', 'hr', 'hu', 'id', 'is', 'it', 'ja', 'ko', 'lt', 'lv', 'nb', 'nl', 'pl', 'pt', 'ro', 'ru', 'sk', 'sl', 'sr', 'sv', 'th', 'tr', 'uk', 'vi', 'zh'})
+

Languages supported by most searxng engines (searx.sxng_locales.sxng_locales).

+
+ +
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/user/about.html b/user/about.html new file mode 100644 index 000000000..26b96ee47 --- /dev/null +++ b/user/about.html @@ -0,0 +1,192 @@ + + + + + + + + About SearXNG — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

About SearXNG

+

SearXNG is a metasearch engine, aggregating the results of other +search engines while not storing information about +its users.

+

The SearXNG project is driven by an open community, come join us on Matrix if +you have questions or just want to chat about SearXNG at #searxng:matrix.org

+

Make SearXNG better.

+
    +

  • You can improve SearXNG translations at Weblate, or…

    • +

  • Track development, send contributions, and report issues at SearXNG sources.

    • +

  • To get further information, visit SearXNG’s project documentation at SearXNG +docs.

    • +
+
+

Why use it?

+
    +

  • SearXNG may not offer you as personalized results as Google, but it doesn’t +generate a profile about you.

    • +

  • SearXNG doesn’t care about what you search for, never shares anything with a +third-party, and it can’t be used to compromise you.

    • +

  • SearXNG is free software, the code is 100% open, and everyone is welcome to +make it better.

    • +
+

If you do care about privacy, want to be a conscious user, or otherwise believe +in digital freedom, make SearXNG your default search engine or run it on your +own server!

+
+
+

How do I set it as the default search engine?

+

SearXNG supports OpenSearch. For more information on changing your default +search engine, see your browser’s documentation:

+
    +

  • Firefox

    • +

  • Microsoft Edge - Behind the link, you will also find some useful instructions +for Chrome and Safari.

    • +

  • Chromium-based browsers only add websites that the user navigates to without +a path.

    • +
+

When adding a search engine, there must be no duplicates with the same name. If +you encounter a problem where you cannot add the search engine, you can either:

+
    +

  • remove the duplicate (default name: SearXNG) or

    • +

  • contact the owner to give the instance a different name than the default.

    • +
+
+
+

How does it work?

+

SearXNG is a fork from the well-known searx metasearch engine which was +inspired by the Seeks project. It provides basic privacy by mixing your +queries with searches on other platforms without storing search data. SearXNG +can be added to your browser’s search bar; moreover, it can be set as the +default search engine.

+

The stats page contains some useful anonymous usage +statistics about the engines used.

+
+
+

How can I make it my own?

+

SearXNG appreciates your concern regarding logs, so take the code from the +SearXNG sources and run it yourself!

+

Add your instance to this list of public +instances to help other people +reclaim their privacy and make the internet freer. The more decentralized the +internet is, the more freedom we have!

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/user/configured_engines.html b/user/configured_engines.html new file mode 100644 index 000000000..1793a31f3 --- /dev/null +++ b/user/configured_engines.html @@ -0,0 +1,2568 @@ + + + + + + + + Configured Engines — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Configured Engines

+ +

SearXNG supports 149 search engines of which +77 are enabled by default.

+

Engines can be assigned to multiple categories. +The UI displays the tabs that are configured in categories_as_tabs. In addition to these UI categories (also +called tabs), engines can be queried by their name or the categories they +belong to, by using a !bing syntax.

+ +
+

tab !general

+
+

group !web

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bing

!bi

bing

y

3.0

1

y

y

y

y

brave

!br

brave

3.0

1

y

y

y

y

duckduckgo

!ddg

duckduckgo

3.0

1

y

y

y

y

google

!go

google

3.0

1

y

y

y

y

mojeek

!mjk

xpath

y

3.0

1

y

qwant

!qw

qwant

3.0

1

y

y

y

startpage

!sp

startpage

y

6.0

1

y

y

y

y

wiby

!wib

json_engine

y

3.0

1

y

yahoo

!yh

yahoo

y

3.0

1

y

y

y

seznam +(CZ)

!szn

seznam

y

3.0

1

goo +(JA)

!goo

xpath

y

4.0

1

y

naver +(KO)

!nvr

xpath

y

3.0

1

y

+
+
+

group !wikimedia

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

wikibooks

!wb

mediawiki

3.0

1

y

wikiquote

!wq

mediawiki

3.0

1

y

wikisource

!ws

mediawiki

3.0

1

y

wikispecies

!wsp

mediawiki

3.0

1

y

wikiversity

!wv

mediawiki

3.0

1

y

wikivoyage

!wy

mediawiki

3.0

1

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

alexandria

!alx

json_engine

y

1.5

1

y

archive is

!ai

xpath

y

7.0

1

crowdview

!cv

json_engine

y

3.0

1

curlie

!cl

xpath

y

3.0

1

y

currency

!cc

currency_convert

3.0

100

not applicable (online_currency)

ddg definitions

!ddd

duckduckgo_definitions

y

3.0

2

dictzone

!dc

dictzone

3.0

100

not applicable (online_dictionary)

lingva

!lv

lingva

3.0

1

not applicable (online_dictionary)

tineye

!tin

tineye

y

9.0

1

not applicable (online_url_search)

wikidata

!wd

wikidata

3.0

2

y

wikipedia

!wp

wikipedia

3.0

1

y

wolframalpha

!wa

wolframalpha_noapi

y

6.0

1

yep

!yep

json_engine

y

3.0

1

tagesschau +(DE)

!ts

tagesschau

y

3.0

1

y

wikimini +(FR)

!wkmn

xpath

y

3.0

1

+
+
+
+

tab !images

+
+

group !web

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bing images

!bii

bing_images

3.0

1

y

y

y

y

brave.images

!brimg

brave

3.0

1

y

y

duckduckgo images

!ddi

duckduckgo_images

y

3.0

1

y

y

y

google images

!goi

google_images

3.0

1

y

y

y

y

qwant images

!qwi

qwant

3.0

1

y

y

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

1x

!1x

www1x

y

3.0

1

artic

!arc

artic

4.0

1

y

deviantart

!da

deviantart

3.0

1

y

y

flickr

!fl

flickr_noapi

3.0

1

y

y

frinkiac

!frk

frinkiac

y

3.0

1

library of congress

!loc

loc

3.0

1

y

openverse

!opv

openverse

3.0

1

y

unsplash

!us

unsplash

3.0

1

y

+
+
+
+

tab !videos

+
+

group !web

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bing videos

!biv

bing_videos

3.0

1

y

y

y

y

brave.videos

!brvid

brave

3.0

1

y

y

google videos

!gov

google_videos

3.0

1

y

y

y

y

qwant videos

!qwv

qwant

3.0

1

y

y

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

ccc-tv

!c3tv

xpath

y

3.0

1

dailymotion

!dm

dailymotion

3.0

1

y

y

y

y

google play movies

!gpm

google_play

y

3.0

1

invidious

!iv

invidious

y

3.0

1

y

y

peertube

!ptb

peertube

y

6.0

1

y

y

y

y

piped

!ppd

piped

3.0

1

y

rumble

!ru

rumble

y

3.0

1

y

sepiasearch

!sep

sepiasearch

3.0

1

y

y

y

y

vimeo

!vm

vimeo

3.0

1

y

youtube

!yt

youtube_noapi

3.0

1

y

y

mediathekviewweb +(DE)

!mvw

mediathekviewweb

y

3.0

1

y

ina +(FR)

!in

ina

y

6.0

1

y

+
+
+
+

tab !news

+
+

group !wikimedia

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

wikinews

!wn

mediawiki

3.0

1

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bing news

!bin

bing_news

3.0

1

y

y

y

brave.news

!brnews

brave

3.0

1

y

y

google news

!gon

google_news

3.0

1

y

y

qwant news

!qwn

qwant

3.0

1

y

y

y

yahoo news

!yhn

yahoo_news

3.0

1

y

tagesschau +(DE)

!ts

tagesschau

y

3.0

1

y

+
+
+
+

tab !map

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

apple maps

!apm

apple_maps

y

5.0

1

openstreetmap

!osm

openstreetmap

3.0

1

photon

!ph

photon

3.0

1

+
+
+

tab !music

+
+

group !lyrics

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

azlyrics

!lyrics

xpath

y

4.0

1

y

genius

!gen

genius

3.0

1

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bandcamp

!bc

bandcamp

3.0

1

y

deezer

!dz

deezer

y

3.0

1

y

gpodder

!gpod

json_engine

y

4.0

1

invidious

!iv

invidious

y

3.0

1

y

y

mixcloud

!mc

mixcloud

3.0

1

y

piped.music

!ppdm

piped

3.0

1

y

soundcloud

!sc

soundcloud

3.0

1

y

youtube

!yt

youtube_noapi

3.0

1

y

y

+
+
+
+

tab !it

+
+

group !packages

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

docker hub

!dh

docker_hub

3.0

1

y

hoogle

!ho

xpath

3.0

1

y

lib.rs

!lrs

xpath

y

3.0

1

metacpan

!cpan

metacpan

y

3.0

1

y

npm

!npm

json_engine

y

5.0

1

y

packagist

!pack

json_engine

y

5.0

1

y

pub.dev

!pd

xpath

y

3.0

1

y

pypi

!pypi

xpath

3.0

1

y

rubygems

!rbg

xpath

y

3.0

1

y

+
+
+

group !q&a

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

askubuntu

!ubuntu

stackexchange

3.0

1

y

stackoverflow

!st

stackexchange

3.0

1

y

superuser

!su

stackexchange

3.0

1

y

+
+
+

group !repos

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

bitbucket

!bb

xpath

y

4.0

1

y

codeberg

!cb

json_engine

y

3.0

1

github

!gh

github

3.0

1

gitlab

!gl

json_engine

y

10.0

1

y

sourcehut

!srht

xpath

y

3.0

1

y

+
+
+

group !software_wikis

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

arch linux wiki

!al

archlinux

3.0

1

y

y

free software directory

!fsd

mediawiki

y

5.0

1

y

gentoo

!ge

gentoo

10.0

1

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

anaconda

!conda

xpath

y

6.0

1

y

framalibre

!frl

framalibre

y

3.0

1

y

habrahabr

!habr

xpath

y

4.0

1

y

lobste.rs

!lo

xpath

y

5.0

1

mankier

!man

json_engine

3.0

1

searchcode code

!scc

searchcode_code

y

3.0

1

y

+
+
+
+

tab !science

+
+

group !scientific_publications

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

arxiv

!arx

arxiv

4.0

1

y

crossref

!cr

crossref

y

30

1

y

google scholar

!gos

google_scholar

3.0

1

y

y

y

pubmed

!pub

pubmed

3.0

1

semantic scholar

!se

semantic_scholar

y

3.0

1

y

+
+
+

group !wikimedia

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

wikispecies

!wsp

mediawiki

3.0

1

y

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

openairedatasets

!oad

json_engine

5.0

1

y

openairepublications

!oap

json_engine

5.0

1

y

pdbe

!pdb

pdbe

3.0

1

+
+
+
+

tab !files

+
+

group !apps

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

apk mirror

!apkm

apkmirror

y

4.0

1

y

apple app store

!aps

apple_app_store

y

3.0

1

y

fdroid

!fd

fdroid

y

3.0

1

y

google play apps

!gpa

google_play

y

3.0

1

+
+
+

without further subgrouping

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

1337x

!1337x

1337x

y

3.0

1

y

annas archive

!aa

annas_archive

y

3.0

1

y

bt4g

!bt4g

bt4g

3.0

1

y

y

btdigg

!bt

btdigg

y

3.0

1

y

kickass

!kc

kickass

y

4.0

1

y

library genesis

!lg

xpath

y

7.0

1

nyaa

!nt

nyaa

y

3.0

1

y

openrepos

!or

xpath

y

4.0

1

y

piratebay

!tpb

piratebay

3.0

1

solidtorrents

!solid

solidtorrents

4.0

1

y

tokyotoshokan

!tt

tokyotoshokan

y

6.0

1

y

z-library

!zlib

zlibrary

7.0

1

y

y

+
+
+
+

tab !social_media

+ ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Engines configured by default (in settings.yml)

Supported features

Name

!bang

Module

Disabled

Timeout

Weight

Paging

Locale

Safe search

Time range

9gag

!9g

9gag

y

3.0

1

y

lemmy comments

!lecom

lemmy

3.0

1

y

lemmy communities

!leco

lemmy

3.0

1

y

lemmy posts

!lepo

lemmy

3.0

1

y

lemmy users

!leus

lemmy

3.0

1

y

reddit

!re

reddit

3.0

1

twitter

!tw

twitter

y

3.0

1

+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/user/index.html b/user/index.html new file mode 100644 index 000000000..0987ec858 --- /dev/null +++ b/user/index.html @@ -0,0 +1,156 @@ + + + + + + + + User information — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

User information

+
+ +
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/user/search-syntax.html b/user/search-syntax.html new file mode 100644 index 000000000..26ddb778e --- /dev/null +++ b/user/search-syntax.html @@ -0,0 +1,209 @@ + + + + + + + + Search syntax — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

Search syntax

+

SearXNG comes with a search syntax by with you can modify the categories, +engines, languages and more. See the preferences for +the list of engines, categories and languages.

+
+

! select engine and category

+

To set category and/or engine names use a ! prefix. To give a few examples:

+
    +

  • search in Wikipedia for paris

    +
      +

    • !wp paris

      • +

    • !wikipedia paris

      • +
    +
    • +

  • search in category map for paris

    +
      +

    • !map paris

      • +
    +
    • +

  • image search

    +
      +

    • !images Wau Holland

      • +
    +
    • +
+

Abbreviations of the engines and languages are also accepted. Engine/category +modifiers are chain able and inclusive. E.g. with !map !ddg !wp paris search in map category and DuckDuckGo and Wikipedia for paris.

+
+
+

: select language

+

To select language filter use a : prefix. To give an example:

+
    +

  • search Wikipedia by a custom language

    +
      +

    • :fr !wp Wau Holland

      • +
    +
    • +
+
+
+

!! external bangs

+

SearXNG supports the external bangs from DuckDuckGo. To directly jump to a +external search page use the !! prefix. To give an example:

+
    +

  • search Wikipedia by a custom language

    +
      +

    • !!wfr Wau Holland

      • +
    +
    • +
+

Please note, your search will be performed directly in the external search +engine, SearXNG cannot protect your privacy on this.

+
+
+

Special Queries

+

In the preferences page you find keywords for +special queries. To give a few examples:

+
    +

  • generate a random UUID

    +
      +

    • random uuid

      • +
    +
    • +

  • find the average

    +
      +

    • avg 123 548 2.04 24.2

      • +
    +
    • +

  • show user agent of your browser (needs to be activated)

    +
      +

    • user-agent

      • +
    +
    • +

  • convert strings to different hash digests (needs to be activated)

    +
      +

    • md5 lorem ipsum

      • +

    • sha512 lorem ipsum

      • +
    +
    • +
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/utils/index.html b/utils/index.html new file mode 100644 index 000000000..b5f796610 --- /dev/null +++ b/utils/index.html @@ -0,0 +1,158 @@ + + + + + + + + DevOps tooling box — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

DevOps tooling box

+

In the folder git://utils/ we maintain some tools useful for administrators +and developers.

+
+ +
+
+

Common command environments

+

The scripts in our tooling box often dispose of common environments:

+
+
FORCE_TIMEOUTenvironment

Sets timeout for interactive prompts. If you want to run a script in batch +job, with defaults choices, set FORCE_TIMEOUT=0. By example; to install a +SearXNG server and nginx proxy on all containers of the SearXNG suite use:

+
sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all
+sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+
+
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/utils/lxc.sh.html b/utils/lxc.sh.html new file mode 100644 index 000000000..acc4c42dd --- /dev/null +++ b/utils/lxc.sh.html @@ -0,0 +1,504 @@ + + + + + + + + utils/lxc.sh — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

utils/lxc.sh

+

With the use of Linux Containers (LXC) we can scale our tasks over a stack of +containers, what we call the: lxc suite. The SearXNG suite config is +loaded by default, every time you start the lxc.sh script (you do not need +to care about).

+ + +
+

Install LXD

+

Before you can start with containers, you need to install and initiate LXD +once:

+
$ snap install lxd
+$ lxd init --auto
+
+
+

To make use of the containers from the SearXNG suite, you have to build the +LXC suite containers initial. But be warned, this might +take some time:

+
$ sudo -H ./utils/lxc.sh build
+
+
+ +

A cup of coffee later, your LXC suite is build up and you can run whatever task +you want / in a selected or even in all LXC suite containers.

+
+

Internet Connectivity & Docker

+ +

There is a conflict in the iptables setup of Docker & LXC. If you have +docker installed, you may find that the internet connectivity of your LXD +containers no longer work.

+

Whenever docker is started (reboot) it sets the iptables policy for the +FORWARD chain to DROP [ref]:

+
$ sudo -H iptables-save | grep FORWARD
+:FORWARD ACCEPT [7048:7851230]
+:FORWARD DROP [7048:7851230]
+
+
+

A handy solution of this problem might be to reset the policy for the +FORWARD chain after the network has been initialized. For this create a +file in the if-up section of the network (/etc/network/if-up.d/iptable) +and insert the following lines:

+
#!/bin/sh
+iptables -F FORWARD
+iptables -P FORWARD ACCEPT
+
+
+

Don’t forget to set the execution bit:

+
sudo chmod ugo+x /etc/network/if-up.d/iptable
+
+
+

Reboot your system and check the iptables rules:

+
$ sudo -H iptables-save | grep FORWARD
+:FORWARD ACCEPT [7048:7851230]
+:FORWARD ACCEPT [7048:7851230]
+
+
+
+
+
+

SearXNG LXC suite

+

The intention of the SearXNG LXC suite is to build up a suite of containers +for development tasks or buildhosts with a very +small set of simple commands. At the end of the --help output the SearXNG +suite from the SearXNG suite config is introduced:

+
$ sudo -H ./utils/lxc.sh --help
+...
+LXC suite: searxng
+  Suite includes installation of SearXNG
+  images:     ubu2004 ubu2204 fedora35 archlinux
+  containers: searxng-ubu2004 searxng-ubu2204 searxng-fedora35 searxng-archlinux
+
+
+

As shown above there are images and containers build up on this images. To show +more info about the containers in the SearXNG LXC suite call show suite. +If this is the first time you make use of the SearXNG LXC suite, no containers +are installed and the output is:

+
$ sudo -H ./utils/lxc.sh show suite
+
+LXC suite (searxng-*)
+=====================
+
++------+-------+------+------+------+-----------+
+| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
++------+-------+------+------+------+-----------+
+
+WARN:  container searxng-ubu2004 does not yet exists
+WARN:  container searxng-ubu2204 does not yet exists
+WARN:  container searxng-fedora35 does not yet exists
+WARN:  container searxng-archlinux does not yet exists
+
+
+

If you do not want to run a command or a build in all containers, you can +build just one. Here by example in the container that is build upon the +archlinux image:

+
$ sudo -H ./utils/lxc.sh build searxng-archlinux
+$ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd
+
+
+

Otherwise, to apply a command to all containers you can use:

+
$ sudo -H ./utils/lxc.sh build
+$ sudo -H ./utils/lxc.sh cmd -- ls -la .
+
+
+
+

Running commands

+

Inside containers, you can run scripts from the DevOps tooling box or run +what ever command you need. By example, to start a bash use:

+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash
+INFO:  [searxng-archlinux] bash
+[root@searxng-archlinux SearXNG]#
+
+
+
+
+

Good to know

+

Each container shares the root folder of the repository and the command +utils/lxc.sh cmd handle relative path names transparent:

+
$ pwd
+/share/SearXNG
+
+$ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd
+INFO:  [searxng-archlinux] pwd
+/share/SearXNG
+
+
+

The path /share/SearXNG will be different on your HOST system. The commands +in the conatiner are executed by the root inside of the container. Compare +output of:

+
$ ls -li Makefile
+47712402 -rw-rw-r-- 1 markus markus 2923 Apr 19 13:52 Makefile
+
+$ sudo -H ./utils/lxc.sh cmd searxng-archlinux ls -li Makefile
+INFO:  [searxng-archlinux] ls -li Makefile
+47712402 -rw-rw-r-- 1 root root 2923 Apr 19 11:52 Makefile
+...
+
+
+

Since the path /share/SearXNG of the HOST system is wrapped into the +container under the same name, the shown Makefile (inode 47712402) in +the ouput is always the identical /share/SearXNG/Makefile from the HOST +system. In the example shown above the owner of the path in the container is +the root user of the conatiner (and the timezone in the container is +different to HOST system).

+
+
+

Install suite

+ +

To install the complete SearXNG suite into all LXC +containers leave the container argument empty and run:

+
$ sudo -H ./utils/lxc.sh build
+$ sudo -H ./utils/lxc.sh install suite
+
+
+

To build & install suite only in one container you can use by example:

+
$ sudo -H ./utils/lxc.sh build searxng-archlinux
+$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
+
+
+

The command above installs a SearXNG suite (see Installation Script). +To install a nginx reverse proxy (or alternatively +use apache):

+
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+
+
+

Same operation just in one container of the suite:

+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+
+
+

The FORCE_TIMEOUT environment is set to zero to run the +script without user interaction.

+

To get the IP (URL) of the SearXNG service in the containers use show suite +command. To test instances from containers just open the URLs in your +WEB-Browser:

+
$ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL
+
+[searxng-ubu2110]      SEARXNG_URL          : http://n.n.n.170/searxng
+[searxng-ubu2004]      SEARXNG_URL          : http://n.n.n.160/searxng
+[searxnggfedora35]     SEARXNG_URL          : http://n.n.n.150/searxng
+[searxng-archlinux]    SEARXNG_URL          : http://n.n.n.140/searxng
+
+
+
+
+

Clean up

+

If there comes the time you want to get rid off all the containers and +clean up local images just type:

+
$ sudo -H ./utils/lxc.sh remove
+$ sudo -H ./utils/lxc.sh remove images
+
+
+
+
+
+

Setup SearXNG buildhost

+

You can install the SearXNG buildhost environment into one or all containers. +The installation procedure to set up a build host takes its +time. Installation in all containers will take more time (time for another cup +of coffee).

+
sudo -H ./utils/lxc.sh cmd -- ./utils/searxng.sh install buildhost
+
+
+

To build (live) documentation inside a archlinux container:

+
sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.clean docs.live
+...
+[I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
+
+
+

To get IP of the container and the port number live docs is listening:

+
$ sudo ./utils/lxc.sh show suite | grep docs.live
+...
+[searxng-archlinux]  INFO:  (eth0) docs.live:  http://n.n.n.140:8080/
+
+
+
+
+

Command Help

+

The --help output of the script is largely self-explanatory:

+
usage::
+  lxc.sh build        [containers|<name>]
+  lxc.sh copy         [images]
+  lxc.sh remove       [containers|<name>|images]
+  lxc.sh [start|stop] [containers|<name>]
+  lxc.sh show         [images|suite|info|config [<name>]]
+  lxc.sh cmd          [--|<name>] '...'
+  lxc.sh install      [suite|base [<name>]]
+
+build
+  :containers:   build, launch all containers and 'install base' packages
+  :<name>:       build, launch container <name>  and 'install base' packages
+copy:
+  :images:       copy remote images of the suite into local storage
+remove
+  :containers:   delete all 'containers' or only <container-name>
+  :images:       delete local images of the suite
+start/stop
+  :containers:   start/stop all 'containers' from the suite
+  :<name>:       start/stop container <name> from suite
+show
+  :info:         show info of all (or <name>) containers from LXC suite
+  :config:       show config of all (or <name>) containers from the LXC suite
+  :suite:        show services of all (or <name>) containers from the LXC suite
+  :images:       show information of local images
+cmd
+  use single quotes to evaluate in container's bash, e.g.: 'echo $(hostname)'
+  --             run command '...' in all containers of the LXC suite
+  :<name>:       run command '...' in container <name>
+install
+  :base:         prepare LXC; install basic packages
+  :suite:        install LXC searxng suite into all (or <name>) containers
+
+LXC suite: searxng
+  Suite includes installation of SearXNG
+  images:     ubu2004 ubu2204 fedora35 archlinux
+  containers: searxng-ubu2004 searxng-ubu2204 searxng-fedora35 searxng-archlinux
+
+
+
+
+

SearXNG suite config

+

The SearXNG suite is defined in the file git://utils/lxc-searxng.env:

+
# -*- coding: utf-8; mode: sh indent-tabs-mode: nil -*-
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# shellcheck shell=bash
+
+# This file is a setup of a LXC suite.  It is sourced from different context, do
+# not manipulate the environment directly, implement functions and manipulate
+# environment only in subshells.
+
+lxc_set_suite_env() {
+
+    export LXC_SUITE_NAME="searxng"
+
+    # name of https://images.linuxcontainers.org
+    export LINUXCONTAINERS_ORG_NAME="${LINUXCONTAINERS_ORG_NAME:-images}"
+    export LXC_HOST_PREFIX="${LXC_SUITE_NAME:-searx}"
+    export LXC_SUITE=(
+
+        # end of standard support see https://wiki.ubuntu.com/Releases
+        "$LINUXCONTAINERS_ORG_NAME:ubuntu/20.04"  "ubu2004" # LTS EOSS April 2025
+        "$LINUXCONTAINERS_ORG_NAME:ubuntu/22.04"  "ubu2204" # LTS EOSS April 2027
+
+        # EOL see https://fedoraproject.org/wiki/Releases
+        "$LINUXCONTAINERS_ORG_NAME:fedora/35"     "fedora35"
+
+        # rolling releases see https://www.archlinux.org/releng/releases/
+        "$LINUXCONTAINERS_ORG_NAME:archlinux"     "archlinux"
+    )
+}
+
+lxc_suite_install_info() {
+    (
+        lxc_set_suite_env
+        cat <<EOF
+LXC suite: ${LXC_SUITE_NAME}
+  Suite includes installation of SearXNG
+  images:     ${LOCAL_IMAGES[*]}
+  containers: ${CONTAINERS[*]}
+EOF
+    )
+}
+
+lxc_suite_install() {
+    (
+        lxc_set_suite_env
+        FORCE_TIMEOUT=0 "${LXC_REPO_ROOT}/utils/searxng.sh" install all
+        rst_title "Suite installation finished ($(hostname))" part
+        if ask_yn "Developer install? (wraps source from HOST into the running instance)" Yn; then
+            "${LXC_REPO_ROOT}/utils/searxng.sh" searxng.install.link_src "$(pwd)"
+        fi
+        lxc_suite_info
+        echo
+    )
+}
+
+lxc_suite_info() {
+    (
+        lxc_set_suite_env
+        for ip in $(global_IPs) ; do
+            if [[ $ip =~ .*:.* ]]; then
+                info_msg "(${ip%|*}) IPv6:       http://[${ip#*|}]"
+            else
+                # IPv4:
+                # shellcheck disable=SC2034,SC2031
+                info_msg "(${ip%|*}) docs-live:  http://${ip#*|}:8080/"
+            fi
+        done
+        "${LXC_REPO_ROOT}/utils/searxng.sh" searxng.instance.env
+    )
+}
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file diff --git a/utils/searxng.sh.html b/utils/searxng.sh.html new file mode 100644 index 000000000..f65188be8 --- /dev/null +++ b/utils/searxng.sh.html @@ -0,0 +1,191 @@ + + + + + + + + utils/searxng.sh — SearXNG Documentation (2023.8.11+905ce2a6f) + + + + + + + + + + + + + +
+
+
+
+ +
+

utils/searxng.sh

+

To simplify the installation and maintenance of a SearXNG instance you can use the +script git://utils/searxng.sh.

+ + +
+

Install

+

In most cases you will install SearXNG simply by running the command:

+
sudo -H ./utils/searx.sh install all
+
+
+

The installation is described in chapter Step by step installation.

+
+
+

Command Help

+

The --help output of the script is largely self-explanatory:

+
usage:
+  searxng.sh install    [all|user|pyenv|settings|uwsgi|redis|nginx|apache|searxng-src|packages|buildhost]
+  searxng.sh remove     [all|user|pyenv|settings|uwsgi|redis|nginx|apache]
+  searxng.sh instance   [cmd|update|check|localtest|inspect]
+install|remove:
+  all           : complete (de-) installation of the SearXNG service
+  user          : service user 'searxng' (/usr/local/searxng)
+  pyenv         : virtualenv (python) in /usr/local/searxng/searx-pyenv
+  settings      : settings from /etc/searxng/settings.yml
+  uwsgi         : SearXNG's uWSGI app searxng.ini
+  redis         : build & install or remove a local redis server /usr/local/searxng-redis/run/redis.sock
+  nginx         : HTTP site /etc/nginx/default.apps-available/searxng.conf
+  apache        : HTTP site /etc/apache2/sites-available/searxng.conf
+install:
+  searxng-src   : clone https://github.com/searxng/searxng into /usr/local/searxng/searxng-src
+  packages      : installs packages from OS package manager required by SearXNG
+  buildhost     : installs packages from OS package manager required by a SearXNG buildhost
+instance:
+  update        : update SearXNG instance (git fetch + reset & update settings.yml)
+  check         : run checks from utils/searxng_check.py in the active installation
+  inspect       : run some small tests and inspect SearXNG's server status and log
+  get_setting   : get settings value from running SearXNG instance
+  cmd           : run command in SearXNG instance's environment (e.g. bash)
+uWSGI:
+  SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket
+environment /usr/local/searxng/searxng-src/utils/brand.env:
+  GIT_URL              : https://github.com/searxng/searxng
+  GIT_BRANCH           : master
+  SEARXNG_URL          : http://fv-az808-537/searxng
+  SEARXNG_PORT         : 8888
+  SEARXNG_BIND_ADDRESS : 127.0.0.1
+
+
+
+
+ + +
+
+
+
+ + +
+
+ + + + + + \ No newline at end of file