searxngRebrandZaclys/dev/engines/offline/sql-engines.html
2023-08-11 10:34:02 +00:00

323 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SQL Engines &#8212; SearXNG Documentation (2023.8.11+905ce2a6f)</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=4f649999" />
<link rel="stylesheet" type="text/css" href="../../../_static/searxng.css?v=52e4ff28" />
<link rel="stylesheet" type="text/css" href="../../../_static/tabs.css?v=a5c4661c" />
<script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js?v=3c88bde0"></script>
<script src="../../../_static/doctools.js?v=888ff710"></script>
<script src="../../../_static/sphinx_highlight.js?v=4825356b"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="Tineye" href="../online_url_search/tineye.html" />
<link rel="prev" title="Local Search APIs" href="search-indexer-engines.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../../../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../../../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="../online_url_search/tineye.html" title="Tineye"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="search-indexer-engines.html" title="Local Search APIs"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../../../index.html">SearXNG Documentation (2023.8.11+905ce2a6f)</a> &#187;</li>
<li class="nav-item nav-item-1"><a href="../../index.html" >Developer documentation</a> &#187;</li>
<li class="nav-item nav-item-2"><a href="../index.html" accesskey="U">Engine Implementations</a> &#187;</li>
<li class="nav-item nav-item-this"><a href="">SQL Engines</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="sql-engines">
<span id="id1"></span><h1>SQL Engines<a class="headerlink" href="#sql-engines" title="Permalink to this heading"></a></h1>
<aside class="sidebar">
<p class="sidebar-title">further read</p>
<ul class="simple">
<li><p><a class="reference external" href="https://www.sqlite.org/index.html">SQLite</a></p></li>
<li><p><a class="reference external" href="https://www.postgresql.org">PostgreSQL</a></p></li>
<li><p><a class="reference external" href="https://www.mysql.com">MySQL</a></p></li>
</ul>
</aside>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#extra-dependencies" id="id9">Extra Dependencies</a></p></li>
<li><p><a class="reference internal" href="#configure-the-engines" id="id10">Configure the engines</a></p>
<ul>
<li><p><a class="reference internal" href="#engine-sqlite" id="id11">SQLite</a></p></li>
<li><p><a class="reference internal" href="#engine-postgresql" id="id12">PostgreSQL</a></p></li>
<li><p><a class="reference internal" href="#engine-mysql-server" id="id13">MySQL</a></p></li>
</ul>
</li>
</ul>
</nav>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<p>Initial sponsored by <a class="reference external" href="https://nlnet.nl/discovery">Search and Discovery Fund</a> of <a class="reference external" href="https://nlnet.nl/">NLnet Foundation</a>.</p>
</aside>
<p>With the <em>SQL engines</em> you can bind SQL databases into SearXNG. The following
Relational Database Management System (RDBMS) are supported:</p>
<ul class="simple">
<li><p><a class="reference internal" href="#engine-sqlite"><span class="std std-ref">SQLite</span></a></p></li>
<li><p><a class="reference internal" href="#engine-postgresql"><span class="std std-ref">PostgreSQL</span></a></p></li>
<li><p><a class="reference internal" href="#engine-mysql-server"><span class="std std-ref">MySQL</span></a></p></li>
</ul>
<p>All of the engines above are just commented out in the <a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/settings.yml">settings.yml</a>, as you have to set the required attributes for the
engines, e.g. <code class="docutils literal notranslate"><span class="pre">database:</span></code></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nv">sqlite|postgresql|mysql_server</span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
<span class="w"> </span><span class="nt">result_template</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nv">template_name</span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
</pre></div>
</div>
<p>By default, the engines use the <code class="docutils literal notranslate"><span class="pre">key-value</span></code> template for displaying results /
see <a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/templates/simple/result_templates/key-value.html">simple</a>
theme. If you are not satisfied with the original result layout, you can use
your own template, set <code class="docutils literal notranslate"><span class="pre">result_template</span></code> attribute to <code class="docutils literal notranslate"><span class="pre">{template_name}</span></code> and
place the templates at:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">searx</span><span class="o">/</span><span class="n">templates</span><span class="o">/</span><span class="p">{</span><span class="n">theme_name</span><span class="p">}</span><span class="o">/</span><span class="n">result_templates</span><span class="o">/</span><span class="p">{</span><span class="n">template_name</span><span class="p">}</span>
</pre></div>
</div>
<p>If you do not wish to expose these engines on a public instance, you can still
add them and limit the access by setting <code class="docutils literal notranslate"><span class="pre">tokens</span></code> as described in section
<a class="reference internal" href="../../../admin/settings/settings_engine.html#private-engines"><span class="std std-ref">Private Engines (tokens)</span></a>.</p>
<section id="extra-dependencies">
<h2><a class="toc-backref" href="#id9" role="doc-backlink">Extra Dependencies</a><a class="headerlink" href="#extra-dependencies" title="Permalink to this heading"></a></h2>
<p>For using <a class="reference internal" href="#engine-postgresql"><span class="std std-ref">PostgreSQL</span></a> or <a class="reference internal" href="#engine-mysql-server"><span class="std std-ref">MySQL</span></a> you need to
install additional packages in Pythons Virtual Environment of your SearXNG
instance. To switch into the environment (<a class="reference internal" href="../../../admin/installation-searxng.html#searxng-src"><span class="std std-ref">Install SearXNG &amp; dependencies</span></a>) you can use
<a class="reference internal" href="../../../utils/searxng.sh.html#searxng-sh"><span class="std std-ref">utils/searxng.sh</span></a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
</pre></div>
</div>
</section>
<section id="configure-the-engines">
<h2><a class="toc-backref" href="#id10" role="doc-backlink">Configure the engines</a><a class="headerlink" href="#configure-the-engines" title="Permalink to this heading"></a></h2>
<p>The configuration of the new database engines are similar. You must put a valid
SQL-SELECT query in <code class="docutils literal notranslate"><span class="pre">query_str</span></code>. At the moment you can only bind at most one
parameter in your query. By setting the attribute <code class="docutils literal notranslate"><span class="pre">limit</span></code> you can define how
many results you want from the SQL server. Basically, it is the same as the
<code class="docutils literal notranslate"><span class="pre">LIMIT</span></code> keyword in SQL.</p>
<p>Please, do not include <code class="docutils literal notranslate"><span class="pre">LIMIT</span></code> or <code class="docutils literal notranslate"><span class="pre">OFFSET</span></code> in your SQL query as the engines
rely on these keywords during paging. If you want to configure the number of
returned results use the option <code class="docutils literal notranslate"><span class="pre">limit</span></code>.</p>
<section id="engine-sqlite">
<span id="id2"></span><h3><a class="toc-backref" href="#id11" role="doc-backlink">SQLite</a><a class="headerlink" href="#engine-sqlite" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/sqlite.py">sqlite.py</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.sqlite"></span><p>SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency.</p>
<section id="example">
<h4>Example<a class="headerlink" href="#example" title="Permalink to this heading"></a></h4>
<p>To demonstrate the power of database engines, here is a more complex example
which reads from a <a class="reference external" href="https://mediathekview.de/">MediathekView</a> (DE) movie database. For this example of the
SQlite engine download the database:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://liste.mediathekview.de/filmliste-v2.db.bz2">https://liste.mediathekview.de/filmliste-v2.db.bz2</a></p></li>
</ul>
<p>and unpack into <code class="docutils literal notranslate"><span class="pre">searx/data/filmliste-v2.db</span></code>. To search the database use e.g
Query to test: <code class="docutils literal notranslate"><span class="pre">!mediathekview</span> <span class="pre">concert</span></code></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mediathekview</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sqlite</span>
<span class="w"> </span><span class="nt">disabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">False</span>
<span class="w"> </span><span class="nt">categories</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">general</span>
<span class="w"> </span><span class="nt">result_template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">default.html</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searx/data/filmliste-v2.db</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">&gt;-</span>
<span class="w"> </span><span class="no">SELECT title || &#39; (&#39; || time(duration, &#39;unixepoch&#39;) || &#39;)&#39; AS title,</span>
<span class="w"> </span><span class="no">COALESCE( NULLIF(url_video_hd,&#39;&#39;), NULLIF(url_video_sd,&#39;&#39;), url_video) AS url,</span>
<span class="w"> </span><span class="no">description AS content</span>
<span class="w"> </span><span class="no">FROM film</span>
<span class="w"> </span><span class="no">WHERE title LIKE :wildcard OR description LIKE :wildcard</span>
<span class="w"> </span><span class="no">ORDER BY duration DESC</span>
</pre></div>
</div>
</section>
<section id="implementations">
<h4>Implementations<a class="headerlink" href="#implementations" title="Permalink to this heading"></a></h4>
</section>
<dl class="py function">
<dt class="sig sig-object py" id="searx.engines.sqlite.sqlite_cursor">
<span class="sig-prename descclassname"><span class="pre">searx.engines.sqlite.</span></span><span class="sig-name descname"><span class="pre">sqlite_cursor</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/searx/engines/sqlite.html#sqlite_cursor"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#searx.engines.sqlite.sqlite_cursor" title="Permalink to this definition"></a></dt>
<dd><p>Implements a <a class="reference external" href="https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager" title="(in Python v3.11)"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Context</span> <span class="pre">Manager</span></code></a> for a
<a class="reference external" href="https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor" title="(in Python v3.11)"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sqlite3.Cursor</span></code></a>.</p>
<p>Open database in read only mode: if the database doesnt exist. The default
mode creates an empty file on the file system. See:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://docs.python.org/3/library/sqlite3.html#sqlite3.connect">https://docs.python.org/3/library/sqlite3.html#sqlite3.connect</a></p></li>
<li><p><a class="reference external" href="https://www.sqlite.org/uri.html">https://www.sqlite.org/uri.html</a></p></li>
</ul>
</dd></dl>
</section>
<section id="engine-postgresql">
<span id="id3"></span><h3><a class="toc-backref" href="#id12" role="doc-backlink">PostgreSQL</a><a class="headerlink" href="#engine-postgresql" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/postgresql.py">postgresql.py</a></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span></code> <a class="reference external" href="psycopg2">psycopg2-binary</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.postgresql"></span><p>PostgreSQL is a powerful and robust open source database. Before configuring
the PostgreSQL engine, you must install the dependency <code class="docutils literal notranslate"><span class="pre">psychopg2</span></code>.</p>
<section id="id4">
<h4>Example<a class="headerlink" href="#id4" title="Permalink to this heading"></a></h4>
<p>Below is an example configuration:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searxng</span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">password</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;SELECT</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">from</span><span class="nv"> </span><span class="s">my_table</span><span class="nv"> </span><span class="s">WHERE</span><span class="nv"> </span><span class="s">my_column</span><span class="nv"> </span><span class="s">=</span><span class="nv"> </span><span class="s">%(query)s&#39;</span>
</pre></div>
</div>
</section>
<section id="id5">
<h4>Implementations<a class="headerlink" href="#id5" title="Permalink to this heading"></a></h4>
</section>
</section>
<section id="engine-mysql-server">
<span id="id6"></span><h3><a class="toc-backref" href="#id13" role="doc-backlink">MySQL</a><a class="headerlink" href="#engine-mysql-server" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/mysql_server.py">mysql_server.py</a></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span></code> <a class="reference external" href="https://pypi.org/project/mysql-connector-python">mysql-connector-python</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.mysql_server"></span><p>MySQL is said to be the most popular open source database. Before enabling
MySQL engine, you must install the package <code class="docutils literal notranslate"><span class="pre">mysql-connector-python</span></code>.</p>
<p>The authentication plugin is configurable by setting <code class="docutils literal notranslate"><span class="pre">auth_plugin</span></code> in the
attributes. By default it is set to <code class="docutils literal notranslate"><span class="pre">caching_sha2_password</span></code>.</p>
<section id="id7">
<h4>Example<a class="headerlink" href="#id7" title="Permalink to this heading"></a></h4>
<p>This is an example configuration for querying a MySQL server:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mysql_server</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searxng</span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">password</span>
<span class="w"> </span><span class="nt">limit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;SELECT</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">from</span><span class="nv"> </span><span class="s">my_table</span><span class="nv"> </span><span class="s">WHERE</span><span class="nv"> </span><span class="s">my_column=%(query)s&#39;</span>
</pre></div>
</div>
</section>
<section id="id8">
<h4>Implementations<a class="headerlink" href="#id8" title="Permalink to this heading"></a></h4>
</section>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<span id="sidebar-top"></span>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../../../index.html">
<img class="logo" src="../../../_static/searxng-wordmark.svg" alt="Logo"/>
</a></p>
<h3><a href="../../../index.html">Table of Contents</a></h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../user/index.html">User information</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../own-instance.html">Why use a private instance?</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../admin/index.html">Administrator documentation</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../index.html">Developer documentation</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../quickstart.html">Development Quickstart</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../contribution_guide.html">How to contribute</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../index.html">Engine Implementations</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../search_api.html">Search API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../plugins.html">Plugins</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../translation.html">Translation</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../lxcdev.html">Developing in Linux Containers</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../makefile.html">Makefile &amp; <code class="docutils literal notranslate"><span class="pre">./manage</span></code></a></li>
<li class="toctree-l2"><a class="reference internal" href="../../reST.html">reST primer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../searxng_extra/index.html">Tooling box <code class="docutils literal notranslate"><span class="pre">searxng_extra</span></code></a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../utils/index.html">DevOps tooling box</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../src/index.html">Source-Code</a></li>
</ul>
<h3>Project Links</h3>
<ul>
<li><a href="https://github.com/searxng/searxng/tree/master">Source</a>
<li><a href="https://github.com/searxng/searxng/wiki">Wiki</a>
<li><a href="https://searx.space">Public instances</a>
<li><a href="https://github.com/searxng/searxng/issues">Issue Tracker</a>
</ul><h3>Navigation</h3>
<ul>
<li><a href="../../../index.html">Overview</a>
<ul>
<li><a href="../../index.html">Developer documentation</a>
<ul>
<li><a href="../index.html">Engine Implementations</a>
<ul>
<li>Previous: <a href="search-indexer-engines.html" title="previous chapter">Local Search APIs</a>
<li>Next: <a href="../online_url_search/tineye.html" title="next chapter">Tineye</a></ul>
</li></ul>
</li>
</ul>
</li>
</ul>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../../../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../../../_sources/dev/engines/offline/sql-engines.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright SearXNG team.
</div>
<script src="../../../_static/version_warning_offset.js"></script>
</body>
</html>