searxngRebrandZaclys/dev/engines/offline/sql-engines.html

323 lines
24 KiB
HTML
Raw Normal View History

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>SQL Engines &#8212; SearXNG Documentation (2023.8.11+905ce2a6f)</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=4f649999" />
<link rel="stylesheet" type="text/css" href="../../../_static/searxng.css?v=52e4ff28" />
<link rel="stylesheet" type="text/css" href="../../../_static/tabs.css?v=a5c4661c" />
<script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js?v=3c88bde0"></script>
<script src="../../../_static/doctools.js?v=888ff710"></script>
<script src="../../../_static/sphinx_highlight.js?v=4825356b"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
<link rel="next" title="Tineye" href="../online_url_search/tineye.html" />
<link rel="prev" title="Local Search APIs" href="search-indexer-engines.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../../../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../../../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="../online_url_search/tineye.html" title="Tineye"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="search-indexer-engines.html" title="Local Search APIs"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../../../index.html">SearXNG Documentation (2023.8.11+905ce2a6f)</a> &#187;</li>
<li class="nav-item nav-item-1"><a href="../../index.html" >Developer documentation</a> &#187;</li>
<li class="nav-item nav-item-2"><a href="../index.html" accesskey="U">Engine Implementations</a> &#187;</li>
<li class="nav-item nav-item-this"><a href="">SQL Engines</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="sql-engines">
<span id="id1"></span><h1>SQL Engines<a class="headerlink" href="#sql-engines" title="Permalink to this heading"></a></h1>
<aside class="sidebar">
<p class="sidebar-title">further read</p>
<ul class="simple">
<li><p><a class="reference external" href="https://www.sqlite.org/index.html">SQLite</a></p></li>
<li><p><a class="reference external" href="https://www.postgresql.org">PostgreSQL</a></p></li>
<li><p><a class="reference external" href="https://www.mysql.com">MySQL</a></p></li>
</ul>
</aside>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#extra-dependencies" id="id9">Extra Dependencies</a></p></li>
<li><p><a class="reference internal" href="#configure-the-engines" id="id10">Configure the engines</a></p>
<ul>
<li><p><a class="reference internal" href="#engine-sqlite" id="id11">SQLite</a></p></li>
<li><p><a class="reference internal" href="#engine-postgresql" id="id12">PostgreSQL</a></p></li>
<li><p><a class="reference internal" href="#engine-mysql-server" id="id13">MySQL</a></p></li>
</ul>
</li>
</ul>
</nav>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<p>Initial sponsored by <a class="reference external" href="https://nlnet.nl/discovery">Search and Discovery Fund</a> of <a class="reference external" href="https://nlnet.nl/">NLnet Foundation</a>.</p>
</aside>
<p>With the <em>SQL engines</em> you can bind SQL databases into SearXNG. The following
Relational Database Management System (RDBMS) are supported:</p>
<ul class="simple">
<li><p><a class="reference internal" href="#engine-sqlite"><span class="std std-ref">SQLite</span></a></p></li>
<li><p><a class="reference internal" href="#engine-postgresql"><span class="std std-ref">PostgreSQL</span></a></p></li>
<li><p><a class="reference internal" href="#engine-mysql-server"><span class="std std-ref">MySQL</span></a></p></li>
</ul>
<p>All of the engines above are just commented out in the <a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/settings.yml">settings.yml</a>, as you have to set the required attributes for the
engines, e.g. <code class="docutils literal notranslate"><span class="pre">database:</span></code></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nv">sqlite|postgresql|mysql_server</span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
<span class="w"> </span><span class="nt">result_template</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{</span><span class="nv">template_name</span><span class="p p-Indicator">}</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">...</span>
</pre></div>
</div>
<p>By default, the engines use the <code class="docutils literal notranslate"><span class="pre">key-value</span></code> template for displaying results /
see <a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/templates/simple/result_templates/key-value.html">simple</a>
theme. If you are not satisfied with the original result layout, you can use
your own template, set <code class="docutils literal notranslate"><span class="pre">result_template</span></code> attribute to <code class="docutils literal notranslate"><span class="pre">{template_name}</span></code> and
place the templates at:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">searx</span><span class="o">/</span><span class="n">templates</span><span class="o">/</span><span class="p">{</span><span class="n">theme_name</span><span class="p">}</span><span class="o">/</span><span class="n">result_templates</span><span class="o">/</span><span class="p">{</span><span class="n">template_name</span><span class="p">}</span>
</pre></div>
</div>
<p>If you do not wish to expose these engines on a public instance, you can still
add them and limit the access by setting <code class="docutils literal notranslate"><span class="pre">tokens</span></code> as described in section
<a class="reference internal" href="../../../admin/settings/settings_engine.html#private-engines"><span class="std std-ref">Private Engines (tokens)</span></a>.</p>
<section id="extra-dependencies">
<h2><a class="toc-backref" href="#id9" role="doc-backlink">Extra Dependencies</a><a class="headerlink" href="#extra-dependencies" title="Permalink to this heading"></a></h2>
<p>For using <a class="reference internal" href="#engine-postgresql"><span class="std std-ref">PostgreSQL</span></a> or <a class="reference internal" href="#engine-mysql-server"><span class="std std-ref">MySQL</span></a> you need to
install additional packages in Pythons Virtual Environment of your SearXNG
instance. To switch into the environment (<a class="reference internal" href="../../../admin/installation-searxng.html#searxng-src"><span class="std std-ref">Install SearXNG &amp; dependencies</span></a>) you can use
<a class="reference internal" href="../../../utils/searxng.sh.html#searxng-sh"><span class="std std-ref">utils/searxng.sh</span></a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
</pre></div>
</div>
</section>
<section id="configure-the-engines">
<h2><a class="toc-backref" href="#id10" role="doc-backlink">Configure the engines</a><a class="headerlink" href="#configure-the-engines" title="Permalink to this heading"></a></h2>
<p>The configuration of the new database engines are similar. You must put a valid
SQL-SELECT query in <code class="docutils literal notranslate"><span class="pre">query_str</span></code>. At the moment you can only bind at most one
parameter in your query. By setting the attribute <code class="docutils literal notranslate"><span class="pre">limit</span></code> you can define how
many results you want from the SQL server. Basically, it is the same as the
<code class="docutils literal notranslate"><span class="pre">LIMIT</span></code> keyword in SQL.</p>
<p>Please, do not include <code class="docutils literal notranslate"><span class="pre">LIMIT</span></code> or <code class="docutils literal notranslate"><span class="pre">OFFSET</span></code> in your SQL query as the engines
rely on these keywords during paging. If you want to configure the number of
returned results use the option <code class="docutils literal notranslate"><span class="pre">limit</span></code>.</p>
<section id="engine-sqlite">
<span id="id2"></span><h3><a class="toc-backref" href="#id11" role="doc-backlink">SQLite</a><a class="headerlink" href="#engine-sqlite" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/sqlite.py">sqlite.py</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.sqlite"></span><p>SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency.</p>
<section id="example">
<h4>Example<a class="headerlink" href="#example" title="Permalink to this heading"></a></h4>
<p>To demonstrate the power of database engines, here is a more complex example
which reads from a <a class="reference external" href="https://mediathekview.de/">MediathekView</a> (DE) movie database. For this example of the
SQlite engine download the database:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://liste.mediathekview.de/filmliste-v2.db.bz2">https://liste.mediathekview.de/filmliste-v2.db.bz2</a></p></li>
</ul>
<p>and unpack into <code class="docutils literal notranslate"><span class="pre">searx/data/filmliste-v2.db</span></code>. To search the database use e.g
Query to test: <code class="docutils literal notranslate"><span class="pre">!mediathekview</span> <span class="pre">concert</span></code></p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mediathekview</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sqlite</span>
<span class="w"> </span><span class="nt">disabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">False</span>
<span class="w"> </span><span class="nt">categories</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">general</span>
<span class="w"> </span><span class="nt">result_template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">default.html</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searx/data/filmliste-v2.db</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">&gt;-</span>
<span class="w"> </span><span class="no">SELECT title || &#39; (&#39; || time(duration, &#39;unixepoch&#39;) || &#39;)&#39; AS title,</span>
<span class="w"> </span><span class="no">COALESCE( NULLIF(url_video_hd,&#39;&#39;), NULLIF(url_video_sd,&#39;&#39;), url_video) AS url,</span>
<span class="w"> </span><span class="no">description AS content</span>
<span class="w"> </span><span class="no">FROM film</span>
<span class="w"> </span><span class="no">WHERE title LIKE :wildcard OR description LIKE :wildcard</span>
<span class="w"> </span><span class="no">ORDER BY duration DESC</span>
</pre></div>
</div>
</section>
<section id="implementations">
<h4>Implementations<a class="headerlink" href="#implementations" title="Permalink to this heading"></a></h4>
</section>
<dl class="py function">
<dt class="sig sig-object py" id="searx.engines.sqlite.sqlite_cursor">
<span class="sig-prename descclassname"><span class="pre">searx.engines.sqlite.</span></span><span class="sig-name descname"><span class="pre">sqlite_cursor</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/searx/engines/sqlite.html#sqlite_cursor"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#searx.engines.sqlite.sqlite_cursor" title="Permalink to this definition"></a></dt>
<dd><p>Implements a <a class="reference external" href="https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager" title="(in Python v3.11)"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Context</span> <span class="pre">Manager</span></code></a> for a
<a class="reference external" href="https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor" title="(in Python v3.11)"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sqlite3.Cursor</span></code></a>.</p>
<p>Open database in read only mode: if the database doesnt exist. The default
mode creates an empty file on the file system. See:</p>
<ul class="simple">
<li><p><a class="reference external" href="https://docs.python.org/3/library/sqlite3.html#sqlite3.connect">https://docs.python.org/3/library/sqlite3.html#sqlite3.connect</a></p></li>
<li><p><a class="reference external" href="https://www.sqlite.org/uri.html">https://www.sqlite.org/uri.html</a></p></li>
</ul>
</dd></dl>
</section>
<section id="engine-postgresql">
<span id="id3"></span><h3><a class="toc-backref" href="#id12" role="doc-backlink">PostgreSQL</a><a class="headerlink" href="#engine-postgresql" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/postgresql.py">postgresql.py</a></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span></code> <a class="reference external" href="psycopg2">psycopg2-binary</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.postgresql"></span><p>PostgreSQL is a powerful and robust open source database. Before configuring
the PostgreSQL engine, you must install the dependency <code class="docutils literal notranslate"><span class="pre">psychopg2</span></code>.</p>
<section id="id4">
<h4>Example<a class="headerlink" href="#id4" title="Permalink to this heading"></a></h4>
<p>Below is an example configuration:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searxng</span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">password</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;SELECT</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">from</span><span class="nv"> </span><span class="s">my_table</span><span class="nv"> </span><span class="s">WHERE</span><span class="nv"> </span><span class="s">my_column</span><span class="nv"> </span><span class="s">=</span><span class="nv"> </span><span class="s">%(query)s&#39;</span>
</pre></div>
</div>
</section>
<section id="id5">
<h4>Implementations<a class="headerlink" href="#id5" title="Permalink to this heading"></a></h4>
</section>
</section>
<section id="engine-mysql-server">
<span id="id6"></span><h3><a class="toc-backref" href="#id13" role="doc-backlink">MySQL</a><a class="headerlink" href="#engine-mysql-server" title="Permalink to this heading"></a></h3>
<aside class="sidebar">
<p class="sidebar-title">info</p>
<ul class="simple">
<li><p><a class="reference external" href="https://github.com/searxng/searxng/blob/master/searx/engines/mysql_server.py">mysql_server.py</a></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span></code> <a class="reference external" href="https://pypi.org/project/mysql-connector-python">mysql-connector-python</a></p></li>
</ul>
</aside>
<span class="target" id="module-searx.engines.mysql_server"></span><p>MySQL is said to be the most popular open source database. Before enabling
MySQL engine, you must install the package <code class="docutils literal notranslate"><span class="pre">mysql-connector-python</span></code>.</p>
<p>The authentication plugin is configurable by setting <code class="docutils literal notranslate"><span class="pre">auth_plugin</span></code> in the
attributes. By default it is set to <code class="docutils literal notranslate"><span class="pre">caching_sha2_password</span></code>.</p>
<section id="id7">
<h4>Example<a class="headerlink" href="#id7" title="Permalink to this heading"></a></h4>
<p>This is an example configuration for querying a MySQL server:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">engine</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mysql_server</span>
<span class="w"> </span><span class="nt">database</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">my_database</span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">searxng</span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">password</span>
<span class="w"> </span><span class="nt">limit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5</span>
<span class="w"> </span><span class="nt">query_str</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;SELECT</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">from</span><span class="nv"> </span><span class="s">my_table</span><span class="nv"> </span><span class="s">WHERE</span><span class="nv"> </span><span class="s">my_column=%(query)s&#39;</span>
</pre></div>
</div>
</section>
<section id="id8">
<h4>Implementations<a class="headerlink" href="#id8" title="Permalink to this heading"></a></h4>
</section>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<span id="sidebar-top"></span>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="../../../index.html">
<img class="logo" src="../../../_static/searxng-wordmark.svg" alt="Logo"/>
</a></p>
<h3><a href="../../../index.html">Table of Contents</a></h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../user/index.html">User information</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../own-instance.html">Why use a private instance?</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../admin/index.html">Administrator documentation</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../index.html">Developer documentation</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../quickstart.html">Development Quickstart</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../contribution_guide.html">How to contribute</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../index.html">Engine Implementations</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../search_api.html">Search API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../plugins.html">Plugins</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../translation.html">Translation</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../lxcdev.html">Developing in Linux Containers</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../makefile.html">Makefile &amp; <code class="docutils literal notranslate"><span class="pre">./manage</span></code></a></li>
<li class="toctree-l2"><a class="reference internal" href="../../reST.html">reST primer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../searxng_extra/index.html">Tooling box <code class="docutils literal notranslate"><span class="pre">searxng_extra</span></code></a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../utils/index.html">DevOps tooling box</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../src/index.html">Source-Code</a></li>
</ul>
<h3>Project Links</h3>
<ul>
<li><a href="https://github.com/searxng/searxng/tree/master">Source</a>
<li><a href="https://github.com/searxng/searxng/wiki">Wiki</a>
<li><a href="https://searx.space">Public instances</a>
<li><a href="https://github.com/searxng/searxng/issues">Issue Tracker</a>
</ul><h3>Navigation</h3>
<ul>
<li><a href="../../../index.html">Overview</a>
<ul>
<li><a href="../../index.html">Developer documentation</a>
<ul>
<li><a href="../index.html">Engine Implementations</a>
<ul>
<li>Previous: <a href="search-indexer-engines.html" title="previous chapter">Local Search APIs</a>
<li>Next: <a href="../online_url_search/tineye.html" title="next chapter">Tineye</a></ul>
</li></ul>
</li>
</ul>
</li>
</ul>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../../../search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../../../_sources/dev/engines/offline/sql-engines.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright SearXNG team.
</div>
<script src="../../../_static/version_warning_offset.js"></script>
</body>
</html>