Commit Graph

17 Commits

Author SHA1 Message Date
Ryan Kaskel e19218c6a9
Copy client connection kwargs before mutating 2022-11-29 12:15:18 +00:00
Alexandre FLAMENT 73b5a58d9e [FIX] Redis initialization
redis.Redis.from_url(url) doesn't check if the url is valid

Before this commit: actual error are detected later when the client is actually used.
With this commit, client() makes sure to return a valid Redis client or None.

Also, the code makes sure not to log the password of the Redis URL
2022-11-07 22:08:32 +00:00
Alexandre FLAMENT e92755d358 Initialize Redis in searx/webapp.py
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"

The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.

It worked before because SearXNG initialized the Redis connection when the limiter started.

In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
2022-11-05 17:45:52 +01:00
Alexandre Flament fe419e355b The checker requires Redis
Remove the abstraction in searx.shared.SharedDict.
Implement a basic and dedicated scheduler for the checker using a Redis script.
2022-11-05 12:04:50 +01:00
Markus Heiser 8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
Alexandre Flament 2babf59adc [fix] pyright repported errors
The errors make pyright usage useless since a new error won't be seen [1].

[1] https://github.com/searxng/searxng/pull/1569

```
  searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]"
    "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]"
    Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues)
  searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str"
    Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
  searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int"
    Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues)
  searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join"
    Type "str" cannot be assigned to type "BytesPath"
      "str" is incompatible with "bytes"
      "str" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['themes']" cannot be assigned to type "BytesPath"
      "Literal['themes']" is incompatible with "bytes"
      "Literal['themes']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "str | Any | None" cannot be assigned to type "BytesPath"
      Type "str" cannot be assigned to type "BytesPath"
        "str" is incompatible with "bytes"
        "str" is incompatible with protocol "PathLike[bytes]"
          "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['img']" cannot be assigned to type "BytesPath"
      "Literal['img']" is incompatible with "bytes"
      "Literal['img']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports)
  searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports)
  searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource)
  searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable)
  searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable)
  searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown
    Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType)
  searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
  searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
```
2022-07-30 18:04:44 +02:00
Markus Heiser 782f73540e [utils/searxng.sh] implement new script to install SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Martin Fischer 640c404844 [pyright:strict] searx.search.checker.background 2022-01-27 22:07:12 +01:00
Martin Fischer 7f89adc5f3 [pyright:strict] searx.shared.shared_abstract 2022-01-27 22:07:12 +01:00
Markus Heiser e85744c254 Revert "[hotfix] interim fix to get docker-build of CI without issues"
This reverts interim fix from commit 50c4b58db and adds requirement
`redis==4.1.1`.

The interim fix was needed by Alpine images (Docker) [1] and has been fixed in
commit [2] merged with the patch series from [3].  In redis-py version 4.1.1
this pach has been released on PyPi [4].

[1] https://github.com/redis/redis-py/issues/1869
[2] https://github.com/redis/redis-py/commit/1fc1233f
[3] https://github.com/redis/redis-py/pull/1854
[4] https://github.com/redis/redis-py/issues/1880

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-17 18:03:29 +01:00
Markus Heiser 50c4b58db6 [hotfix] interim fix to get docker-build of CI without issues
There is an issue with redis v4.1.0 [1] / for the interim lets remove this
python dependency.

[1] https://github.com/searxng/searxng/issues/741

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-12 09:31:12 +01:00
Markus Heiser dca83944b5 [fix] redis: don't create a new connection at each client() call
Suggested-by: @dalf https://github.com/searxng/searxng/pull/686#pullrequestreview-844942973
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-07 17:33:40 +01:00
Markus Heiser a6cfab93fa [enh] add redis connector searx/shared/redisdb.py
Add a redis connector, the default DB connector is a socket at::

    unix:///usr/local/searxng-redis/run/redis.sock?db=0

To set up a redis instance simply use::

    $ ./manage redis.build
    $ sudo -H ./manage redis.install

A hint for developers:

To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::

    $ sudo -H ./manage redis.addgrp "${USER}"
    # don't forget to logout & login to get member of group

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 20:56:39 +01:00
Markus Heiser 3d96a9839a [format.python] initial formatting of the python code
This patch was generated by black [1]::

    make format.python

[1] https://github.com/psf/black

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 09:26:22 +01:00
Alexandre Flament 912c7e975c [fix] checker: don't run the checker when uwsgi is not properly configured
Before this commit, even with the scheduler disabled, the checker was running
at least once for each uwsgi worker.
2021-01-13 14:07:39 +01:00
Alexandre Flament a0c8b413a6 [mod] searx.shared: minor tweaks
searx.shared.shared_abstract.SharedDict inherit from abc.ABC
searx.shared.shared_uwsgi.schedule can schedule multiple functions without issue
2021-01-12 11:47:17 +01:00
Alexandre Flament 6e2872f436 [enh] add searx.shared
shared dictionary between the workers (UWSGI or werkzeug)
scheduler: run a task once every x seconds (UWSGI or werkzeug)
2021-01-12 11:47:17 +01:00