e92755d358
Initialize Redis in searx/webapp.py
...
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"
The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.
It worked before because SearXNG initialized the Redis connection when the limiter started.
In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
2022-11-05 17:45:52 +01:00
32e8c2cf09
searx.network: add "verify" option to the networks
...
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.
SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-10-14 13:59:22 +00:00
8d5653e60d
Merge branch 'searxng:master' into master
2022-09-30 23:06:54 +00:00
ba8959ad7c
[fix] typos / reported by @kianmeng in searx PR-3366
...
[PR-3366] https://github.com/searx/searx/pull/3366
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
5832c70680
correct sci-hub links/ add `.ru` and remove other 3rd party domains.
2022-09-24 11:03:57 -04:00
caebafdd06
[fix] typo in crossref settings: disable --> disabled
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 08:12:36 +02:00
d6446be38f
[mod] science category: various update of about PR 1705
2022-09-23 20:52:55 +02:00
e36f85b836
Science category: update the engines
...
* use the paper.html template
* fetch more data from the engines
* add crossref.py
2022-09-23 20:45:58 +02:00
bef3984d03
Merge pull request #1728 from liimee/eng-ddw
...
add duckduckgo weather engine
2022-09-23 18:14:09 +02:00
d3fec1388c
Merge pull request #1624 from liimee/eng-wttr
...
Add wttr.in engine
2022-09-23 18:13:37 +02:00
33b43763b9
Brave engine: fix BrotliDecoderDecompressStream error
2022-09-18 22:08:38 +00:00
3f72a79088
add yandex to autocomplete backends settings
2022-09-09 23:50:58 +03:00
ad8ffd222c
[mod] option 'ui: cache_url:' to configure internet cache or archive
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:42:20 +02:00
dd0887be18
xpath engine: change raise_for_httperror to no_result_for_http_status
...
no_result_for_http_status contains a list of HTTP status.
These HTTP status are seen an empty result list.
In other cases an exception is thrown as usual.
Previously raise_for_httperror were ignoring all HTTP error,
which make defective engines invisible in the stats.
2022-09-04 09:07:28 +02:00
a15dfa5ee1
[fix] engine woxikon.de - don't raise exception on empty result list
...
Woxikon expects a word in German, so with query "foo" the site finds nothing and
respons a 404:
httpx.HTTPStatusError: Client error '404 Not Found' \
for url 'https://synonyme.woxikon.de/synonyme/foo.php '
[1] https://github.com/searxng/searxng/issues/1543#issuecomment-1193317054
Closes: https://github.com/searxng/searxng/issues/1543
Suggested-by: @allendema [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:07:28 +02:00
8e9fb0b435
Merge pull request #1647 from return42/deepl-engine
...
[mod] add deepl translation engine
2022-09-02 14:09:22 +02:00
fcccf39030
Disable brave by default
...
Brave is too unstable and will often not work by default. As seen in many issues: https://github.com/searxng/searxng/issues?q=is%3Aissue++sort%3Aupdated-desc+brave+label%3Abug+
2022-08-31 15:47:56 +02:00
12f7d4a46b
add duckduckgo weather engine
2022-08-31 17:29:32 +07:00
341ad46303
settings.yml: set default values for result_proxy
...
* initialize result_proxy with searx/settings_defaults.py
* allow result_proxy.key to be a string
this commit supersedes #1522
2022-08-28 09:27:53 +00:00
56000d5162
Merge pull request #1699 from liimee/eng-app-store
...
add apple app store engine
2022-08-27 07:43:23 +02:00
44bc94c36e
Merge pull request #1700 from liimee/eng-ddm
...
add apple maps engine
2022-08-27 07:36:16 +02:00
5dce299b22
add apple maps engine
2022-08-25 17:05:40 +07:00
e5c1b64b1d
add the apple app store engine
...
The Apple App Store is the digital app distribution platform for iOS & iPadOS.
2022-08-24 17:27:36 +07:00
dd9127492f
add 9gag engine
...
9GAG is a social media website where users upload and share user-generated images and videos
2022-08-22 17:35:07 +07:00
5ed40af3ba
Merge pull request #1661 from liimee/eng-tw
...
Add twitter engine
2022-08-21 15:21:18 +02:00
75bb8c45d0
[mod] decouple qwant's categories from SearXNG's categories
...
By using new property `qwant_categ:` the category of qwant is no longer bound to
the category of SearXNG.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-14 10:26:54 +02:00
96ea355a1f
add twitter engine
2022-08-14 08:39:41 +07:00
8fa84ee9f4
Fix time range support and add soft_max_redirects
...
Add custom time_range_url and time_range_map
Set soft_max_redirects = 2 to prevent "ErrorContext('searx/search/processors/online.py', 116, 'count_error(', None, '2 redirects, maximum: 0', ('200', 'OK', 'neeva.com')) True"
2022-08-13 07:35:15 +07:00
d6dfada1a9
add neeva web engine
...
Neeva is "the world's first ad-free, private search engine" and uses data from Apple, Bing, Yelp and "others".
They claim to crawl "hundreds of millions" of URLs a day (https://twitter.com/Neeva/status/1536447373903335426 ).
2022-08-13 07:26:36 +07:00
e0215e0cc8
add sourcehut engine
2022-08-10 21:48:12 +02:00
90bdb23756
add lib.rs engine
2022-08-10 12:17:55 +02:00
d4acbcfe63
[mod] add deepl translation engine
...
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API. This api only needs a to-lang, from-lang is a fake
by now.
There is a free option to use [1].
[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
2022-08-10 09:14:36 +02:00
925f3ff487
Merge pull request #1627 from allendema/add-marginalia
...
[enh] Initial Marginalia.nu support (foss)
2022-08-08 06:53:02 +02:00
58f630d312
Merge pull request #1623 from return42/mod-yep.com
...
[mod] engine yep.com: show all 100 results yep.com has
2022-08-08 06:52:15 +02:00
5b815b3966
[enh] initial Marginalia.nu support (foss)
...
Currently it uses a public api_key `/public/` [1]
The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.
0: popular
1: blogs
2: big_sites
3: default
4: experimental
'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.
[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626
Closes: https://github.com/searxng/searxng/issues/1620
2022-08-08 06:31:04 +02:00
8b58cd1167
[fix] typo in uWSGI cache: searxcache --> searxngcache
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
8aa018db95
add wttr.in engine
2022-08-07 13:04:18 +07:00
647e6187d7
[mod] engine yep.com: show all 100 results yep.com has
...
yep.com is still in beta, the api.yep.com does not have paging support. There
is only a 'limit' argument with a maximum of 100 results.
yep.com seems fast; there is nor need for a timeout of 12 sec.
The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.
FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.
Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-06 13:21:28 +02:00
58369e41d6
[wip] brave time range support
2022-08-04 04:47:25 +02:00
8df1f0c47e
[mod] add 'Accept-Language' HTTP header to online processores
...
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).
- add new engine option: send_accept_language_header
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
4231a5770b
[fix] sjp engine - convert enginename to a latin1 compliance name
...
The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more. Unicode characters in the name of an engine could
cause various issues.
Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-24 21:10:55 +02:00
6d646129c3
[mod] add tor_check plugin - convenient tor checking trough searxng
2022-07-19 07:34:54 +02:00
4e05197444
Merge pull request #1475 from return42/Emojipedia
...
[mod] Add engine for Emojipedia
2022-07-15 09:30:40 +02:00
10edcbe3c2
[mod] Add engine for Emojipedia
...
Emojipedia is an emoji reference website which documents the meaning and
common usage of emoji characters in the Unicode Standard. It is owned by Zedge
since 2021. Emojipedia is a voting member of The Unicode Consortium.[1]
Cherry picked from @james-still [2[3] and slightly modified to fit SearXNG's
quality gates.
[1] https://en.wikipedia.org/wiki/Emojipedia
[2] 2fc01eb20fhttps://github.com/searx/searx/pull/3278
2022-07-15 09:26:44 +02:00
84e2a3bd3f
Add infinite scroll as a setting in settings.yml
2022-07-09 17:33:25 +00:00
df837d8b1b
Merge pull request #1428 from return42/fix-center_aligment
...
fix typo and document preference 'center_alignment' in the 'ui' section
2022-07-07 09:43:12 +02:00
14756a2674
[mod] Adds Lingva translate engine
...
Add the lingva engine (which grabs data from google translate). Results from
Lingva are added to the infobox results.
2022-07-04 19:06:45 +02:00
241bde5ae3
[fix] typo: add missing 'n' in center_aligment --> center_alignment
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-03 17:35:54 +02:00
f8f239fe1f
Donation link: default value to searxng.org, can be hidden or custom
...
Add a new setting: general.donation_url
By default the value is https://docs.searxng.org/donate.html
When the value is false, the link is hidden
When the value is true, the link goes to the infopage donation,
the administrator can create a custom page.
2022-07-02 11:29:21 +02:00
ed1ff4b8bc
Merge pull request #1357 from liimee/eng-3
...
Add curlie.org engine
2022-06-22 18:29:36 +02:00
7c457fcf7a
add curlie engine
2022-06-22 11:46:33 +07:00
8883aed132
[fix] google play apps engine: implement engines/google_play_apps.py
2022-06-18 16:02:39 +02:00
8a85d1b0c7
Theme: add a preference to center the results (Oscar) (2nd edition)
...
related to
* #1169
* #1281
* #1286
2022-06-18 08:54:24 +02:00
a7b0b2ecbf
Merge pull request #1329 from samsaptidev/feature/privacypolicy_url
...
Add privacypolicy_url option
2022-06-18 07:26:40 +02:00
6bdc66b70e
[fix] mojeek engine: XPath
...
Update the XPath of the Mojeek engine.
2022-06-17 09:32:53 +02:00
31005595c9
Add privacypolicy_url option
2022-06-16 11:56:24 +02:00
e9cc6ab0e7
add pub.dev engine
2022-06-15 12:55:37 +02:00
fd9a13a3e5
[enh] Initial no paging support for Yep.com
...
Upstream example query:
https://yep.com/web?q=test
https://yep.com/about
2022-06-11 14:17:44 +02:00
98c4156f12
Fixed typo in settings.yml
2022-06-09 17:08:14 +02:00
9723cef99a
Revert "Theme: add a preference to center the results (Oscar)"
...
This reverts commit 4237f5fd50
2022-06-05 23:03:26 +02:00
4237f5fd50
Theme: add a preference to center the results (Oscar)
...
related to #1169
2022-06-05 19:59:49 +02:00
43dc9eb7d6
[enh] Initial Petalsearch Images support
...
Upstream example query:
https://petalsearch.com/search?query=test&channel=image&ps=50&pn=1®ion=de-de&ss_mode=off&ss_type=normal
Depending on locale it will internally use some/all results from other
engines. See:
https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#general-indexing-search-engines
2022-06-02 14:32:37 +02:00
167f635924
[engine] petal search news
...
Add Petal Search News to SearXNG via XPath
2022-05-29 22:49:41 +02:00
f09e8abd76
[engine] petal search
...
Add Petal Search to SearXNG via XPath
2022-05-29 22:23:30 +02:00
7c8903c809
[fix] mojeek engine: xpath
2022-05-28 18:09:25 +02:00
13f4dcafc1
[fix] [new] Brave - Update xpath
...
Last merge request was messed up with git.
2022-05-26 18:44:11 +02:00
c24864a9ad
Enable duckduckgo by default
...
see https://github.com/searxng/searxng/pull/1219#discussion_r879543174
2022-05-24 23:33:12 +02:00
cea7b71d14
Disable the bing engine
...
see
* https://github.com/searxng/searxng/discussions/1209
* https://github.com/searxng/searxng/issues/941
* https://github.com/searxng/searxng/pull/1219
2022-05-24 23:33:08 +02:00
6ef24ffd94
[enh] Add alexandria.org engine
...
engine: json_engine
official_api_documentation: https://github.com/alexandria-org/alexandria-api/raw/master/README.md
2022-05-15 11:10:15 +02:00
444b1e70db
[mod] add setting: search.autocomplete_min
...
Minimun characters to type before autocompleter starts.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 17:58:30 +02:00
e8541b6006
[theme] peel out oscar from SearXNG development
...
This is the first step of removing oscar theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
02026bcc76
[fix] library genesis engine: xpath and url
...
Changed the xpath to use the official download mirror.
Changed the URL to libgen.fun as this is the official instance. (https://en.wikipedia.org/wiki/Library_Genesis )
2022-04-27 13:11:34 +02:00
624b58764a
[enh] Add pagination to Brave
...
Also added ```&spellcheck=1``` because now it is disabled by default, not returning any ```suggestion_xpath```.
2022-04-23 14:06:53 +02:00
f039873e1a
[mod] Make a default domain for ebay
...
Which can be changed.
2022-04-17 10:37:12 +02:00
de4af2fefd
[enh] add seznam autocomplete
2022-04-14 03:02:05 +02:00
19fa0095a0
(fix) satisfy the linter, and btw reduce timeout
2022-04-01 09:23:24 -04:00
934ae4e086
(feat) add jisho.org
...
Closes #1016
2022-03-31 14:45:39 -04:00
dec04c0ed6
Merge pull request #999 from return42/fix-doc-settings2
...
[docs] document settings.yml: search.languages
2022-03-26 17:09:07 +01:00
a521d4985a
[docs] document settings.yml: search.languages
...
Requested-by: @dalf https://github.com/searxng/searxng/pull/996#discussion_r830858139
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-26 13:32:42 +01:00
7653e8bf16
Merge pull request #995 from return42/fix-932
...
[interim fix] static files can't be delivered by HTTP.
2022-03-25 06:56:11 +01:00
32a35b4654
[docs] improve settings.yml documentation (admin/engines/settings)
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-21 08:48:06 +01:00
fd5fcdedce
[interim fix] static files can't be delivered by HTTP.
...
Since PR 932 [1][2] static files can't be delivered by HTTP server any longer.
This patch makes the hash paramter in the URL of static files:
/static/themes/simple/css/searxng.min.css?5fde34a74bc438c7b56ec8c6501e131cc9914bd8
optional. By default the hash parameter is disabled.
HINT:
Instances that do not deliver static files by their HTTP server and have a
long expire time [3] should enable this option.
----
This is only a interim solution, on the long run:
make static.build.commit
creates files including the file name:
css/searxng-5fde34a74bc438c7b56ec8c6501e131cc9914bd8.min.css
and a mapping.json with this content[4]
[1] https://github.com/searxng/searxng/issues/964
[2] https://github.com/searxng/searxng/pull/932#issuecomment-1067039518
[3] 5583336440https://github.com/searxng/searxng/pull/932#issuecomment-1067216426
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-20 16:28:14 +01:00
e4a2bfba38
[fix] settings.yml: ui.results_on_new_tab
...
Closes: https://github.com/searxng/searxng/issues/987
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-19 23:06:21 +01:00
20f4538e13
[fix] engine: Semantic Scholar (Science) // rework & fix
...
Closes: https://github.com/searxng/searxng/issues/939
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-05 11:53:41 +01:00
8230603f48
Merge pull request #916 from dalf/pref_infinite_scroll2
...
Convert the infinite_scroll plugin as a preference (second version)
2022-02-21 12:49:04 +01:00
56e34947a6
[mod] infinite_scroll as preference
...
* oscar theme: code from searx/plugins/infinite_scroll.py
* simple theme: new implementation
Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
2022-02-20 22:58:51 +01:00
5ec9eb8030
[mod] switch default theme from oscar to simple
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-20 18:36:41 +01:00
ae8b174e93
[fix] replace peer.tube by peertube.biz
...
More peertube instances are listed at [1]
[1] https://instances.joinpeertube.org/instances
Closes: https://github.com/searxng/searxng/issues/881
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-13 22:48:09 +01:00
f2e16904a5
[mod] disable deezer engine by default
...
To play content from deezer a account is needed, the majority of the SearXNG
won't have.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-07 22:58:05 +01:00
e2ec6b4211
[fix] invidious engine: store random base_url in param
...
Two different threads ( = two different user queries) can call the request
function in a row and then the response function. The namespace will be same
since this is the same engine.
To keep exactly the same value ``base_url`` must be stored in params and then
retrieve using ``resp.search_params["base_url"]``.
Suggested-by: @dalf https://github.com/searxng/searxng/pull/862#discussion_r799324861
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-04 15:42:06 +01:00
d92b3d96fd
[fix] solidtorrents engine: JSON API no longer exists
...
The API endpoint, we where using does not exist anymore. This patch is a
rewrite that parses the HTML page.
Related: https://github.com/paulgoio/searxng/issues/17
Closes: https://github.com/searxng/searxng/issues/858
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-04 14:53:37 +01:00
f79b0fce06
[enh] limiter plugin
...
can replace filtron:
* rate limite the number of request per IP and per (IP, User-Agent)
* block some bots
use Redis
data stored in Redis never contains the IP addresses, only HMAC using the secret_key
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-02 09:15:59 +01:00
ebd3013a1a
[mod] tineye engine: minor changes
...
* remove "disable: false" in settings.yml
* use the json() method from httpx.Response (faster character encoding detection)
2022-01-30 20:49:22 +01:00
a6b879f19c
[mod] tineye engine: set engine_type to 'online_url_search'
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-30 16:30:52 +01:00
880555e263
[enh] engine - add Tineye reverse image search
...
Other optional parameter ..
`&sort=crawl_date`
can be appended to search_string to sort results by date.
`&domain=example.org`
can be implemented to search_string to get results from just one domain.
Public instances could get relatively fast timed-out for 3600s.
--
Merged from @allendema's commit [1] and slightly modfied / see [2].
Related-to: [1] 455b2b4460https://github.com/searx/searx/pull/3040
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-28 09:06:44 +01:00
ccffab3a43
Merge pull request #813 from MarcAbonce/change-wolfram-category
...
Move WolframAlpha away from Science category
2022-01-26 14:10:19 +01:00
10c9b9775d
[mod] improve setup of invidious engine
...
- My experience is, that a timeout of 5 sec is not need, I got fast response
less than a second.
- https://invidious.tube/ redirects to http://ww25.invidious.tube/
- in SearXNG defaults the http protocol is unsafe and raise an error
- https://ww25.invidious.tube has SSL_ERROR_UNSAFE_NEGOTIATION
Related-to: https://github.com/searxng/searxng/issues/821
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-25 19:02:11 +01:00
1311745f24
move WolframAlpha away from Science category
2022-01-23 23:24:02 -06:00
977e9a4330
Merge pull request #686 from return42/lib_redis
...
Add redis DB and connector
2022-01-11 19:55:14 +01:00
37baf46ece
[fix] Rename ccengine engine to openverse
...
The CC engine was merged with WordPress and renamed to Openverse
Source: https://wordpress.org/news/2021/05/welcome-to-openverse/
2022-01-07 13:06:05 +01:00
Alexandre FLAMENT