b5a8ddfec8
Revert "[fix] limit maximum page number of a search query to page 50." ( #2979 )
...
This reverts commit 7e2e335dd1
2023-11-01 16:55:20 +00:00
d13a8f6453
[mod] document server:public_instance & remove it out of the botdetection
...
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
in botdetection module
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
fd814aac86
[mod] isolation of botdetection from the limiter
...
This patch was inspired by the discussion around PR-2882 [2]. The goals of this
patch are:
1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
enabled (Redis is available).
This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.
Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin. The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.
[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891
To test:
- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
- http://0.0.0.0:8000/admin/searx.limiter.html
- http://0.0.0.0:8000/src/searx.botdetection.html#botdetection
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
b05a15540e
Check `public_instance` in simple theme
...
Fix #2975
2023-10-31 10:40:19 +01:00
7e2e335dd1
[fix] limit maximum page number of a search query to page 50.
...
To test this PR run a local instance and try to query page 51:
http://127.0.0.1:8888/search?q=foo&pageno=51
A parameter exception will be raised:
searx.exceptions.SearxParameterException: Invalid value "51" for parameter pageno
And the client will receive a HTTP 400 (Bad request).
Closes https://github.com/searxng/searxng/issues/2972
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-29 16:17:25 +01:00
b63841f0a1
Update searx.data - update_engine_traits.py
2023-10-29 09:04:24 +01:00
a9d778ee0f
Update searx.data - update_wikidata_units.py
2023-10-29 08:45:15 +01:00
92efb941d8
Update searx.data - update_firefox_version.py
2023-10-29 08:44:59 +01:00
fa494b9be3
Update searx.data - update_currencies.py
2023-10-29 08:44:35 +01:00
1b92ea4e70
Update searx.data - update_engine_descriptions.py
2023-10-29 08:44:17 +01:00
55e7cb3e33
Update searx.data - update_ahmia_blacklist.py
2023-10-29 08:17:29 +01:00
6af8644ead
[translations] update from Weblate
...
4e5e5db44e1a8d350884bddfb89d67a4114d62fe8e3286e37ab9752cdab3247cf7ea72340ea31389322151e440e4eaf42b6
2023-10-27 09:23:23 +02:00
ef56e1d684
[fix] HTMLParser: undocumented not implemented method
...
In python versions <py3.10 there is an issue with an undocumented method
HTMLParser.error() [1][2] that was deprecated in Python 3.4 and removed
in Python 3.5.
To be compatible to higher versions (>=py3.10) an error method is implemented
which throws an AssertionError exception like the higher Python versions do [3].
[1] https://github.com/python/cpython/issues/76025
[2] https://bugs.python.org/issue31844
[3] https://github.com/python/cpython/pull/8562
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-22 10:35:02 +02:00
01b5b9cb8e
[translations] update from Weblate
...
2325f15835090c6a8e3a38219d894a9f4164bdbeb4b30f9b483f487f1ca1997c5a701dc4
2023-10-20 10:44:11 +02:00
b013cbb4cc
[build] /static
2023-10-13 12:56:29 +02:00
df2e41c150
[fix] search.js: crash on homepage when setting form listeners
2023-10-13 12:56:29 +02:00
4280318fc5
fixing results parsing brave
2023-10-13 11:47:30 +02:00
ac82daf58d
[translations] update from Weblate
...
74e401e68897dd8db16ed046a90815ecb33665d9a0c2f3ec249ef953dc6c108
2023-10-13 09:20:54 +02:00
efada7cba2
[fix] hackernews keyerror problem
2023-10-13 08:16:47 +02:00
af071121de
[fix] imgur - incorrect wikidata id
2023-10-12 09:14:00 +02:00
14323d683f
[fix] ddg-lite & ddg-extra: don't send empty vqd value
...
DDG's bot detection is sensitive to the vqd value. For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.
If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.
Requests from IPs in this block list run into timeouts.
Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).
Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.
Related:
- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12 08:52:28 +02:00
3388441917
[fix] ddg-lite vqd value: some search terms do not have a vqd value
...
Some search terms do not have results and therefore no vqd value
BTW: remove a leftover from 9197efa
2023-10-10 09:12:30 +02:00
9197efa2a7
[fix] duckduckgo lite engine: set HTTP header 'Referer'
...
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].
From reverse engineering:
- The Referer ``https://google.com/ `` was set in commt 257dc7d6c4https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 08:40:53 +02:00
fa5b2a7948
[mod] yacy: use official instance by default and fix crashes
2023-10-09 20:50:24 +02:00
5d53aac20f
[mod] add hotkeys option to settings.yml
...
The change in the hotkey mechanism introduced in 317db5b04Closes : #2898
2023-10-09 18:13:00 +02:00
ff78b1a902
[feat] implement hackernews engine - news.ycombinator.com
2023-10-09 14:00:04 +02:00
213cb74378
[fix] matrixrooms add proper MRS integration
...
Related:
- https://github.com/searxng/searxng/issues/2918
2023-10-09 13:25:13 +02:00
48cb58bd2e
[feat] duckduckgo: support for videos and news
2023-10-09 06:53:43 +02:00
c3ab49cd90
[fix] kickass: crash when no results
2023-10-07 11:48:23 +02:00
f22daf8b47
[mod] piped: always show video length if available
2023-10-07 11:45:46 +02:00
ce270961e8
[feat] engine: implementation of mastodon
2023-10-06 10:58:23 +02:00
48f8ac48d1
[translations] update from Weblate
...
68d74328142f091b7f2479c0d7ba4e6cd5929d4e5f5c3b79d44775746291184f24d7e8b16140911f9
2023-10-06 09:19:00 +02:00
fd1422a670
[mod] engine - simplify region & lang handling, make filters configurable
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-05 10:55:08 +02:00
3e2ae756f0
[feat] engine: implementation of radio-browser.info
2023-10-05 10:55:08 +02:00
d56b31d674
[fix] limiter / botdetection: remove http_connection method
...
Related:
- https://github.com/searxng/searxng/issues/2892#issuecomment-1742153932
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 20:24:22 +02:00
e509cb7c45
[typo] solved a typo in yahoo error message.
2023-10-01 08:29:06 +02:00
d4d9f2073e
[fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866
...
Co-authored-by: Blair Noctis <n@sail.ng>
2023-10-01 08:29:06 +02:00
fe9386b58d
[fix] emojipedia: fix engine
2023-10-01 08:19:45 +02:00
32a4ea350e
[fix] Revision of the Bing engines
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 08:01:38 +02:00
079636c079
[fix] engine - bing fix search, pagination, remove safesearch
2023-10-01 08:01:38 +02:00
5ce1792432
[feat] engine: implementation of pinterest
2023-09-30 15:01:45 +02:00
6096457e4d
[fix] matrixrooms.info: pagination not working properly
2023-09-30 14:51:07 +02:00
e1a8b8189f
[fix] engine - moviepilot instead of thumbnail use img_src
...
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.
Related:
- b4e0d2eedc (r128785388)
2023-09-30 11:29:19 +02:00
159629c588
[mod] tagesschau: add option to only use tagesschau urls
2023-09-30 11:00:11 +02:00
2ca60a19fc
[feat] engine: implementation of matrixrooms.info
2023-09-30 09:09:23 +02:00
d1a3fc5be6
[feat] engine: implementation of tootfinder
2023-09-29 23:00:30 +02:00
fc4a20f734
[mod] add movies category for tmdb, imdb and moviepilot
2023-09-29 22:37:51 +02:00
0330648b98
[mod] warn to info for failed ping limiter
2023-09-29 22:20:56 +02:00
26fed56d51
[mod] settings.yml: remove plugin settings for plugins that don't exist anymore
2023-09-29 11:26:49 +02:00
9a065832e0
[translations] update from Weblate
...
80bbf74ef5d554ef71771fd1b90d412012c81ac171d0aa004c40cc47e729f89f85b28cc475ce5d58140ce0600921f343f714c80f80fdef3c859e4
2023-09-29 09:12:12 +02:00
ae2762eb78
Update searx.data - update_firefox_version.py
2023-09-29 06:58:11 +02:00
18da3d65c5
Update searx.data - update_ahmia_blacklist.py
2023-09-29 06:57:40 +02:00
4a66fa3e7d
Update searx.data - update_wikidata_units.py
2023-09-29 06:56:50 +02:00
9be6e20256
Update searx.data - update_engine_traits.py
2023-09-29 06:56:10 +02:00
d821ec44dd
Update searx.data - update_currencies.py
2023-09-29 06:55:04 +02:00
cdee5aa3b9
Update searx.data - update_engine_descriptions.py
2023-09-29 06:53:05 +02:00
75587d03a6
[fix] categories.html: can't select social media category using search on category select
2023-09-28 10:45:49 +02:00
e37d775fa2
[fix] engine - currency fix and simplify
2023-09-28 08:29:38 +02:00
ae28d429c9
[fix] bilibili new api used
2023-09-28 08:24:51 +02:00
1a66d74673
[fix] engine - kickass update url, fix parsing, use multiple mirrors
2023-09-27 10:19:41 +02:00
0c39064d60
Update searx.data - update_engine_traits.py
...
Update searx.data - update_engine_traits.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
b428ccc5a0
[fix] engine brave - fetch traits (modified settings menu)
...
Brave has changed it settings menu fundamental. Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:
https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
3a456b1282
[fix] engine annas archive - fetch traits (modified xpath selectors)
...
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
1f835a9f1f
solve #2839
2023-09-25 23:01:14 +02:00
3c6549a17f
[fix] update user agent
2023-09-25 22:46:22 +02:00
ffec8d1b4c
add warning about enabling public_instance setting
2023-09-25 22:31:14 +02:00
47721a3485
add new parameter called server.public_instance
...
for enabling by default advanced limiter functions
in the future allow us to add features just for the public instances
2023-09-25 22:31:14 +02:00
1df4588279
[build] /static
2023-09-25 17:45:21 +02:00
98366bcf90
[fix] settings icon invisble on mobile devices
2023-09-25 17:45:21 +02:00
19383921d7
/config: expose limiter configuration
2023-09-24 11:01:11 +02:00
1851f27154
[mod] remove twitter ( #2843 )
2023-09-24 08:32:53 +00:00
597c68b4aa
[doc] move dosc of botdetection from developer to admin section
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-23 14:00:03 +02:00
75c9de02d1
[feat] engine: implementation of imgur
2023-09-22 20:50:53 +02:00
fb72f71f0a
[fix] internet archive scholar: crash when there's no title
2023-09-22 18:49:39 +02:00
0660d20e1d
[translations] update from Weblate
...
f27a3bbd4a97336d0a91e8946b319f53731edaf10bd75ab004b8b8cdedd99c9
2023-09-22 18:05:02 +02:00
95885e7f9f
[build] /static
2023-09-22 17:54:55 +02:00
8b0b6731fc
[fix] search.less: start page search view on mobile devices
2023-09-22 17:54:55 +02:00
71358e9c67
Revert "[fix] engine - duckduckgo vqd edge-case"
...
This reverts commit 102502a4f0
2023-09-22 09:31:25 +02:00
51236ae47a
[feat] engine: implementation of chefkoch.de
2023-09-21 17:23:59 +02:00
8bcca0e620
[fix] engine - brave don't show ads
2023-09-21 16:55:39 +02:00
b729542a66
[fix] engine - google images error when no results
2023-09-21 16:38:37 +02:00
f0df53673e
[mod] search-syntax.md: add tutorial about the feeling lucky feature
2023-09-21 15:17:23 +02:00
cc2e0537a3
[feat] engine: implementation of google icons/material design icons
2023-09-21 15:16:49 +02:00
c999cfb422
[feat] engine: implementation of wallhaven
2023-09-21 14:25:43 +02:00
102502a4f0
[fix] engine - duckduckgo vqd edge-case
2023-09-20 20:05:06 +02:00
d013f51a25
[fix] Search on category select without JavaScript PR #2740
...
Small addendum to #2740 ; search_on_category_select is now no longer a plugin.
Related: https://github.com/searxng/searxng/pull/2740#issuecomment-1725437584
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 21:03:02 +02:00
0ce10046c4
[build] /static
2023-09-19 20:15:51 +02:00
a76c8e4101
Improve the layout of the search page on mobile
2023-09-19 20:15:51 +02:00
33496e7f51
Switch back to qwant web normal API
2023-09-19 17:07:22 +02:00
043dcbf7c5
[fix] engine qwant (web-lite) - ignore advertising adds
...
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 17:06:56 +02:00
3ac7c40b6a
[build] /static
2023-09-19 11:15:25 +02:00
33e722f83b
better error message when no results found
2023-09-19 11:10:48 +02:00
ad725ce7d7
wikipedia wikidata infobox + disable wikisource ( #2806 )
...
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 10:31:02 +02:00
dcee823345
[feat] implement feeling lucky feature
2023-09-19 09:40:57 +02:00
71508abcbf
[build] /static
2023-09-19 08:06:06 +02:00
2ec77ef813
[mod] reformatted keyBindingLayouts in JS client (default, vim)
2023-09-19 08:06:06 +02:00
317db5b04f
[mod] preferences: implement drop-down menu for hotkeys (default, vim)
...
Replace the on/off checkbox of the vim-hotkeys in the preferences by a drop-down
menu.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 08:06:06 +02:00
09935e2897
[build] /static
2023-09-18 21:29:11 +02:00
a55e0ac553
[feat] search on category select without JS
...
Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-09-18 21:29:11 +02:00
efd3a2d6d1
[feat] engine: implementation of internet archive scholar
2023-09-18 18:12:00 +02:00
Markus Heiser