ZaclysIndustries
/
searxngRebrandZaclys
forked from zaclys/searxng
2
0
Fork 0
Code Issues Pull Requests Activity
5,525 Commits 3 Branches 17 Tags 53 MiB
Commit Graph

273 Commits

Author SHA1 Message Date
Alexandre Flament 333e54943d
[fix] fix monkey patch in test_webapp.py (#1667) 
at the end of test_webapp.py, the monkey patch of searx.search.Search was not revert which lead to side effects on other tests
close #1663
 2019-08-03 13:23:36 +02:00
Alexandre Flament 72029d27de
[enh] Add timeout limit per request (#1640) 
The new url parameter "timeout_limit" set timeout limit defined in second.
Example "timeout_limit=1.5" means the timeout limit is 1.5 seconds.

In addition, the query can start with <[number] to set the timeout limit.

For number between 0 and 99, the unit is the second :
Example: "<30 searx" means the timeout limit is 3 seconds

For number above 100, the unit is the millisecond:
Example: "<850 searx" means the timeout is 850 milliseconds.

In addition, there is a new optional setting: outgoing.max_request_timeout.
If not set, the user timeout can't go above searx configuration (as before: the max timeout of selected engine for a query).

If the value is set, the user can set a timeout between 0 and max_request_timeout using
<[number] or timeout_limit query parameter.

Related to #1077
Updated version of PR #1413 from @isj-privacore
 2019-08-02 13:50:51 +02:00
Alexandre Flament 2179079a91
[fix] fix flickr_noapi decoding (#1655) 
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
 2019-08-02 13:37:13 +02:00
Dalf 6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection. 
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
 2019-07-29 07:39:39 +02:00
Frank de Lange cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571) 
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
 2019-07-28 10:42:00 +02:00
Frank de Lange 204a2cbbf0 [fix] bing_videos (#1579) 
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
 2019-07-27 17:49:30 +02:00
Frank de Lange 11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568) 
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
 2019-07-27 08:22:02 +02:00
volth eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 2019-07-25 08:40:48 +02:00
rachmadani haryono 3b1122c5fa [fix] fix duden engine (#1594) 2019-07-25 08:17:45 +02:00
Alexandre Flament 554a21e1d0
[enh] Add Server-Timing header (#1637) 
Server Timing specification: https://www.w3.org/TR/server-timing/

In the browser Dev Tools, focus on the main request, there are the responses per engine in the Timing tab.
 2019-07-17 10:38:45 +02:00
rachmadani haryono ec88fb8a0f [fix] secret_key can be bytes instead of a string (#1602) 
Fix #1600
In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
 2019-07-17 10:09:09 +02:00
rachmadani haryono 8f44014627 [fix] preference query parameter decoding (#1599) 
Fix issue #1598
 2019-07-17 09:42:40 +02:00
rachmadani haryono ac357b12e3
Merge branch 'master' into feature/fix-config 2019-05-28 19:16:58 +08:00
Dalf ffe0972f91 Remove some engines : subtitleseeker, seedpeer, swisscows 
http://www.subtitleseeker.com and http://www.seedpeer.eu don't exist anymore.
https://swisscows.ch/ has change : the engine needs to be updated
 2019-05-28 04:06:35 +02:00
rachmadaniHaryono 9afc1b1e83 new: dev: test for config api 2019-05-18 18:58:05 +08:00
Marc Abonce Seguin 3e1c2153f7 [fix] duckduckgo images requests 2019-04-13 00:38:37 -05:00
Marc Abonce Seguin f2d49a6971 [fix] get youtube results from js object 
Results are not appearing in the html document anymore,
instead they are found inside an object embedded in a script.
 2019-03-26 21:09:15 -06:00
d-tux f1814079f0
Merge branch 'master' into engines/unsplash 2019-01-14 13:40:57 +01:00
Marc Abonce Seguin 626a8e9ac9 [fix] unicode error with WolframAlpha API engine 2019-01-08 21:02:23 -06:00
d-tux 329172f66e
Merge branch 'master' into engines/unsplash 2019-01-08 09:24:45 +01:00
Noémi Ványi 97351a2c72 fix after rebase 2019-01-07 21:28:58 +01:00
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages" 
This reverts commit 4d1770398a.
 2019-01-07 21:19:00 +01:00
Marc Abonce Seguin 0169b63e84 [fix] fetch google's supported languages 2019-01-06 21:31:45 -06:00
Marc Abonce Seguin 5568f24d6c [fix] check language aliases when setting search language 2019-01-06 20:31:57 -06:00
Noémi Ványi 1ea56576dc
Merge branch 'master' into devel_google_videos 2019-01-04 22:27:05 +01:00
Noémi Ványi 5b81f7c2cc
Merge branch 'master' into bugfix_startpage 2019-01-04 22:03:00 +01:00
Venca24 0e493db2fb [fix] google videos test 2019-01-04 16:04:05 +01:00
Léo Bourrel 73ec2ba74e Update bing image engine with new image json format 2018-12-19 11:12:25 +01:00
Michael Pfitzner 83a06d276d change rest of test_startpage.py 2018-12-14 22:29:57 +01:00
Michael Pfitzner 5a20d9ecd1 change the html in test_startpage.py to look more like the actual startpage site 2018-12-14 22:18:21 +01:00
Venca24 cee15f0375 [fix] google videos test 2018-11-22 13:18:18 +01:00
Denis Wernert b9ada93b3a Removes what looks like tracking parameters 2018-10-08 14:56:20 +02:00
Denis Wernert ee07a5e750 Adds a unit test for the unsplash engine 2018-10-08 14:01:55 +02:00
Adam Tauber baacfac32b [fix] remove publicly unavailable 500px engine - #1338 2018-08-19 15:06:59 +02:00
dadosch b575f898c0 duden.de engine 2018-08-18 19:24:02 +02:00
Léo Bourrel 0a37f90990 Fix wikidata tests with updated path to media 2018-07-06 11:15:43 +02:00
Adam Tauber a3b83b73cb [fix] remove obsolete test 2018-06-14 11:47:42 +02:00
Noémi Ványi 7fd7bd75ff follow up tests 2018-05-21 18:15:49 +02:00
rinpatch 059c2ccb95
Fix acgsou encoding error on python2 and add unicode characters to unit testt 2018-05-05 23:41:13 +03:00
rinpatch b7eb05dac6
Fix unit test url 2018-05-03 15:20:48 +03:00
rinpatch a79c676f27
Remove Japanese characters 2018-04-27 17:38:52 +03:00
rinpatch 1ac0c90037
Fix unit test 2018-04-27 17:33:23 +03:00
rinpatch dcc9fdb47f
Added unit test 2018-04-27 15:36:15 +03:00
rinpatch 06e070aee2 Fix string length 2018-04-23 12:55:25 +03:00
rinpatch c40d8e1d15 Unit test 2018-04-23 11:02:32 +03:00
Marc Abonce Seguin f7f9c50393 [fix] force English results in Google when using en-US 2018-04-18 23:29:48 -05:00
Marc Abonce Seguin 9687786226 update unit tests for google news and wikidata 2018-04-08 23:31:24 -05:00
Marc Abonce Seguin 772c048d01 refactor engine's search language handling 
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
 2018-03-27 00:08:03 -06:00
Noémi Ványi 3ef8533f4d fix unit tests 2018-02-10 20:03:55 +01:00
marc 4d1770398a remove 'all' option from search languages 2017-12-06 01:20:15 -06:00
Adam Tauber 9e79ad4bde [fix] pdbe test 2017-12-01 20:58:40 +01:00
Adam Tauber 060c797739 [fix] change domain name in google engine test 2017-11-22 23:06:24 +01:00
pyrrh0n1c a1d1a50149 Fixed unittests. 2017-11-07 14:49:10 +00:00
pyrrh0n1c 4340c0b16c Fixed typos. 2017-11-07 14:14:20 +00:00
Adam Tauber 00a7041daa
Merge pull request #1068 from Apply55gx/genius 
Continuation of PR #1004 (seems abandoned)
 2017-11-03 20:21:25 +01:00
Noémi Ványi 5954a8e16a minor fix of BASE engine 2017-11-01 17:02:38 +01:00
jibe-b df0d915806 [add] pubmed engine 2017-11-01 17:02:38 +01:00
Noémi Ványi 9c2b7a82f0 minor fixes of arxiv 
Closes #1050
 2017-11-01 14:22:22 +01:00
jibe-b 3e3672e079 [add] arxiv engine 2017-11-01 11:54:16 +01:00
Apply55gx d800e3fcfa Merge pull request #1 from asciimoo/master 
-
 2017-10-25 10:44:28 +02:00
marc 54f0ab1f3b test fetch langauges function on bing images 2017-10-10 18:37:24 -05:00
marc 44085e31d0 update engines_languages.json and languages.py 
Also, fix fetch_languages.py so it can run on python3.
 2017-10-10 16:53:28 -05:00
marc a524dbb823 [fix] language support for bing images and videos 2017-10-10 16:49:49 -05:00
jibe-b d081eee3f8 [add] unit_test for base engine 2017-09-22 22:26:37 +02:00
misnyo c3232b0e1a Merge branch 'master' into nyaa 2017-09-04 17:48:25 +02:00
misnyo 01330f71cd [fix] nyaa.si fixed 2017-08-31 21:32:30 +02:00
misnyo 3182ba7069 [fix] google news dom xpath fix 2017-08-31 17:48:07 +02:00
Adam Tauber e74aaa781e Merge pull request #1016 from misnyo/blekko 
[mod]blekko images removed
 2017-08-30 21:48:36 +02:00
misnyo 0607b167f8 [fix] faroo json api and image layout fixed 2017-08-30 21:09:03 +02:00
misnyo f139f9cfdb [mod]blekko images removed 2017-08-30 16:47:56 +02:00
misnyo bf3f9a91fa [fix]torrentz unit_tests timezone fix 2017-08-29 17:54:41 +02:00
misnyo 2098e1b7ff [fix]torrentz unit_tests timezone fix 2017-08-25 01:11:02 +02:00
misnyo d14e2781b2 [fix]torrentz search engine fixed for new version 2017-08-25 00:52:35 +02:00
woorst 636b760196 remove unicode characters 2017-08-21 11:45:23 -05:00
woorst 62b2f79ce7 fix line lengths for pep8 standards 2017-08-20 21:10:51 -05:00
woorst 2434c29dc5 New engine: Genius (lyrics) 2017-08-19 19:05:17 -05:00
marc 3ca9cad927 add bing videos engine 2017-08-05 20:35:57 -05:00
marc 856dfc3018 add google videos 2017-07-26 15:49:16 +02:00
Noémi Ványi 2242000bd4 change unresponsive_engines to a set to eliminate duplication of errors 2017-07-20 13:30:56 +02:00
Noémi Ványi 243d3e4298 show engine errors in infobox && add new error alert 2017-07-12 16:29:11 +02:00
marc 405e5c8f24 [fix] duckduckgo images doesn't fail with countryless language 2017-07-10 18:41:50 +02:00
marc 3b950929b1 [fix] fix duckduckgo's offset 
First page now starts with 0 offset,
rather than starting on the 30th result.

DuckDuckGo returns 30 results on each page.
 2017-05-27 21:12:39 -05:00
marc c65a409f0d add duckduckgo images engine 2017-05-23 20:07:09 +02:00
Adam Tauber 8db527c1d2 [fix] use raw response with etree.parsefromstring - Unicode strings with encoding declaration are not supported 2017-05-22 15:36:52 +02:00
marc 9ee8e552da [fix] bing images 2017-05-17 15:45:55 +02:00
Alexandre Flament f5128c7cb9 [mod] add/modify image fetching for bing_news, qwant and twitter engines 2017-05-15 17:28:21 +02:00
Adam Tauber 52e615dede [enh] py3 compatibility 2017-05-15 12:02:30 +02:00
François Revol 45d15bd6f0 Add framalibre engine 
framalibre.org is a catalogue of Free Software, edited by Framasoft.

For now we pass the thumbnail as img_src as it doesn't seem to be used
for IT...
 2017-03-22 00:18:23 +01:00
marc 805fb02ed1 add language support for qwant 
closes issue #863
 2017-03-18 23:44:21 +01:00
Alexandre Flament e67dfaaac7 Merge branch 'master' into flask_perimeter 2017-01-27 13:47:59 +00:00
Alexandre Flament 7fdfeca3a4 [mod] add a __common__ template that can't be selected but that provides a common place for shared templates. 
What has been moved into this template :
* opensearch*.xml is always the same whatever the themes.
* the text inside */about.html
 2017-01-20 15:40:38 +01:00
David A Roberts 1d30141c20 [enh] show spelling corrections 2017-01-16 13:31:16 +10:00
Alexandre Flament 84a2c97a65 [mod] searx uses flask framework only in webapp.py. Make migration to another framework easier. 2017-01-02 12:06:04 +01:00
Adam Tauber 6bf9c398a7 [fix] use english as default language in bing 
If no language is specified, bing returns results with multiple languages
for one query which isn't really useful. Setting english as default
insted if nothing.
 2016-12-30 18:17:14 +01:00
marc 1175b3906f change language list to only include languages with a minimum of engines 
that support them.
users can still query lesser supported through the :lang_code bang.
 2016-12-29 01:55:30 -06:00
marc af35eee10b tests for _fetch_supported_languages in engines 
and refactor method to make it testable without making requests
 2016-12-15 00:40:21 -06:00
marc e0c270bd72 tests for language support in engines 2016-12-13 23:51:15 -06:00
marc f62ce21f50 [mod] fetch supported languages for several engines 
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
 2016-12-13 19:58:10 -06:00
marc a11948c71b Add language support for more engines. 2016-12-13 19:32:43 -06:00
Noémi Ványi b034356825 add year filter to engines with time range support && tests 
Following engines does not support "Last year":
 * Bing News
 * DeviantArt
 * DuckDuckGo
 * Yahoo
 * YouTube (noapi)
 2016-12-11 16:58:31 +01:00
Noémi Ványi c59c76e6ee add year to time range to engines which support "Last year" 
Engines:
 * Bing images
 * Flickr (noapi)
 * Google
 * Google Images
 * Google News
 2016-12-11 16:58:31 +01:00
Adam Tauber 9b2bd6847f [fix] 500px unit tests 2016-12-11 03:30:13 +01:00
Adam Tauber f6e9c074bb [fix] vimeo engine change follow-up 2016-12-11 02:33:04 +01:00
Adam Tauber a2c94895c1 [fix] google news engine change follow-up 2016-12-11 01:03:52 +01:00
Adam Tauber 8116e341cc [fix] kickass test html escaping 2016-12-09 19:20:55 +01:00
Adam Tauber fdf63940e8 [fix] tests ++ flickr error 2016-12-09 19:11:22 +01:00
Adam Tauber 971ed0abd1 [enh] add quick answer functionality with an example answerer 2016-11-19 20:53:51 +01:00
Adam Tauber 044809e298 [fix] search mocking in webapp test 2016-11-14 22:21:19 +01:00
Adam Tauber 16ff8d06c7 [fix] bing paging and language support 
see https://msdn.microsoft.com/en-us/library/ff795620.aspx for bing
specific search operators

closes #755
 2016-11-07 22:30:20 +01:00
dalf fbb080f358 Change plugin API : 
- pre_search(request, search)
- post_search(request, search)
- on_result(request, search, result)

with
- request is the Flask request
- search a searx.Search instance
- result a searx result as usual
 2016-11-02 14:22:16 +01:00
dalf 67e11c42b9 Clean up the architecture 
Purposes :
- isolate the plugins calls
- distinction between parsing the web request and running the search (Search class). To be able to test code easily, to run searx code outside a web server, to filter the search query parameters with plugins more easily, etc...

Details :
- request.request_data contains request.form or request.args (initialize inside pre_request() function)
- Query class is renamed RawTextQuery
- SearchQuery class defines all search parameters
- get_search_query_from_webapp create a SearchQuery instance (basically the previous Search.__init__ code)
- Search class and SearchWithPlugins class takes a SearchQuery instance as class constructor parameter
- SearchWithPlugins class inherites from Search class, and run plugins
- A dedicated function search_with_plugins executes plugins to have a well define locals() (which is used by the plugins code).
- All plugins code is executed inside the try...except block (webapp.py, index function)
- advanced_search HTTP parameter value stays in webapp.py (it is only part of UI)
- multiple calls to result_container.get_ordered_results() doesn't compute the order multiple time (note : this method was call only once before)
- paging value is stored in the result_container class (compute in the extend method)
- test about engine.suspend_end_time is done during search method call (instead of __init__)
- check that the format parameter value is one of these : html, rss, json, rss (before the html value was assumed but some text formatting wasn't not done)
 2016-11-02 14:22:16 +01:00
Noémi Ványi 1490d6bc93 add time range search for flickr 2016-11-01 17:58:29 +01:00
Noémi Ványi 5c02b9ef31 add time range support for bing images 2016-11-01 17:58:29 +01:00
Noémi Ványi d2e0cbafb2 add time range search for youtube noapi 2016-11-01 17:58:29 +01:00
Noémi Ványi f70d405202 add time range search for bing news 2016-11-01 17:58:29 +01:00
Noémi Ványi 53c9fde992 fix kickass torrents engine 2016-10-22 05:27:18 +02:00
Alexander Minges 3c5883408c initial commit of pdbe engine 
Adds support for queries to the Protein Data Bank Europe (PDBe).
 2016-10-13 00:40:38 +02:00
Pydo 55a5b686ed Merge branch 'master' of https://github.com/asciimoo/searx into feature/seedpeer-engine-integration 
Resolved conflict searx/settings.yml
 2016-10-01 10:46:18 -04:00
Adam Tauber bee7b497a3 [mod] rename "default" theme to "legacy" 2016-09-22 23:51:07 +02:00
marc d1d4ed4376 [fix] results with digbit don't truncate anymore 2016-09-20 16:11:33 -05:00
firebovine f67db1e107 fix tests to reflect: Wolfram|Alpha (input) response 2016-09-10 18:04:24 -04:00
marc 09ee2aa69d [fix] Result text in Wolfram|Alpha (#607) 2016-09-10 17:42:04 -04:00
Pydo ec4a03628e Put html fixture in file to be pep8 line length compliant 2016-09-05 15:37:20 -04:00
Pydo aa85045a7d Added seedpeer unitests 2016-09-05 14:50:26 -04:00
Noemi Vanyi aa227180e2 remove self info answers from pages after first one 2016-08-27 16:45:25 +02:00
Noemi Vanyi 6dd5f7a8c9 add digbt unittest 2016-08-13 16:27:16 +02:00
Adam Tauber eb57481450 [fix] google images paging - closes #571 2016-08-13 01:13:41 +02:00
marc ad58b14be7 [fix] merge infoboxes based on weight 
also minor changes in attributes and images from wikidata
 2016-08-05 23:51:04 -05:00
marc a0a1284998 wikidata refactor and more attributes (see issue #560) 2016-08-05 23:51:04 -05:00
Adam Tauber 350a84520d [fix] time range detection 2016-07-26 00:28:48 +02:00
Noemi Vanyi ba590de7f1 [fix] deviantart engine xpaths 2016-07-19 09:37:02 +02:00
Adam Tauber aa09f963eb Merge pull request #621 from stepshal/anomalous-backslash-in-string 
Fix anomalous backslash in string
 2016-07-18 22:27:17 +02:00
Adam Tauber c75d0d1532 [fix] result count tests 2016-07-16 21:53:42 +02:00
Adam Tauber f7a3e9771d Merge pull request #615 from mmuman/scanr 
Add ScanR structures search engine
 2016-07-15 13:02:29 +02:00
a01200356 8fc4b8156a [fix] multilingual duckduckgo 
only works if both country and language are set
 2016-07-12 23:44:39 -05:00
François Revol 1dba6dcbac Add ScanR structures search engine 
In theory ScanR should also search for projects but the API is different,
so we'd need another engine.
 2016-07-12 15:47:04 +02:00
stepshal b3ab221b98 Fix anomalous backslash in string 2016-07-11 23:53:13 +07:00
stepshal cd9b494cb5 Fix quantity of blank lines after code object. 2016-07-10 21:44:27 +07:00
François Revol b538de568a Add INA search engine 2016-07-09 22:09:54 +02:00
Adam Tauber 257e52954a [enh][fix] update to latest dependencies ++ fix tests & travis test runner 
WARNING: dependency changes
 2016-07-04 22:46:43 +02:00
Maxqia 56e53c9297 [fix] currency_convert: make unit tests pass 
also make regex more efficent and normalize 's'es
 2016-05-03 18:59:30 -07:00
a01200356 6dca14e95d [enh] multilingual duckduckgo_definitions 2016-04-17 16:34:46 -05:00
a01200356 8d335dbdae [enh] wikipedia infobox 
creates simple multilingual infobox using wikipedia's api
 2016-04-17 16:22:19 -05:00
Kirill Isakov d98024c26e Update Reddit unit test -- check for img_src 2016-04-16 16:45:49 +06:00
Adam Tauber 85c0351dca Merge pull request #526 from ukwt/anime 
Add a few search engines
 2016-04-14 10:59:31 +02:00
Kirill Isakov d54e82dfb9 Revert to using random UA in Reddit search engine 2016-04-13 22:06:00 +06:00
Adam Tauber 5544fdb756 Merge pull request #531 from guyou/add-doku-engine 
Add doku engine
 2016-04-12 13:54:12 +02:00
Noemi Vanyi f0fd9ad628 add unit && robot tests 2016-04-09 01:09:26 +02:00
Noemi Vanyi fe691a0988 new preferences handling 
Preferences class was introduced in order to handle user preferences. Right now
it parses cookies and the form in preferences. Also it can retrieve settings
based on the name of the setting.

ATTENTION
Please note that engine preferences are handled differently from now on. So it
introduces incompatible changes. Every user who has saved preferences should reset and
save his/her settings again.

This change was needed, because everytime a default disabled engine was
added saved user preferences would broke. Now engine setting tracking is
fixed.
 2016-04-09 01:08:44 +02:00
Adam Tauber 9331fc28a8 [fix] broken google images parsing 2016-04-07 08:07:17 +02:00