zaclys
/
searxng
mirror of https://github.com/searxng/searxng.git
1
0
Fork 1
Code Issues Packages Projects Releases Wiki Activity
3,601 Commits 5 Branches 17 Tags 71 MiB
Commit Graph

967 Commits

Author SHA1 Message Date
Emilien Devos 8f51430f5c [fix] Force Google old UI with a new user agent 2019-11-22 23:01:41 +01:00
lorddavidiii 5e5ff0cbf8 webapp.py: use html.escape if cgi.escape is not available 
- cgi.escape was removed in python 3.8
- also use html.escape in framalibre.py
 2019-11-16 21:58:25 +01:00
Dalf 85b3723345 [mod] speed optimization 
compile XPath only once
avoid redundant call to urlparse
get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
 2019-11-15 09:33:15 +01:00
Noémi Ványi 5796dc60c9 fix pep 8 check 2019-10-16 15:52:48 +02:00
Noémi Ványi a6f20caf32 add initial support for offline engines && command engine 2019-10-16 15:52:48 +02:00
Adam Tauber 7d8fd4b95e [fix] pep8 2019-10-16 15:32:03 +02:00
Adam Tauber bbe4442a86 [fix] update gigablast engine 2019-10-16 15:27:37 +02:00
Adam Tauber 1057e42cfd [fix] update digg engine 2019-10-16 15:11:42 +02:00
Adam Tauber 7177c9e12f [fix] update devianart engine 2019-10-16 14:42:31 +02:00
Adam Tauber 6ca1622378 [fix] update 1x engine 2019-10-16 13:27:05 +02:00
Adam Tauber c98a2df36d [fix] enable paging support for arxiv engine 2019-10-16 13:12:17 +02:00
Adam Tauber ed1c1bdb04 [fix] pep8 2019-10-14 15:09:39 +02:00
Adam Tauber 77a70fe541 [fix] update startpage engine - closes #1601 2019-10-14 14:18:41 +02:00
Adam Tauber 94ea9d6622 [fix] duckduckgo paging - closes #1677 2019-10-14 13:52:15 +02:00
Marc Abonce Seguin bb4d223770 [fix] google images 2019-08-26 21:54:01 -07:00
Léo Bourrel 88261e111c Fix bing engine results count (#1387) 
This PR fixes the result count from bing which was throwing an (hidden) error and add a validation to avoid reading more results than avalaible.

For example :
If there is 100 results from some search and we try to get results from 120 to 130, Bing will send back the results from 0 to 10 and no error. If we compare results count with the first parameter of the request we can avoid this "invalid" results.
 2019-08-05 16:15:40 +02:00
Dalf 1cee2c1796 [fix] bing engine 
before this commit, sometimes there are no results
use a generic user-agent instead of one with the OS "Windows NT 6.3; WOW64"
 2019-08-05 15:46:40 +02:00
Dalf fcc9587ee9 [fix] fdroid engine 2019-08-05 15:44:02 +02:00
Dalf fbf6b689dd [fix] dictzone engine 2019-08-05 15:43:21 +02:00
Dalf 9ff5001816 [fix] arxiv engine 2019-08-05 15:43:01 +02:00
Alexandre Flament 2179079a91
[fix] fix flickr_noapi decoding (#1655) 
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
 2019-08-02 13:37:13 +02:00
cy8aer 4dc792e1e2 [enh] add invidious engine. (#1657) 
closes #1372
 2019-08-02 13:25:25 +02:00
0xhtml b2e1ee8d35 Fix some more errors with none/wrong credentials 2019-07-31 21:09:02 +02:00
0xhtml 275b37cc7c Fix error if the user hasn't set api credentials 2019-07-31 21:01:24 +02:00
0xhtml c329ea135e Fix spotify engine 2019-07-31 20:44:41 +02:00
Dalf 0c032c8429 [fix] youtube_noapi engine: fix the title 2019-07-31 08:39:40 +02:00
Dalf 8b7ac56669 [fix] google_videos engine: some results don't a thumbnail 2019-07-31 08:38:45 +02:00
Dalf d44677e226 [fix] dailymotion engine: remove HTML tags from the description 2019-07-31 08:37:51 +02:00
Dalf 6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection. 
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
 2019-07-29 07:39:39 +02:00
Frank de Lange cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571) 
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
 2019-07-28 10:42:00 +02:00
Frank de Lange 204a2cbbf0 [fix] bing_videos (#1579) 
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
 2019-07-27 17:49:30 +02:00
Dalf 23611897ec [fix] make sure then engine name is lower case 
Minor fix: "%s engine initialized" display the right engine name
 2019-07-27 08:52:30 +02:00
Frank de Lange 11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568) 
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
 2019-07-27 08:22:02 +02:00
Alexandre Flament f34b5cedb1
[fix] fixes google play engines (#1651) 
update commit 87baa74a86
 2019-07-25 09:31:47 +02:00
volth eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 2019-07-25 08:40:48 +02:00
rachmadani haryono 3b1122c5fa [fix] fix duden engine (#1594) 2019-07-25 08:17:45 +02:00
Venca24 87baa74a86 [fix] fixes google play engines and adds thumbnails to their results (#1612) 
fix google play apps, google play apps, google play music engines

xpath engine: thumbnail_xpath can define an optional thumbnail
 2019-07-25 07:46:41 +02:00
Dalf da0ce5880f [fix] fix soundcloud engine, speed up searx start time 2019-07-19 08:59:11 +02:00
Dalf 45702b77ca embedded iframe (youtube, dailymotion, vimeo): use https 2019-07-13 15:49:59 +02:00
Emilien Devos cbd1ebdce8 [fix] Force Google old UI (#1597) 2019-05-29 10:05:57 +09:00
Frank de Lange 4b7332286a Use string formatter to create source and img_format labels (#1566) 
google_images :  use JSON embedded in HTML (engine expected pure JSON)
 2019-05-28 12:33:31 +09:00
Dalf ffe0972f91 Remove some engines : subtitleseeker, seedpeer, swisscows 
http://www.subtitleseeker.com and http://www.seedpeer.eu don't exist anymore.
https://swisscows.ch/ has change : the engine needs to be updated
 2019-05-28 04:06:35 +02:00
Alexandre Flament df2b9a76f7
Merge branch 'master' into ne/fix-google-image-search 2019-04-14 13:55:00 +02:00
Nick Espig 1c6ab79b9f
Fix google image search 
- Because there is not full image url in the dom, we replace "image_url" with the same url as the "url" (url of source).
  See example HTML https://gist.github.com/Nachtalb/2dea8a4d2c723c49226ad9645838121f
- Remove unused import
- Fix google image search title
- Keep google image safe value up to date
 2019-04-14 12:03:25 +02:00
Marc Abonce Seguin 3e1c2153f7 [fix] duckduckgo images requests 2019-04-13 00:38:37 -05:00
Marc Abonce Seguin f2d49a6971 [fix] get youtube results from js object 
Results are not appearing in the html document anymore,
instead they are found inside an object embedded in a script.
 2019-03-26 21:09:15 -06:00
Jonas Zohren f7bdd827c4 [enh] adds apkmirror search engine 2019-02-13 00:37:29 +01:00
Léo Bourrel bf4a38ad66 Remove asksteem 2019-01-28 11:02:43 +01:00
d-tux f1814079f0
Merge branch 'master' into engines/unsplash 2019-01-14 13:40:57 +01:00
Marc Abonce Seguin 626a8e9ac9 [fix] unicode error with WolframAlpha API engine 2019-01-08 21:02:23 -06:00
d-tux 329172f66e
Merge branch 'master' into engines/unsplash 2019-01-08 09:24:45 +01:00
Noémi Ványi 97351a2c72 fix after rebase 2019-01-07 21:28:58 +01:00
Noémi Ványi b63d645a52 Revert "remove 'all' option from search languages" 
This reverts commit 4d1770398a.
 2019-01-07 21:19:00 +01:00
Marc Abonce Seguin 0169b63e84 [fix] fetch google's supported languages 2019-01-06 21:31:45 -06:00
Marc Abonce Seguin 51111c2594 [fix] always set langauge_aliases even if it's empty 2019-01-06 20:49:56 -06:00
Marc Abonce Seguin 5568f24d6c [fix] check language aliases when setting search language 2019-01-06 20:31:57 -06:00
Venca24 2456b8f571 [mod] google videos 2019-01-05 12:12:09 +01:00
Noémi Ványi 1ea56576dc
Merge branch 'master' into devel_google_videos 2019-01-04 22:27:05 +01:00
Noémi Ványi aeb6dab187
Merge branch 'master' into master 2019-01-04 22:14:40 +01:00
Noémi Ványi 5b81f7c2cc
Merge branch 'master' into bugfix_startpage 2019-01-04 22:03:00 +01:00
Venca24 cf26aba93b [FIX] google videos thumbnails 2019-01-04 15:48:22 +01:00
Léo Bourrel 73ec2ba74e Update bing image engine with new image json format 2018-12-19 11:12:25 +01:00
Michael Pfitzner 44ce51f0c5 restore startpage search results 2018-12-14 21:38:48 +01:00
d-tux 4a127b19de
Merge branch 'master' into engines/unsplash 2018-12-14 07:06:19 +01:00
dimqua 0d86ed9c7e update startpage.py 2018-12-11 21:45:47 +03:00
dimqua 442772a6c5 remove findx engine (#1452) 2018-12-02 22:39:04 +01:00
Venca24 0bdf090ed7 [fix] google videos engine 2018-11-22 13:00:34 +01:00
Denis Wernert b9ada93b3a Removes what looks like tracking parameters 2018-10-08 14:56:20 +02:00
Denis Wernert 72d063d27d Uses the raw url for the image result, rather than the full 
size result.
 2018-10-08 14:01:35 +02:00
Denis Wernert cd7849ffc8 Adds the Unsplash image engine 2018-10-08 09:18:42 +02:00
Adam Tauber 052a71d05a
Merge pull request #1368 from aliceinwire/gentoo_engine 
Add Gentoo wiki search engine
 2018-08-21 16:24:34 +02:00
Alice Ferrazzi d24c58cf4e Fix pep8 errors 2018-08-21 22:55:51 +09:00
Alice Ferrazzi ff2eac9f6b Fix choosing languges function 
Searching for english is now giving all pages results.
Instead searching for a specific language different from english
is correctly returning only translated pages for the selected language.
 2018-08-21 22:55:39 +09:00
Adam Tauber baacfac32b [fix] remove publicly unavailable 500px engine - #1338 2018-08-19 15:06:59 +02:00
dadosch b575f898c0 duden.de engine 2018-08-18 19:24:02 +02:00
Adam Tauber 845ba79e75 [fix] update qwant search url 
thx to @dalf for the fix (#1365)
 2018-08-13 12:38:19 +02:00
Alice Ferrazzi 5f0888b8f4 added languages to Gentoo engine 2018-08-12 23:00:57 +09:00
Alice Ferrazzi b7886bb15c gentoo engine test 2018-08-12 23:00:57 +09:00
Adam Tauber d51732c0e5
Merge pull request #1303 from MarcAbonce/bing 
Fix bing "garbage" results
 2018-07-09 11:00:37 +02:00
Léo Bourrel 7a474db61b Fix formatting 2018-07-06 10:31:01 +02:00
Léo Bourrel acaef6600e Update path to wikidata image 2018-07-05 10:11:45 +02:00
Adam Tauber f82ead3e30 [fix] add py3 compatibility to base engine - closes #1319 2018-06-14 11:50:03 +02:00
Adam Tauber 57e7e9da98 [fix] use html result page in google images (previous endpoint stopped working) 2018-06-14 11:40:39 +02:00
Noémi Ványi 16915116e0 fix piratebay url 
Closes #1229
 2018-05-21 16:58:39 +02:00
Marc Abonce Seguin 75b276f408 fix bing "garbage" results (issue #1275) 2018-05-20 18:13:32 -05:00
Noémi Ványi b843c1351a add missing parameter to gigablast search url 
Closes #1293
 2018-05-16 22:26:32 +02:00
rinpatch 059c2ccb95
Fix acgsou encoding error on python2 and add unicode characters to unit testt 2018-05-05 23:41:13 +03:00
rinpatch 3b89ce9bfa
Switch acgsou to http 2018-05-03 08:34:21 +03:00
Adam Tauber 1a1f9852f1
Merge pull request #1283 from rinpatch/acgsou-engine 
[Feature] Acgsou as a searchable engine
 2018-04-28 22:42:35 +02:00
rinpatch f6695cea59
Fix nyaa to use https 2018-04-27 17:55:42 +03:00
rinpatch f17e54a396
Forgot to remove print command used for debugging 2018-04-27 15:40:42 +03:00
rinpatch dcc9fdb47f
Added unit test 2018-04-27 15:36:15 +03:00
rinpatch e3bd2670d7
Traling whitespace 2018-04-27 08:36:11 +03:00
rinpatch 86f47689fe
Hopefully fix code style errors(again)(again) 2018-04-27 08:32:43 +03:00
rinpatch b8b23d8006
Hopefully fix code style errors(again) 2018-04-27 08:29:17 +03:00
rinpatch fb364ffae7
Hopefully fix code style errors 2018-04-27 08:23:58 +03:00
rinpatch 3ec35a06f7
Added Engine 2018-04-26 22:42:31 +03:00
rinpatch cd1374d2f2 added whitespaces 2018-04-22 14:12:32 +03:00
rinpatch a305dae9d8 switched currency convert to duckduckgo and added Bitcoin to currency list 2018-04-22 13:46:13 +03:00
Noémi Ványi 25b48143ea fix findx engine 2018-04-21 20:02:52 +02:00
Marc Abonce Seguin f7f9c50393 [fix] force English results in Google when using en-US 2018-04-18 23:29:48 -05:00
Marc Abonce Seguin b12857a70d [fix] make search requests on wikidata more accurate 2018-04-08 21:17:00 -05:00
Marc Abonce Seguin 835d1edd58 [fix] google news xpath 2018-04-08 20:56:05 -05:00
Marc Abonce Seguin 343e555ee9 [fix] append http if no scheme is provided in xpath's extact_url 
This solves a bug with Yahoo where some results don't specify
a protocol.
 2018-04-08 20:35:34 -05:00
Adam Tauber 283f6c9053
Merge pull request #1252 from MarcAbonce/search-languages 
[mod] Refactor engine's search language handling
 2018-04-05 17:27:07 +02:00
Marc Abonce Seguin 772c048d01 refactor engine's search language handling 
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
 2018-03-27 00:08:03 -06:00
Noémi Ványi 8cc529e9a3 forward category to engine without highlighting on the ui 2018-03-22 11:02:24 +01:00
Noémi Ványi d04e471ce5 add findx engine for general, images and videos 2018-03-22 10:02:37 +01:00
Marc Abonce Seguin d1eae9359f fix fetch_langauges to be more accurate 
Add languages supported by either all default general engines or 10 engines.
 2018-03-20 17:58:20 -06:00
Marc Abonce Seguin 216716db70 [fix] change duckduckgo url to avoid error response 2018-03-14 23:18:42 -06:00
Adam Tauber 03d7a8f065 [fix] pep8 2018-02-18 23:24:59 +01:00
Adam Tauber b4c657d227 [enh] add asksteem engine - closes #1163 2018-02-18 22:42:32 +01:00
Noémi Ványi 988cf38196 fix Microsoft Academic engine 2018-02-17 21:39:49 +01:00
Adam Tauber 2f69eaeb2f [fix] fix engine initialization 2018-02-17 14:30:06 +01:00
Noémi Ványi c645915171 fix bing videos engine 2018-02-10 19:44:07 +01:00
Marc Abonce Seguin 829032f306 [fix] read utf-8 files (settings, languages, currency) with python3.5 
Related to discussion in #1124
The io.open import is necessary for python2
 2018-01-16 23:26:10 -06:00
Adam Tauber 8511e64f35
Merge pull request #1124 from JosephKiranBabu/python3-unicode-support 
Make Python 3 able to read settings files with Unicode characters
 2018-01-12 10:58:23 +01:00
Noémi Ványi f3aa5ea47d fix yacy result url 
Closes #1064
 2018-01-06 14:52:14 +01:00
Joseph Nuthalapati bdc803e185
Make Python 3 able to read settings files with Unicode characters 
SearX currently doesn't start up when run with Python 3 as it tries to parse the
settings.yml file with ASCII codecs.
There are similar problems with engines_languages.json and currencies.json
Python 3 requires that files with Unicode characters be read with a 'b' flag.
This also works with Python 2 and hence can be integrated into the main source
code.

Tested with the latest Python 3.6.4rc1 on Debian unstable.

Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
 2017-12-21 17:33:19 +05:30
Noémi Ványi 2d5eed9b59 send constant cookie with query to Google 2017-12-18 21:38:52 +01:00
Adam Tauber 7752b0a807 [fix] gigablast url parameter change - fixes #1107 2017-12-09 21:00:08 +01:00
marc 4d1770398a remove 'all' option from search languages 2017-12-06 01:20:15 -06:00
Adam Tauber 1613c6319e [fix] handle /sorry redirects 2017-12-05 20:38:34 +01:00
Adam Tauber 1088c2a75d [fix] do not crash if publication date is missing in pubmed engine 2017-12-01 20:54:12 +01:00
Adam Tauber 801b5a56ad [fix] multiple fixes in pdbe engine 2017-12-01 20:54:12 +01:00
Adam Tauber 0969e50c5b [fix] convert json engine result attributes to string - closes #1006 2017-12-01 20:54:12 +01:00
Adam Tauber 6eb9503896 [fix] use english in google engine if no language was set - this prevents guessing the language by the IP of the instance 2017-11-22 22:56:47 +01:00
Adam Tauber 6fdb6640d9 [fix] revert language changes to prevent CAPTCHAs 2017-11-22 22:50:48 +01:00
Adam Tauber 9ab8536479 [fix] fix language support of google 2017-11-21 16:28:53 +01:00
pyrrh0n1c 4340c0b16c Fixed typos. 2017-11-07 14:14:20 +00:00
pyrrh0n1c 2231b02add Fixed the currency_convert engine. 2017-11-07 13:29:17 +00:00
Adam Tauber 00a7041daa
Merge pull request #1068 from Apply55gx/genius 
Continuation of PR #1004 (seems abandoned)
 2017-11-03 20:21:25 +01:00
Adam Tauber 7a9b18e9e9 [mod] add more error handling to json engine II. 2017-11-02 01:08:15 +01:00
Adam Tauber 181f1c6305 [mod] add more error handling to json engine 2017-11-02 00:43:29 +01:00
Noémi Ványi 5954a8e16a minor fix of BASE engine 2017-11-01 17:02:38 +01:00
Noémi Ványi d20bba6dc7 minor fixes of pubmed engine 
Closes #1045
 2017-11-01 17:02:38 +01:00
jibe-b df0d915806 [add] pubmed engine 2017-11-01 17:02:38 +01:00
Noémi Ványi 9c2b7a82f0 minor fixes of arxiv 
Closes #1050
 2017-11-01 14:22:22 +01:00
jibe-b e391b2d970 [fix] remove .encode for python3 compatibility 2017-11-01 11:54:55 +01:00
jibe-b 5278fa666c [enh] use format to concatenate strings 2017-11-01 11:54:38 +01:00
jibe-b 3e3672e079 [add] arxiv engine 2017-11-01 11:54:16 +01:00
Apply55gx d800e3fcfa Merge pull request #1 from asciimoo/master 
-
 2017-10-25 10:44:28 +02:00
Apply55gx 18a4e7035f removed unused indicies array 2017-10-25 10:42:37 +02:00
marc 44085e31d0 update engines_languages.json and languages.py 
Also, fix fetch_languages.py so it can run on python3.
 2017-10-10 16:53:28 -05:00
marc a524dbb823 [fix] language support for bing images and videos 2017-10-10 16:49:49 -05:00
misnyo 33fd938016 [mod] int_or_zero refactored to searx_utils 2017-09-04 20:05:04 +02:00
misnyo c3232b0e1a Merge branch 'master' into nyaa 2017-09-04 17:48:25 +02:00
Adam Tauber 234366b900 Merge pull request #1018 from misnyo/generalfile 
[mod]generalfile engine removed
 2017-09-03 23:09:49 +02:00
Adam Tauber c505c3c1ca Merge pull request #1019 from misnyo/gigablast 
[fix] gigablast api fix
 2017-09-03 23:09:33 +02:00
misnyo b91772eff1 [fix] gigablast pep8 fix 2017-08-31 21:49:19 +02:00