[enh] add raise_for_httperror

check HTTP response:
* detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time.
* otherwise raise HTTPError as before

the check is done in poolrequests.py (was before in search.py).

update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
This commit is contained in:
Alexandre Flament 2020-12-09 21:23:20 +01:00
parent 033f39bff7
commit d703119d3a
11 changed files with 179 additions and 56 deletions

View file

@ -647,11 +647,6 @@ engines:
shortcut : qwn
categories : news
- name : qwant social
engine : qwant
shortcut : qws
categories : social media
# - name: library
# engine: recoll
# shortcut: lib
@ -817,12 +812,13 @@ engines:
# Or you can use the html non-stable engine, activated by default
engine : youtube_noapi
- name : yggtorrent
engine : yggtorrent
shortcut : ygg
url: https://www2.yggtorrent.si/
disabled : True
timeout : 4.0
# tmp suspended: Cloudflare CAPTCHA
#- name : yggtorrent
# engine : yggtorrent
# shortcut : ygg
# url: https://www2.yggtorrent.si/
# disabled : True
# timeout : 4.0
- name : dailymotion
engine : dailymotion