mirror of
				https://github.com/searxng/searxng
				synced 2024-01-01 19:24:07 +01:00 
			
		
		
		
	[fix] duckduckgo lite engine: set HTTP header 'Referer'
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].
From reverse engineering:
- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!
- The 'Referer' header is only set on second and follow up pages but not on the
  first page
- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.
Related to 'Referer' header & ddg requests:
[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
			
			
This commit is contained in:
		
							parent
							
								
									fa5b2a7948
								
							
						
					
					
						commit
						9197efa2a7
					
				
					 1 changed files with 4 additions and 1 deletions
				
			
		|  | @ -222,7 +222,6 @@ def request(query, params): | ||||||
|     # link again and again .. |     # link again and again .. | ||||||
| 
 | 
 | ||||||
|     params['headers']['Content-Type'] = 'application/x-www-form-urlencoded' |     params['headers']['Content-Type'] = 'application/x-www-form-urlencoded' | ||||||
|     params['headers']['Referer'] = 'https://google.com/' |  | ||||||
| 
 | 
 | ||||||
|     # initial page does not have an offset |     # initial page does not have an offset | ||||||
|     if params['pageno'] == 2: |     if params['pageno'] == 2: | ||||||
|  | @ -248,6 +247,10 @@ def request(query, params): | ||||||
|         params['data']['nextParams'] = form_data.get('nextParams', '') |         params['data']['nextParams'] = form_data.get('nextParams', '') | ||||||
|         params['data']['v'] = form_data.get('v', 'l') |         params['data']['v'] = form_data.get('v', 'l') | ||||||
| 
 | 
 | ||||||
|  |         # request needs a vqd argument | ||||||
|  |         params['data']['vqd'] = get_vqd(query) | ||||||
|  |         params['headers']['Referer'] = 'https://lite.duckduckgo.com/' | ||||||
|  | 
 | ||||||
|     params['data']['kl'] = eng_region |     params['data']['kl'] = eng_region | ||||||
|     params['cookies']['kl'] = eng_region |     params['cookies']['kl'] = eng_region | ||||||
| 
 | 
 | ||||||
|  |  | ||||||
		Loading…
	
	Add table
		
		Reference in a new issue
	
	 Markus Heiser
						Markus Heiser