searxng/docs/src
Markus Heiser 2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
..
index.rst Correct my small mistake 2023-01-04 20:07:51 +01:00
searx.babel_extract.rst [mod] add i18n infrastructure for SearXNG message files (searxng.msg) 2022-03-16 09:55:53 +01:00
searx.engine.duckduckgo.rst [mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searx.enginelib.rst [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
searx.engines.bing.rst [doc] add a description of bing engines (web, news, video, images) 2023-03-24 10:37:42 +01:00
searx.engines.demo_offline.rst [docs] add engine "Demo Offline Engine" 2021-06-04 15:04:38 +02:00
searx.engines.demo_online.rst [docs] add engine "Demo Online Engine" 2021-06-04 15:05:58 +02:00
searx.engines.google.rst [mod] Google: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searx.engines.peertube.rst [mod] Peertube: re-engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searx.engines.rst [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
searx.engines.startpage.rst [mod] Startpage: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searx.engines.tineye.rst [mod] tineye - add some documentation 2022-01-28 09:06:44 +01:00
searx.engines.wikipedia.rst [mod] wikipedia & wikidata: upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
searx.engines.yahoo.rst [fix] engine - yahoo: rewrite and fix issues 2021-10-16 20:05:26 +00:00
searx.exceptions.rst [doc] Add doc-strings to searx.exceptions 2023-01-29 19:06:19 +01:00
searx.infopage.rst [mod] replace /help by /info pages and include pages in project docs 2022-03-12 11:36:31 +01:00
searx.locales.rst [mod] replace searx.languages by searx.sxng_locales 2023-03-24 10:37:42 +01:00
searx.plugins.limiter.rst [doc] add some documentation about the limiter plugin (and redis) 2022-03-25 11:26:10 +01:00
searx.plugins.tor_check.rst [mod] add tor_check plugin - convenient tor checking trough searxng 2022-07-19 07:34:54 +02:00
searx.redisdb.rst move searx.shared.redisdb to searx.redisdb 2022-12-10 09:26:38 +01:00
searx.redislib.rst [mod] collection of redis functions and lua scripts (initial) 2022-05-30 11:10:30 +02:00
searx.search.processors.rst [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
searx.search.rst [doc] update docs/dev/plugins.rst 2021-09-10 10:58:22 +02:00
searx.utils.rst [mod] add documentation about searx.utils 2022-01-29 22:49:42 +01:00