searxng_extra/update/

[source]

Scripts to update static data in git://searx/data/

update_ahmia_blacklist.py

[source]

This script saves Ahmia’s blacklist for onion sites.

Output file: git://searx/data/ahmia_blacklist.txt (CI Update data …).

update_currencies.py

[source]

Fetch currencies from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/currencies.json (CI Update data …).

update_engine_descriptions.py

[source]

Fetch website description from websites and from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/engine_descriptions.json.

searxng_extra.update.update_engine_descriptions.get_output()[source]

From descriptions[engine][language] = [description, source] To

  • output[language][engine] = description_and_source

  • description_and_source can be:

    • [description, source]

    • description (if source = “wikipedia”)

    • [f”engine:lang”, “ref”] (reference to another existing description)

update_external_bangs.py

[source]

Update git://searx/data/external_bangs.json using the duckduckgo bangs (CI Update data …).

https://duckduckgo.com/newbang loads:

This script loads the javascript, then the bangs.

The javascript URL may change in the future ( for example https://duckduckgo.com/bv2.js ), but most probably it will requires to update RE_BANG_VERSION

searxng_extra.update.update_external_bangs.merge_when_no_leaf(node)[source]

Minimize the number of nodes

A -> B -> C

  • B is child of A

  • C is child of B

If there are no C equals to <LEAF_KEY>, then each C are merged into A. For example (5 nodes):

d -> d -> g -> <LEAF_KEY> (ddg)
  -> i -> g -> <LEAF_KEY> (dig)

becomes (3 noodes):

d -> dg -> <LEAF_KEY>
  -> ig -> <LEAF_KEY>

update_firefox_version.py

[source]

Fetch firefox useragent signatures

Output file: git://searx/data/useragents.json (CI Update data …).

update_engine_traits.py

[source]

Update searx.enginelib.traits.EngineTraitsMap and git://searx/languages.py

searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE:

Persistence of engines traits, fetched from the engines.

git://searx/languages.py

Is generated from intersecting each engine’s supported traits.

The script git://searxng_extra/update/update_engine_traits.py is called in the CI Update data …

class searxng_extra.update.update_engine_traits.UnicodeEscape[source]

Escape unicode string in pprint.pformat

searxng_extra.update.update_engine_traits.fetch_traits_map()[source]

Fetchs supported languages for each engine and writes json file with those.

searxng_extra.update.update_engine_traits.filter_locales(traits_map: EngineTraitsMap)[source]

Filter language & region tags by a threshold.

searxng_extra.update.update_engine_traits.get_unicode_flag(locale: Locale)[source]

Determine a unicode flag (emoji) that fits to the locale

update_osm_keys_tags.py

[source]

Fetch OSM keys and tags.

To get the i18n names, the scripts uses Wikidata Query Service instead of for example OSM tags API (sidenote: the actual change log from map.atownsend.org.uk might be useful to normalize OSM tags).

Output file: git://searx/data/osm_keys_tags (CI Update data …).

SPARQL_TAGS_REQUEST :

Wikidata SPARQL query that returns type-categories and types. The returned tag is Tag:{category}={type} (see get_tags()). Example:

SPARQL_KEYS_REQUEST :

Wikidata SPARQL query that returns keys. Example with “payment”:

rdfs:label get all the labels without language selection (as opposed to SERVICE wikibase:label).

update_pygments.py

[source]

Update pygments style

Call this script after each upgrade of pygments

update_wikidata_units.py

[source]

Fetch units from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/wikidata_units.json (CI Update data …).