searxng_extra/update/

[source]

Scripts to update static data in git://searx/data/

update_ahmia_blacklist.py

[source]

This script saves Ahmia’s blacklist for onion sites.

Output file: git://searx/data/ahmia_blacklist.txt (CI Update data …).

update_currencies.py

[source]

Fetch currencies from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/currencies.json (CI Update data …).

update_engine_descriptions.py

[source]

Fetch website description from websites and from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/engine_descriptions.json.

searxng_extra.update.update_engine_descriptions.get_output()

From descriptions[engine][language] = [description, source] To

• output[language][engine] = description_and_source

• description_and_source can be:

  • [description, source]

  • description (if source = “wikipedia”)

  • [f”engine:lang”, “ref”] (reference to another existing description)

update_external_bangs.py

[source]

Update git://searx/data/external_bangs.json using the duckduckgo bangs (CI Update data …).

https://duckduckgo.com/newbang loads:

• a javascript which provides the bang version ( https://duckduckgo.com/bv1.js )

• a JSON file which contains the bangs ( https://duckduckgo.com/bang.v260.js for example )

This script loads the javascript, then the bangs.

The javascript URL may change in the future ( for example https://duckduckgo.com/bv2.js ), but most probably it will requires to update RE_BANG_VERSION

searxng_extra.update.update_external_bangs.merge_when_no_leaf(node)

Minimize the number of nodes

A -> B -> C

• B is child of A

• C is child of B

If there are no C equals to <LEAF_KEY>, then each C are merged into A. For example (5 nodes):

d -> d -> g -> <LEAF_KEY> (ddg)
  -> i -> g -> <LEAF_KEY> (dig)

becomes (3 nodes):

d -> dg -> <LEAF_KEY>
  -> ig -> <LEAF_KEY>

update_firefox_version.py

[source]

Fetch firefox useragent signatures

Output file: git://searx/data/useragents.json (CI Update data …).

update_engine_traits.py

[source]

Update searx.enginelib.traits.EngineTraitsMap and git://searx/languages.py

searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE:

Persistence of engines traits, fetched from the engines.

git://searx/languages.py

Is generated from intersecting each engine’s supported traits.

The script git://searxng_extra/update/update_engine_traits.py is called in the CI Update data …

class searxng_extra.update.update_engine_traits.UnicodeEscape

Escape unicode string in pprint.pformat

searxng_extra.update.update_engine_traits.fetch_traits_map()

Fetchs supported languages for each engine and writes json file with those.

searxng_extra.update.update_engine_traits.filter_locales(traits_map: EngineTraitsMap)

Filter language & region tags by a threshold.

searxng_extra.update.update_engine_traits.get_unicode_flag(locale: Locale)

Determine a unicode flag (emoji) that fits to the locale

update_osm_keys_tags.py

[source]

Fetch OSM keys and tags.

To get the i18n names, the scripts uses Wikidata Query Service instead of for example OSM tags API (side note: the actual change log from map.atownsend.org.uk might be useful to normalize OSM tags).

Output file: git://searx/data/osm_keys_tags (CI Update data …).

SPARQL_TAGS_REQUEST :

Wikidata SPARQL query that returns type-categories and types. The returned tag is Tag:{category}={type} (see get_tags()). Example:

• https://taginfo.openstreetmap.org/tags/building=house#overview

• https://wiki.openstreetmap.org/wiki/Tag:building%3Dhouse at the bottom of the infobox (right side), there is a link to wikidata: https://www.wikidata.org/wiki/Q3947 see property “OpenStreetMap tag or key” (P1282)

• https://wiki.openstreetmap.org/wiki/Tag%3Abuilding%3Dbungalow https://www.wikidata.org/wiki/Q850107

SPARQL_KEYS_REQUEST :

Wikidata SPARQL query that returns keys. Example with “payment”:

• https://wiki.openstreetmap.org/wiki/Key%3Apayment at the bottom of infobox (right side), there is a link to wikidata: https://www.wikidata.org/wiki/Q1148747 link made using the “OpenStreetMap tag or key” property (P1282) to be confirm: there is a one wiki page per key ?

• https://taginfo.openstreetmap.org/keys/payment#values

• https://taginfo.openstreetmap.org/keys/payment:cash#values

rdfs:label get all the labels without language selection (as opposed to SERVICE wikibase:label).

update_pygments.py

[source]

Update pygments style

Call this script after each upgrade of pygments

class searxng_extra.update.update_pygments.Formatter(**options)

update_wikidata_units.py

[source]

Fetch units from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/wikidata_units.json (CI Update data …).