Other optional parameter ..
`&sort=crawl_date`
can be appended to search_string to sort results by date.
`&domain=example.org`
can be implemented to search_string to get results from just one domain.
Public instances could get relatively fast timed-out for 3600s.
--
Merged from @allendema's commit [1] and slightly modfied / see [2].
Related-to: [1] 455b2b4460
Related-to: [2] https://github.com/searx/searx/pull/3040
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Check 'using_tor_proxy' for each engine individually instead of checking globally
[fix] searx.network: update _rdns test to the last httpx version
Co-authored-by: Alexandre Flament <alex@al-f.net>
The macro "checkbox" in macros.html uses the macro "icon_small"
from icons.html
The commit imports icon_small in macros.html to fix the issue.
It works because the macros in macros.html are imported with the Jinja2 context.
See https://jinja.palletsprojects.com/en/3.0.x/templates/#import-visibilityclose#819
Engine description can be configured, this is needed e.g. by custom search
engines. Here is an example of a command engine with a description in the about
section::
- name: locate
engine: command
command: ['locate', '{{QUERY}}']
disabled: true
categories: files
about:
description: local files
website: 'https://www.man7.org/linux/man-pages/man1/locate.1.html'
delimiter:
chars: ' '
keys: ['line']
Closes: https://github.com/searxng/searxng/issues/788
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Use httpx.Response.json() to avoid charset_normalizer issues:
DEBUG charset_normalizer : override steps (5) and chunk_size (512) as content does not fit (153 byte(s) given) parameters.
INFO charset_normalizer : ascii passed initial chaos probing. Mean measured chaos is 0.000000 %
DEBUG charset_normalizer : ascii should target any language(s) of ['Latin Based']
INFO charset_normalizer : ascii is most likely the one. Stopping the process.
[1] https://www.python-httpx.org/api/#response
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Currently we have two kinds of user documentation:
* the about page[1] which is written in HTML and part of the web
application and can therefore link instance-specific pages
(like e.g. the preferences) via Jinja variables
* the Sphinx documentation[2] which is written in reStructuredText
and cannot link instance-specific pages since it doesn't know
which instance the user is using
The plan is to integrate the user documentation currently in Sphinx
into the application, so that it can also link instance specific pages.
We also want to enable the user documentation to be translated.
This commit implements the first step in this endeavor (see #722).
[1]: searx/templates/__common__/about.html
[2]: docs/user/ (currently served at https://docs.searxng.org/user/)
Since https://github.com/searxng/searxng/pull/354
the searx.network.stream(...) returns a tuple
This commits update the checker code according to
this function signature change.
webapp.py monkey-patches the Flask request global.
This commit adds a type cast so that e.g. Pyright[1]
doesn't show "Cannot access member" errors everywhere.
[1]: https://github.com/microsoft/pyright
* mirror all inline SVGs so that direction SVGs display correctly on RTL
* set the bold list element in info box to RTL so the colon gets displayed on the right side
* set correct .ltr function for the left border on the search button in #q
* move text to the right in autocomplete
* move search form in lign with result article on RTL
* add the correct padding for img thumbnails in categories like music on RTL
* apply RTL to result table for map results
* align text in tables part of /preferences on RTL
* move burger menu on index page to the left on RTL
* fix positioning of drop down arrow on select boxes on RTL
* align result URL on the right (written LTR)
* align vim hotkeys help on the left since it is not translated
* image detail:
* labels (author, format, URL, etc...) are written on the right,
values are on the left.
* URL are written LTR and overflow on the right
The less grunt runner silently ignore missing files and continue with the build[1]::
Running "less:production" (less) task
>> Destination css/searxng.min.css not written because no source files were found.
>> 1 stylesheet created.
>> 1 sourcemap created.
Add filter function that calls grunt.fail() if the scr file does not exists.
[1] https://github.com/searxng/searxng/pull/750#discussion_r784357031
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Bangs with a `*` suffix (e.g. `!!d*`) overwrite Bangs with the same
prefix (e.g. `!!d`) [1]. This can be avoid when a non printable character is
used to tag a LEAF_KEY.
[1] https://github.com/searxng/searxng/pull/740#issuecomment-1010411888
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
There is an issue with redis v4.1.0 [1] / for the interim lets remove this
python dependency.
[1] https://github.com/searxng/searxng/issues/741
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
An ambiguous bang like `!!d` raises an exception in function get_bang_url(). A
bang is only unique when the bang_definition from get_bang_definition_and_ac() is
a string / for a ambiguous bang the returned bang_definition is a dictionary.
Reported-by: user prg at #searxng:matrix.org on 2022/01/11
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In case of CAPTCHA raise a SearxEngineCaptchaException and suspend for 7 days.
When get_sc_code() fails raise a SearxEngineResponseException and suspend for 7
days.
[1] https://github.com/searxng/searxng/pull/695
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Startpage has introduced new anti-scraping measures that make SearXNG instances
run into captchas:
1. some arguments has been removed and a new `sc` has been added.
2. search path changed from `do/search` to `sp/search`
3. POST request is no longer needed
Closes: https://github.com/searxng/searxng/issues/692
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:
/v1/images? -->/ v1/images/?
otherwise it redirects, here is the debug log:
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
WARNING searx.engines.openverse : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation of the etools engine is poor. No date-range support, no
language support and it is broken by a CAPTCHA.
etools is a metasearch engine, the major search engines it supports (google,
bing, wikipedia, Yahoo) are already available in SeaarXNG.
While etools does support several engines we currently don't support directly,
support for them should be added directly to SearXNG if there is demand.
In practice: in SearXNG the worse etools results will be mixed with good results
from other engines we have (as long as there is no captcha).
At best case, what we win with etools is in e.g. results from de.ask.com in a
query from a german request .. in all other cases worse results are bubble up in
SearXNG's result list.
[1] https://github.com/searxng/searxng/issues/696#issuecomment-1005855499
Closes: https://github.com/searxng/searxng/issues/696
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The previous implementation used two hash sets and a list.
... that's not necessary ... a single hash map suffices.
And it's also less error prone ... because the previous data structure
allowed a setting to be enabled and disabled at the same time.
Previously the default_value was abused for the cookie name.
Having SwitchableSetting subclass Setting doesn't even make sense
in the first place since none of the Setting methods apply.
The ? search operator has been broken for some time and
currently only raises the question why it's still there.
## Context ##
The query "Paris !images" searches for "Paris" in the "images" category.
Once upon a time Searx supported "Paris ?images" to search for "Paris"
in the currently enabled categories and the "images" category.
The feature makes sense ... the ? syntax does not.
We will hopefully introduce a +!images syntax in the future.
Fixes#702.
* allow not to record metrics (response time, etc...)
* this commit doesn't change the UI. If the metrics are disabled
/stats and /stats/errors will return empty response.
in /preferences, the columns response time and reliability will be empty.
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:
categories_as_tabs:
general:
icon: search-outline
These dictionaries are no longer part of the general category,
so they're no longer queried by default -> we can enable them
by default without degrading general query performance.
The general category is the category that is searched by default.
From a privacy standpoint it doesn't make sense to send all general
queries to specialized search engines that cannot deal with those
queries anyway.
Previously we didn't have a good place to put search engines that don't
fit into any of the tab categories. This commit automatically puts
search engines that don't belong to any tab category in an "other"
category, that is only displayed in the user preferences (and not above
search results).
Add a redis connector, the default DB connector is a socket at::
unix:///usr/local/searxng-redis/run/redis.sock?db=0
To set up a redis instance simply use::
$ ./manage redis.build
$ sudo -H ./manage redis.install
A hint for developers:
To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.
This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs can still be searched with !bangs.
Add event listener to query selector::
'#urls img.image'
From the user point of view, I think it is better to hide the image:
img_load_error.svg is helplful in the image category because it still allows to
select the image. IMO, in the news category, the fact there is a missing image
won't help to choose the links. From a developer point of view, the place holder
is signal that may be the engine needs to be updated (at least give a look). The
browser console should show the same information too, but it requires some
additional steps. [1]
[1] https://github.com/searxng/searxng/pull/610#issuecomment-997640132
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix pylint issues from commit (3d96a983)
[format.python] initial formatting of the python code
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Follow up queries for the pages needed to be fixed.
- Split search-term in one for initial query and one for following queries.
- Set some headers in HTTP requests, bing needs for paging support.
- IMO //div[@class="sa_cc"] does no longer match in a bing response.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix remarks from pylint and improved code-style. In preparation for a bug-fix
of the Bing (Web) engine I add this engine to the pylint-list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Images should include dimension attributes. Without `width` and `height`
specified, image dimensions are 0×0 pixels at first. ... In this case the
browser determines that all of them are visible to the user and decides to load
everything [1].
In CSS the `width` is set to a value and the `height` is unsed to scale the image
proportional in both dimensions.
[1] https://web.dev/browser-level-image-lazy-loading/#images-should-include-dimension-attributes
[2] https://caniuse.com/loading-lazy-attr
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::
make themes.simple
make themes.simple.test
In the github CI the `themes.all` target enforce a `themes.simple`.
BTW: Remove 'jshint' left overs from 0ee316f3d
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
automatically fix some of the problems reported by eslint rules::
$ ./manage nvm.bash
nvm-env$ npm --prefix searx/static/themes/simple run eslint-fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All three filters (`language`, `time_range` and `safesearch`) are rendered in
one line. A size of 45% for `language` and `time_range` left only 10% for the
`safesearch` filter. Solution: drop with from `language` and `time_range`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The CSP issue is, that the `_Position` function in the autocomplete-js set the
style attributes by `setAttribute("style", ...)`. Using `setAttribute` to set
the style attribute invokes the HTML parser and CSP is triggered [1].
This patch overwrite the `_Position` function of autocomplete-js.
BTW: remove trailing whitespace
[1] https://stackoverflow.com/a/57633533
Closes: https://github.com/searxng/searxng/issues/352
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Include the source files directly in the `.map` files [1] for two reasons:
1. In the simple theme, modules from `./node_modules` are only available on
developer's build host (a `make node.env` is needed to get the sources). By
example; in the `searxng.min.js.map` file, the `autocomplete.js` is linked at
URL:
../node_modules/autocomplete-js/dist/autocomplete.js
2. Later, when client sources has been moved out of the `./static` URL [2] to
the `<repo-roo>/src` folder, all JS sources are no longer available.
Drawback: `.map` files grow up in size, but this should not bother, because the
files are only loaded in debug mode.
[1] https://github.com/gruntjs/grunt-contrib-uglify#sourcemapincludesources
[2] https://github.com/searxng/searxng/issues/141
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
remove:
* searx/static/themes/simple/js/searxng.head.js
* searx/static/themes/simple/js/searxng.js
* searx/static/themes/simple/css/searxng-rtl.css
* searx/static/themes/simple/css/searxng.css
These files are never used.
About the js files: the .map files references the sources instead of the concated version.
In window.searxng.theme the theme data can be passed through to JS
implementations.
Initial the window.searxng.theme namespace starts with a value for
`img_load_error`::
// image that is displayed if load of <img src='...'> failed
img_load_error: 'img/img_load_error.svg'
The searx/static/themes/__common__/js/image_layout.js is patched to uses the
value, if the theme defines a value for img_load_error in this namespace.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The *background color* of an transparent image is the color of the color of
wallpaper on which all images are shown on.
Set variable `--color-result-image-background` and `--color-base-background` to
the same value.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* move `searx/static/themes/simple/img/searxng.svg` to `src/brand/searxng.svg`
* README.rst can use it without a reference to a theme.
* the simple theme can create `searx/static/themes/simple/img/searxng.png` using
the svg2png task
Suggested-by: @dalf https://github.com/searxng/searxng/pull/561#issuecomment-981747902
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This is a follow up of 9a3253fc escaping `<` and `>` in all themes. This patch
fix issue in oscar theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- remove unused 'path' constant
- fix indentation
- Put all image tasks at the end, where it is easy to list which image is processed.
Suggested -by: @dalf https://github.com/searxng/searxng/pull/541#discussion_r756247157
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the video search, google also sometimes includes news. E.g. in the DE
language when you search for `!gov paris`, google adds an article from a german
newspaper (FAZ), I assume these are sponsored link (not tagged advertisement?)
Those links do not have an image / this patch ignores *video links* wqithout an
image ID.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Strings like::
'Query in the page\'s title'
are hard to read / remove escape sequence by using double quotation marks for
strings ::
"Query in the page's title"
BTW: remove a leading dot in the simple theme [1].
[1] 80fb77476f (r756112716)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Support for the prefixed version (-moz-border-radius) was removed in Gecko
13.0 (Firefox 13.0 / Thunderbird 13.0 / SeaMonkey 2.10) [1].
[1] https://developer.mozilla.org/pt-BR/docs/Web/CSS/border-radius
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* rework selection UI in pref (fix based on: 78643e9f43)
* moved search filters underneath categories
* cut params from url and replace / with ›
* make h3 and url in article bigger
* add safe search select to search filter (this will not override settings and only be valid while on result page in a session)
* make search form button not overlap each other when js is disabled
* 1rem padding around preview image and thumbnail in default article template
Sometimes there is no href in the `<a ..>` tag of a *link_node* [1].
[1] https://github.com/searxng/searxng/issues/532
Reported-by: @TheEssem
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Folder src/brand is intended to place logo, wordmark and more in. This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder
About the new created src/ folder:
On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since digg no longer works, we do nat have a active engine in the social-media
category. Enable reddit by default to have at least one engine back in this
category.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
when on the top of the page, the image is just below the search fields.
this commit slightly aligh the top of the image detail with the bottom
of the search fields.
the width of #main_results #results.image-detail-open.only_template_images
was set to 59.25rem. On mobile, the images overflowed on right side of
the viewport, which creates a horizontal scroll.
This commit set the value to min(98%, 59.25rem) whatever the max-width is.
* update from ionicons-npm to ionicons ver.5
* drop the webfont built by grunt for icons
* built icons.html template for inlining icons with jinja2 into html
* update icon to use mostly the outline version
* add icons to categories and do not display them on mobile to save space
* remove all legacy ion icon font files from simple theme
* icons.html is added in this commit since make statc.build.restore requires git to know the file already
* cleanup error-dialog
new bash function convert_if_newer() / usage::
convert_if_newer <origfile> <outfile> [<options>, ...]
convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64
Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>. The command line is to convert is::
convert <origfile> [<options>, ...] <outfile>
PNG 'searx/static/themes/simple/img/favicon.png' has been created by::
$ make themes.simple
CONVERT searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* remove hover effect from vim help modal
* remove bold font weight from active item in prefenreces tab bar
* remove margin from answer box to unify with rest of result page
* disable by default
* settings.yml: ui.query_in_title
* in /preferences: privacy tab
when enabled, the result page's title contains the user query.
previously:
* oscar theme: the query was always included
* simple theme: the query was included with the GET method
Before this commit, the default click event on an image result is prevented,
this include clicks inside the detail.
This commit makes sure the click happends outside the detail to prevent the default event.
buttons:
* previous & next in the result page
* save, restore, & back in the preferences
* back to top
<select> input in Chrom* browsers:
* fix the white text with a white background issue
It is not possible to use CSS variable in a SVG when this is in a background.
This commit adds two .svg files, less converts them into data URL.
The two files are indentical except the fill color.
* remove vim arrow
* add 1rem padding to results
* add 0.2rem left border to vim selected article
* set column gap to 1.2rem and make search bar in line with results
* put 10px border-radius selected article
* result article: 0.125rem margin on tablet and esktop; 1rem margin on phone
* index page: margin top is 24% of the view port.
avoid to scroll a small screen,
center the content in the middle of the screen
* link to preferences at the same height same the input fields
* increase the category tab heighs
* increase the margin bottom of the query field
* in the results, change the h3 margin top and bottom to 0.4rem (6px)
* move the back to top button slightly on the right when the results are only images
When an image is selected, the detail with the full size image is displayed
on the right side of the screen (or full screen on tablet and phone).
When Javascript is disabled, the thumbnail is a linked to the full size image,
as it was before.
When the image proxy is enabled, the full size image is also proxied,
in consequence this commit increases the bandwidth usage of instances.
The detail can be closed by the close button or the Esc key.
It is possible to go to the next and previous images using the j and k keys
or the button on the top right of the screen.
* url in article head is clickable
* url is bigger now 0.96em font
* url is now left floating on tablet and phone
* there is a 8px top and bottom margin on h3 result heading
* rework search form as grid layout
* remove various paddings and margins
* add logo and link to index to search form
* make categories bigger on phone
Languages are supported by mapping the language to a domain. If domain is not
found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used.
BTW: fix issue reported at https://github.com/searx/searx/issues/3020
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
before this commit, when the user picks "Chinese, China" or "Chinese, Taiwan",
the locale fallbacks to English.
This commit fixes this issue (the translations branch must be updated)
close#339
* clean up vars in defenition
* results look now the same on mobile and desktop
* reworked results on mobile
* new color theme with more vibrant colors
* remove vars and add elements to base and btn vars
* change default border radius to 10px and padding to 0.7em
* put border radius and padding on search input form, infoxbox and buttons
* remove unused .help class in #categories_container
* remove active background from tabs to straemline design
* redo search form: 10px padding
* 2rem margin on search results on desktop
* fix modal pacement of engine reliability in prefs
* use darker accent colors
* streamline autocomplete with more padding and a hover effect
The key of the dictionary 'searx.data.ENGINES_LANGUAGES' is the *engine name*
configured in settings.xml. When multiple engines are configured to use the
same origin engine (e.g. `engine: google`)::
- name: google
engine: google
use_mobile_ui: false
...
- name: google italian
engine: google
use_mobile_ui: false
language: it
...
- name: google mobile ui
engine: google
shortcut: gomui
use_mobile_ui: true
There exists no entry for ENGINES_LANGUAGES[engine.name] (e.g. `name: google
mobile ui` or `name: google italian`). This issue can be solved by recreate the
ENGINES_LANGUAGES::
make data.languages
But this is nothing an SearXNG admin would like to do when just configuring
additional engines, since this just doubles entries in ENGINES_LANGUAGES and
BTW: `make data.languages` has various external requirements which might be not
installed or not available, on a production host.
With this patch, if engine.name fails, ENGINES_LANGUAGES[engine.engine] is used
to get the engine.supported_languages (e.g. `google` for the engine named
`google mobile`).
For an engine, when there is `language: ...` in the YAML settings, the engine
supports only one language, in this case engine.supported_languages should
contains this value defined in settings.yml (e.g. `it` for the engine named
`google italian`).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Closes: https://github.com/searxng/searxng/issues/384
The utils.load_module() function is used to load a python file (aka module) and
return the module's namespace. SearXNG uses this function to load *engines and
answerers* from arbitrary locations with arbitrary modifications. These are not
real python modules and it is not intended to mix this *engines and answerers*
with the python modules registered in sys.modules.
Closes: https://github.com/searxng/searxng/issues/312
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/312
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Implement a scrapper for DuckDuckGo-Lite [1]. The existing DuckDuckGo [2]
engine does not support paging. DuckDuckgo-Lite is much faster, less verbose
and does have a paging option (reversed engineered from the input form of [1]).
[1] https://lite.duckduckgo.com/lite
[2] https://duckduckgo.com/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* download images using the "image_proxy" network (HTTP/1 instead of HTTP/2)
* don't cache data: URL (reduce memory usage)
* after each test: purge image URL cache then call garbage collector
* download only the first 64kb of images
Martin Fischer