Commit Graph

5575 Commits

Author SHA1 Message Date
Markus Heiser
25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00
Markus Heiser
03d5d14d98 [mod] utils/manage_static.sh: remove git_log_current_branch
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
80710c666f [mod] utils/manage_static.sh: simplify git restore actions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Markus Heiser
2cb1f35086 [mod] utils/manage_static.sh: add script to shellcheck
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)

No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:13:31 +02:00
Alexandre Flament
0606cde105 [mod] add make targets to manage the build files in the /searx/static directory
The idea is to avoid conflict when there are differents branches with changes are made on the static files.
A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies.
So the solution in this commit keep the sources and the build files in the same git repository.

In one branch, the modification of the source (*.less, *.js) are commited without the built files.
The built files are commited in a uniq commit, with a commit message "Static build"
In case of merge or rebase, this commit can be dropped.

New make targets:
* static.build.commit.drop:
    drop the last "Static build".
    The command checks that there are only build files in the commit.
* static.build.commit :
    call "make static.build.commit.drop"
    call "make themes.all"
    commit the files
* static.git.restore.staged:
    git restore --staged <build files>
* static.git.restore:
    git restore <build files>

Related to https://github.com/searxng/searxng/issues/137
2021-06-26 13:13:31 +02:00
Alexandre Flament
4b60c557a9
Merge pull request #177 from unixfox/patch-1
Set use_mobile_ui to false for Google engine
2021-06-26 12:19:21 +02:00
Émilien Devos
63ebe00554
Set use_mobile_ui to false for Google engine 2021-06-26 09:58:35 +00:00
Alexandre Flament
560e503285
README.rst: remove searx logo. 2021-06-25 11:10:54 +02:00
Markus Heiser
d19869b9f7
Merge pull request #173 from return42/fix-lxc-iptables
[mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
2021-06-25 08:23:40 +00:00
Alexandre Flament
c91603dae9
Merge pull request #168 from return42/stylelint
Lint the less files of the simple theme
2021-06-25 09:02:35 +02:00
Markus Heiser
a3f58a3a3d [emacs] .dir-locals.el: add setup for js-mode
- move jshint option from gruntfile to .jshintrc
- remove trailing-whitespace from gruntfile and
- add jshint esversion: 6
- .dir-locals.el add locals for js-mode to use JSHint from the simple theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser
f73a00dcee [stylelint] disable role 'no-descending-specificity'
This patch disables role 'no-descending-specificity'.  IMO it is better to have
this rule active (see below [1]), but it is hard to rewrite the less files to
pass this rule, so for the first I chose to disable this rule.

---

Source order is important in CSS, and when two selectors have the same
specificity, the one that occurs last will take priority. However, the situation
is different when one of the selectors has a higher specificity. In that case,
source order does not matter: the selector with higher specificity will win out
even if it comes first.

The clashes of these two mechanisms for prioritization, source order and
specificity, can cause some confusion when reading stylesheets. If a selector
with higher specificity comes before the selector it overrides, we have to think
harder to understand it, because it violates the source order
expectation. Stylesheets are most legible when overriding selectors always come
after the selectors they override. That way both mechanisms, source order and
specificity, work together nicely.

This rule enforces that practice as best it can, reporting fewer errors than it
should. It cannot catch every actual overriding selector, but it can catch
certain common mistakes.

[1] https://stylelint.io/user-guide/rules/list/no-descending-specificity/

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser
0b7d03c694 [stylelint] simple theme: fix some errors reported by stylelint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser
14b09c15c4 [fix] simple theme: use stylint to fix common lint errors
This fix was autogenerated by::

     npx stylelint -f unix --fix 'searx/static/themes/simple/src/less/**/*.less'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser
dfc5c3bc15 [mod] add stylelint process to target themes.simple
Before build, do lint the LESS files using stylelint [1].  The configuration
'stylelint-config-standard' [2] is added to the root of the simple theme [3].

[1] https://stylelint.io/
[2] https://github.com/stylelint/stylelint-config-standard
[3] https://stylelint.io/user-guide/configure

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Alexandre Flament
e73a96b5f9
Merge pull request #172 from return42/fix-searx.sh
[fix] utils/searx.sh: signature of YAML settings has been changed
2021-06-23 15:02:08 +02:00
Markus Heiser
c8a6548592 [mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC.  With this patch:

- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
  suggestion for a solution a solution

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 12:47:52 +02:00
Markus Heiser
b5f5ddd5a6 [fix] utils/searx.sh: signature of YAML settings has been changed
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this
into account.

Related-to: 5a7b12ee [yamllint] searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23 09:56:59 +02:00
Markus Heiser
056550d523
Merge pull request #169 from return42/tidy-ignorelists
[fix] tidy up ignore lists .gitignore & .dockerignore
2021-06-22 20:21:30 +00:00
Markus Heiser
98d1661f72
Merge pull request #171 from searxng/build_static_cleanup
Build static cleanup
2021-06-22 19:59:01 +00:00
Alexandre Flament
5742267e35 [fix] update .gitattributes 2021-06-22 20:34:39 +02:00
Alexandre Flament
915f3b0bdd [fix] Makefile: remove themes.bootstrap 2021-06-22 20:34:25 +02:00
Markus Heiser
5081cd4ed7
Merge pull request #170 from return42/emacs
[emacs] .dir-locals.el: add setup for json-mode
2021-06-22 15:48:18 +00:00
Markus Heiser
5a8fed48c9 [emacs] .dir-locals.el: add setup for json-mode
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 17:46:46 +02:00
Markus Heiser
7ef6ffa803 [fix] tidy up ignore lists .gitignore & .dockerignore
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 16:55:30 +02:00
Alexandre Flament
2530576f24
Merge pull request #157 from searxng/mod_static_build
static build: move generated files to searx/static/themes/*/src/generated
2021-06-22 10:36:19 +02:00
Markus Heiser
c1f72b71ba [fix] .gitattributes: bootstrap.css & .js has been moved to oscar
see 86912e22 [mod] oscar: get bootstrap and typeahead from NPM

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:24:05 +02:00
Markus Heiser
1559ed386e [fix] pygments.less - remove pyenv_OK condition
BTW: add pygments.less to the help message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:22:33 +02:00
Alexandre Flament
04492b76de [themes] ignore packages-lock.json 2021-06-22 08:17:06 +02:00
Alexandre Flament
2bc9b00cd7 Static build 2021-06-22 08:15:17 +02:00
Alexandre Flament
dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Alexandre Flament
d20f6a1f19 [mod] make themes.all update pygments*.less
Add a searx/static/themes/*/src/generated empty folder
2021-06-22 08:15:11 +02:00
Alexandre Flament
fb2f4ba902
Merge pull request #164 from searxng/docker_upgrade_alpine
Docker: upgrade to alpine 3.14
2021-06-22 07:27:19 +02:00
Markus Heiser
089b97156a
Merge pull request #167 from return42/fix-166
[fix] UnicodeEncodeError for preferences ULR
2021-06-21 23:37:09 +00:00
Markus Heiser
5973491701 [fix] UnicodeEncodeError for preferences ULR
Closes: https://github.com/searxng/searxng/issues/166

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 01:35:54 +02:00
Alexandre Flament
f3e56836d6
Merge pull request #165 from return42/patch-google
improve & document google engine
2021-06-21 20:12:07 +02:00
Markus Heiser
0ef6aa5126 [docs] add documentation from the sources of the google engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 18:25:52 +02:00
Alexandre Flament
ef12c41206
Docker: upgrade to alpine 3.14
Uses Python version 3.9 instead of 3.8
2021-06-21 18:03:54 +02:00
Markus Heiser
05e90f2e57 [fix] google answers: normalize space of the answers.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:25 +02:00
Markus Heiser
f096d68ec6 [mod] google engine: reduce mobile UI parameters to what is needed
Reverse engineering shows that not all of the parameters used by google's mobile
UI (aka "more results" button) are needed [1].

[1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:23 +02:00
Alexandre Flament
f4da4bafa7
Merge pull request #160 from searxng/google_mobile_ui
[experimental] google: use the mobile UI
2021-06-21 14:56:50 +02:00
Alexandre Flament
bf5f505d74
Merge pull request #161 from return42/fix-typo-manage
[fix] py.build - fix small typo in the build message
2021-06-21 14:55:28 +02:00
Alexandre Flament
7a5c36408a [mod] google: add "use_mobile_ui" parameter to use mobile endpoint.
disable by default, it has to be enabled in settings.yml

related to  #159
2021-06-21 14:52:04 +02:00
Markus Heiser
f41b7a7204 [fix] py.build - fix small typo in the build message
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 12:52:26 +02:00
Alexandre Flament
bb3e67f72b [mod] simple theme: normalize.css becomes a packages.json dependency 2021-06-21 08:28:29 +02:00
Alexandre Flament
0e3a87b73a
Merge pull request #158 from return42/fix-node.clean
[fix] node.clean - ignore npm dependencies when npm is not installed
2021-06-20 07:20:38 +02:00
Markus Heiser
a62e1123e2 [fix] node.clean - ignore npm dependencies when npm is not installed
error pattern::

    $ make clean
    CLEAN     pyenv
    PYENV     [virtualenv] drop local/py3
    CLEAN     docs -- build/docs dist/docs
    CLEAN     locally installed npm dependencies
    ./manage: line 318: npm: command not found
    ERROR: node.clean exit with error (127)
    make: *** [Makefile:90: node.clean] Error 127

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-19 16:58:18 +02:00
Alexandre Flament
eb6832eb95
Merge pull request #156 from return42/fix-google-news
Fix google news
2021-06-18 14:01:03 +02:00
Markus Heiser
9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 13:21:20 +02:00
Markus Heiser
dd7b53d369 [fix] google-news engine - KeyError: 'hl in request
Since we added

- 1c67b6aec [enh] google engine: supports "default language"

there is a KeyError: 'hl in request,error pattern::

    ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
    Traceback (most recent call last):
      File "searx/search/processors/online.py", line 144, in search
        search_results = self._search_basic(query, params)
      File "searx/search/processors/online.py", line 118, in _search_basic
        self.engine.request(query, params)
      File "searx/engines/google_news.py", line 97, in request
        if lang_info['hl'] == 'en':
      KeyError: 'hl'

Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 11:34:11 +02:00