From f09459b98ae877d7dfd40d85c0edf058c65913d5 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Wed, 18 Dec 2019 16:11:05 +0100 Subject: [PATCH] doc: describe Makefile targets With the aim to simplify development cycles, started with PR #1756 a Makefile based boilerplate was added. This patch adds the missing developer documentation. Signed-off-by: Markus Heiser --- Makefile | 4 + docs/conf.py | 1 + docs/dev/contribution_guide.rst | 15 ++- docs/dev/index.rst | 1 + docs/dev/makefile.rst | 217 ++++++++++++++++++++++++++++++++ docs/dev/quickstart.rst | 26 +++- 6 files changed, 257 insertions(+), 7 deletions(-) create mode 100644 docs/dev/makefile.rst diff --git a/Makefile b/Makefile index b69202ba2..f35b86c41 100644 --- a/Makefile +++ b/Makefile @@ -8,6 +8,9 @@ PYOBJECTS = searx DOC = docs PY_SETUP_EXTRAS ?= \[test\] +PYDIST=./dist/py +PYBUILD=./build/py + include utils/makefile.include include utils/makefile.python include utils/makefile.sphinx @@ -23,6 +26,7 @@ help: @echo ' install - developer install (./local)' @echo ' uninstall - uninstall (./local)' @echo ' gh-pages - build docs & deploy on gh-pages branch' + @echo ' clean - drop builds and environments' @echo '' @$(MAKE) -s -f utils/makefile.include make-help @echo '' diff --git a/docs/conf.py b/docs/conf.py index c0bd246ac..64c7a715f 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -32,6 +32,7 @@ extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://') extlinks['patch'] = (GIT_URL + '/commit/%s', '#') extlinks['search'] = (SEARX_URL + '/%s', '#') extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ') +extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: ') extensions = [ 'sphinx.ext.extlinks', diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 064f28e63..6d8d3924d 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -79,6 +79,8 @@ Translation currently takes place on :ref:`transifex `. Please, do not update translation files in the repo. +.. _contrib docs: + Documentation ============= @@ -91,7 +93,7 @@ Documentation The documentation is built using Sphinx_. So in order to be able to generate the required files, you have to install it on your system. Much easier, use -Makefile our targets. +our :ref:`makefile`. Here is an example which makes a complete rebuild: @@ -101,6 +103,7 @@ Here is an example which makes a complete rebuild: ... The HTML pages are in dist/docs. +.. _make docs-live: live build ---------- @@ -110,9 +113,10 @@ live build It is recommended to assert a complete rebuild before deploying (use ``docs-clean``). -Live build is like WYSIWYG, If you want to edit the documentation, its -recommended to use. The Makefile target ``docs-live`` builds the docs, opens URL -in your favorite browser and rebuilds every time a reST file has been changed. +Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target ``docs-live`` builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed. .. code:: sh @@ -123,12 +127,13 @@ in your favorite browser and rebuilds every time a reST file has been changed. ... Start watching changes +.. _deploy on github.io: deploy on github.io ------------------- To deploy documentation at :docs:`github.io <.>` use Makefile target -``gh-pages``, which will builds the documentation, clones searx into a sub +:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub folder ``gh-pages``, cleans it, copies the doc build into and runs all the needed git add, commit and push: diff --git a/docs/dev/index.rst b/docs/dev/index.rst index 8e18066ca..93340c1db 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -11,3 +11,4 @@ Developer documentation search_api plugins translation + makefile diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst new file mode 100644 index 000000000..65cddd3ee --- /dev/null +++ b/docs/dev/makefile.rst @@ -0,0 +1,217 @@ +.. _makefile: + +================ +Makefile Targets +================ + +.. sidebar:: build environment + + Before looking deeper at the targets, first read about :ref:`makefile setup` + and :ref:`make pyenv`. + +With the aim to simplify development cycles, started with :pull:`1756` a +``Makefile`` based boilerplate was added. + +The usage is simple, just type ``make {target-name}`` to *build* a target. +Calling the ``help`` target gives a first overview:: + + $ make help + test - run developer tests + docs - build documentation + docs-live - autobuild HTML documentation while editing + run - run developer instance + install - developer install (./local) + uninstall - uninstall (./local) + gh-pages - build docs & deploy on gh-pages branch + clean - drop builds and environments + ... + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +.. _makefile setup: + +Setup +===== + +.. _git stash: https://git-scm.com/docs/git-stash + +The main setup is done in the :origin:`Makefile`:: + + export GIT_URL=https://github.com/asciimoo/searx + export SEARX_URL=https://searx.me + export DOCS_URL=https://asciimoo.github.io/searx + +.. sidebar:: fork & upstream + + Commit changes in your (local) branch, fork or whatever, but do not push them + upstream / `git stash`_ is your friend. + +:GIT_URL: Changes this, to point to your searx fork. + +:SEARX_URL: Changes this, to point to your searx instance. + +:DOCS_URL: If you host your own (branded) documentation, change this URL. + +.. _make pyenv: + +Python environment +================== + +.. sidebar:: activate environment + + ``source ./local/py3/bin/activate`` + +With Makefile we do no longer need to build up the virualenv manually (as +described in the :ref:`devquickstart` guide). Jump into your git working tree +and release a ``make pyenv``: + +.. code:: sh + + $ cd ~/searx-clone + $ make pyenv + PYENV usage: source ./local/py3/bin/activate + ... + +With target ``pyenv`` a development environment (aka virtualenv) was build up in +``./local/py3/``. To make a *developer install** of searx (:origin:`setup.py`) +into this environment make target ``install`` + +.. code:: sh + + $ make install + PYENV usage: source ./local/py3/bin/activate + PYENV using virtualenv from ./local/py3 + PYENV install . + +You have never to think about intermediate targets like ``pyenv`` or +``install``, the ``Makefile`` chains them as requisites. Just run your main +target. + +.. sidebar:: drop environment + + To get rid of the existing environment before re-build use :ref:`clean target + ` first. + +If you think, something goes wrong with your ./local environment or you change +the :origin:`setup.py` file (or the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call +:ref:`make clean`. + + +.. _make run: + +``make run`` +============ + +To get up a running a developer instance simply call ``make run``. This enables +*debug* option in :origin:`searx/settings.yml`, starts ``./searx/webapp.py`` +instance, disables *debug* option and opens the site (xdg-open): + +.. code:: sh + + $ make run + PYENV usage: source ./local/py3/bin/activate + PYENV install . + ./local/py3/bin/python ./searx/webapp.py + ... + INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) + ... + +.. _make clean: + +``make clean`` +============== + +Drop all intermediate files, all builds, but keep sources untouched. Includes +target ``pyclean`` which drops ./local environment. Before calling ``make +clean`` stop all processes using :ref:`make pyenv`. + +.. code:: sh + + $ make clean + CLEAN pyclean + CLEAN clean + +.. _make docs: + +``make docs docs-live docs-clean`` +================================== + +We describe the usage of the ``doc*`` targets in the :ref:`How to contribute / +Documentation ` section. If you want to edit the documentation +read our :ref:`make docs-live` section. If you are working in your own brand, +adjust your :ref:`Makefile setup `. + + +.. _make gh-pages: + +``make gh-pages`` +================= + +To deploy on github.io first adjust your :ref:`Makefile setup `. For any further read :ref:`deploy on github.io`. + +.. _make test: + +``make test`` +============= + +Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does +additional :ref:`pylint checks `. You can run tests selective, +e.g.: + +.. code:: sh + + $ make test.pep8 test.unit + . ./local/py3/bin/activate; ./manage.sh pep8_check + [!] Running pep8 check + . ./local/py3/bin/activate; ./manage.sh unit_tests + [!] Running unit tests + +.. _make pylint: + +``make pylint`` +=============== + +.. _Pylint: https://www.pylint.org/ + +Before commiting its recommend to do some (more) linting. Pylint_ is known as +one of the best source-code, bug and quality checker for the Python programming +language. Pylint_ is not yet a quality gate within our searx project (like +:ref:`test.pep8 ` it is), but Pylint_ can help to improve code +quality anyway. The pylint profile we use at searx project is found in +project's root folder :origin:`.pylintrc`. + +Code quality is a ongoing process. Don't try to fix all messages from Pylint, +run Pylint and check if your changed lines are bringing up new messages. If so, +fix it. By this, code quality gets incremental better and if there comes the +day, the linting is balanced out, we might decide to add Pylint as a quality +gate. + + +``make pybuild`` +================ + +.. _PyPi: https://pypi.org/ +.. _twine: https://twine.readthedocs.io/en/latest/ + +Build Python packages in ``./dist/py``. + +.. code:: sh + + $ make pybuild + ... + BUILD pybuild + running sdist + running egg_info + ... + $ ls ./dist/py/ + searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz + +To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs +twine_ to be installed. Since you are not the owner of :pypi:`searx` you will +never need the latter. diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index a4a37a266..e40772b3b 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -4,15 +4,23 @@ Development Quickstart ====================== +.. sidebar:: :ref:`makefile` + + For additional developer purpose there are :ref:`makefile`. + This quickstart guide gets your environment set up with searx. Furthermore, it gives a short introduction to the ``manage.sh`` script. How to setup your development environment ========================================= +.. sidebar:: :ref:`make pyenv ` + + Alternatively use the :ref:`make pyenv`. + First, clone the source code of searx to the desired folder. In this case the source is cloned to ``~/myprojects/searx``. Then create and activate the -searx-ve virtualenv and install the required packages using manage.sh. +searx-ve virtualenv and install the required packages using ``manage.sh``. .. code:: sh @@ -27,6 +35,10 @@ searx-ve virtualenv and install the required packages using manage.sh. How to run tests ================ +.. sidebar:: :ref:`make test.unit ` + + Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets. + Tests can be run using the ``manage.sh`` script. Following tests and checks are available: @@ -41,7 +53,8 @@ For example unit tests are run with the command below: ./manage.sh unit_tests -For further test options, please consult the help of the ``manage.sh`` script. +For further test options, please consult the help of the ``manage.sh`` script or +read :ref:`make test`. How to compile styles and javascript @@ -97,6 +110,11 @@ After installing grunt, the files can be built using the following command: Tips for debugging/development ============================== +.. sidebar:: :ref:`make run` + + Makefile target ``run`` already enables debug option for your developer + session / see :ref:`make run`. + Turn on debug logging Whether you are working on a new engine or trying to eliminate a bug, it is always a good idea to turn on debug logging. When debug logging is enabled a @@ -104,6 +122,10 @@ Turn on debug logging message. It can be turned on by setting ``debug: False`` to ``debug: True`` in :origin:`settings.yml `. +.. sidebar:: :ref:`make test` + + Alternatively use the :ref:`make test` targets. + Run ``./manage.sh tests`` before creating a PR. Failing build on Travis is common because of PEP8 checks. So a new commit must be created containing these format fixes. This phase can be skipped if