[doc] rearranges Settings & Engines docs for better readability

We have built up detailed documentation of the *settings* and the *engines* over
the past few years.  However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.

This patch rearranges the Settings & Engines documentation for better
readability.

To review new ordered docs::

   make docs.clean docs.live

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

docs/admin/buildhosts.rst

@@ -9,7 +9,7 @@ Buildhosts
    If you have any contribution send us your :pull:`PR <../pulls>`, see
    :ref:`how to contribute`.
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/engines/command-line-engines.rst

@@ -1,79 +0,0 @@
-.. _engine command:
-
-====================
-Command Line Engines
-====================
-
-.. sidebar:: info
-
-   - :origin:`command.py <searx/engines/command.py>`
-   - :ref:`offline engines`
-
-With *command engines* administrators can run engines to integrate arbitrary
-shell commands.
-
-When creating and enabling a ``command`` engine on a public instance, you must
-be careful to avoid leaking private data.  The easiest solution is to limit the
-access by setting ``tokens`` as described in section :ref:`private engines`.
-
-The engine base is flexible.  Only your imagination can limit the power of this
-engine (and maybe security concerns).  The following options are available:
-
-``command``:
-  A comma separated list of the elements of the command.  A special token
-  ``{{QUERY}}`` tells where to put the search terms of the user. Example:
-
-  .. code:: yaml
-
-     ['ls', '-l', '-h', '{{QUERY}}']
-
-``delimiter``:
-  A mapping containing a delimiter ``char`` and the *titles* of each element in
-  ``keys``.
-
-``parse_regex``:
-  A dict containing the regular expressions for each result key.
-
-``query_type``:
-
-  The expected type of user search terms.  Possible values: ``path`` and
-  ``enum``.
-
-  ``path``:
-    Checks if the user provided path is inside the working directory.  If not,
-    the query is not executed.
-
-  ``enum``:
-    Is a list of allowed search terms.  If the user submits something which is
-    not included in the list, the query returns an error.
-
-``query_enum``:
-  A list containing allowed search terms if ``query_type`` is set to ``enum``.
-
-``working_dir``:
-
-  The directory where the command has to be executed.  Default: ``./``
-
-``result_separator``:
-  The character that separates results. Default: ``\n``
-
-The example engine below can be used to find files with a specific name in the
-configured working directory:
-
-.. code:: yaml
-
-  - name: find
-    engine: command
-    command: ['find', '.', '-name', '{{QUERY}}']
-    query_type: path
-    shortcut: fnd
-    delimiter:
-        chars: ' '
-        keys: ['line']
-
-
-Acknowledgment
-==============
-
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/index.rst

@@ -1,26 +0,0 @@
-.. _engines and settings:
-
-==================
-Engines & Settings
-==================
-
-.. sidebar:: Further reading ..
-
-   - :ref:`settings engine`
-   - :ref:`engine settings` & :ref:`engine file`
-
-.. toctree::
-   :maxdepth: 3
-
-   settings
-
-.. toctree::
-   :maxdepth: 1
-
-   private-engines
-   recoll
-   sql-engines
-   nosql-engines
-   search-indexer-engines
-   command-line-engines
-   searx.engines.xpath

docs/admin/engines/private-engines.rst

@@ -1,49 +0,0 @@
-.. _private engines:
-
-============================
-Private Engines (``tokens``)
-============================
-
-Administrators might find themselves wanting to limit access to some of the
-enabled engines on their instances. It might be because they do not want to
-expose some private information through :ref:`offline engines`.  Or they would
-rather share engines only with their trusted friends or colleagues.
-
-To solve this issue the concept of *private engines* exists.
-
-
-A new option was added to engines named `tokens`. It expects a list of
-strings. If the user making a request presents one of the tokens of an engine,
-they can access information about the engine and make search requests.
-
-Example configuration to restrict access to the Arch Linux Wiki engine:
-
-.. code:: yaml
-
-  - name: arch linux wiki
-    engine: archlinux
-    shortcut: al
-    tokens: [ 'my-secret-token' ]
-
-
-Unless a user has configured the right token, the engine is going
-to be hidden from him/her. It is not going to be included in the
-list of engines on the Preferences page and in the output of
-`/config` REST API call.
-
-Tokens can be added to one's configuration on the Preferences page
-under "Engine tokens". The input expects a comma separated list of
-strings.
-
-The distribution of the tokens from the administrator to the users
-is not carved in stone. As providing access to such engines
-implies that the admin knows and trusts the user, we do not see
-necessary to come up with a strict process. Instead,
-we would like to add guidelines to the documentation of the feature.
-
-
-Acknowledgment
-==============
-
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/recoll.rst

@@ -1,50 +0,0 @@
-.. _engine recoll:
-
-=============
-Recoll Engine
-=============
-
-.. sidebar:: info
-
-   - `Recoll <https://www.lesbonscomptes.com/recoll/>`_
-   - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
-   - :origin:`searx/engines/recoll.py`
-
-Recoll_ is a desktop full-text search tool based on Xapian.  By itself Recoll_
-does not offer WEB or API access, this can be achieved using recoll-webui_
-
-
-Configuration
-=============
-
-You must configure the following settings:
-
-``base_url``:
-  Location where recoll-webui can be reached.
-
-``mount_prefix``:
-  Location where the file hierarchy is mounted on your *local* filesystem.
-
-``dl_prefix``:
-  Location where the file hierarchy as indexed by recoll can be reached.
-
-``search_dir``:
-  Part of the indexed file hierarchy to be search, if empty the full domain is
-  searched.
-
-
-Example
-=======
-
-Scenario:
-
-#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
-#. the Recoll search interface can be reached at https://recoll.example.org/ and
-#. the contents of this filesystem can be reached though https://download.example.org/reference
-
-.. code:: yaml
-
-   base_url: https://recoll.example.org/
-   mount_prefix: /export/documents
-   dl_prefix: https://download.example.org
-   search_dir: ''

docs/admin/engines/search-indexer-engines.rst

@@ -1,136 +0,0 @@
-====================
-Local Search Engines
-====================
-
-.. sidebar:: further read
-
-   - `Comparison to alternatives
-     <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
-
-Administrators might find themselves wanting to integrate locally running search
-engines.  The following ones are supported for now:
-
-* `Elasticsearch`_
-* `Meilisearch`_
-* `Solr`_
-
-Each search engine is powerful, capable of full-text search.  All of the engines
-above are added to ``settings.yml`` just commented out, as you have to
-``base_url`` for all them.
-
-Please note that if you are not using HTTPS to access these engines, you have to enable
-HTTP requests by setting ``enable_http`` to ``True``.
-
-Furthermore, if you do not want to expose these engines on a public instance, you
-can still add them and limit the access by setting ``tokens`` as described in
-section :ref:`private engines`.
-
-.. _engine meilisearch:
-
-MeiliSearch
-===========
-
-.. sidebar:: info
-
-   - :origin:`meilisearch.py <searx/engines/meilisearch.py>`
-   - `MeiliSearch <https://www.meilisearch.com>`_
-   - `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
-   - `Install MeiliSearch
-     <https://docs.meilisearch.com/learn/getting_started/installation.html>`_
-
-MeiliSearch_ is aimed at individuals and small companies.  It is designed for
-small-scale (less than 10 million documents) data collections.  E.g. it is great
-for storing web pages you have visited and searching in the contents later.
-
-The engine supports faceted search, so you can search in a subset of documents
-of the collection.  Furthermore, you can search in MeiliSearch_ instances that
-require authentication by setting ``auth_token``.
-
-Here is a simple example to query a Meilisearch instance:
-
-.. code:: yaml
-
-  - name: meilisearch
-    engine: meilisearch
-    shortcut: mes
-    base_url: http://localhost:7700
-    index: my-index
-    enable_http: true
-
-
-.. _engine elasticsearch:
-
-Elasticsearch
-=============
-
-.. sidebar:: info
-
-   - :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
-   - `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
-   - `Elasticsearch Guide
-     <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
-   - `Install Elasticsearch
-     <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
-
-Elasticsearch_ supports numerous ways to query the data it is storing.  At the
-moment the engine supports the most popular search methods (``query_type``):
-
-- ``match``,
-- ``simple_query_string``,
-- ``term`` and
-- ``terms``.
-
-If none of the methods fit your use case, you can select ``custom`` query type
-and provide the JSON payload to submit to Elasticsearch in
-``custom_query_json``.
-
-The following is an example configuration for an Elasticsearch_ instance with
-authentication configured to read from ``my-index`` index.
-
-.. code:: yaml
-
-  - name: elasticsearch
-    shortcut: es
-    engine: elasticsearch
-    base_url: http://localhost:9200
-    username: elastic
-    password: changeme
-    index: my-index
-    query_type: match
-    # custom_query_json: '{ ... }'
-    enable_http: true
-
-.. _engine solr:
-
-Solr
-====
-
-.. sidebar:: info
-
-   - :origin:`solr.py <searx/engines/solr.py>`
-   - `Solr <https://solr.apache.org>`_
-   - `Solr Resources <https://solr.apache.org/resources.html>`_
-   - `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
-
-Solr_ is a popular search engine based on Lucene, just like Elasticsearch_.  But
-instead of searching in indices, you can search in collections.
-
-This is an example configuration for searching in the collection
-``my-collection`` and get the results in ascending order.
-
-.. code:: yaml
-
-  - name: solr
-    engine: solr
-    shortcut: slr
-    base_url: http://localhost:8983
-    collection: my-collection
-    sort: asc
-    enable_http: true
-
-
-Acknowledgment
-==============
-
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/settings.rst

@@ -1,778 +0,0 @@
-.. _settings.yml:
-
-================
-``settings.yml``
-================
-
-This page describe the options possibilities of the :origin:`searx/settings.yml`
-file.
-
-.. sidebar:: Further reading ..
-
-   - :ref:`use_default_settings.yml`
-   - :ref:`search API`
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. _settings location:
-
-settings.yml location
-=====================
-
-The initial ``settings.yml`` we be load from these locations:
-
-1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
-2. ``/etc/searxng/settings.yml``
-
-If these files don't exist (or are empty or can't be read), SearXNG uses the
-:origin:`searx/settings.yml` file.  Read :ref:`settings use_default_settings` to
-see how you can simplify your *user defined* ``settings.yml``.
-
-
-.. _settings global:
-
-Global Settings
-===============
-
-.. _settings brand:
-
-``brand:``
-----------
-
-.. code:: yaml
-
-   brand:
-     issue_url: https://github.com/searxng/searxng/issues
-     docs_url: https://docs.searxng.org
-     public_instances: https://searx.space
-     wiki_url: https://github.com/searxng/searxng/wiki
-
-``issue_url`` :
-  If you host your own issue tracker change this URL.
-
-``docs_url`` :
-  If you host your own documentation change this URL.
-
-``public_instances`` :
-  If you host your own https://searx.space change this URL.
-
-``wiki_url`` :
-  Link to your wiki (or ``false``)
-
-.. _settings general:
-
-``general:``
-------------
-
-.. code:: yaml
-
-   general:
-     debug: false
-     instance_name:  "SearXNG"
-     privacypolicy_url: false
-     donation_url: false
-     contact_url: false
-     enable_metrics: true
-
-``debug`` : ``$SEARXNG_DEBUG``
-  Allow a more detailed log if you run SearXNG directly. Display *detailed* error
-  messages in the browser too, so this must be deactivated in production.
-
-``donation_url`` :
-  Set value to ``true`` to use your own donation page written in the
-  :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
-  the donation link altogether.
-
-``privacypolicy_url``:
-  Link to privacy policy.
-
-``contact_url``:
-  Contact ``mailto:`` address or WEB form.
-
-``enable_metrics``:
-  Enabled by default. Record various anonymous metrics availabled at ``/stats``,
-  ``/stats/errors`` and ``/preferences``.
-
-.. _settings search:
-
-``search:``
------------
-
-.. code:: yaml
-
-   search:
-     safe_search: 0
-     autocomplete: ""
-     default_lang: ""
-     ban_time_on_fail: 5
-     max_ban_time_on_fail: 120
-     suspended_times:
-       SearxEngineAccessDenied: 86400
-       SearxEngineCaptcha: 86400
-       SearxEngineTooManyRequests: 3600
-       cf_SearxEngineCaptcha: 1296000
-       cf_SearxEngineAccessDenied: 86400
-       recaptcha_SearxEngineCaptcha: 604800
-     formats:
-       - html
-
-``safe_search``:
-  Filter results.
-
-  - ``0``: None
-  - ``1``: Moderate
-  - ``2``: Strict
-
-``autocomplete``:
-  Existing autocomplete backends, leave blank to turn it off.
-
-  - ``dbpedia``
-  - ``duckduckgo``
-  - ``google``
-  - ``startpage``
-  - ``swisscows``
-  - ``qwant``
-  - ``wikipedia``
-
-``default_lang``:
-  Default search language - leave blank to detect from browser information or
-  use codes from :origin:`searx/languages.py`.
-
-``languages``:
-  List of available languages - leave unset to use all codes from
-  :origin:`searx/languages.py`.  Otherwise list codes of available languages.
-  The ``all`` value is shown as the ``Default language`` in the user interface
-  (in most cases, it is meant to send the query without a language parameter ;
-  in some cases, it means the English language) Example:
-
-  .. code:: yaml
-
-     languages:
-       - all
-       - en
-       - en-US
-       - de
-       - it-IT
-       - fr
-       - fr-BE
-
-``ban_time_on_fail``:
-  Ban time in seconds after engine errors.
-
-``max_ban_time_on_fail``:
-  Max ban time in seconds after engine errors.
-
-``suspended_times``:
-  Engine suspension time after error (in seconds; set to 0 to disable)
-
-  ``SearxEngineAccessDenied``: 86400
-    For error "Access denied" and "HTTP error [402, 403]"
-
-  ``SearxEngineCaptcha``: 86400
-    For error "CAPTCHA"
-
-  ``SearxEngineTooManyRequests``: 3600
-    For error "Too many request" and "HTTP error 429"
-
-  Cloudflare CAPTCHA:
-     - ``cf_SearxEngineCaptcha``: 1296000
-     - ``cf_SearxEngineAccessDenied``: 86400
-
-  Google CAPTCHA:
-    - ``recaptcha_SearxEngineCaptcha``: 604800
-
-``formats``:
-  Result formats available from web, remove format to deny access (use lower
-  case).
-
-  - ``html``
-  - ``csv``
-  - ``json``
-  - ``rss``
-
-
-.. _settings server:
-
-``server:``
------------
-
-.. code:: yaml
-
-   server:
-       base_url: http://example.org/location  # change this!
-       port: 8888
-       bind_address: "127.0.0.1"
-       secret_key: "ultrasecretkey"           # change this!
-       limiter: false
-       image_proxy: false
-       default_http_headers:
-         X-Content-Type-Options : nosniff
-         X-XSS-Protection : 1; mode=block
-         X-Download-Options : noopen
-         X-Robots-Tag : noindex, nofollow
-         Referrer-Policy : no-referrer
-
-
-``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
-  The base URL where SearXNG is deployed.  Used to create correct inbound links.
-  If you change the value, don't forget to rebuild instance's environment
-  (:ref:`utils/brand.env <make buildenv>`)
-
-``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
-  Port number and *bind address* of the SearXNG web application if you run it
-  directly using ``python searx/webapp.py``.  Doesn't apply to a SearXNG
-  services running behind a proxy and using socket communications.  If you
-  change the value, don't forget to rebuild instance's environment
-  (:ref:`utils/brand.env <make buildenv>`)
-
-``secret_key`` : ``$SEARXNG_SECRET``
-  Used for cryptography purpose.
-
-.. _limiter:
-
-``limiter`` :
-  Rate limit the number of request on the instance, block some bots.  The
-  :ref:`limiter src` requires a :ref:`settings redis` database.
-
-.. _image_proxy:
-
-``image_proxy`` :
-  Allow your instance of SearXNG of being able to proxy images.  Uses memory space.
-
-.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
-
-``default_http_headers`` :
-  Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
-
-
-.. _settings ui:
-
-``ui:``
--------
-
-.. _cache busting:
-   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
-
-.. code:: yaml
-
-   ui:
-     static_use_hash: false
-     default_locale: ""
-     query_in_title: false
-     infinite_scroll: false
-     center_alignment: false
-     cache_url: https://web.archive.org/web/
-     default_theme: simple
-     theme_args:
-       simple_style: auto
-
-.. _static_use_hash:
-
-``static_use_hash`` :
-  Enables `cache busting`_ of static files.
-
-``default_locale`` :
-  SearXNG interface language.  If blank, the locale is detected by using the
-  browser language.  If it doesn't work, or you are deploying a language
-  specific instance of searx, a locale can be defined using an ISO language
-  code, like ``fr``, ``en``, ``de``.
-
-``query_in_title`` :
-  When true, the result page's titles contains the query it decreases the
-  privacy, since the browser can records the page titles.
-
-``infinite_scroll``:
-  When true, automatically loads the next page when scrolling to bottom of the current page.
-
-``center_alignment`` : default ``false``
-  When enabled, the results are centered instead of being in the left (or RTL)
-  side of the screen.  This setting only affects the *desktop layout*
-  (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
-
-.. cache_url:
-
-``cache_url`` : ``https://web.archive.org/web/``
-  URL prefix of the internet archive or cache, don't forgett trailing slash (if
-  needed).  The default is https://web.archive.org/web/ alternatives are:
-
-  - https://webcache.googleusercontent.com/search?q=cache:
-  - https://archive.today/
-
-``default_theme`` :
-  Name of the theme you want to use by default on your SearXNG instance.
-
-``theme_args.simple_style``:
-  Style of simple theme: ``auto``, ``light``, ``dark``
-
-``results_on_new_tab``:
-  Open result links in a new tab by default.
-
-.. _settings redis:
-
-``redis:``
-----------
-
-.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
-
-A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
-will find a description to test your redis connection in SerXNG.  When using
-sockets, don't forget to check the access rights on the socket::
-
-  ls -la /usr/local/searxng-redis/run/redis.sock
-  srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
-
-In this example read/write access is given to the *searxng-redis* group.  To get
-access rights to redis instance (the socket), your SearXNG (or even your
-developer) account needs to be added to the *searxng-redis* group.
-
-``url`` : ``$SEARXNG_REDIS_URL``
-  URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
-
-    redis://[[username]:[password]]@localhost:6379/0
-    rediss://[[username]:[password]]@localhost:6379/0
-    unix://[[username]:[password]]@/path/to/socket.sock?db=0
-
-.. admonition:: Tip for developers
-
-   To set up a local redis instance, first set the socket path of the Redis DB
-   in your YAML setting:
-
-   .. code:: yaml
-
-      redis:
-        url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
-
-   Then use the following commands to install the redis instance ::
-
-     $ ./manage redis.build
-     $ sudo -H ./manage redis.install
-     $ sudo -H ./manage redis.addgrp "${USER}"
-     # don't forget to logout & login to get member of group
-
-
-.. _settings outgoing:
-
-``outgoing:``
--------------
-
-Communication with search engines.
-
-.. code:: yaml
-
-   outgoing:
-     request_timeout: 2.0       # default timeout in seconds, can be override by engine
-     max_request_timeout: 10.0  # the maximum timeout in seconds
-     useragent_suffix: ""       # information like an email address to the administrator
-     pool_connections: 100      # Maximum number of allowable connections, or null
-                                # for no limits. The default is 100.
-     pool_maxsize: 10           # Number of allowable keep-alive connections, or null
-                                # to always allow. The default is 10.
-     enable_http2: true         # See https://www.python-httpx.org/http2/
-     # uncomment below section if you want to use a custom server certificate
-     # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
-     # and https://www.python-httpx.org/compatibility/#ssl-configuration
-     #  verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
-     #
-     # uncomment below section if you want to use a proxyq see: SOCKS proxies
-     #   https://2.python-requests.org/en/latest/user/advanced/#proxies
-     # are also supported: see
-     #   https://2.python-requests.org/en/latest/user/advanced/#socks
-     #
-     #  proxies:
-     #    all://:
-     #      - http://proxy1:8080
-     #      - http://proxy2:8080
-     #
-     #  using_tor_proxy: true
-     #
-     # Extra seconds to add in order to account for the time taken by the proxy
-     #
-     #  extra_proxy_timeout: 10.0
-     #
-
-``request_timeout`` :
-  Global timeout of the requests made to others engines in seconds.  A bigger
-  timeout will allow to wait for answers from slow engines, but in consequence
-  will slow SearXNG reactivity (the result page may take the time specified in the
-  timeout to load).  Can be override by ``timeout`` in the :ref:`settings engine`.
-
-``useragent_suffix`` :
-  Suffix to the user-agent SearXNG uses to send requests to others engines.  If an
-  engine wish to block you, a contact info here may be useful to avoid that.
-
-.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
-
-``pool_maxsize``:
-  Number of allowable keep-alive connections, or ``null`` to always allow.  The
-  default is 10.  See ``max_keepalive_connections`` `Pool limit configuration`_.
-
-``pool_connections`` :
-  Maximum number of allowable connections, or ``null`` # for no limits.  The
-  default is 100.  See ``max_connections`` `Pool limit configuration`_.
-
-``keepalive_expiry`` :
-  Number of seconds to keep a connection in the pool.  By default 5.0 seconds.
-  See ``keepalive_expiry`` `Pool limit configuration`_.
-
-
-.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
-
-``proxies`` :
-  Define one or more proxies you wish to use, see `httpx proxies`_.
-  If there are more than one proxy for one protocol (http, https),
-  requests to the engines are distributed in a round-robin fashion.
-
-``source_ips`` :
-  If you use multiple network interfaces, define from which IP the requests must
-  be made. Example:
-
-  * ``0.0.0.0`` any local IPv4 address.
-  * ``::`` any local IPv6 address.
-  * ``192.168.0.1``
-  * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
-  * ``fe80::60a2:1691:e5a2:ee1f``
-  * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
-  * ``[ 192.168.0.1, fe80::/126 ]``
-
-``retries`` :
-  Number of retry in case of an HTTP error.  On each retry, SearXNG uses an
-  different proxy and source ip.
-
-``enable_http2`` :
-  Enable by default. Set to ``false`` to disable HTTP/2.
-
-.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
-.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
-
-``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
-  Allow to specify a path to certificate.
-  see `httpx verification defaults`_.
-
-  In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
-  ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
-  see `httpx ssl configuration`_.
-
-``max_redirects`` :
-  30 by default. Maximum redirect before it is an error.
-
-``using_tor_proxy`` :
-  Using tor proxy (``true``) or not (``false``) for all engines.  The default is
-  ``false`` and can be overwritten in the :ref:`settings engine`
-
-
-
-.. _settings categories_as_tabs:
-
-``categories_as_tabs:``
------------------------
-
-A list of the categories that are displayed as tabs in the user interface.
-Categories not listed here can still be searched with the :ref:`search-syntax`.
-
-.. code-block:: yaml
-
-  categories_as_tabs:
-    general:
-    images:
-    videos:
-    news:
-    map:
-    music:
-    it:
-    science:
-    files:
-    social media:
-
-Engines are added to ``categories:`` (compare :ref:`engine categories`), the
-categories listed in ``categories_as_tabs`` are shown as tabs in the UI.  If
-there are no active engines in a category, the tab is not displayed (e.g. if a
-user disables all engines in a category).
-
-On the preferences page (``/preferences``) -- under *engines* -- there is an
-additional tab, called *other*.  In this tab are all engines listed that are not
-in one of the UI tabs (not included in ``categories_as_tabs``).
-
-.. _settings engine:
-
-Engine settings
-===============
-
-.. sidebar:: Further reading ..
-
-   - :ref:`configured engines`
-   - :ref:`engines-dev`
-
-In the code example below a *full fledged* example of a YAML setup from a dummy
-engine is shown.  Most of the options have a default value or even are optional.
-
-.. code:: yaml
-
-   - name: example engine
-     engine: example
-     shortcut: demo
-     base_url: 'https://{language}.example.com/'
-     send_accept_language_header: false
-     categories: general
-     timeout: 3.0
-     api_key: 'apikey'
-     disabled: false
-     language: en_US
-     tokens: [ 'my-secret-token' ]
-     weight: 1
-     display_error_messages: true
-     about:
-        website: https://example.com
-        wikidata_id: Q306656
-        official_api_documentation: https://example.com/api-doc
-        use_official_api: true
-        require_api_key: true
-        results: HTML
-
-     # overwrite values from section 'outgoing:'
-     enable_http2: false
-     retries: 1
-     max_connections: 100
-     max_keepalive_connections: 10
-     keepalive_expiry: 5.0
-     using_tor_proxy: false
-     proxies:
-       http:
-         - http://proxy1:8080
-         - http://proxy2:8080
-       https:
-         - http://proxy1:8080
-         - http://proxy2:8080
-         - socks5://user:password@proxy3:1080
-         - socks5h://user:password@proxy4:1080
-
-     # other network settings
-     enable_http: false
-     retry_on_http_error: true # or 403 or [404, 429]
-
-
-``name`` :
-  Name that will be used across SearXNG to define this engine.  In settings, on
-  the result page...
-
-``engine`` :
-  Name of the python file used to handle requests and responses to and from this
-  search engine.
-
-``shortcut`` :
-  Code used to execute bang requests (in this case using ``!bi``)
-
-``base_url`` : optional
-  Part of the URL that should be stable across every request.  Can be useful to
-  use multiple sites using only one engine, or updating the site URL without
-  touching at the code.
-
-``send_accept_language_header`` :
-  Several engines that support languages (or regions) deal with the HTTP header
-  ``Accept-Language`` to build a response that fits to the locale.  When this
-  option is activated, the language (locale) that is selected by the user is used
-  to build and send a ``Accept-Language`` header in the request to the origin
-  search engine.
-
-.. _engine categories:
-
-``categories`` : optional
-  Specifies to which categories the engine should be added.  Engines can be
-  assigned to multiple categories.
-
-  Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
-  UI.  A search in a tab (in the UI) will query all engines that are active in
-  this tab.  In the preferences page (``/preferences``) -- under *engines* --
-  users can select what engine should be active when querying in this tab.
-
-  Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
-  in a category, regardless of whether they are active or not, or whether they
-  are in a tab of the UI or not.  For example, ``!dictionaries`` can be used to
-  query all search engines in that category (group).
-
-``timeout`` : optional
-  Timeout of the search with the current search engine.  Overwrites
-  ``request_timeout`` from :ref:`settings outgoing`.  **Be careful, it will
-  modify the global timeout of SearXNG.**
-
-``api_key`` : optional
-  In a few cases, using an API needs the use of a secret key.  How to obtain them
-  is described in the file.
-
-``disabled`` : optional
-  To disable by default the engine, but not deleting it.  It will allow the user
-  to manually activate it in the settings.
-
-``inactive``: optional
-  Remove the engine from the settings (*disabled & removed*).
-
-``language`` : optional
-  If you want to use another language for a specific engine, you can define it
-  by using the ISO code of language (and region), like ``fr``, ``en-US``,
-  ``de-DE``.
-
-``tokens`` : optional
-  A list of secret tokens to make this engine *private*, more details see
-  :ref:`private engines`.
-
-``weight`` : default ``1``
-  Weighting of the results of this engine.
-
-``display_error_messages`` : default ``true``
-  When an engine returns an error, the message is displayed on the user interface.
-
-``network`` : optional
-  Use the network configuration from another engine.
-  In addition, there are two default networks:
-
-  - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
-  - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
-
-``enable_http`` : optional
-  Enable HTTP for this engine (by default only HTTPS is enabled).
-
-``retry_on_http_error`` : optional
-  Retry request on some HTTP status code.
-
-  Example:
-
-  * ``true`` : on HTTP status code between 400 and 599.
-  * ``403`` : on HTTP status code 403.
-  * ``[403, 429]``: on HTTP status code 403 and 429.
-
-``proxies`` :
-  Overwrites proxy settings from :ref:`settings outgoing`.
-
-``using_tor_proxy`` :
-  Using tor proxy (``true``) or not (``false``) for this engine.  The default is
-  taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
-
-``max_keepalive_connection#s`` :
-  `Pool limit configuration`_, overwrites value ``pool_maxsize`` from
-   :ref:`settings outgoing` for this engine.
-
-``max_connections`` :
-  `Pool limit configuration`_, overwrites value ``pool_connections`` from
-  :ref:`settings outgoing` for this engine.
-
-``keepalive_expiry`` :
-  `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
-  :ref:`settings outgoing` for this engine.
-
-.. note::
-
-   A few more options are possible, but they are pretty specific to some
-   engines, and so won't be described here.
-
-
-Example: Multilingual Search
-----------------------------
-
-SearXNG does not support true multilingual search.  You have to use the language
-prefix in your search query when searching in a different language.
-
-But there is a workaround: By adding a new search engine with a different
-language, SearXNG will search in your default and other language.
-
-Example configuration in settings.yml for a German and English speaker:
-
-.. code-block:: yaml
-
-    search:
-        default_lang : "de"
-        ...
-
-    engines:
-      - name : google english
-        engine : google
-        language : en
-        ...
-
-When searching, the default google engine will return German results and
-"google english" will return English results.
-
-
-.. _settings use_default_settings:
-
-use_default_settings
-====================
-
-.. sidebar:: ``use_default_settings: true``
-
-   - :ref:`settings location`
-   - :ref:`use_default_settings.yml`
-   - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
-
-The user defined ``settings.yml`` is loaded from the :ref:`settings location`
-and can relied on the default configuration :origin:`searx/settings.yml` using:
-
- ``use_default_settings: true``
-
-``server:``
-  In the following example, the actual settings are the default settings defined
-  in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
-  the ``bind_address``:
-
-  .. code-block:: yaml
-
-    use_default_settings: true
-    server:
-        secret_key: "ultrasecretkey"   # change this!
-        bind_address: "0.0.0.0"
-
-``engines:``
-  With ``use_default_settings: true``, each settings can be override in a
-  similar way, the ``engines`` section is merged according to the engine
-  ``name``.  In this example, SearXNG will load all the default engines, will
-  enable the ``bing`` engine and define a :ref:`token <private engines>` for
-  the arch linux engine:
-
-  .. code-block:: yaml
-
-    use_default_settings: true
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: arch linux wiki
-        tokens: ['$ecretValue']
-      - name: bing
-        disabled: false
-
-
-``engines:`` / ``remove:``
-  It is possible to remove some engines from the default settings. The following
-  example is similar to the above one, but SearXNG doesn't load the the google
-  engine:
-
-  .. code-block:: yaml
-
-    use_default_settings:
-      engines:
-        remove:
-          - google
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: arch linux wiki
-        tokens: ['$ecretValue']
-
-``engines:`` / ``keep_only:``
-  As an alternative, it is possible to specify the engines to keep. In the
-  following example, SearXNG has only two engines:
-
-  .. code-block:: yaml
-
-    use_default_settings:
-      engines:
-        keep_only:
-          - google
-          - duckduckgo
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: google
-        tokens: ['$ecretValue']
-      - name: duckduckgo
-        tokens: ['$ecretValue']

docs/admin/index.rst

@@ -4,8 +4,8 @@ Administrator documentation
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
 
+   settings/index
    installation
    installation-docker
    installation-scripts
@@ -15,7 +15,6 @@ Administrator documentation
    installation-apache
    update-searxng
    answer-captcha
-   engines/index
    api
    architecture
    plugins

docs/admin/installation-apache.rst

@@ -61,7 +61,7 @@ section might give you some guidance.
    - `Apache Fedora`_
    - `Apache directives`_
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/installation-nginx.rst

@@ -41,7 +41,7 @@ section might give you some guidance.
    - `uWSGI support from nginx`_
 
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/installation-searxng.rst

@@ -4,7 +4,7 @@
 Step by step installation
 =========================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry
@@ -73,7 +73,7 @@ Configuration
 
 .. sidebar:: ``use_default_settings: True``
 
-   - :ref:`settings global`
+   - :ref:`settings.yml`
    - :ref:`settings location`
    - :ref:`settings use_default_settings`
    - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`

docs/admin/installation-uwsgi.rst

@@ -9,7 +9,7 @@ uWSGI
    - `systemd.unit`_
    - `uWSGI Emperor`_
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/settings/index.rst

@@ -0,0 +1,25 @@
+========
+Settings
+========
+
+.. sidebar:: Further reading ..
+
+   - :ref:`engine settings`
+   - :ref:`engine file`
+
+.. toctree::
+   :maxdepth: 2
+
+   settings
+   settings_engine
+   settings_brand
+   settings_general
+   settings_search
+   settings_server
+   settings_ui
+   settings_redis
+   settings_outgoing
+   settings_categories_as_tabs
+
+
+

docs/admin/settings/settings.rst

@@ -0,0 +1,117 @@
+.. _settings.yml:
+
+================
+``settings.yml``
+================
+
+This page describe the options possibilities of the :origin:`searx/settings.yml`
+file.
+
+.. sidebar:: Further reading ..
+
+   - :ref:`use_default_settings.yml`
+   - :ref:`search API`
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. _settings location:
+
+settings.yml location
+=====================
+
+The initial ``settings.yml`` we be load from these locations:
+
+1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
+2. ``/etc/searxng/settings.yml``
+
+If these files don't exist (or are empty or can't be read), SearXNG uses the
+:origin:`searx/settings.yml` file.  Read :ref:`settings use_default_settings` to
+see how you can simplify your *user defined* ``settings.yml``.
+
+
+
+.. _settings use_default_settings:
+
+use_default_settings
+====================
+
+.. sidebar:: ``use_default_settings: true``
+
+   - :ref:`settings location`
+   - :ref:`use_default_settings.yml`
+   - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
+
+The user defined ``settings.yml`` is loaded from the :ref:`settings location`
+and can relied on the default configuration :origin:`searx/settings.yml` using:
+
+ ``use_default_settings: true``
+
+``server:``
+  In the following example, the actual settings are the default settings defined
+  in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
+  the ``bind_address``:
+
+  .. code:: yaml
+
+    use_default_settings: true
+    server:
+        secret_key: "ultrasecretkey"   # change this!
+        bind_address: "0.0.0.0"
+
+``engines:``
+  With ``use_default_settings: true``, each settings can be override in a
+  similar way, the ``engines`` section is merged according to the engine
+  ``name``.  In this example, SearXNG will load all the default engines, will
+  enable the ``bing`` engine and define a :ref:`token <private engines>` for
+  the arch linux engine:
+
+  .. code:: yaml
+
+    use_default_settings: true
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: arch linux wiki
+        tokens: ['$ecretValue']
+      - name: bing
+        disabled: false
+
+
+``engines:`` / ``remove:``
+  It is possible to remove some engines from the default settings. The following
+  example is similar to the above one, but SearXNG doesn't load the the google
+  engine:
+
+  .. code:: yaml
+
+    use_default_settings:
+      engines:
+        remove:
+          - google
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: arch linux wiki
+        tokens: ['$ecretValue']
+
+``engines:`` / ``keep_only:``
+  As an alternative, it is possible to specify the engines to keep. In the
+  following example, SearXNG has only two engines:
+
+  .. code:: yaml
+
+    use_default_settings:
+      engines:
+        keep_only:
+          - google
+          - duckduckgo
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: google
+        tokens: ['$ecretValue']
+      - name: duckduckgo
+        tokens: ['$ecretValue']

docs/admin/settings/settings_brand.rst

@@ -0,0 +1,25 @@
+.. _settings brand:
+
+==========
+``brand:``
+==========
+
+.. code:: yaml
+
+   brand:
+     issue_url: https://github.com/searxng/searxng/issues
+     docs_url: https://docs.searxng.org
+     public_instances: https://searx.space
+     wiki_url: https://github.com/searxng/searxng/wiki
+
+``issue_url`` :
+  If you host your own issue tracker change this URL.
+
+``docs_url`` :
+  If you host your own documentation change this URL.
+
+``public_instances`` :
+  If you host your own https://searx.space change this URL.
+
+``wiki_url`` :
+  Link to your wiki (or ``false``)

docs/admin/settings/settings_categories_as_tabs.rst

@@ -0,0 +1,31 @@
+.. _settings categories_as_tabs:
+
+=======================
+``categories_as_tabs:``
+=======================
+
+A list of the categories that are displayed as tabs in the user interface.
+Categories not listed here can still be searched with the :ref:`search-syntax`.
+
+.. code:: yaml
+
+  categories_as_tabs:
+    general:
+    images:
+    videos:
+    news:
+    map:
+    music:
+    it:
+    science:
+    files:
+    social media:
+
+Engines are added to ``categories:`` (compare :ref:`engine categories`), the
+categories listed in ``categories_as_tabs`` are shown as tabs in the UI.  If
+there are no active engines in a category, the tab is not displayed (e.g. if a
+user disables all engines in a category).
+
+On the preferences page (``/preferences``) -- under *engines* -- there is an
+additional tab, called *other*.  In this tab are all engines listed that are not
+in one of the UI tabs (not included in ``categories_as_tabs``).

docs/admin/settings/settings_engine.rst

@@ -0,0 +1,244 @@
+.. _settings engine:
+
+===========
+``engine:``
+===========
+
+.. sidebar:: Further reading ..
+
+   - :ref:`configured engines`
+   - :ref:`engines-dev`
+
+In the code example below a *full fledged* example of a YAML setup from a dummy
+engine is shown.  Most of the options have a default value or even are optional.
+
+.. hint::
+
+   A few more options are possible, but they are pretty specific to some
+   engines (:ref:`engine implementations`).
+
+.. code:: yaml
+
+   - name: example engine
+     engine: example
+     shortcut: demo
+     base_url: 'https://{language}.example.com/'
+     send_accept_language_header: false
+     categories: general
+     timeout: 3.0
+     api_key: 'apikey'
+     disabled: false
+     language: en_US
+     tokens: [ 'my-secret-token' ]
+     weight: 1
+     display_error_messages: true
+     about:
+        website: https://example.com
+        wikidata_id: Q306656
+        official_api_documentation: https://example.com/api-doc
+        use_official_api: true
+        require_api_key: true
+        results: HTML
+
+     # overwrite values from section 'outgoing:'
+     enable_http2: false
+     retries: 1
+     max_connections: 100
+     max_keepalive_connections: 10
+     keepalive_expiry: 5.0
+     using_tor_proxy: false
+     proxies:
+       http:
+         - http://proxy1:8080
+         - http://proxy2:8080
+       https:
+         - http://proxy1:8080
+         - http://proxy2:8080
+         - socks5://user:password@proxy3:1080
+         - socks5h://user:password@proxy4:1080
+
+     # other network settings
+     enable_http: false
+     retry_on_http_error: true # or 403 or [404, 429]
+
+
+``name`` :
+  Name that will be used across SearXNG to define this engine.  In settings, on
+  the result page...
+
+``engine`` :
+  Name of the python file used to handle requests and responses to and from this
+  search engine.
+
+``shortcut`` :
+  Code used to execute bang requests (in this case using ``!bi``)
+
+``base_url`` : optional
+  Part of the URL that should be stable across every request.  Can be useful to
+  use multiple sites using only one engine, or updating the site URL without
+  touching at the code.
+
+``send_accept_language_header`` :
+  Several engines that support languages (or regions) deal with the HTTP header
+  ``Accept-Language`` to build a response that fits to the locale.  When this
+  option is activated, the language (locale) that is selected by the user is used
+  to build and send a ``Accept-Language`` header in the request to the origin
+  search engine.
+
+.. _engine categories:
+
+``categories`` : optional
+  Specifies to which categories the engine should be added.  Engines can be
+  assigned to multiple categories.
+
+  Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
+  UI.  A search in a tab (in the UI) will query all engines that are active in
+  this tab.  In the preferences page (``/preferences``) -- under *engines* --
+  users can select what engine should be active when querying in this tab.
+
+  Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
+  in a category, regardless of whether they are active or not, or whether they
+  are in a tab of the UI or not.  For example, ``!dictionaries`` can be used to
+  query all search engines in that category (group).
+
+``timeout`` : optional
+  Timeout of the search with the current search engine.  Overwrites
+  ``request_timeout`` from :ref:`settings outgoing`.  **Be careful, it will
+  modify the global timeout of SearXNG.**
+
+``api_key`` : optional
+  In a few cases, using an API needs the use of a secret key.  How to obtain them
+  is described in the file.
+
+``disabled`` : optional
+  To disable by default the engine, but not deleting it.  It will allow the user
+  to manually activate it in the settings.
+
+``inactive``: optional
+  Remove the engine from the settings (*disabled & removed*).
+
+``language`` : optional
+  If you want to use another language for a specific engine, you can define it
+  by using the ISO code of language (and region), like ``fr``, ``en-US``,
+  ``de-DE``.
+
+``tokens`` : optional
+  A list of secret tokens to make this engine *private*, more details see
+  :ref:`private engines`.
+
+``weight`` : default ``1``
+  Weighting of the results of this engine.
+
+``display_error_messages`` : default ``true``
+  When an engine returns an error, the message is displayed on the user interface.
+
+``network`` : optional
+  Use the network configuration from another engine.
+  In addition, there are two default networks:
+
+  - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
+  - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
+
+``enable_http`` : optional
+  Enable HTTP for this engine (by default only HTTPS is enabled).
+
+``retry_on_http_error`` : optional
+  Retry request on some HTTP status code.
+
+  Example:
+
+  * ``true`` : on HTTP status code between 400 and 599.
+  * ``403`` : on HTTP status code 403.
+  * ``[403, 429]``: on HTTP status code 403 and 429.
+
+``proxies`` :
+  Overwrites proxy settings from :ref:`settings outgoing`.
+
+``using_tor_proxy`` :
+  Using tor proxy (``true``) or not (``false``) for this engine.  The default is
+  taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
+
+.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
+
+``max_keepalive_connection#s`` :
+  `Pool limit configuration`_, overwrites value ``pool_maxsize`` from
+   :ref:`settings outgoing` for this engine.
+
+``max_connections`` :
+  `Pool limit configuration`_, overwrites value ``pool_connections`` from
+  :ref:`settings outgoing` for this engine.
+
+``keepalive_expiry`` :
+  `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
+  :ref:`settings outgoing` for this engine.
+
+
+.. _private engines:
+
+Private Engines (``tokens``)
+============================
+
+Administrators might find themselves wanting to limit access to some of the
+enabled engines on their instances.  It might be because they do not want to
+expose some private information through :ref:`offline engines`.  Or they would
+rather share engines only with their trusted friends or colleagues.
+
+.. sidebar:: info
+
+   Initial sponsored by `Search and Discovery Fund
+   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+
+To solve this issue the concept of *private engines* exists.
+
+A new option was added to engines named `tokens`.  It expects a list of strings.
+If the user making a request presents one of the tokens of an engine, they can
+access information about the engine and make search requests.
+
+Example configuration to restrict access to the Arch Linux Wiki engine:
+
+.. code:: yaml
+
+  - name: arch linux wiki
+    engine: archlinux
+    shortcut: al
+    tokens: [ 'my-secret-token' ]
+
+Unless a user has configured the right token, the engine is going to be hidden
+from him/her.  It is not going to be included in the list of engines on the
+Preferences page and in the output of `/config` REST API call.
+
+Tokens can be added to one's configuration on the Preferences page under "Engine
+tokens".  The input expects a comma separated list of strings.
+
+The distribution of the tokens from the administrator to the users is not carved
+in stone.  As providing access to such engines implies that the admin knows and
+trusts the user, we do not see necessary to come up with a strict process.
+Instead, we would like to add guidelines to the documentation of the feature.
+
+
+Example: Multilingual Search
+============================
+
+SearXNG does not support true multilingual search.  You have to use the language
+prefix in your search query when searching in a different language.
+
+But there is a workaround: By adding a new search engine with a different
+language, SearXNG will search in your default and other language.
+
+Example configuration in settings.yml for a German and English speaker:
+
+.. code-block:: yaml
+
+    search:
+        default_lang : "de"
+        ...
+
+    engines:
+      - name : google english
+        engine : google
+        language : en
+        ...
+
+When searching, the default google engine will return German results and
+"google english" will return English results.
+

docs/admin/settings/settings_general.rst

@@ -0,0 +1,34 @@
+.. _settings general:
+
+============
+``general:``
+============
+
+.. code:: yaml
+
+   general:
+     debug: false
+     instance_name:  "SearXNG"
+     privacypolicy_url: false
+     donation_url: false
+     contact_url: false
+     enable_metrics: true
+
+``debug`` : ``$SEARXNG_DEBUG``
+  Allow a more detailed log if you run SearXNG directly. Display *detailed* error
+  messages in the browser too, so this must be deactivated in production.
+
+``donation_url`` :
+  Set value to ``true`` to use your own donation page written in the
+  :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
+  the donation link altogether.
+
+``privacypolicy_url``:
+  Link to privacy policy.
+
+``contact_url``:
+  Contact ``mailto:`` address or WEB form.
+
+``enable_metrics``:
+  Enabled by default. Record various anonymous metrics availabled at ``/stats``,
+  ``/stats/errors`` and ``/preferences``.

docs/admin/settings/settings_outgoing.rst

@@ -0,0 +1,110 @@
+.. _settings outgoing:
+
+=============
+``outgoing:``
+=============
+
+Communication with search engines.
+
+.. code:: yaml
+
+   outgoing:
+     request_timeout: 2.0       # default timeout in seconds, can be override by engine
+     max_request_timeout: 10.0  # the maximum timeout in seconds
+     useragent_suffix: ""       # information like an email address to the administrator
+     pool_connections: 100      # Maximum number of allowable connections, or null
+                                # for no limits. The default is 100.
+     pool_maxsize: 10           # Number of allowable keep-alive connections, or null
+                                # to always allow. The default is 10.
+     enable_http2: true         # See https://www.python-httpx.org/http2/
+     # uncomment below section if you want to use a custom server certificate
+     # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
+     # and https://www.python-httpx.org/compatibility/#ssl-configuration
+     #  verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
+     #
+     # uncomment below section if you want to use a proxyq see: SOCKS proxies
+     #   https://2.python-requests.org/en/latest/user/advanced/#proxies
+     # are also supported: see
+     #   https://2.python-requests.org/en/latest/user/advanced/#socks
+     #
+     #  proxies:
+     #    all://:
+     #      - http://proxy1:8080
+     #      - http://proxy2:8080
+     #
+     #  using_tor_proxy: true
+     #
+     # Extra seconds to add in order to account for the time taken by the proxy
+     #
+     #  extra_proxy_timeout: 10.0
+     #
+
+``request_timeout`` :
+  Global timeout of the requests made to others engines in seconds.  A bigger
+  timeout will allow to wait for answers from slow engines, but in consequence
+  will slow SearXNG reactivity (the result page may take the time specified in the
+  timeout to load).  Can be override by ``timeout`` in the :ref:`settings engine`.
+
+``useragent_suffix`` :
+  Suffix to the user-agent SearXNG uses to send requests to others engines.  If an
+  engine wish to block you, a contact info here may be useful to avoid that.
+
+.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
+
+``pool_maxsize``:
+  Number of allowable keep-alive connections, or ``null`` to always allow.  The
+  default is 10.  See ``max_keepalive_connections`` `Pool limit configuration`_.
+
+``pool_connections`` :
+  Maximum number of allowable connections, or ``null`` # for no limits.  The
+  default is 100.  See ``max_connections`` `Pool limit configuration`_.
+
+``keepalive_expiry`` :
+  Number of seconds to keep a connection in the pool.  By default 5.0 seconds.
+  See ``keepalive_expiry`` `Pool limit configuration`_.
+
+.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
+
+``proxies`` :
+  Define one or more proxies you wish to use, see `httpx proxies`_.
+  If there are more than one proxy for one protocol (http, https),
+  requests to the engines are distributed in a round-robin fashion.
+
+``source_ips`` :
+  If you use multiple network interfaces, define from which IP the requests must
+  be made. Example:
+
+  * ``0.0.0.0`` any local IPv4 address.
+  * ``::`` any local IPv6 address.
+  * ``192.168.0.1``
+  * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
+  * ``fe80::60a2:1691:e5a2:ee1f``
+  * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
+  * ``[ 192.168.0.1, fe80::/126 ]``
+
+``retries`` :
+  Number of retry in case of an HTTP error.  On each retry, SearXNG uses an
+  different proxy and source ip.
+
+``enable_http2`` :
+  Enable by default. Set to ``false`` to disable HTTP/2.
+
+.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
+.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
+
+``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
+  Allow to specify a path to certificate.
+  see `httpx verification defaults`_.
+
+  In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
+  ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
+  see `httpx ssl configuration`_.
+
+``max_redirects`` :
+  30 by default. Maximum redirect before it is an error.
+
+``using_tor_proxy`` :
+  Using tor proxy (``true``) or not (``false``) for all engines.  The default is
+  ``false`` and can be overwritten in the :ref:`settings engine`
+
+

docs/admin/settings/settings_redis.rst

@@ -0,0 +1,43 @@
+.. _settings redis:
+
+==========
+``redis:``
+==========
+
+.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
+
+A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
+will find a description to test your redis connection in SerXNG.  When using
+sockets, don't forget to check the access rights on the socket::
+
+  ls -la /usr/local/searxng-redis/run/redis.sock
+  srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
+
+In this example read/write access is given to the *searxng-redis* group.  To get
+access rights to redis instance (the socket), your SearXNG (or even your
+developer) account needs to be added to the *searxng-redis* group.
+
+``url`` : ``$SEARXNG_REDIS_URL``
+  URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
+
+    redis://[[username]:[password]]@localhost:6379/0
+    rediss://[[username]:[password]]@localhost:6379/0
+    unix://[[username]:[password]]@/path/to/socket.sock?db=0
+
+.. admonition:: Tip for developers
+
+   To set up a local redis instance, first set the socket path of the Redis DB
+   in your YAML setting:
+
+   .. code:: yaml
+
+      redis:
+        url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+   Then use the following commands to install the redis instance ::
+
+     $ ./manage redis.build
+     $ sudo -H ./manage redis.install
+     $ sudo -H ./manage redis.addgrp "${USER}"
+     # don't forget to logout & login to get member of group
+

docs/admin/settings/settings_search.rst

@@ -0,0 +1,97 @@
+.. _settings search:
+
+===========
+``search:``
+===========
+
+.. code:: yaml
+
+   search:
+     safe_search: 0
+     autocomplete: ""
+     default_lang: ""
+     ban_time_on_fail: 5
+     max_ban_time_on_fail: 120
+     suspended_times:
+       SearxEngineAccessDenied: 86400
+       SearxEngineCaptcha: 86400
+       SearxEngineTooManyRequests: 3600
+       cf_SearxEngineCaptcha: 1296000
+       cf_SearxEngineAccessDenied: 86400
+       recaptcha_SearxEngineCaptcha: 604800
+     formats:
+       - html
+
+``safe_search``:
+  Filter results.
+
+  - ``0``: None
+  - ``1``: Moderate
+  - ``2``: Strict
+
+``autocomplete``:
+  Existing autocomplete backends, leave blank to turn it off.
+
+  - ``dbpedia``
+  - ``duckduckgo``
+  - ``google``
+  - ``startpage``
+  - ``swisscows``
+  - ``qwant``
+  - ``wikipedia``
+
+``default_lang``:
+  Default search language - leave blank to detect from browser information or
+  use codes from :origin:`searx/languages.py`.
+
+``languages``:
+  List of available languages - leave unset to use all codes from
+  :origin:`searx/languages.py`.  Otherwise list codes of available languages.
+  The ``all`` value is shown as the ``Default language`` in the user interface
+  (in most cases, it is meant to send the query without a language parameter ;
+  in some cases, it means the English language) Example:
+
+  .. code:: yaml
+
+     languages:
+       - all
+       - en
+       - en-US
+       - de
+       - it-IT
+       - fr
+       - fr-BE
+
+``ban_time_on_fail``:
+  Ban time in seconds after engine errors.
+
+``max_ban_time_on_fail``:
+  Max ban time in seconds after engine errors.
+
+``suspended_times``:
+  Engine suspension time after error (in seconds; set to 0 to disable)
+
+  ``SearxEngineAccessDenied``: 86400
+    For error "Access denied" and "HTTP error [402, 403]"
+
+  ``SearxEngineCaptcha``: 86400
+    For error "CAPTCHA"
+
+  ``SearxEngineTooManyRequests``: 3600
+    For error "Too many request" and "HTTP error 429"
+
+  Cloudflare CAPTCHA:
+     - ``cf_SearxEngineCaptcha``: 1296000
+     - ``cf_SearxEngineAccessDenied``: 86400
+
+  Google CAPTCHA:
+    - ``recaptcha_SearxEngineCaptcha``: 604800
+
+``formats``:
+  Result formats available from web, remove format to deny access (use lower
+  case).
+
+  - ``html``
+  - ``csv``
+  - ``json``
+  - ``rss``

docs/admin/settings/settings_server.rst

@@ -0,0 +1,54 @@
+.. _settings server:
+
+===========
+``server:``
+===========
+
+.. code:: yaml
+
+   server:
+       base_url: http://example.org/location  # change this!
+       port: 8888
+       bind_address: "127.0.0.1"
+       secret_key: "ultrasecretkey"           # change this!
+       limiter: false
+       image_proxy: false
+       default_http_headers:
+         X-Content-Type-Options : nosniff
+         X-XSS-Protection : 1; mode=block
+         X-Download-Options : noopen
+         X-Robots-Tag : noindex, nofollow
+         Referrer-Policy : no-referrer
+
+
+``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
+  The base URL where SearXNG is deployed.  Used to create correct inbound links.
+  If you change the value, don't forget to rebuild instance's environment
+  (:ref:`utils/brand.env <make buildenv>`)
+
+``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
+  Port number and *bind address* of the SearXNG web application if you run it
+  directly using ``python searx/webapp.py``.  Doesn't apply to a SearXNG
+  services running behind a proxy and using socket communications.  If you
+  change the value, don't forget to rebuild instance's environment
+  (:ref:`utils/brand.env <make buildenv>`)
+
+``secret_key`` : ``$SEARXNG_SECRET``
+  Used for cryptography purpose.
+
+.. _limiter:
+
+``limiter`` :
+  Rate limit the number of request on the instance, block some bots.  The
+  :ref:`limiter src` requires a :ref:`settings redis` database.
+
+.. _image_proxy:
+
+``image_proxy`` :
+  Allow your instance of SearXNG of being able to proxy images.  Uses memory space.
+
+.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
+
+``default_http_headers`` :
+  Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
+

docs/admin/settings/settings_ui.rst

@@ -0,0 +1,62 @@
+.. _settings ui:
+
+=======
+``ui:``
+=======
+
+.. _cache busting:
+   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
+
+.. code:: yaml
+
+   ui:
+     static_use_hash: false
+     default_locale: ""
+     query_in_title: false
+     infinite_scroll: false
+     center_alignment: false
+     cache_url: https://web.archive.org/web/
+     default_theme: simple
+     theme_args:
+       simple_style: auto
+
+.. _static_use_hash:
+
+``static_use_hash`` :
+  Enables `cache busting`_ of static files.
+
+``default_locale`` :
+  SearXNG interface language.  If blank, the locale is detected by using the
+  browser language.  If it doesn't work, or you are deploying a language
+  specific instance of searx, a locale can be defined using an ISO language
+  code, like ``fr``, ``en``, ``de``.
+
+``query_in_title`` :
+  When true, the result page's titles contains the query it decreases the
+  privacy, since the browser can records the page titles.
+
+``infinite_scroll``:
+  When true, automatically loads the next page when scrolling to bottom of the current page.
+
+``center_alignment`` : default ``false``
+  When enabled, the results are centered instead of being in the left (or RTL)
+  side of the screen.  This setting only affects the *desktop layout*
+  (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
+
+.. cache_url:
+
+``cache_url`` : ``https://web.archive.org/web/``
+  URL prefix of the internet archive or cache, don't forgett trailing slash (if
+  needed).  The default is https://web.archive.org/web/ alternatives are:
+
+  - https://webcache.googleusercontent.com/search?q=cache:
+  - https://archive.today/
+
+``default_theme`` :
+  Name of the theme you want to use by default on your SearXNG instance.
+
+``theme_args.simple_style``:
+  Style of simple theme: ``auto``, ``light``, ``dark``
+
+``results_on_new_tab``:
+  Open result links in a new tab by default.

docs/admin/update-searxng.rst

@@ -9,7 +9,7 @@ SearXNG maintenance
    - :ref:`toolboxing`
    - :ref:`uWSGI maintenance`
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/dev/contribution_guide.rst

@@ -4,7 +4,7 @@
 How to contribute
 =================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engines.demo_offline.rst → docs/dev/engines/demo/demo_offline.rst

@@ -4,6 +4,11 @@
 Demo Offline Engine
 ===================
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.demo_offline
   :members:
 

docs/src/searx.engines.demo_online.rst → docs/dev/engines/demo/demo_online.rst

@@ -4,6 +4,11 @@
 Demo Online Engine
 ==================
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.demo_online
   :members:
 

docs/dev/engine_overview.rst → docs/dev/engines/engine_overview.rst

@@ -4,6 +4,11 @@
 Engine Overview
 ===============
 
+.. contents::
+   :depth: 3
+   :local:
+   :backlinks: entry
+
 .. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
 
 .. sidebar:: Further reading ..
@@ -11,10 +16,6 @@ Engine Overview
    - :ref:`configured engines`
    - :ref:`settings engine`
 
-.. contents::
-   :depth: 3
-   :backlinks: entry
-
 SearXNG is a metasearch-engine_, so it uses different search engines to provide
 better results.
 
@@ -49,12 +50,12 @@ Engine File
    categories              list        categories, in which the engine is working
    paging                  boolean     support multiple pages
    time_range_support      boolean     support search time range
-   engine_type             str         - ``online`` :ref:`[ref] <demo online engine>` by
+   engine_type             str         - ``online`` :ref:`[ref] <online engines>` by
                                          default, other possibles values are:
                                        - ``offline`` :ref:`[ref] <offline engines>`
-                                       - ``online_dictionary``
-                                       - ``online_currency``
-                                       - ``online_url_search``
+                                       - ``online_dictionary`` :ref:`[ref] <online dictionary>`
+                                       - ``online_currency`` :ref:`[ref] <online currency>`
+                                       - ``online_url_search`` :ref:`[ref] <online url search>`
    ======================= =========== ========================================================
 
 .. _engine settings:
@@ -239,12 +240,18 @@ following parameters can be used to specify a search request:
 .. _engine results:
 .. _engine media types:
 
-Media Types
-===========
+Result Types (``template``)
+===========================
 
 Each result item of an engine can be of different media-types.  Currently the
-following media-types are supported.  To set another media-type as ``default``,
-the parameter ``template`` must be set to the desired type.
+following media-types are supported.  To set another media-type as
+:ref:`template default`, the parameter ``template`` must be set to the desired
+type.
+
+.. _template default:
+
+``default``
+-----------
 
 .. table::  Parameter of the **default** media type:
    :width: 100%
@@ -259,6 +266,11 @@ the parameter ``template`` must be set to the desired type.
    ========================= =====================================================
 
 
+.. _template images:
+
+``images``
+----------
+
 .. table::  Parameter of the **images** media type:
    :width: 100%
 
@@ -277,6 +289,11 @@ the parameter ``template`` must be set to the desired type.
    ========================= =====================================================
 
 
+.. _template videos:
+
+``videos``
+----------
+
 .. table::  Parameter of the **videos** media type:
    :width: 100%
 
@@ -292,6 +309,12 @@ the parameter ``template`` must be set to the desired type.
    thumbnail                 string, url to a small-preview image
    ========================= =====================================================
 
+
+.. _template torrent:
+
+``torrent``
+-----------
+
 .. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
 
 .. table::  Parameter of the **torrent** media type:
@@ -315,6 +338,12 @@ the parameter ``template`` must be set to the desired type.
    torrentfile               string, torrentfile of the result
    ========================= =====================================================
 
+
+.. _template map:
+
+``map``
+-------
+
 .. table::  Parameter of the **map** media type:
    :width: 100%
 
@@ -342,6 +371,12 @@ the parameter ``template`` must be set to the desired type.
    address.country           country of object
    ========================= =====================================================
 
+
+.. _template paper:
+
+``paper``
+---------
+
 .. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
 .. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
 
@@ -430,3 +465,4 @@ the parameter ``template`` must be set to the desired type.
    * - html_url
      - :py:class:`str`
      - URL to full article, HTML version
+

docs/src/searx.enginelib.rst → docs/dev/engines/enginelib.rst

@@ -1,15 +1,20 @@
 .. _searx.enginelib:
 
-============
-Engine model
-============
+==============
+Engine Library
+==============
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
 
 .. automodule:: searx.enginelib
   :members:
 
 .. _searx.enginelib.traits:
 
-=============
+
 Engine traits
 =============
 

docs/dev/engines/engines.rst

@@ -0,0 +1,9 @@
+.. _searx.engines loader:
+
+========================
+SearXNG's engines loader
+========================
+
+.. automodule:: searx.engines
+  :members:
+

docs/dev/engines/index.rst

@@ -0,0 +1,97 @@
+.. _engine implementations:
+
+======================
+Engine Implementations
+======================
+
+Framework Components
+====================
+
+.. toctree::
+   :maxdepth: 2
+
+   enginelib
+   engines
+   engine_overview
+
+
+Engine Types
+============
+
+The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine
+determines which :ref:`search processor <searx.search.processors>` is used by
+the engine.
+
+In this section a list of the enignes that are documented is given, a complete
+list of the engines can be found in the source under: :origin:`searx/engines`.
+
+.. _online engines:
+
+Online Engines
+--------------
+
+.. sidebar:: info
+
+   - :py:obj:`processors.online <searx.search.processors.online>`
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   demo/demo_online
+   xpath
+   online/*
+
+.. _offline engines:
+
+Offline Engines
+---------------
+
+.. sidebar:: info
+
+   - :py:obj:`processors.offline <searx.search.processors.offline>`
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   offline_concept
+   demo/demo_offline
+   offline/*
+
+.. _online url search:
+
+Online URL Search
+-----------------
+
+.. sidebar:: info
+
+   - :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>`
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   online_url_search/*
+
+.. _online currency:
+
+Online Currency
+---------------
+
+.. sidebar:: info
+
+   - :py:obj:`processors.online_currency <searx.search.processors.online_currency>`
+
+*no engine of this type is documented yet / comming soon*
+
+.. _online dictionary:
+
+Online Dictionary
+-----------------
+
+.. sidebar:: info
+
+   - :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>`
+
+*no engine of this type is documented yet / comming soon*

docs/dev/engines/offline/command-line-engines.rst

@@ -0,0 +1,23 @@
+.. _engine command:
+
+====================
+Command Line Engines
+====================
+
+.. sidebar:: info
+
+   - :origin:`command.py <searx/engines/command.py>`
+   - :ref:`offline engines`
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. sidebar:: info
+
+   Initial sponsored by `Search and Discovery Fund
+   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+
+.. automodule:: searx.engines.command
+  :members:

docs/admin/engines/nosql-engines.rst → docs/dev/engines/offline/nosql-engines.rst

@@ -1,3 +1,5 @@
+.. _nosql engines:
+
 ===============
 NoSQL databases
 ===============
@@ -8,6 +10,16 @@ NoSQL databases
    - `redis.io <https://redis.io/>`_
    - `MongoDB <https://www.mongodb.com>`_
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. sidebar:: info
+
+   Initial sponsored by `Search and Discovery Fund
+   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+
 The following `NoSQL databases`_ are supported:
 
 - :ref:`engine redis_server`
@@ -30,15 +42,8 @@ can still add them and limit the access by setting ``tokens`` as described in
 section :ref:`private engines`.
 
 
-Configure the engines
-=====================
-
-`NoSQL databases`_ are used for storing arbitrary data without first defining
-their structure.
-
-
 Extra Dependencies
-------------------
+==================
 
 For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
 install additional packages in Python's Virtual Environment of your SearXNG
@@ -49,6 +54,13 @@ instance.  To switch into the environment (:ref:`searxng-src`) you can use
   (searxng-pyenv)$ pip install ...
 
 
+Configure the engines
+=====================
+
+`NoSQL databases`_ are used for storing arbitrary data without first defining
+their structure.
+
+
 .. _engine redis_server:
 
 Redis Server
@@ -62,29 +74,9 @@ Redis Server
    - redis.io_
    - :origin:`redis_server.py <searx/engines/redis_server.py>`
 
+.. automodule:: searx.engines.redis_server
+  :members:
 
-Redis is an open source (BSD licensed), in-memory data structure (key value
-based) store.  Before configuring the ``redis_server`` engine, you must install
-the dependency redis_.
-
-Select a database to search in and set its index in the option ``db``.  You can
-either look for exact matches or use partial keywords to find what you are
-looking for by configuring ``exact_match_only``.  You find an example
-configuration below:
-
-.. code:: yaml
-
-  # Required dependency: redis
-
-  - name: myredis
-    shortcut : rds
-    engine: redis_server
-    exact_match_only: false
-    host: '127.0.0.1'
-    port: 6379
-    enable_http: true
-    password: ''
-    db: 0
 
 .. _engine mongodb:
 
@@ -99,37 +91,7 @@ MongoDB
    - MongoDB_
    - :origin:`mongodb.py <searx/engines/mongodb.py>`
 
-MongoDB_ is a document based database program that handles JSON like data.
-Before configuring the ``mongodb`` engine, you must install the dependency
-redis_.
-
-In order to query MongoDB_, you have to select a ``database`` and a
-``collection``.  Furthermore, you have to select a ``key`` that is going to be
-searched.  MongoDB_ also supports the option ``exact_match_only``, so configure
-it as you wish.  Below is an example configuration for using a MongoDB
-collection:
-
-.. code:: yaml
-
-  # MongoDB engine
-  # Required dependency: pymongo
-
-  - name: mymongo
-    engine: mongodb
-    shortcut: md
-    exact_match_only: false
-    host: '127.0.0.1'
-    port: 27017
-    enable_http: true
-    results_per_page: 20
-    database: 'business'
-    collection: 'reviews'  # name of the db collection
-    key: 'name'            # key in the collection to search for
-
-
-Acknowledgment
-==============
 
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+.. automodule:: searx.engines.mongodb
+  :members:
 

docs/dev/engines/offline/search-indexer-engines.rst

@@ -0,0 +1,62 @@
+=================
+Local Search APIs
+=================
+
+.. sidebar:: further read
+
+   - `Comparison to alternatives
+     <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
+
+.. contents::
+   :depth: 1
+   :local:
+   :backlinks: entry
+
+.. sidebar:: info
+
+   Initial sponsored by `Search and Discovery Fund
+   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+
+Administrators might find themselves wanting to integrate locally running search
+engines.  The following ones are supported for now:
+
+* `Elasticsearch`_
+* `Meilisearch`_
+* `Solr`_
+
+Each search engine is powerful, capable of full-text search.  All of the engines
+above are added to ``settings.yml`` just commented out, as you have to
+``base_url`` for all them.
+
+Please note that if you are not using HTTPS to access these engines, you have to
+enable HTTP requests by setting ``enable_http`` to ``True``.
+
+Furthermore, if you do not want to expose these engines on a public instance,
+you can still add them and limit the access by setting ``tokens`` as described
+in section :ref:`private engines`.
+
+.. _engine meilisearch:
+
+MeiliSearch
+===========
+
+.. automodule:: searx.engines.meilisearch
+  :members:
+
+
+.. _engine elasticsearch:
+
+Elasticsearch
+=============
+
+.. automodule:: searx.engines.elasticsearch
+  :members:
+
+.. _engine solr:
+
+Solr
+====
+
+.. automodule:: searx.engines.solr
+  :members:
+

docs/admin/engines/sql-engines.rst → docs/dev/engines/offline/sql-engines.rst

@@ -10,6 +10,16 @@ SQL Engines
    - `PostgreSQL <https://www.postgresql.org>`_
    - `MySQL <https://www.mysql.com>`_
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. sidebar:: info
+
+   Initial sponsored by `Search and Discovery Fund
+   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+
 With the *SQL engines* you can bind SQL databases into SearXNG.  The following
 Relational Database Management System (RDBMS) are supported:
 
@@ -42,6 +52,18 @@ add them and limit the access by setting ``tokens`` as described in section
 :ref:`private engines`.
 
 
+Extra Dependencies
+==================
+
+For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
+install additional packages in Python's Virtual Environment of your SearXNG
+instance.  To switch into the environment (:ref:`searxng-src`) you can use
+:ref:`searxng.sh`::
+
+  $ sudo utils/searxng.sh instance cmd bash
+  (searxng-pyenv)$ pip install ...
+
+
 Configure the engines
 =====================
 
@@ -64,45 +86,8 @@ SQLite
 
    - :origin:`sqlite.py <searx/engines/sqlite.py>`
 
-.. _MediathekView: https://mediathekview.de/
-
-SQLite is a small, fast and reliable SQL database engine.  It does not require
-any extra dependency.  To demonstrate the power of database engines, here is a
-more complex example which reads from a MediathekView_ (DE) movie database.  For
-this example of the SQlite engine download the database:
-
-- https://liste.mediathekview.de/filmliste-v2.db.bz2
-
-and unpack into ``searx/data/filmliste-v2.db``.  To search the database use e.g
-Query to test: ``!mediathekview concert``
-
-.. code:: yaml
-
-   - name: mediathekview
-     engine: sqlite
-     disabled: False
-     categories: general
-     result_template: default.html
-     database: searx/data/filmliste-v2.db
-     query_str:  >-
-       SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
-              COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
-              description AS content
-         FROM film
-        WHERE title LIKE :wildcard OR description LIKE :wildcard
-        ORDER BY duration DESC
-
-
-Extra Dependencies
-------------------
-
-For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
-install additional packages in Python's Virtual Environment of your SearXNG
-instance.  To switch into the environment (:ref:`searxng-src`) you can use
-:ref:`searxng.sh`::
-
-  $ sudo utils/searxng.sh instance cmd bash
-  (searxng-pyenv)$ pip install ...
+.. automodule:: searx.engines.sqlite
+  :members:
 
 
 .. _engine postgresql:
@@ -115,20 +100,10 @@ PostgreSQL
 .. sidebar:: info
 
    - :origin:`postgresql.py <searx/engines/postgresql.py>`
-   - ``pip install`` psycopg2_
-
-PostgreSQL is a powerful and robust open source database.  Before configuring
-the PostgreSQL engine, you must install the dependency ``psychopg2``.  You can
-find an example configuration below:
+   - ``pip install`` `psycopg2-binary <psycopg2>`_
 
-.. code:: yaml
-
-   - name: my_database
-     engine: postgresql
-     database: my_database
-     username: searxng
-     password: password
-     query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
+.. automodule:: searx.engines.postgresql
+  :members:
 
 .. _engine mysql_server:
 
@@ -140,27 +115,7 @@ MySQL
    - :origin:`mysql_server.py <searx/engines/mysql_server.py>`
    - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
 
-MySQL is said to be the most popular open source database. Before enabling MySQL
-engine, you must install the package ``mysql-connector-python``.
-
-The authentication plugin is configurable by setting ``auth_plugin`` in the
-attributes.  By default it is set to ``caching_sha2_password``.  This is an
-example configuration for querying a MySQL server:
-
-.. code:: yaml
-
-   - name: my_database
-     engine: mysql_server
-     database: my_database
-     username: searxng
-     password: password
-     limit: 5
-     query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
-
-
-Acknowledgment
-==============
 
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+.. automodule:: searx.engines.mysql_server
+  :members:
 

docs/dev/offline_engines.rst → docs/dev/engines/offline_concept.rst

@@ -1,15 +1,14 @@
-.. _offline engines:
-
 ===============
-Offline Engines
+Offline Concept
 ===============
 
 .. sidebar:: offline engines
 
    - :ref:`demo offline engine`
-   - :ref:`sql engines`
    - :ref:`engine command`
-   - :origin:`Redis <searx/engines/redis_server.py>`
+   - :ref:`sql engines`
+   - :ref:`nosql engines`
+   - :py:obj:`searx.search.processors.offline`
 
 To extend the functionality of SearXNG, offline engines are going to be
 introduced.  An offline engine is an engine which does not need Internet
@@ -31,7 +30,6 @@ Programming Interface
   in advance.
 
 :py:func:`search(query, params) <searx.engines.demo_offline.searc>`
-
   Each offline engine has a function named ``search``.  This function is
   responsible to perform a search and return the results in a presentable
   format. (Where *presentable* means presentable by the selected result
@@ -69,10 +67,3 @@ administrators can set token(s) for each of the :ref:`private engines`.  If a
 query contains a valid token, then SearXNG performs the requested private
 search.  If not, requests from an offline engines return errors.
 
-
-Acknowledgement
-===============
-
-This development was sponsored by `Search and Discovery Fund
-<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .
-

docs/dev/engines/online/annas_archive.rst

@@ -0,0 +1,13 @@
+.. _annas_archive engine:
+
+==============
+Anna's Archive
+==============
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. automodule:: searx.engines.annas_archive
+   :members:

docs/src/searx.engine.archlinux.rst → docs/dev/engines/online/archlinux.rst

@@ -4,6 +4,11 @@
 Arch Linux
 ==========
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.archlinux
   :members:
 

docs/src/searx.engines.bing.rst → docs/dev/engines/online/bing.rst

@@ -4,7 +4,7 @@
 Bing Engines
 ============
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engine.dailymotion.rst → docs/dev/engines/online/dailymotion.rst

@@ -4,5 +4,10 @@
 Dailymotion
 ===========
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.dailymotion
   :members:

docs/src/searx.engine.duckduckgo.rst → docs/dev/engines/online/duckduckgo.rst

@@ -1,10 +1,10 @@
 .. _duckduckgo engines:
 
 =================
-DukcDukGo engines
+DukcDukGo Engines
 =================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engines.google.rst → docs/dev/engines/online/google.rst

@@ -4,7 +4,7 @@
 Google Engines
 ==============
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engines.peertube.rst → docs/dev/engines/online/peertube.rst

@@ -4,7 +4,7 @@
 Peertube Engines
 ================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/dev/engines/online/recoll.rst

@@ -0,0 +1,13 @@
+.. _engine recoll:
+
+=============
+Recoll Engine
+=============
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. automodule:: searx.engines.recoll
+   :members:

docs/src/searx.engines.startpage.rst → docs/dev/engines/online/startpage.rst

@@ -1,10 +1,10 @@
 .. _startpage engines:
 
 =================
-Startpage engines
+Startpage Engines
 =================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/dev/engines/online/torznab.rst

@@ -0,0 +1,13 @@
+.. _torznab engine:
+
+==============
+Torznab WebAPI
+==============
+
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. automodule:: searx.engines.torznab
+   :members:

docs/src/searx.engines.wikipedia.rst → docs/dev/engines/online/wikipedia.rst

@@ -4,7 +4,7 @@
 Wikimedia
 =========
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engines.yahoo.rst → docs/dev/engines/online/yahoo.rst

@@ -4,5 +4,10 @@
 Yahoo Engine
 ============
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.yahoo
   :members:

docs/src/searx.engines.tineye.rst → docs/dev/engines/online_url_search/tineye.rst

@@ -4,6 +4,11 @@
 Tineye
 ======
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.tineye
   :members:
 

docs/admin/engines/searx.engines.xpath.rst → docs/dev/engines/xpath.rst

@@ -4,6 +4,11 @@
 XPath Engine
 ============
 
+.. contents::
+   :depth: 2
+   :local:
+   :backlinks: entry
+
 .. automodule:: searx.engines.xpath
   :members:
 

docs/dev/index.rst

@@ -4,12 +4,10 @@ Developer documentation
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
 
    quickstart
    contribution_guide
-   engine_overview
-   offline_engines
+   engines/index
    search_api
    plugins
    translation

docs/dev/lxcdev.rst

@@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in
    section :ref:`internet connectivity docker`.
 
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/dev/makefile.rst

@@ -22,7 +22,7 @@ Calling the ``help`` target gives a first overview (``make help``):
 
 .. program-output:: bash -c "cd ..; make --no-print-directory help"
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry
@@ -243,14 +243,14 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
 We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
 Documentation <contrib docs>` section.  If you want to edit the documentation
 read our :ref:`make docs.live` section.  If you are working in your own brand,
-adjust your :ref:`settings global`.
+adjust your :ref:`settings brand`.
 
 .. _make docs.gh-pages:
 
 ``make docs.gh-pages``
 ======================
 
-To deploy on github.io first adjust your :ref:`settings global`.  For any
+To deploy on github.io first adjust your :ref:`settings brand`.  For any
 further read :ref:`deploy on github.io`.
 
 .. _make test:

docs/dev/reST.rst

@@ -37,7 +37,7 @@ docs.live <make docs.live>` to build HTML while editing.
    - DOT_, `Graphviz's dot`_, Graphviz_
 
 
-.. contents:: Contents
+.. contents::
    :depth: 3
    :local:
    :backlinks: entry

docs/dev/searxng_extra/index.rst

@@ -9,7 +9,6 @@ developers.
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
 
    update
    standalone_searx.py

docs/index.rst

@@ -43,7 +43,6 @@ If you don't trust anyone, you can set up your own, see :ref:`installation`.
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
 
    user/index
    own-instance

docs/own-instance.rst

@@ -7,7 +7,7 @@ Why use a private instance?
   \.\. is a common question among SearXNG users.  Before answering this
   question, see what options a SearXNG user has.
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/index.rst

@@ -8,7 +8,6 @@ every item from the source code, but we will add documentation when requested.
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
    :glob:
 
    searx.*

docs/src/searx.botdetection.rst

@@ -4,7 +4,7 @@
 Bot Detection
 =============
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.engines.annas_archive.rst

@@ -1,2 +0,0 @@
-.. automodule:: searx.engines.annas_archive
-   :members:

docs/src/searx.engines.rst

@@ -1,8 +0,0 @@
-.. _searx.engines:
-
-=================
-SearXNG's engines
-=================
-
-.. automodule:: searx.engines
-  :members:

docs/src/searx.engines.torznab.rst

@@ -1,2 +0,0 @@
-.. automodule:: searx.engines.torznab
-   :members:

docs/src/searx.locales.rst

@@ -4,7 +4,7 @@
 Locales
 =======
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/src/searx.search.processors.rst

@@ -4,7 +4,7 @@
 Search processors
 =================
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry
@@ -34,7 +34,7 @@ Online currency processor
 .. automodule:: searx.search.processors.online_currency
   :members:
 
-Online Dictionary processor
+Online dictionary processor
 ===========================
 
 .. automodule:: searx.search.processors.online_dictionary

docs/user/configured_engines.rst

@@ -22,7 +22,7 @@ Configured Engines
    called *tabs*), engines can be queried by their name or the categories they
    belong to, by using a :ref:`\!bing syntax <search-syntax>`.
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/user/index.rst

@@ -2,7 +2,7 @@
 User information
 ================
 
-.. contents:: Contents
+.. contents::
    :depth: 3
    :local:
    :backlinks: entry

docs/utils/index.rst

@@ -10,7 +10,6 @@ and developers.
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents
 
    searxng.sh
    lxc.sh

docs/utils/lxc.sh.rst

@@ -26,7 +26,7 @@ to care about*).
    - `LXC/LXD Image Server`_
    - `LXD@github`_
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

docs/utils/searxng.sh.rst

@@ -15,7 +15,7 @@ script :origin:`utils/searxng.sh`.
    - :ref:`installation nginx`
    - :ref:`installation apache`
 
-.. contents:: Contents
+.. contents::
    :depth: 2
    :local:
    :backlinks: entry

searx/enginelib/__init__.py

@@ -1,18 +1,15 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""Engine related implementations
+"""Implementations of the framework for the SearXNG engines.
 
-.. note::
+.. hint::
 
-   The long term goal is to modularize all relevant implementations to the
-   engines here in this Python package.  In addition to improved modularization,
-   this will also be necessary in part because the probability of circular
-   imports will increase due to the increased typification of implementations in
-   the future.
+   The long term goal is to modularize all implementations of the engine
+   framework here in this Python package.  ToDo:
 
-   ToDo:
+   - move implementations of the :ref:`searx.engines loader` to a new module in
+     the :py:obj:`searx.enginelib` namespace.
 
-   - move :py:obj:`searx.engines.load_engine` to a new module `searx.enginelib`.
 """
 
 
@@ -36,7 +33,7 @@ class Engine:  # pylint: disable=too-few-public-methods
     # Common options in the engine module
 
     engine_type: str
-    """Type of the engine (:origin:`searx/search/processors`)"""
+    """Type of the engine (:ref:`searx.search.processors`)"""
 
     paging: bool
     """Engine supports multiple pages."""

searx/engines/__init__.py

@@ -1,8 +1,6 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""This module implements the engine loader.
-
-Load and initialize the ``engines``, see :py:func:`load_engines` and register
+"""Load and initialize the ``engines``, see :py:func:`load_engines` and register
 :py:obj:`engine_shortcuts`.
 
 usage::

searx/engines/annas_archive.py

@@ -1,24 +1,12 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-""".. _annas_archive engine:
-
-==============
-Anna's Archive
-==============
+"""`Anna's Archive`_ is a free non-profit online shadow library metasearch
+engine providing access to a variety of book resources (also via IPFS), created
+by a team of anonymous archivists (AnnaArchivist_).
 
 .. _Anna's Archive: https://annas-archive.org/
 .. _AnnaArchivist: https://annas-software.org/AnnaArchivist/annas-archive
 
-`Anna's Archive`_ is a free non-profit online shadow library metasearch engine
-providing access to a variety of book resources (also via IPFS), created by a
-team of anonymous archivists (AnnaArchivist_).
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-
 Configuration
 =============
 
@@ -41,7 +29,6 @@ for *newest* articles and journals (PDF) / by shortcut ``!aaa <search-term>``.
     aa_ext: 'pdf'
     aa_sort: 'newest'
 
-
 Implementations
 ===============
 

searx/engines/command.py

@@ -1,6 +1,77 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
-"""
- Command (offline)
+"""With *command engines* administrators can run engines to integrate arbitrary
+shell commands.
+
+.. attention::
+
+   When creating and enabling a ``command`` engine on a public instance, you
+   must be careful to avoid leaking private data.
+
+The easiest solution is to limit the access by setting ``tokens`` as described
+in section :ref:`private engines`.  The engine base is flexible.  Only your
+imagination can limit the power of this engine (and maybe security concerns).
+
+Configuration
+=============
+
+The following options are available:
+
+``command``:
+  A comma separated list of the elements of the command.  A special token
+  ``{{QUERY}}`` tells where to put the search terms of the user. Example:
+
+  .. code:: yaml
+
+     ['ls', '-l', '-h', '{{QUERY}}']
+
+``delimiter``:
+  A mapping containing a delimiter ``char`` and the *titles* of each element in
+  ``keys``.
+
+``parse_regex``:
+  A dict containing the regular expressions for each result key.
+
+``query_type``:
+
+  The expected type of user search terms.  Possible values: ``path`` and
+  ``enum``.
+
+  ``path``:
+    Checks if the user provided path is inside the working directory.  If not,
+    the query is not executed.
+
+  ``enum``:
+    Is a list of allowed search terms.  If the user submits something which is
+    not included in the list, the query returns an error.
+
+``query_enum``:
+  A list containing allowed search terms if ``query_type`` is set to ``enum``.
+
+``working_dir``:
+  The directory where the command has to be executed.  Default: ``./``.
+
+``result_separator``:
+  The character that separates results. Default: ``\\n``.
+
+Example
+=======
+
+The example engine below can be used to find files with a specific name in the
+configured working directory:
+
+.. code:: yaml
+
+  - name: find
+    engine: command
+    command: ['find', '.', '-name', '{{QUERY}}']
+    query_type: path
+    shortcut: fnd
+    delimiter:
+        chars: ' '
+        keys: ['line']
+
+Implementations
+===============
 """
 
 import re

searx/engines/elasticsearch.py

@@ -1,6 +1,44 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
-"""
- Elasticsearch
+""".. sidebar:: info
+
+   - :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
+   - `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
+   - `Elasticsearch Guide
+     <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
+   - `Install Elasticsearch
+     <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
+
+Elasticsearch_ supports numerous ways to query the data it is storing.  At the
+moment the engine supports the most popular search methods (``query_type``):
+
+- ``match``,
+- ``simple_query_string``,
+- ``term`` and
+- ``terms``.
+
+If none of the methods fit your use case, you can select ``custom`` query type
+and provide the JSON payload to submit to Elasticsearch in
+``custom_query_json``.
+
+Example
+=======
+
+The following is an example configuration for an Elasticsearch_ instance with
+authentication configured to read from ``my-index`` index.
+
+.. code:: yaml
+
+  - name: elasticsearch
+    shortcut: es
+    engine: elasticsearch
+    base_url: http://localhost:9200
+    username: elastic
+    password: changeme
+    index: my-index
+    query_type: match
+    # custom_query_json: '{ ... }'
+    enable_http: true
+
 """
 
 from json import loads, dumps

searx/engines/meilisearch.py

@@ -1,7 +1,35 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""
- Meilisearch
+""".. sidebar:: info
+
+   - :origin:`meilisearch.py <searx/engines/meilisearch.py>`
+   - `MeiliSearch <https://www.meilisearch.com>`_
+   - `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
+   - `Install MeiliSearch
+     <https://docs.meilisearch.com/learn/getting_started/installation.html>`_
+
+MeiliSearch_ is aimed at individuals and small companies.  It is designed for
+small-scale (less than 10 million documents) data collections.  E.g. it is great
+for storing web pages you have visited and searching in the contents later.
+
+The engine supports faceted search, so you can search in a subset of documents
+of the collection.  Furthermore, you can search in MeiliSearch_ instances that
+require authentication by setting ``auth_token``.
+
+Example
+=======
+
+Here is a simple example to query a Meilisearch instance:
+
+.. code:: yaml
+
+  - name: meilisearch
+    engine: meilisearch
+    shortcut: mes
+    base_url: http://localhost:7700
+    index: my-index
+    enable_http: true
+
 """
 
 # pylint: disable=global-statement

searx/engines/mongodb.py

@@ -1,11 +1,53 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""MongoDB engine (Offline)
+"""MongoDB_ is a document based database program that handles JSON like data.
+Before configuring the ``mongodb`` engine, you must install the dependency
+pymongo_.
+
+Configuration
+=============
+
+In order to query MongoDB_, you have to select a ``database`` and a
+``collection``.  Furthermore, you have to select a ``key`` that is going to be
+searched.  MongoDB_ also supports the option ``exact_match_only``, so configure
+it as you wish.
+
+Example
+=======
+
+Below is an example configuration for using a MongoDB collection:
+
+.. code:: yaml
+
+  # MongoDB engine
+  # Required dependency: pymongo
+
+  - name: mymongo
+    engine: mongodb
+    shortcut: md
+    exact_match_only: false
+    host: '127.0.0.1'
+    port: 27017
+    enable_http: true
+    results_per_page: 20
+    database: 'business'
+    collection: 'reviews'  # name of the db collection
+    key: 'name'            # key in the collection to search for
+
+Implementations
+===============
 
 """
 
 import re
-from pymongo import MongoClient  # pyright: ignore # pylint: disable=import-error
+
+try:
+    from pymongo import MongoClient  # type: ignore
+except ImportError:
+    # import error is ignored because the admin has to install pymongo manually
+    # to use the engine
+    pass
+
 
 engine_type = 'offline'
 

searx/engines/mysql_server.py

@@ -1,12 +1,37 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""MySQL database (offline)
+"""MySQL is said to be the most popular open source database.  Before enabling
+MySQL engine, you must install the package ``mysql-connector-python``.
+
+The authentication plugin is configurable by setting ``auth_plugin`` in the
+attributes.  By default it is set to ``caching_sha2_password``.
+
+Example
+=======
+
+This is an example configuration for querying a MySQL server:
+
+.. code:: yaml
+
+   - name: my_database
+     engine: mysql_server
+     database: my_database
+     username: searxng
+     password: password
+     limit: 5
+     query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
+
+Implementations
+===============
 
 """
 
-# import error is ignored because the admin has to install mysql manually to use
-# the engine
-import mysql.connector  # pyright: ignore # pylint: disable=import-error
+try:
+    import mysql.connector  # type: ignore
+except ImportError:
+    # import error is ignored because the admin has to install mysql manually to use
+    # the engine
+    pass
 
 engine_type = 'offline'
 auth_plugin = 'caching_sha2_password'

searx/engines/postgresql.py

@@ -1,12 +1,33 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""PostgreSQL database (offline)
+"""PostgreSQL is a powerful and robust open source database.  Before configuring
+the PostgreSQL engine, you must install the dependency ``psychopg2``.
+
+Example
+=======
+
+Below is an example configuration:
+
+.. code:: yaml
+
+   - name: my_database
+     engine: postgresql
+     database: my_database
+     username: searxng
+     password: password
+     query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
+
+Implementations
+===============
 
 """
 
-# import error is ignored because the admin has to install mysql manually to use
-# the engine
-import psycopg2  # pyright: ignore # pylint: disable=import-error
+try:
+    import psycopg2  # type: ignore
+except ImportError:
+    # import error is ignored because the admin has to install postgresql
+    # manually to use the engine.
+    pass
 
 engine_type = 'offline'
 host = "127.0.0.1"

searx/engines/recoll.py

@@ -1,6 +1,51 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
-"""
- Recoll (local search engine)
+# lint: pylint
+""".. sidebar:: info
+
+   - `Recoll <https://www.lesbonscomptes.com/recoll/>`_
+   - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
+   - :origin:`searx/engines/recoll.py`
+
+Recoll_ is a desktop full-text search tool based on Xapian.  By itself Recoll_
+does not offer WEB or API access, this can be achieved using recoll-webui_
+
+Configuration
+=============
+
+You must configure the following settings:
+
+``base_url``:
+  Location where recoll-webui can be reached.
+
+``mount_prefix``:
+  Location where the file hierarchy is mounted on your *local* filesystem.
+
+``dl_prefix``:
+  Location where the file hierarchy as indexed by recoll can be reached.
+
+``search_dir``:
+  Part of the indexed file hierarchy to be search, if empty the full domain is
+  searched.
+
+Example
+=======
+
+Scenario:
+
+#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
+#. the Recoll search interface can be reached at https://recoll.example.org/ and
+#. the contents of this filesystem can be reached though https://download.example.org/reference
+
+.. code:: yaml
+
+   base_url: https://recoll.example.org/
+   mount_prefix: /export/documents
+   dl_prefix: https://download.example.org
+   search_dir: ''
+
+Implementations
+===============
+
 """
 
 from datetime import date, timedelta
@@ -33,7 +78,7 @@ embedded_url = '<{ttype} controls height="166px" ' + 'src="{url}" type="{mtype}"
 
 # helper functions
 def get_time_range(time_range):
-    sw = {'day': 1, 'week': 7, 'month': 30, 'year': 365}
+    sw = {'day': 1, 'week': 7, 'month': 30, 'year': 365}  # pylint: disable=invalid-name
 
     offset = sw.get(time_range, 0)
     if not offset:

searx/engines/redis_server.py

@@ -1,6 +1,37 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""Redis engine (offline)
+"""Redis is an open source (BSD licensed), in-memory data structure (key value
+based) store.  Before configuring the ``redis_server`` engine, you must install
+the dependency redis_.
+
+Configuration
+=============
+
+Select a database to search in and set its index in the option ``db``.  You can
+either look for exact matches or use partial keywords to find what you are
+looking for by configuring ``exact_match_only``.
+
+Example
+=======
+
+Below is an example configuration:
+
+.. code:: yaml
+
+  # Required dependency: redis
+
+  - name: myredis
+    shortcut : rds
+    engine: redis_server
+    exact_match_only: false
+    host: '127.0.0.1'
+    port: 6379
+    enable_http: true
+    password: ''
+    db: 0
+
+Implementations
+===============
 
 """
 

searx/engines/solr.py

@@ -1,7 +1,31 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""
- Solr
+""".. sidebar:: info
+
+   - :origin:`solr.py <searx/engines/solr.py>`
+   - `Solr <https://solr.apache.org>`_
+   - `Solr Resources <https://solr.apache.org/resources.html>`_
+   - `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
+
+Solr_ is a popular search engine based on Lucene, just like Elasticsearch_.  But
+instead of searching in indices, you can search in collections.
+
+Example
+=======
+
+This is an example configuration for searching in the collection
+``my-collection`` and get the results in ascending order.
+
+.. code:: yaml
+
+  - name: solr
+    engine: solr
+    shortcut: slr
+    base_url: http://localhost:8983
+    collection: my-collection
+    sort: asc
+    enable_http: true
+
 """
 
 # pylint: disable=global-statement

searx/engines/sqlite.py

@@ -1,7 +1,40 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
+"""SQLite is a small, fast and reliable SQL database engine.  It does not require
+any extra dependency.
 
-"""SQLite database (Offline)
+Example
+=======
+
+.. _MediathekView: https://mediathekview.de/
+
+To demonstrate the power of database engines, here is a more complex example
+which reads from a MediathekView_ (DE) movie database.  For this example of the
+SQlite engine download the database:
+
+- https://liste.mediathekview.de/filmliste-v2.db.bz2
+
+and unpack into ``searx/data/filmliste-v2.db``.  To search the database use e.g
+Query to test: ``!mediathekview concert``
+
+.. code:: yaml
+
+   - name: mediathekview
+     engine: sqlite
+     disabled: False
+     categories: general
+     result_template: default.html
+     database: searx/data/filmliste-v2.db
+     query_str:  >-
+       SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
+              COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
+              description AS content
+         FROM film
+        WHERE title LIKE :wildcard OR description LIKE :wildcard
+        ORDER BY duration DESC
+
+Implementations
+===============
 
 """
 
@@ -26,14 +59,15 @@ def init(engine_settings):
 
 @contextlib.contextmanager
 def sqlite_cursor():
-    """Implements a `Context Manager`_ for a :py:obj:`sqlite3.Cursor`.
+    """Implements a :py:obj:`Context Manager <contextlib.contextmanager>` for a
+    :py:obj:`sqlite3.Cursor`.
 
-    Open database in read only mode: if the database doesn't exist.
-    The default mode creates an empty file on the file system.
+    Open database in read only mode: if the database doesn't exist.  The default
+    mode creates an empty file on the file system.  See:
 
-    see:
     * https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
     * https://www.sqlite.org/uri.html
+
     """
     uri = 'file:' + database + '?mode=ro'
     with contextlib.closing(sqlite3.connect(uri, uri=True)) as connect:

searx/engines/torznab.py

@@ -1,17 +1,6 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-""".. _torznab engine:
-
-==============
-Torznab WebAPI
-==============
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-Torznab_ is an API specification that provides a standardized way to query
+"""Torznab_ is an API specification that provides a standardized way to query
 torrent site for content. It is used by a number of torrent applications,
 including Prowlarr_ and Jackett_.
 
@@ -55,7 +44,6 @@ The engine has the following settings:
 .. _Jackett-categories:
    https://github.com/Jackett/Jackett/wiki/Jackett-Categories
 
-
 Implementations
 ===============
 

searx/engines/xpath.py

@@ -3,8 +3,55 @@
 """The XPath engine is a *generic* engine with which it is possible to configure
 engines in the settings.
 
-Here is a simple example of a XPath engine configured in the
-:ref:`settings engine` section, further read :ref:`engines-dev`.
+.. _XPath selector: https://quickref.me/xpath.html#xpath-selectors
+
+Configuration
+=============
+
+Request:
+
+- :py:obj:`search_url`
+- :py:obj:`lang_all`
+- :py:obj:`soft_max_redirects`
+- :py:obj:`cookies`
+- :py:obj:`headers`
+
+Paging:
+
+- :py:obj:`paging`
+- :py:obj:`page_size`
+- :py:obj:`first_page_num`
+
+Time Range:
+
+- :py:obj:`time_range_support`
+- :py:obj:`time_range_url`
+- :py:obj:`time_range_map`
+
+Safe-Search:
+
+- :py:obj:`safe_search_support`
+- :py:obj:`safe_search_map`
+
+Response:
+
+- :py:obj:`no_result_for_http_status`
+
+`XPath selector`_:
+
+- :py:obj:`results_xpath`
+- :py:obj:`url_xpath`
+- :py:obj:`title_xpath`
+- :py:obj:`content_xpath`
+- :py:obj:`thumbnail_xpath`
+- :py:obj:`suggestion_xpath`
+
+
+Example
+=======
+
+Here is a simple example of a XPath engine configured in the :ref:`settings
+engine` section, further read :ref:`engines-dev`.
 
 .. code:: yaml
 
@@ -16,6 +63,9 @@ Here is a simple example of a XPath engine configured in the
     title_xpath : //article[@class="repo-summary"]//a[@class="repo-link"]
     content_xpath : //article[@class="repo-summary"]/p
 
+Implementations
+===============
+
 """
 
 from urllib.parse import urlencode
@@ -26,7 +76,7 @@ from searx.network import raise_for_httperror
 
 search_url = None
 """
-Search URL of the engine. Example::
+Search URL of the engine.  Example::
 
     https://example.org/?search={query}&page={pageno}{time_range}{safe_search}
 
@@ -74,30 +124,33 @@ soft_max_redirects = 0
 '''Maximum redirects, soft limit. Record an error but don't stop the engine'''
 
 results_xpath = ''
-'''XPath selector for the list of result items'''
+'''`XPath selector`_ for the list of result items'''
 
 url_xpath = None
-'''XPath selector of result's ``url``.'''
+'''`XPath selector`_ of result's ``url``.'''
 
 content_xpath = None
-'''XPath selector of result's ``content``.'''
+'''`XPath selector`_ of result's ``content``.'''
 
 title_xpath = None
-'''XPath selector of result's ``title``.'''
+'''`XPath selector`_ of result's ``title``.'''
 
 thumbnail_xpath = False
-'''XPath selector of result's ``img_src``.'''
+'''`XPath selector`_ of result's ``img_src``.'''
 
 suggestion_xpath = ''
-'''XPath selector of result's ``suggestion``.'''
+'''`XPath selector`_ of result's ``suggestion``.'''
 
 cached_xpath = ''
 cached_url = ''
 
 cookies = {}
+'''Some engines might offer different result based on cookies.
+Possible use-case: To set safesearch cookie.'''
+
 headers = {}
-'''Some engines might offer different result based on cookies or headers.
-Possible use-case: To set safesearch cookie or header to moderate.'''
+'''Some engines might offer different result based headers.  Possible use-case:
+To set header to moderate.'''
 
 paging = False
 '''Engine supports paging [True or False].'''