filtron.rst 4.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184
  1. .. _searx filtron:
  2. ==========================
  3. How to protect an instance
  4. ==========================
  5. .. sidebar:: further reading
  6. - :ref:`filtron.sh`
  7. - :ref:`nginx searx site`
  8. .. contents:: Contents
  9. :depth: 2
  10. :local:
  11. :backlinks: entry
  12. .. _filtron: https://github.com/asciimoo/filtron
  13. Searx depends on external search services. To avoid the abuse of these services
  14. it is advised to limit the number of requests processed by searx.
  15. An application firewall, filtron_ solves exactly this problem. Filtron is just
  16. a middleware between your web server (nginx, apache, ...) and searx, we describe
  17. such infratructures in chapter: :ref:`architecture`.
  18. filtron & go
  19. ============
  20. .. _Go: https://golang.org/
  21. .. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md
  22. Filtron needs Go_ installed. If Go_ is preinstalled, filtron_ is simply
  23. installed by ``go get`` package management (see `filtron README`_). If you use
  24. filtron as middleware, a more isolated setup is recommended. To simplify such
  25. an installation and the maintenance of, use our script :ref:`filtron.sh`.
  26. Sample configuration of filtron
  27. ===============================
  28. .. sidebar:: Tooling box
  29. - :origin:`/etc/filtron/rules.json <utils/templates/etc/filtron/rules.json>`
  30. An example configuration can be find below. This configuration limits the access
  31. of:
  32. - scripts or applications (roboagent limit)
  33. - webcrawlers (botlimit)
  34. - IPs which send too many requests (IP limit)
  35. - too many json, csv, etc. requests (rss/json limit)
  36. - the same UserAgent of if too many requests (useragent limit)
  37. .. code:: json
  38. [
  39. { "name": "search request",
  40. "filters": [
  41. "Param:q",
  42. "Path=^(/|/search)$"
  43. ],
  44. "interval": "<time-interval-in-sec (int)>",
  45. "limit": "<max-request-number-in-interval (int)>",
  46. "subrules": [
  47. {
  48. "name": "roboagent limit",
  49. "interval": "<time-interval-in-sec (int)>",
  50. "limit": "<max-request-number-in-interval (int)>",
  51. "filters": [
  52. "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"
  53. ],
  54. "actions": [
  55. { "name": "log"},
  56. { "name": "block",
  57. "params": {
  58. "message": "Rate limit exceeded"
  59. }
  60. }
  61. ]
  62. },
  63. {
  64. "name": "botlimit",
  65. "limit": 0,
  66. "stop": true,
  67. "filters": [
  68. "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"
  69. ],
  70. "actions": [
  71. { "name": "log"},
  72. { "name": "block",
  73. "params": {
  74. "message": "Rate limit exceeded"
  75. }
  76. }
  77. ]
  78. },
  79. {
  80. "name": "IP limit",
  81. "interval": "<time-interval-in-sec (int)>",
  82. "limit": "<max-request-number-in-interval (int)>",
  83. "stop": true,
  84. "aggregations": [
  85. "Header:X-Forwarded-For"
  86. ],
  87. "actions": [
  88. { "name": "log"},
  89. { "name": "block",
  90. "params": {
  91. "message": "Rate limit exceeded"
  92. }
  93. }
  94. ]
  95. },
  96. {
  97. "name": "rss/json limit",
  98. "interval": "<time-interval-in-sec (int)>",
  99. "limit": "<max-request-number-in-interval (int)>",
  100. "stop": true,
  101. "filters": [
  102. "Param:format=(csv|json|rss)"
  103. ],
  104. "actions": [
  105. { "name": "log"},
  106. { "name": "block",
  107. "params": {
  108. "message": "Rate limit exceeded"
  109. }
  110. }
  111. ]
  112. },
  113. {
  114. "name": "useragent limit",
  115. "interval": "<time-interval-in-sec (int)>",
  116. "limit": "<max-request-number-in-interval (int)>",
  117. "aggregations": [
  118. "Header:User-Agent"
  119. ],
  120. "actions": [
  121. { "name": "log"},
  122. { "name": "block",
  123. "params": {
  124. "message": "Rate limit exceeded"
  125. }
  126. }
  127. ]
  128. }
  129. ]
  130. }
  131. ]
  132. .. _filtron route request:
  133. Route request through filtron
  134. =============================
  135. Filtron can be started using the following command:
  136. .. code:: sh
  137. $ filtron -rules rules.json
  138. It listens on ``127.0.0.1:4004`` and forwards filtered requests to
  139. ``127.0.0.1:8888`` by default.
  140. Use it along with ``nginx`` with the following example configuration.
  141. .. code:: nginx
  142. location / {
  143. proxy_pass http://127.0.0.1:4004/;
  144. proxy_set_header Host $http_host;
  145. proxy_set_header X-Real-IP $remote_addr;
  146. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  147. proxy_set_header X-Scheme $scheme;
  148. }
  149. Requests are coming from port 4004 going through filtron and then forwarded to
  150. port 8888 where a searx is being run. For a complete setup see: :ref:`nginx
  151. searx site`.