You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

UPGRADING.rst 20KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341
  1. **************************
  2. Upgrading to Graylog 3.0.x
  3. **************************
  4. .. _upgrade-from-24-to-30:
  5. This file only contains the upgrade note for the upcoming release.
  6. Please see `our documentation <http://docs.graylog.org/en/latest/pages/upgrade.html>`_
  7. for the complete upgrade notes.
  8. Elasticsearch Version Requirements
  9. ==================================
  10. Graylog 3.0 drops support for Elasticsearch versions before 5.6.x. That means you have to upgrade Elasticsearch to at least version 5.6.5 before upgrading Graylog to version 3.0. Make sure to read the Elasticsearch upgrade guides before doing that.
  11. Simplified HTTP interface configuration
  12. =======================================
  13. Graylog used to have a lot of different settings regarding the various HTTP interfaces it provides, namely the Graylog REST API and the Graylog web interface.
  14. This mostly originates from the fact that Graylog used to consist of two components before Graylog 2.0.0, a server component and a separate web interface.
  15. The changes in this release finally merge the HTTP listeners for the Graylog REST API and web interface into a single HTTP listener, which should make the initial configuration of Graylog simpler and reduce errors caused by conflicting settings.
  16. The path of the Graylog REST API is now hard-coded to ``/api``, so if you're still using the legacy URI on port 12900/tcp or have been using a custom path (via the ``rest_listen_uri`` or ``rest_transport_uri`` settings), you'll have to update the URI used to access the Graylog REST API.
  17. If you are using a reverse proxy in front of Graylog (like nginx) and configured it to set the ``X-Graylog-Server-URL`` HTTP header, you have to remove the ``api/`` suffix because that is now the default. (as mentioned above)
  18. Example::
  19. # This nginx setting in Graylog <3.0 ...
  20. header_upstream X-Graylog-Server-URL http://{host}/api
  21. # ... needs to be changed to the following with Graylog 3.0
  22. header_upstream X-Graylog-Server-URL http://{host}/
  23. For a more detailed description of the new HTTP settings, please consult the annotated `Graylog configuration file <https://github.com/Graylog2/graylog2-server/blob/d9bb656275eeac7027e3fe12d9ee1b6a0905dcd1/misc/graylog.conf#L79-L81>`__.
  24. Overview of removed Graylog REST API settings:
  25. +----------------------------------+----------------------------------+--------------------------------+
  26. | Removed Setting | New Setting | Default |
  27. +==================================+==================================+================================+
  28. | ``rest_listen_uri`` | ``http_bind_address`` | ``127.0.0.1:9000`` |
  29. +----------------------------------+----------------------------------+--------------------------------+
  30. | ``rest_transport_uri`` | ``http_publish_uri`` | ``http://$http_bind_address/`` |
  31. +----------------------------------+----------------------------------+--------------------------------+
  32. | ``web_endpoint_uri`` | ``http_external_uri`` | ``$http_publish_uri`` |
  33. +----------------------------------+----------------------------------+--------------------------------+
  34. | ``rest_enable_cors`` | ``http_enable_cors`` | ``true`` |
  35. +----------------------------------+----------------------------------+--------------------------------+
  36. | ``rest_enable_gzip`` | ``http_enable_gzip`` | ``true`` |
  37. +----------------------------------+----------------------------------+--------------------------------+
  38. | ``rest_max_header_size`` | ``http_max_header_size`` | ``8192`` |
  39. +----------------------------------+----------------------------------+--------------------------------+
  40. | ``rest_max_initial_line_length`` | ``http_max_initial_line_length`` | ``4096`` |
  41. +----------------------------------+----------------------------------+--------------------------------+
  42. | ``rest_thread_pool_size`` | ``http_thread_pool_size`` | ``16`` |
  43. +----------------------------------+----------------------------------+--------------------------------+
  44. | ``rest_enable_tls`` | ``http_enable_tls`` | ``false`` |
  45. +----------------------------------+----------------------------------+--------------------------------+
  46. | ``rest_tls_cert_file`` | ``http_tls_cert_file`` | Empty |
  47. +----------------------------------+----------------------------------+--------------------------------+
  48. | ``rest_tls_key_file`` | ``http_tls_key_file`` | Empty |
  49. +----------------------------------+----------------------------------+--------------------------------+
  50. | ``rest_tls_key_password`` | ``http_tls_key_password`` | Empty |
  51. +----------------------------------+----------------------------------+--------------------------------+
  52. Overview of removed Graylog web interface settings:
  53. +---------------------------------+----------------------------------+--------------------+
  54. | Removed Setting | New Setting | Default |
  55. +=================================+==================================+====================+
  56. | ``web_enable`` | None | |
  57. +---------------------------------+----------------------------------+--------------------+
  58. | ``web_listen_uri`` | ``http_bind_address`` | ``127.0.0.1:9000`` |
  59. +---------------------------------+----------------------------------+--------------------+
  60. | ``web_enable_cors`` | ``http_enable_cors`` | ``true`` |
  61. +---------------------------------+----------------------------------+--------------------+
  62. | ``web_enable_gzip`` | ``http_enable_gzip`` | ``true`` |
  63. +---------------------------------+----------------------------------+--------------------+
  64. | ``web_max_header_size`` | ``http_max_header_size`` | ``8192`` |
  65. +---------------------------------+----------------------------------+--------------------+
  66. | ``web_max_initial_line_length`` | ``http_max_initial_line_length`` | ``4096`` |
  67. +---------------------------------+----------------------------------+--------------------+
  68. | ``web_thread_pool_size`` | ``http_thread_pool_size`` | ``16`` |
  69. +---------------------------------+----------------------------------+--------------------+
  70. | ``web_enable_tls`` | ``http_enable_tls`` | ``false`` |
  71. +---------------------------------+----------------------------------+--------------------+
  72. | ``web_tls_cert_file`` | ``http_tls_cert_file`` | Empty |
  73. +---------------------------------+----------------------------------+--------------------+
  74. | ``web_tls_key_file`` | ``http_tls_key_file`` | Empty |
  75. +---------------------------------+----------------------------------+--------------------+
  76. | ``web_tls_key_password`` | ``http_tls_key_password`` | Empty |
  77. +---------------------------------+----------------------------------+--------------------+
  78. Plugins merged into the Graylog server
  79. ======================================
  80. Starting with Graylog 3.0.0, the following official plugins were merged into the Graylog server:
  81. - `Beats Input <https://github.com/Graylog2/graylog-plugin-beats>`_
  82. - `CEF Input <https://github.com/Graylog2/graylog-plugin-cef>`_
  83. - `Collector Plugin <https://github.com/Graylog2/graylog-plugin-collector>`_
  84. - `Enterprise Integration Page <https://github.com/Graylog2/graylog-plugin-enterprise-integration>`_
  85. - `Map Widget <https://github.com/Graylog2/graylog-plugin-map-widget>`_
  86. - `NetFlow Input <https://github.com/Graylog2/graylog-plugin-netflow>`_
  87. - `Pipeline Processor <https://github.com/Graylog2/graylog-plugin-pipeline-processor>`_
  88. That means these plugins are not available as separate plugins anymore. If you manually update your Graylog installation (without using operating system packages), make sure to remove all old plugin files from the `plugin_dir <http://docs.graylog.org/en/3.0/pages/configuration/server.conf.html>`_ folder.
  89. The old issues in these repositories are still available for reference but new issues should only be created in the `Graylog server issue tracker <https://github.com/Graylog2/graylog2-server/issues>`_.
  90. The following HTTP API paths changed due to the plugin merge:
  91. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  92. | Old Path | New Path |
  93. +=============================================================================================+===============================================+
  94. | ``/plugins/org.graylog.plugins.map/mapdata`` | ``/search/mapdata`` |
  95. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  96. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline`` | ``/system/pipelines/pipeline`` |
  97. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  98. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline/parse`` | ``/system/pipelines/pipeline/parse`` |
  99. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  100. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule`` | ``/system/pipelines/rule`` |
  101. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  102. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/functions`` | ``/system/pipelines/rule/functions`` |
  103. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  104. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/multiple`` | ``/system/pipelines/rule/multiple`` |
  105. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  106. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/parse`` | ``/system/pipelines/rule/parse`` |
  107. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  108. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections`` | ``/system/pipelines/connections`` |
  109. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  110. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_stream`` | ``/system/pipelines/connections/to_stream`` |
  111. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  112. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_pipeline`` | ``/system/pipelines/connections/to_pipeline`` |
  113. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  114. | ``/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/simulate`` | ``/system/pipelines/simulate`` |
  115. +---------------------------------------------------------------------------------------------+-----------------------------------------------+
  116. New "bin_dir" and "data_dir" configuration parameters
  117. =====================================================
  118. We introduced two new configuration parameters related to file system paths.
  119. - ``bin_dir`` config option points to the directory that contains scripts like ``graylogctl``.
  120. - ``data_dir`` option configures the base directory for Graylog server state.
  121. Please check the updated default ``graylog.conf`` configuration file for required changes to your existing file.
  122. Removed support for Drools-based filters
  123. ========================================
  124. For a long time, Graylog allowed to use `Drools <https://www.drools.org/>`_ to filter messages. Unfortunately, using Drools to perform complex filter logic came with a performance penalty and wasn't as flexible as we would have liked it to be.
  125. Starting with Graylog 3.0.0, the support for Drools-based message filters has been removed from Graylog. The ``rules_file`` configuration setting has been removed accordingly.
  126. We recommend migrating the Drools-based logic to `Processing Pipelines <http://docs.graylog.org/en/3.0/pages/pipelines.html>`_.
  127. Drools-based blacklist
  128. ----------------------
  129. Graylog provided undocumented blacklist-functionality based on Drools. This blacklist could only be modified via the Graylog REST API on the ``/filters/blacklist`` resource.
  130. If you've been using this functionality, you'll have to migrate these blacklist rules to the `Processing Pipelines <http://docs.graylog.org/en/3.0/pages/pipelines.html>`_.
  131. To check if you're using the Drools-based blacklist in Graylog prior to version 3.0.0, you can run the following command::
  132. # curl -u admin:password -H 'Accept: application/json' 'http://graylog.example.com/api/filters/blacklist?pretty=true'
  133. String-based blacklist rule
  134. ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  135. Old blacklist rule::
  136. {
  137. "id" : "54e300001234123412340001",
  138. "type" : "string",
  139. "name" : "String Blacklist",
  140. "description" : "Drop messages based on case-insensitive string comparison",
  141. "fieldName" : "custom_field",
  142. "pattern" : "EXAMPLE pattern",
  143. "creator_user_id" : "admin",
  144. "created_at" : "2018-04-04T12:00:00.000Z"
  145. }
  146. New pipeline rule::
  147. rule "string-blacklist"
  148. when
  149. has_field("custom_field") &&
  150. lowercase(to_string($message.custom_field)) == "example pattern"
  151. then
  152. drop_message();
  153. end
  154. See also:
  155. * `has_field() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#has-field>`_
  156. * `lowercase() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#lowercase>`_
  157. * `drop_message() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#drop-message>`_
  158. Regex-based blacklist rule
  159. ^^^^^^^^^^^^^^^^^^^^^^^^^^
  160. Old blacklist rule::
  161. {
  162. "id" : "54e300001234123412340002",
  163. "type" : "regex",
  164. "name" : "Regex Blacklist",
  165. "description" : "Drop messages based on regular expression",
  166. "fieldName" : "custom_field",
  167. "pattern" : "^EXAMPLE.*",
  168. "creator_user_id" : "admin",
  169. "created_at" : "2018-04-04T12:00:00.000Z"
  170. }
  171. New pipeline rule::
  172. rule "regex-blacklist"
  173. when
  174. has_field("custom_field") &&
  175. regex("^EXAMPLE.*", to_string($message.custom_field)).matches == true
  176. then
  177. drop_message();
  178. end
  179. See also:
  180. * `has_field() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#has-field>`_
  181. * `regex() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#regex>`_
  182. * `drop_message() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#drop-message>`_
  183. IP Range-based blacklist rule
  184. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  185. Old blacklist rule::
  186. {
  187. "id" : "54e300001234123412340003",
  188. "type" : "iprange",
  189. "name" : "IP Blacklist",
  190. "description" : "Drop messages based on IP address",
  191. "fieldName" : "custom_field",
  192. "pattern" : "192.168.0.0/16",
  193. "creator_user_id" : "admin",
  194. "created_at" : "2018-04-04T12:00:00.000Z"
  195. }
  196. New pipeline rule::
  197. rule "ip-blacklist"
  198. when
  199. has_field("custom_field") &&
  200. cidr_match("192.168.0.0/16", to_ip($message.custom_field))
  201. then
  202. drop_message();
  203. end
  204. See also:
  205. * `has_field() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#has-field>`_
  206. * `to_ip() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#to-ip>`_
  207. * `cidr_match() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#cidr-match>`_
  208. * `drop_message() <http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#drop-message>`_
  209. Changed metrics name for stream rules
  210. =====================================
  211. The name of the metrics for stream rules have been changed to include the stream ID which helps identifying the actual stream they are related to.
  212. Old metric name::
  213. org.graylog2.plugin.streams.StreamRule.${stream-rule-id}.executionTime
  214. New metric name::
  215. org.graylog2.plugin.streams.Stream.${stream-id}.StreamRule.${stream-rule-id}.executionTime
  216. Email alarm callback default settings
  217. =====================================
  218. The defaults of the configuration settings for the email alarm callback with regard to encrypted connections have been changed.
  219. +-----------------------------+-------------+-------------+
  220. | Setting | Old default | New default |
  221. +=============================+=============+=============+
  222. | ``transport_email_use_tls`` | ``false`` | ``true`` |
  223. +-----------------------------+-------------+-------------+
  224. | ``transport_email_use_ssl`` | ``true`` | ``false`` |
  225. +-----------------------------+-------------+-------------+
  226. Furthermore, it's not possible anymore to enable both settings (SMTP with STARTTLS and SMTP over SSL) at the same time because this led to errors at runtime when Graylog tried to upgrade the connection to TLS with STARTTLS in an already existing SMTPS connection.
  227. Most SMTP services prefer SMTP with STARTTLS to provide an encrypted connection.
  228. Setting initial configuration on widget's configurationCreateComponent
  229. ======================================================================
  230. Widget plugins that want to customize the create modal by adding some custom inputs need to additionally set the
  231. initial configuration for them. Before, we accessed the component's ``getInitialConfiguration()`` when opening the
  232. creation modal form, but this is now not possible due to performance improvements.
  233. In 3.0, setting the initial widget configuration on the create component can be achieved in two different ways:
  234. Setting ``initialConfiguration`` class property
  235. ---------------------------------------------
  236. This is the preferred method, and it should be used every time configuration does not depend on any external state
  237. or props. Example::
  238. static initialConfiguration = { shouldShowChart: true, description: 'Initial description' };
  239. Calling the ``setInitialConfiguration`` prop
  240. --------------------------------------------
  241. ``WidgetCreationModal`` passes a function called ``setInitialConfiguration`` to the ``configurationCreateComponent``
  242. defined for the widget. That function can be called on the ``constructor`` or ``componentDidMount`` of the custom
  243. component to set the initial configuration values if any of them is derived from state or other props.
  244. Note that any configuration key set through ``setInitialConfiguration`` will have precedence over configuration keys
  245. set by ``initialConfiguration`` and will override existing configuration keys.
  246. Example::
  247. static initialConfiguration = { key: value, test: false };
  248. constructor(props) {
  249. super(props);
  250. props.setInitialConfiguration({ field: props.fields[0], test: true });
  251. }
  252. /* The effective initial configuration would be: { key: value, field: props.fields[0], test: true } */