You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

931 lines

  1. [[chapter_ha_manager]]
  2. ifdef::manvolnum[]
  3. ha-manager(1)
  4. =============
  5. :pve-toplevel:
  6. NAME
  7. ----
  8. ha-manager - Proxmox VE HA Manager
  10. --------
  11. include::ha-manager.1-synopsis.adoc[]
  13. -----------
  14. endif::manvolnum[]
  15. ifndef::manvolnum[]
  16. High Availability
  17. =================
  18. :pve-toplevel:
  19. endif::manvolnum[]
  20. Our modern society depends heavily on information provided by
  21. computers over the network. Mobile devices amplified that dependency,
  22. because people can access the network any time from anywhere. If you
  23. provide such services, it is very important that they are available
  24. most of the time.
  25. We can mathematically define the availability as the ratio of (A) the
  26. total time a service is capable of being used during a given interval
  27. to (B) the length of the interval. It is normally expressed as a
  28. percentage of uptime in a given year.
  29. .Availability - Downtime per Year
  30. [width="60%",cols="<d,d",options="header"]
  31. |===========================================================
  32. |Availability % |Downtime per year
  33. |99 |3.65 days
  34. |99.9 |8.76 hours
  35. |99.99 |52.56 minutes
  36. |99.999 |5.26 minutes
  37. |99.9999 |31.5 seconds
  38. |99.99999 |3.15 seconds
  39. |===========================================================
  40. There are several ways to increase availability. The most elegant
  41. solution is to rewrite your software, so that you can run it on
  42. several host at the same time. The software itself need to have a way
  43. to detect errors and do failover. This is relatively easy if you just
  44. want to serve read-only web pages. But in general this is complex, and
  45. sometimes impossible because you cannot modify the software
  46. yourself. The following solutions works without modifying the
  47. software:
  48. * Use reliable ``server'' components
  49. +
  50. NOTE: Computer components with same functionality can have varying
  51. reliability numbers, depending on the component quality. Most vendors
  52. sell components with higher reliability as ``server'' components -
  53. usually at higher price.
  54. * Eliminate single point of failure (redundant components)
  55. ** use an uninterruptible power supply (UPS)
  56. ** use redundant power supplies on the main boards
  57. ** use ECC-RAM
  58. ** use redundant network hardware
  59. ** use RAID for local storage
  60. ** use distributed, redundant storage for VM data
  61. * Reduce downtime
  62. ** rapidly accessible administrators (24/7)
  63. ** availability of spare parts (other nodes in a {pve} cluster)
  64. ** automatic error detection (provided by `ha-manager`)
  65. ** automatic failover (provided by `ha-manager`)
  66. Virtualization environments like {pve} make it much easier to reach
  67. high availability because they remove the ``hardware'' dependency. They
  68. also support to setup and use redundant storage and network
  69. devices. So if one host fail, you can simply start those services on
  70. another host within your cluster.
  71. Even better, {pve} provides a software stack called `ha-manager`,
  72. which can do that automatically for you. It is able to automatically
  73. detect errors and do automatic failover.
  74. {pve} `ha-manager` works like an ``automated'' administrator. First, you
  75. configure what resources (VMs, containers, ...) it should
  76. manage. `ha-manager` then observes correct functionality, and handles
  77. service failover to another node in case of errors. `ha-manager` can
  78. also handle normal user requests which may start, stop, relocate and
  79. migrate a service.
  80. But high availability comes at a price. High quality components are
  81. more expensive, and making them redundant duplicates the costs at
  82. least. Additional spare parts increase costs further. So you should
  83. carefully calculate the benefits, and compare with those additional
  84. costs.
  85. TIP: Increasing availability from 99% to 99.9% is relatively
  86. simple. But increasing availability from 99.9999% to 99.99999% is very
  87. hard and costly. `ha-manager` has typical error detection and failover
  88. times of about 2 minutes, so you can get no more than 99.999%
  89. availability.
  90. Requirements
  91. ------------
  92. You must meet the following requirements before you start with HA:
  93. * at least three cluster nodes (to get reliable quorum)
  94. * shared storage for VMs and containers
  95. * hardware redundancy (everywhere)
  96. * use reliable “server” components
  97. * hardware watchdog - if not available we fall back to the
  98. linux kernel software watchdog (`softdog`)
  99. * optional hardware fencing devices
  100. [[ha_manager_resources]]
  101. Resources
  102. ---------
  103. We call the primary management unit handled by `ha-manager` a
  104. resource. A resource (also called ``service'') is uniquely
  105. identified by a service ID (SID), which consists of the resource type
  106. and an type specific ID, e.g.: `vm:100`. That example would be a
  107. resource of type `vm` (virtual machine) with the ID 100.
  108. For now we have two important resources types - virtual machines and
  109. containers. One basic idea here is that we can bundle related software
  110. into such a VM or container, so there is no need to compose one big
  111. service from other services, like it was done with `rgmanager`. In
  112. general, a HA managed resource should not depend on other resources.
  113. Management Tasks
  114. ----------------
  115. This section provides a short overview of common management tasks. The
  116. first step is to enable HA for a resource. This is done by adding the
  117. resource to the HA resource configuration. You can do this using the
  118. GUI, or simply use the command line tool, for example:
  119. ----
  120. # ha-manager add vm:100
  121. ----
  122. The HA stack now tries to start the resources and keeps it
  123. running. Please note that you can configure the ``requested''
  124. resources state. For example you may want the HA stack to stop the
  125. resource:
  126. ----
  127. # ha-manager set vm:100 --state stopped
  128. ----
  129. and start it again later:
  130. ----
  131. # ha-manager set vm:100 --state started
  132. ----
  133. You can also use the normal VM and container management commands. They
  134. automatically forward the commands to the HA stack, so
  135. ----
  136. # qm start 100
  137. ----
  138. simply sets the requested state to `started`. Same applied to `qm
  139. stop`, which sets the requested state to `stopped`.
  140. NOTE: The HA stack works fully asynchronous and needs to communicate
  141. with other cluster members. So it takes some seconds until you see
  142. the result of such actions.
  143. To view the current HA resource configuration use:
  144. ----
  145. # ha-manager config
  146. vm:100
  147. state stopped
  148. ----
  149. And you can view the actual HA manager and resource state with:
  150. ----
  151. # ha-manager status
  152. quorum OK
  153. master node1 (active, Wed Nov 23 11:07:23 2016)
  154. lrm elsa (active, Wed Nov 23 11:07:19 2016)
  155. service vm:100 (node1, started)
  156. ----
  157. You can also initiate resource migration to other nodes:
  158. ----
  159. # ha-manager migrate vm:100 node2
  160. ----
  161. This uses online migration and tries to keep the VM running. Online
  162. migration needs to transfer all used memory over the network, so it is
  163. sometimes faster to stop VM, then restart it on the new node. This can be
  164. done using the `relocate` command:
  165. ----
  166. # ha-manager relocate vm:100 node2
  167. ----
  168. Finally, you can remove the resource from the HA configuration using
  169. the following command:
  170. ----
  171. # ha-manager remove vm:100
  172. ----
  173. NOTE: This does not start or stop the resource.
  174. But all HA related tasks can be done in the GUI, so there is no need to
  175. use the command line at all.
  176. How It Works
  177. ------------
  178. This section provides a detailed description of the {PVE} HA manager
  179. internals. It describes all involved daemons and how they work
  180. together. To provide HA, two daemons run on each node:
  181. `pve-ha-lrm`::
  182. The local resource manager (LRM), which controls the services running on
  183. the local node. It reads the requested states for its services from
  184. the current manager status file and executes the respective commands.
  185. `pve-ha-crm`::
  186. The cluster resource manager (CRM), which makes the cluster wide
  187. decisions. It sends commands to the LRM, processes the results,
  188. and moves resources to other nodes if something fails. The CRM also
  189. handles node fencing.
  190. .Locks in the LRM & CRM
  191. [NOTE]
  192. Locks are provided by our distributed configuration file system (pmxcfs).
  193. They are used to guarantee that each LRM is active once and working. As an
  194. LRM only executes actions when it holds its lock, we can mark a failed node
  195. as fenced if we can acquire its lock. This lets us then recover any failed
  196. HA services securely without any interference from the now unknown failed node.
  197. This all gets supervised by the CRM which holds currently the manager master
  198. lock.
  199. Service States
  200. ~~~~~~~~~~~~~~
  201. The CRM use a service state enumeration to record the current service
  202. state. We display this state on the GUI and you can query it using
  203. the `ha-manager` command line tool:
  204. ----
  205. # ha-manager status
  206. quorum OK
  207. master elsa (active, Mon Nov 21 07:23:29 2016)
  208. lrm elsa (active, Mon Nov 21 07:23:22 2016)
  209. service ct:100 (elsa, stopped)
  210. service ct:102 (elsa, started)
  211. service vm:501 (elsa, started)
  212. ----
  213. Here is the list of possible states:
  214. stopped::
  215. Service is stopped (confirmed by LRM). If the LRM detects a stopped
  216. service is still running, it will stop it again.
  217. request_stop::
  218. Service should be stopped. The CRM waits for confirmation from the
  219. LRM.
  220. stopping::
  221. Pending stop request. But the CRM did not get the request so far.
  222. started::
  223. Service is active an LRM should start it ASAP if not already running.
  224. If the Service fails and is detected to be not running the LRM
  225. restarts it
  226. (see xref:ha_manager_start_failure_policy[Start Failure Policy]).
  227. starting::
  228. Pending start request. But the CRM has not got any confirmation from the
  229. LRM that the service is running.
  230. fence::
  231. Wait for node fencing (service node is not inside quorate cluster
  232. partition). As soon as node gets fenced successfully the service will
  233. be recovered to another node, if possible
  234. (see xref:ha_manager_fencing[Fencing]).
  235. freeze::
  236. Do not touch the service state. We use this state while we reboot a
  237. node, or when we restart the LRM daemon
  238. (see xref:ha_manager_package_updates[Package Updates]).
  239. ignored::
  240. Act as if the service were not managed by HA at all.
  241. Useful, when full control over the service is desired temporarily,
  242. without removing it from the HA configuration.
  243. migrate::
  244. Migrate service (live) to other node.
  245. error::
  246. Service is disabled because of LRM errors. Needs manual intervention
  247. (see xref:ha_manager_error_recovery[Error Recovery]).
  248. queued::
  249. Service is newly added, and the CRM has not seen it so far.
  250. disabled::
  251. Service is stopped and marked as `disabled`
  252. Local Resource Manager
  253. ~~~~~~~~~~~~~~~~~~~~~~
  254. The local resource manager (`pve-ha-lrm`) is started as a daemon on
  255. boot and waits until the HA cluster is quorate and thus cluster wide
  256. locks are working.
  257. It can be in three states:
  258. wait for agent lock::
  259. The LRM waits for our exclusive lock. This is also used as idle state if no
  260. service is configured.
  261. active::
  262. The LRM holds its exclusive lock and has services configured.
  263. lost agent lock::
  264. The LRM lost its lock, this means a failure happened and quorum was lost.
  265. After the LRM gets in the active state it reads the manager status
  266. file in `/etc/pve/ha/manager_status` and determines the commands it
  267. has to execute for the services it owns.
  268. For each command a worker gets started, these workers are running in
  269. parallel and are limited to at most 4 by default. This default setting
  270. may be changed through the datacenter configuration key `max_worker`.
  271. When finished the worker process gets collected and its result saved for
  272. the CRM.
  273. .Maximum Concurrent Worker Adjustment Tips
  274. [NOTE]
  275. The default value of at most 4 concurrent workers may be unsuited for
  276. a specific setup. For example may 4 live migrations happen at the same
  277. time, which can lead to network congestions with slower networks and/or
  278. big (memory wise) services. Ensure that also in the worst case no congestion
  279. happens and lower the `max_worker` value if needed. On the contrary, if you
  280. have a particularly powerful high end setup you may also want to increase it.
  281. Each command requested by the CRM is uniquely identifiable by a UID, when
  282. the worker finishes its result will be processed and written in the LRM
  283. status file `/etc/pve/nodes/<nodename>/lrm_status`. There the CRM may collect
  284. it and let its state machine - respective the commands output - act on it.
  285. The actions on each service between CRM and LRM are normally always synced.
  286. This means that the CRM requests a state uniquely marked by a UID, the LRM
  287. then executes this action *one time* and writes back the result, also
  288. identifiable by the same UID. This is needed so that the LRM does not
  289. execute an outdated command.
  290. With the exception of the `stop` and the `error` command,
  291. those two do not depend on the result produced and are executed
  292. always in the case of the stopped state and once in the case of
  293. the error state.
  294. .Read the Logs
  295. [NOTE]
  296. The HA Stack logs every action it makes. This helps to understand what
  297. and also why something happens in the cluster. Here its important to see
  298. what both daemons, the LRM and the CRM, did. You may use
  299. `journalctl -u pve-ha-lrm` on the node(s) where the service is and
  300. the same command for the pve-ha-crm on the node which is the current master.
  301. Cluster Resource Manager
  302. ~~~~~~~~~~~~~~~~~~~~~~~~
  303. The cluster resource manager (`pve-ha-crm`) starts on each node and
  304. waits there for the manager lock, which can only be held by one node
  305. at a time. The node which successfully acquires the manager lock gets
  306. promoted to the CRM master.
  307. It can be in three states:
  308. wait for agent lock::
  309. The CRM waits for our exclusive lock. This is also used as idle state if no
  310. service is configured
  311. active::
  312. The CRM holds its exclusive lock and has services configured
  313. lost agent lock::
  314. The CRM lost its lock, this means a failure happened and quorum was lost.
  315. Its main task is to manage the services which are configured to be highly
  316. available and try to always enforce the requested state. For example, a
  317. service with the requested state 'started' will be started if its not
  318. already running. If it crashes it will be automatically started again.
  319. Thus the CRM dictates the actions the LRM needs to execute.
  320. When an node leaves the cluster quorum, its state changes to unknown.
  321. If the current CRM then can secure the failed nodes lock, the services
  322. will be 'stolen' and restarted on another node.
  323. When a cluster member determines that it is no longer in the cluster
  324. quorum, the LRM waits for a new quorum to form. As long as there is no
  325. quorum the node cannot reset the watchdog. This will trigger a reboot
  326. after the watchdog then times out, this happens after 60 seconds.
  327. HA Simulator
  328. ------------
  329. [thumbnail="screenshot/gui-ha-manager-status.png"]
  330. By using the HA simulator you can test and learn all functionalities of the
  331. Proxmox VE HA solutions.
  332. By default, the simulator allows you to watch and test the behaviour of a
  333. real-world 3 node cluster with 6 VMs. You can also add or remove additional VMs
  334. or Container.
  335. You do not have to setup or configure a real cluster, the HA simulator runs out
  336. of the box.
  337. Install with apt:
  338. ----
  339. apt install pve-ha-simulator
  340. ----
  341. You can even install the package on any Debian based system without any
  342. other Proxmox VE packages. For that you will need to download the package and
  343. copy it to the system you want to run it on for installation. When you install
  344. the package with apt from the local file system it will also resolve the
  345. required dependencies for you.
  346. To start the simulator on a remote machine you must have a X11 redirection to
  347. your current system.
  348. If you are on a Linux machine you can use:
  349. ----
  350. ssh root@<IPofPVE> -Y
  351. ----
  352. On Windows it is working with[mobaxterm].
  353. After either connecting to a existing {pve} with the simulator installed, or
  354. installing it on your local Debian based system manually you can try it out as
  355. follows.
  356. First you need to create a working directory where the simulator saves it's
  357. current state and writes its the default config:
  358. ----
  359. mkdir working
  360. ----
  361. Then, simply pass the created directory as parameter to 'pve-ha-simulator':
  362. ----
  363. pve-ha-simulator working/
  364. ----
  365. You can then start, stop, migrate the simulated HA services, or even check out
  366. what happens on a node failure.
  367. Configuration
  368. -------------
  369. The HA stack is well integrated into the {pve} API. So, for example,
  370. HA can be configured via the `ha-manager` command line interface, or
  371. the {pve} web interface - both interfaces provide an easy way to
  372. manage HA. Automation tools can use the API directly.
  373. All HA configuration files are within `/etc/pve/ha/`, so they get
  374. automatically distributed to the cluster nodes, and all nodes share
  375. the same HA configuration.
  376. [[ha_manager_resource_config]]
  377. Resources
  378. ~~~~~~~~~
  379. [thumbnail="screenshot/gui-ha-manager-status.png"]
  380. The resource configuration file `/etc/pve/ha/resources.cfg` stores
  381. the list of resources managed by `ha-manager`. A resource configuration
  382. inside that list looks like this:
  383. ----
  384. <type>: <name>
  385. <property> <value>
  386. ...
  387. ----
  388. It starts with a resource type followed by a resource specific name,
  389. separated with colon. Together this forms the HA resource ID, which is
  390. used by all `ha-manager` commands to uniquely identify a resource
  391. (example: `vm:100` or `ct:101`). The next lines contain additional
  392. properties:
  393. include::ha-resources-opts.adoc[]
  394. Here is a real world example with one VM and one container. As you see,
  395. the syntax of those files is really simple, so it is even possible to
  396. read or edit those files using your favorite editor:
  397. .Configuration Example (`/etc/pve/ha/resources.cfg`)
  398. ----
  399. vm: 501
  400. state started
  401. max_relocate 2
  402. ct: 102
  403. # Note: use default settings for everything
  404. ----
  405. [thumbnail="screenshot/gui-ha-manager-add-resource.png"]
  406. Above config was generated using the `ha-manager` command line tool:
  407. ----
  408. # ha-manager add vm:501 --state started --max_relocate 2
  409. # ha-manager add ct:102
  410. ----
  411. [[ha_manager_groups]]
  412. Groups
  413. ~~~~~~
  414. [thumbnail="screenshot/gui-ha-manager-groups-view.png"]
  415. The HA group configuration file `/etc/pve/ha/groups.cfg` is used to
  416. define groups of cluster nodes. A resource can be restricted to run
  417. only on the members of such group. A group configuration look like
  418. this:
  419. ----
  420. group: <group>
  421. nodes <node_list>
  422. <property> <value>
  423. ...
  424. ----
  425. include::ha-groups-opts.adoc[]
  426. [thumbnail="screenshot/gui-ha-manager-add-group.png"]
  427. A common requirement is that a resource should run on a specific
  428. node. Usually the resource is able to run on other nodes, so you can define
  429. an unrestricted group with a single member:
  430. ----
  431. # ha-manager groupadd prefer_node1 --nodes node1
  432. ----
  433. For bigger clusters, it makes sense to define a more detailed failover
  434. behavior. For example, you may want to run a set of services on
  435. `node1` if possible. If `node1` is not available, you want to run them
  436. equally split on `node2` and `node3`. If those nodes also fail the
  437. services should run on `node4`. To achieve this you could set the node
  438. list to:
  439. ----
  440. # ha-manager groupadd mygroup1 -nodes "node1:2,node2:1,node3:1,node4"
  441. ----
  442. Another use case is if a resource uses other resources only available
  443. on specific nodes, lets say `node1` and `node2`. We need to make sure
  444. that HA manager does not use other nodes, so we need to create a
  445. restricted group with said nodes:
  446. ----
  447. # ha-manager groupadd mygroup2 -nodes "node1,node2" -restricted
  448. ----
  449. Above commands created the following group configuration fils:
  450. .Configuration Example (`/etc/pve/ha/groups.cfg`)
  451. ----
  452. group: prefer_node1
  453. nodes node1
  454. group: mygroup1
  455. nodes node2:1,node4,node1:2,node3:1
  456. group: mygroup2
  457. nodes node2,node1
  458. restricted 1
  459. ----
  460. The `nofailback` options is mostly useful to avoid unwanted resource
  461. movements during administration tasks. For example, if you need to
  462. migrate a service to a node which hasn't the highest priority in the
  463. group, you need to tell the HA manager to not move this service
  464. instantly back by setting the `nofailback` option.
  465. Another scenario is when a service was fenced and it got recovered to
  466. another node. The admin tries to repair the fenced node and brings it
  467. up online again to investigate the failure cause and check if it runs
  468. stable again. Setting the `nofailback` flag prevents that the
  469. recovered services move straight back to the fenced node.
  470. [[ha_manager_fencing]]
  471. Fencing
  472. -------
  473. On node failures, fencing ensures that the erroneous node is
  474. guaranteed to be offline. This is required to make sure that no
  475. resource runs twice when it gets recovered on another node. This is a
  476. really important task, because without, it would not be possible to
  477. recover a resource on another node.
  478. If a node did not get fenced, it would be in an unknown state where
  479. it may have still access to shared resources. This is really
  480. dangerous! Imagine that every network but the storage one broke. Now,
  481. while not reachable from the public network, the VM still runs and
  482. writes to the shared storage.
  483. If we then simply start up this VM on another node, we would get a
  484. dangerous race conditions because we write from both nodes. Such
  485. condition can destroy all VM data and the whole VM could be rendered
  486. unusable. The recovery could also fail if the storage protects from
  487. multiple mounts.
  488. How {pve} Fences
  489. ~~~~~~~~~~~~~~~~
  490. There are different methods to fence a node, for example, fence
  491. devices which cut off the power from the node or disable their
  492. communication completely. Those are often quite expensive and bring
  493. additional critical components into a system, because if they fail you
  494. cannot recover any service.
  495. We thus wanted to integrate a simpler fencing method, which does not
  496. require additional external hardware. This can be done using
  497. watchdog timers.
  498. .Possible Fencing Methods
  499. - external power switches
  500. - isolate nodes by disabling complete network traffic on the switch
  501. - self fencing using watchdog timers
  502. Watchdog timers are widely used in critical and dependable systems
  503. since the beginning of micro controllers. They are often independent
  504. and simple integrated circuits which are used to detect and recover
  505. from computer malfunctions.
  506. During normal operation, `ha-manager` regularly resets the watchdog
  507. timer to prevent it from elapsing. If, due to a hardware fault or
  508. program error, the computer fails to reset the watchdog, the timer
  509. will elapse and triggers a reset of the whole server (reboot).
  510. Recent server motherboards often include such hardware watchdogs, but
  511. these need to be configured. If no watchdog is available or
  512. configured, we fall back to the Linux Kernel 'softdog'. While still
  513. reliable, it is not independent of the servers hardware, and thus has
  514. a lower reliability than a hardware watchdog.
  515. Configure Hardware Watchdog
  516. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  517. By default, all hardware watchdog modules are blocked for security
  518. reasons. They are like a loaded gun if not correctly initialized. To
  519. enable a hardware watchdog, you need to specify the module to load in
  520. '/etc/default/pve-ha-manager', for example:
  521. ----
  522. # select watchdog module (default is softdog)
  524. ----
  525. This configuration is read by the 'watchdog-mux' service, which load
  526. the specified module at startup.
  527. Recover Fenced Services
  528. ~~~~~~~~~~~~~~~~~~~~~~~
  529. After a node failed and its fencing was successful, the CRM tries to
  530. move services from the failed node to nodes which are still online.
  531. The selection of nodes, on which those services gets recovered, is
  532. influenced by the resource `group` settings, the list of currently active
  533. nodes, and their respective active service count.
  534. The CRM first builds a set out of the intersection between user selected
  535. nodes (from `group` setting) and available nodes. It then choose the
  536. subset of nodes with the highest priority, and finally select the node
  537. with the lowest active service count. This minimizes the possibility
  538. of an overloaded node.
  539. CAUTION: On node failure, the CRM distributes services to the
  540. remaining nodes. This increase the service count on those nodes, and
  541. can lead to high load, especially on small clusters. Please design
  542. your cluster so that it can handle such worst case scenarios.
  543. [[ha_manager_start_failure_policy]]
  544. Start Failure Policy
  545. ---------------------
  546. The start failure policy comes in effect if a service failed to start on a
  547. node one or more times. It can be used to configure how often a restart
  548. should be triggered on the same node and how often a service should be
  549. relocated so that it gets a try to be started on another node.
  550. The aim of this policy is to circumvent temporary unavailability of shared
  551. resources on a specific node. For example, if a shared storage isn't available
  552. on a quorate node anymore, e.g. network problems, but still on other nodes,
  553. the relocate policy allows then that the service gets started nonetheless.
  554. There are two service start recover policy settings which can be configured
  555. specific for each resource.
  556. max_restart::
  557. Maximum number of tries to restart a failed service on the actual
  558. node. The default is set to one.
  559. max_relocate::
  560. Maximum number of tries to relocate the service to a different node.
  561. A relocate only happens after the max_restart value is exceeded on the
  562. actual node. The default is set to one.
  563. NOTE: The relocate count state will only reset to zero when the
  564. service had at least one successful start. That means if a service is
  565. re-started without fixing the error only the restart policy gets
  566. repeated.
  567. [[ha_manager_error_recovery]]
  568. Error Recovery
  569. --------------
  570. If after all tries the service state could not be recovered it gets
  571. placed in an error state. In this state the service won't get touched
  572. by the HA stack anymore. The only way out is disabling a service:
  573. ----
  574. # ha-manager set vm:100 --state disabled
  575. ----
  576. This can also be done in the web interface.
  577. To recover from the error state you should do the following:
  578. * bring the resource back into a safe and consistent state (e.g.:
  579. kill its process if the service could not be stopped)
  580. * disable the resource to remove the error flag
  581. * fix the error which led to this failures
  582. * *after* you fixed all errors you may request that the service starts again
  583. [[ha_manager_package_updates]]
  584. Package Updates
  585. ---------------
  586. When updating the ha-manager you should do one node after the other, never
  587. all at once for various reasons. First, while we test our software
  588. thoughtfully, a bug affecting your specific setup cannot totally be ruled out.
  589. Updating one node after the other and checking the functionality of each node
  590. after finishing the update helps to recover from eventual problems, while
  591. updating all at once could result in a broken cluster and is generally not
  592. good practice.
  593. Also, the {pve} HA stack uses a request acknowledge protocol to perform
  594. actions between the cluster and the local resource manager. For restarting,
  595. the LRM makes a request to the CRM to freeze all its services. This prevents
  596. that they get touched by the Cluster during the short time the LRM is restarting.
  597. After that the LRM may safely close the watchdog during a restart.
  598. Such a restart happens normally during a package update and, as already stated,
  599. an active master CRM is needed to acknowledge the requests from the LRM. If
  600. this is not the case the update process can take too long which, in the worst
  601. case, may result in a reset triggered by the watchdog.
  602. Node Maintenance
  603. ----------------
  604. It is sometimes possible to shutdown or reboot a node to do maintenance tasks.
  605. Either to replace hardware, or simply to install a new kernel image.
  606. This is also true when using the HA stack. The behaviour of the HA stack during
  607. a shutdown can be configured.
  608. [[ha_manager_shutdown_policy]]
  609. Shutdown Policy
  610. ~~~~~~~~~~~~~~~
  611. Below you will find a description of the different HA policies for a node
  612. shutdown. Currently 'Conditional' is the default due to backward compatibility.
  613. Some users may find that the 'Migrate' behaves more as expected.
  614. Migrate
  615. ^^^^^^^
  616. Once the Local Resource manager (LRM) gets a shutdown request and this policy
  617. is enabled, it will mark it self as unavailable for the current HA manager.
  618. This triggers a migration of all HA Services currently located on this node.
  619. Until all running Services got moved away, the LRM will try to delay the
  620. shutdown process. But, this expects that the running services *can* be migrated
  621. to another node. In other words, the service must not be locally bound, for
  622. example by using hardware passthrough. As non-group member nodes are considered
  623. as runnable target if no group member is available, this policy can still be
  624. used when making use of HA groups with only some nodes selected. But, marking a
  625. group as 'restricted' tells the HA manager that the service cannot run outside
  626. of the chosen set of nodes, if all of those nodes are unavailable the shutdown
  627. will hang until you manually intervene. Once the shut down node comes back
  628. online again, the previously displaced services will be moved back, if they did
  629. not get migrated manually in-between.
  630. NOTE: The watchdog is still active during the migration process on shutdown.
  631. If the node loses quorum it will be fenced and the services will be recovered.
  632. If you start a (previously stopped) service on a node which is currently being
  633. maintained, the node needs to be fenced to ensure that the service can be moved
  634. and started on another, available, node.
  635. Failover
  636. ^^^^^^^^
  637. This mode ensures that all services get stopped, but that they will also be
  638. recovered, if the current node is not online soon. It can be useful when doing
  639. maintenance on a cluster scale, were live-migrating VMs may not be possible if
  640. to many nodes are powered-off at a time, but you still want to ensure HA
  641. services get recovered and started again as soon as possible.
  642. Freeze
  643. ^^^^^^
  644. This mode ensures that all services get stopped and frozen, so that they won't
  645. get recovered until the current node is online again.
  646. Conditional
  647. ^^^^^^^^^^^
  648. The 'Conditional' shutdown policy automatically detects if a shutdown or a
  649. reboot is requested, and changes behaviour accordingly.
  650. .Shutdown
  651. A shutdown ('poweroff') is usually done if the node is planned to stay down for
  652. some time. The LRM stops all managed services in that case. This means that
  653. other nodes will take over those service afterwards.
  654. NOTE: Recent hardware has large amounts of memory (RAM). So we stop all
  655. resources, then restart them to avoid online migration of all that RAM. If you
  656. want to use online migration, you need to invoke that manually before you
  657. shutdown the node.
  658. .Reboot
  659. Node reboots are initiated with the 'reboot' command. This is usually done
  660. after installing a new kernel. Please note that this is different from
  661. ``shutdown'', because the node immediately starts again.
  662. The LRM tells the CRM that it wants to restart, and waits until the CRM puts
  663. all resources into the `freeze` state (same mechanism is used for
  664. xref:ha_manager_package_updates[Package Updates]). This prevents that those
  665. resources are moved to other nodes. Instead, the CRM start the resources after
  666. the reboot on the same node.
  667. Manual Resource Movement
  668. ^^^^^^^^^^^^^^^^^^^^^^^^
  669. Last but not least, you can also move resources manually to other nodes before
  670. you shutdown or restart a node. The advantage is that you have full control,
  671. and you can decide if you want to use online migration or not.
  672. NOTE: Please do not 'kill' services like `pve-ha-crm`, `pve-ha-lrm` or
  673. `watchdog-mux`. They manage and use the watchdog, so this can result in a
  674. immediate node reboot or even reset.
  675. ifdef::manvolnum[]
  676. include::pve-copyright.adoc[]
  677. endif::manvolnum[]