You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

pvesr.adoc 9.7KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290
  1. [[chapter_pvesr]]
  2. ifdef::manvolnum[]
  3. pvesr(1)
  4. ========
  5. :pve-toplevel:
  6. NAME
  7. ----
  8. pvesr - Proxmox VE Storage Replication
  9. SYNOPSIS
  10. --------
  11. include::pvesr.1-synopsis.adoc[]
  12. DESCRIPTION
  13. -----------
  14. endif::manvolnum[]
  15. ifndef::manvolnum[]
  16. Storage Replication
  17. ===================
  18. :pve-toplevel:
  19. endif::manvolnum[]
  20. The `pvesr` command line tool manages the {PVE} storage replication
  21. framework. Storage replication brings redundancy for guests using
  22. local storage and reduces migration time.
  23. It replicates guest volumes to another node so that all data is available
  24. without using shared storage. Replication uses snapshots to minimize traffic
  25. sent over the network. Therefore, new data is sent only incrementally after
  26. an initial full sync. In the case of a node failure, your guest data is
  27. still available on the replicated node.
  28. The replication will be done automatically in configurable intervals.
  29. The minimum replication interval is one minute and the maximal interval is
  30. once a week. The format used to specify those intervals is a subset of
  31. `systemd` calendar events, see
  32. xref:pvesr_schedule_time_format[Schedule Format] section:
  33. Every guest can be replicated to multiple target nodes, but a guest cannot
  34. get replicated twice to the same target node.
  35. Each replications bandwidth can be limited, to avoid overloading a storage
  36. or server.
  37. Virtual guest with active replication cannot currently use online migration.
  38. Offline migration is supported in general. If you migrate to a node where
  39. the guests data is already replicated only the changes since the last
  40. synchronisation (so called `delta`) must be sent, this reduces the required
  41. time significantly. In this case the replication direction will also switch
  42. nodes automatically after the migration finished.
  43. For example: VM100 is currently on `nodeA` and gets replicated to `nodeB`.
  44. You migrate it to `nodeB`, so now it gets automatically replicated back from
  45. `nodeB` to `nodeA`.
  46. If you migrate to a node where the guest is not replicated, the whole disk
  47. data must send over. After the migration the replication job continues to
  48. replicate this guest to the configured nodes.
  49. [IMPORTANT]
  50. ====
  51. High-Availability is allowed in combination with storage replication, but it
  52. has the following implications:
  53. * redistributing services after a more preferred node comes online will lead
  54. to errors.
  55. * recovery works, but there may be some data loss between the last synced
  56. time and the time a node failed.
  57. ====
  58. Supported Storage Types
  59. -----------------------
  60. .Storage Types
  61. [width="100%",options="header"]
  62. |============================================
  63. |Description |PVE type |Snapshots|Stable
  64. |ZFS (local) |zfspool |yes |yes
  65. |============================================
  66. [[pvesr_schedule_time_format]]
  67. Schedule Format
  68. ---------------
  69. {pve} has a very flexible replication scheduler. It is based on the systemd
  70. time calendar event format.footnote:[see `man 7 systemd.time` for more information]
  71. Calendar events may be used to refer to one or more points in time in a
  72. single expression.
  73. Such a calendar event uses the following format:
  74. ----
  75. [day(s)] [[start-time(s)][/repetition-time(s)]]
  76. ----
  77. This allows you to configure a set of days on which the job should run.
  78. You can also set one or more start times, it tells the replication scheduler
  79. the moments in time when a job should start.
  80. With this information we could create a job which runs every workday at 10
  81. PM: `'mon,tue,wed,thu,fri 22'` which could be abbreviated to: `'mon..fri
  82. 22'`, most reasonable schedules can be written quite intuitive this way.
  83. NOTE: Hours are set in 24h format.
  84. To allow easier and shorter configuration one or more repetition times can
  85. be set. They indicate that on the start-time(s) itself and the start-time(s)
  86. plus all multiples of the repetition value replications will be done. If
  87. you want to start replication at 8 AM and repeat it every 15 minutes until
  88. 9 AM you would use: `'8:00/15'`
  89. Here you see also that if no hour separation (`:`) is used the value gets
  90. interpreted as minute. If such a separation is used the value on the left
  91. denotes the hour(s) and the value on the right denotes the minute(s).
  92. Further, you can use `*` to match all possible values.
  93. To get additional ideas look at
  94. xref:pvesr_schedule_format_examples[more Examples below].
  95. Detailed Specification
  96. ~~~~~~~~~~~~~~~~~~~~~~
  97. days:: Days are specified with an abbreviated English version: `sun, mon,
  98. tue, wed, thu, fri and sat`. You may use multiple days as a comma-separated
  99. list. A range of days can also be set by specifying the start and end day
  100. separated by ``..'', for example `mon..fri`. Those formats can be also
  101. mixed. If omitted `'*'` is assumed.
  102. time-format:: A time format consists of hours and minutes interval lists.
  103. Hours and minutes are separated by `':'`. Both, hour and minute, can be list
  104. and ranges of values, using the same format as days.
  105. First come hours then minutes, hours can be omitted if not needed, in this
  106. case `'*'` is assumed for the value of hours.
  107. The valid range for values is `0-23` for hours and `0-59` for minutes.
  108. [[pvesr_schedule_format_examples]]
  109. Examples:
  110. ~~~~~~~~~
  111. .Schedule Examples
  112. [width="100%",options="header"]
  113. |==============================================================================
  114. |Schedule String |Alternative |Meaning
  115. |mon,tue,wed,thu,fri |mon..fri |Every working day at 0:00
  116. |sat,sun |sat..sun |Only on weekends at 0:00
  117. |mon,wed,fri |-- |Only on Monday, Wednesday and Friday at 0:00
  118. |12:05 |12:05 |Every day at 12:05 PM
  119. |*/5 |0/5 |Every five minutes
  120. |mon..wed 30/10 |mon,tue,wed 30/10 |Monday, Tuesday, Wednesday 30, 40 and 50 minutes after every full hour
  121. |mon..fri 8..17,22:0/15 |-- |Every working day every 15 minutes between 8 AM and 6 PM and between 10 PM and 11 PM
  122. |fri 12..13:5/20 |fri 12,13:5/20 |Friday at 12:05, 12:25, 12:45, 13:05, 13:25 and 13:45
  123. |12,14,16,18,20,22:5 |12/2:5 |Every day starting at 12:05 until 22:05, every 2 hours
  124. |* |*/1 |Every minute (minimum interval)
  125. |==============================================================================
  126. Error Handling
  127. --------------
  128. If a replication job encounters problems it will be placed in error state.
  129. In this state the configured replication intervals get suspended
  130. temporarily. Then we retry the failed replication in a 30 minute interval,
  131. once this succeeds the original schedule gets activated again.
  132. Possible issues
  133. ~~~~~~~~~~~~~~~
  134. This represents only the most common issues possible, depending on your
  135. setup there may be also another cause.
  136. * Network is not working.
  137. * No free space left on the replication target storage.
  138. * Storage with same storage ID available on target node
  139. NOTE: You can always use the replication log to get hints about a problems
  140. cause.
  141. Migrating a guest in case of Error
  142. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  143. // FIXME: move this to better fitting chapter (sysadmin ?) and only link to
  144. // it here
  145. In the case of a grave error a virtual guest may get stuck on a failed
  146. node. You then need to move it manually to a working node again.
  147. Example
  148. ~~~~~~~
  149. Lets assume that you have two guests (VM 100 and CT 200) running on node A
  150. and replicate to node B.
  151. Node A failed and can not get back online. Now you have to migrate the guest
  152. to Node B manually.
  153. - connect to node B over ssh or open its shell via the WebUI
  154. - check if that the cluster is quorate
  155. +
  156. ----
  157. # pvecm status
  158. ----
  159. - If you have no quorum we strongly advise to fix this first and make the
  160. node operable again. Only if this is not possible at the moment you may
  161. use the following command to enforce quorum on the current node:
  162. +
  163. ----
  164. # pvecm expected 1
  165. ----
  166. WARNING: If expected votes are set avoid changes which affect the cluster
  167. (for example adding/removing nodes, storages, virtual guests) at all costs.
  168. Only use it to get vital guests up and running again or to resolve the quorum
  169. issue itself.
  170. - move both guest configuration files form the origin node A to node B:
  171. +
  172. ----
  173. # mv /etc/pve/nodes/A/qemu-server/100.conf /etc/pve/nodes/B/qemu-server/100.conf
  174. # mv /etc/pve/nodes/A/lxc/200.conf /etc/pve/nodes/B/lxc/200.conf
  175. ----
  176. - Now you can start the guests again:
  177. +
  178. ----
  179. # qm start 100
  180. # pct start 200
  181. ----
  182. Remember to replace the VMIDs and node names with your respective values.
  183. Managing Jobs
  184. -------------
  185. [thumbnail="screenshot/gui-qemu-add-replication-job.png"]
  186. You can use the web GUI to create, modify and remove replication jobs
  187. easily. Additionally the command line interface (CLI) tool `pvesr` can be
  188. used to do this.
  189. You can find the replication panel on all levels (datacenter, node, virtual
  190. guest) in the web GUI. They differ in what jobs get shown: all, only node
  191. specific or only guest specific jobs.
  192. Once adding a new job you need to specify the virtual guest (if not already
  193. selected) and the target node. The replication
  194. xref:pvesr_schedule_time_format[schedule] can be set if the default of `all
  195. 15 minutes` is not desired. You may also impose rate limiting on a
  196. replication job, this can help to keep the storage load acceptable.
  197. A replication job is identified by an cluster-wide unique ID. This ID is
  198. composed of the VMID in addition to an job number.
  199. This ID must only be specified manually if the CLI tool is used.
  200. Command Line Interface Examples
  201. -------------------------------
  202. Create a replication job which will run every 5 minutes with limited bandwidth of
  203. 10 mbps (megabytes per second) for the guest with guest ID 100.
  204. ----
  205. # pvesr create-local-job 100-0 pve1 --schedule "*/5" --rate 10
  206. ----
  207. Disable an active job with ID `100-0`
  208. ----
  209. # pvesr disable 100-0
  210. ----
  211. Enable a deactivated job with ID `100-0`
  212. ----
  213. # pvesr enable 100-0
  214. ----
  215. Change the schedule interval of the job with ID `100-0` to once a hour
  216. ----
  217. # pvesr update 100-0 --schedule '*/00'
  218. ----
  219. ifdef::manvolnum[]
  220. include::pve-copyright.adoc[]
  221. endif::manvolnum[]