You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

pveceph.adoc 28KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810
  1. [[chapter_pveceph]]
  2. ifdef::manvolnum[]
  3. pveceph(1)
  4. ==========
  5. :pve-toplevel:
  6. NAME
  7. ----
  8. pveceph - Manage Ceph Services on Proxmox VE Nodes
  9. SYNOPSIS
  10. --------
  11. include::pveceph.1-synopsis.adoc[]
  12. DESCRIPTION
  13. -----------
  14. endif::manvolnum[]
  15. ifndef::manvolnum[]
  16. Manage Ceph Services on Proxmox VE Nodes
  17. ========================================
  18. :pve-toplevel:
  19. endif::manvolnum[]
  20. [thumbnail="screenshot/gui-ceph-status.png"]
  21. {pve} unifies your compute and storage systems, i.e. you can use the same
  22. physical nodes within a cluster for both computing (processing VMs and
  23. containers) and replicated storage. The traditional silos of compute and
  24. storage resources can be wrapped up into a single hyper-converged appliance.
  25. Separate storage networks (SANs) and connections via network attached storages
  26. (NAS) disappear. With the integration of Ceph, an open source software-defined
  27. storage platform, {pve} has the ability to run and manage Ceph storage directly
  28. on the hypervisor nodes.
  29. Ceph is a distributed object store and file system designed to provide
  30. excellent performance, reliability and scalability.
  31. .Some advantages of Ceph on {pve} are:
  32. - Easy setup and management with CLI and GUI support
  33. - Thin provisioning
  34. - Snapshots support
  35. - Self healing
  36. - Scalable to the exabyte level
  37. - Setup pools with different performance and redundancy characteristics
  38. - Data is replicated, making it fault tolerant
  39. - Runs on economical commodity hardware
  40. - No need for hardware RAID controllers
  41. - Open source
  42. For small to mid sized deployments, it is possible to install a Ceph server for
  43. RADOS Block Devices (RBD) directly on your {pve} cluster nodes, see
  44. xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]. Recent
  45. hardware has plenty of CPU power and RAM, so running storage services
  46. and VMs on the same node is possible.
  47. To simplify management, we provide 'pveceph' - a tool to install and
  48. manage {ceph} services on {pve} nodes.
  49. .Ceph consists of a couple of Daemons footnote:[Ceph intro https://docs.ceph.com/docs/{ceph_codename}/start/intro/], for use as a RBD storage:
  50. - Ceph Monitor (ceph-mon)
  51. - Ceph Manager (ceph-mgr)
  52. - Ceph OSD (ceph-osd; Object Storage Daemon)
  53. TIP: We highly recommend to get familiar with Ceph's architecture
  54. footnote:[Ceph architecture https://docs.ceph.com/docs/{ceph_codename}/architecture/]
  55. and vocabulary
  56. footnote:[Ceph glossary https://docs.ceph.com/docs/{ceph_codename}/glossary].
  57. Precondition
  58. ------------
  59. To build a hyper-converged Proxmox + Ceph Cluster there should be at least
  60. three (preferably) identical servers for the setup.
  61. Check also the recommendations from
  62. https://docs.ceph.com/docs/{ceph_codename}/start/hardware-recommendations/[Ceph's website].
  63. .CPU
  64. Higher CPU core frequency reduce latency and should be preferred. As a simple
  65. rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
  66. provide enough resources for stable and durable Ceph performance.
  67. .Memory
  68. Especially in a hyper-converged setup, the memory consumption needs to be
  69. carefully monitored. In addition to the intended workload from virtual machines
  70. and container, Ceph needs enough memory available to provide good and stable
  71. performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory
  72. will be used by an OSD. OSD caching will use additional memory.
  73. .Network
  74. We recommend a network bandwidth of at least 10 GbE or more, which is used
  75. exclusively for Ceph. A meshed network setup
  76. footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
  77. is also an option if there are no 10 GbE switches available.
  78. The volume of traffic, especially during recovery, will interfere with other
  79. services on the same network and may even break the {pve} cluster stack.
  80. Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
  81. link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate
  82. 10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwith
  83. will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
  84. even 100 GBps are possible.
  85. .Disks
  86. When planning the size of your Ceph cluster, it is important to take the
  87. recovery time into consideration. Especially with small clusters, the recovery
  88. might take long. It is recommended that you use SSDs instead of HDDs in small
  89. setups to reduce recovery time, minimizing the likelihood of a subsequent
  90. failure event during recovery.
  91. In general SSDs will provide more IOPs than spinning disks. This fact and the
  92. higher cost may make a xref:pve_ceph_device_classes[class based] separation of
  93. pools appealing. Another possibility to speedup OSDs is to use a faster disk
  94. as journal or DB/**W**rite-**A**head-**L**og device, see
  95. xref:pve_ceph_osds[creating Ceph OSDs]. If a faster disk is used for multiple
  96. OSDs, a proper balance between OSD and WAL / DB (or journal) disk must be
  97. selected, otherwise the faster disk becomes the bottleneck for all linked OSDs.
  98. Aside from the disk type, Ceph best performs with an even sized and distributed
  99. amount of disks per node. For example, 4 x 500 GB disks with in each node is
  100. better than a mixed setup with a single 1 TB and three 250 GB disk.
  101. One also need to balance OSD count and single OSD capacity. More capacity
  102. allows to increase storage density, but it also means that a single OSD
  103. failure forces ceph to recover more data at once.
  104. .Avoid RAID
  105. As Ceph handles data object redundancy and multiple parallel writes to disks
  106. (OSDs) on its own, using a RAID controller normally doesn’t improve
  107. performance or availability. On the contrary, Ceph is designed to handle whole
  108. disks on it's own, without any abstraction in between. RAID controller are not
  109. designed for the Ceph use case and may complicate things and sometimes even
  110. reduce performance, as their write and caching algorithms may interfere with
  111. the ones from Ceph.
  112. WARNING: Avoid RAID controller, use host bus adapter (HBA) instead.
  113. NOTE: Above recommendations should be seen as a rough guidance for choosing
  114. hardware. Therefore, it is still essential to adapt it to your specific needs,
  115. test your setup and monitor health and performance continuously.
  116. [[pve_ceph_install_wizard]]
  117. Initial Ceph installation & configuration
  118. -----------------------------------------
  119. [thumbnail="screenshot/gui-node-ceph-install.png"]
  120. With {pve} you have the benefit of an easy to use installation wizard
  121. for Ceph. Click on one of your cluster nodes and navigate to the Ceph
  122. section in the menu tree. If Ceph is not already installed you will be
  123. offered to do so now.
  124. The wizard is divided into different sections, where each needs to be
  125. finished successfully in order to use Ceph. After starting the installation
  126. the wizard will download and install all required packages from {pve}'s ceph
  127. repository.
  128. After finishing the first step, you will need to create a configuration.
  129. This step is only needed once per cluster, as this configuration is distributed
  130. automatically to all remaining cluster members through {pve}'s clustered
  131. xref:chapter_pmxcfs[configuration file system (pmxcfs)].
  132. The configuration step includes the following settings:
  133. * *Public Network:* You should setup a dedicated network for Ceph, this
  134. setting is required. Separating your Ceph traffic is highly recommended,
  135. because it could lead to troubles with other latency dependent services,
  136. e.g., cluster communication may decrease Ceph's performance, if not done.
  137. [thumbnail="screenshot/gui-node-ceph-install-wizard-step2.png"]
  138. * *Cluster Network:* As an optional step you can go even further and
  139. separate the xref:pve_ceph_osds[OSD] replication & heartbeat traffic
  140. as well. This will relieve the public network and could lead to
  141. significant performance improvements especially in big clusters.
  142. You have two more options which are considered advanced and therefore
  143. should only changed if you are an expert.
  144. * *Number of replicas*: Defines the how often a object is replicated
  145. * *Minimum replicas*: Defines the minimum number of required replicas
  146. for I/O to be marked as complete.
  147. Additionally you need to choose your first monitor node, this is required.
  148. That's it, you should see a success page as the last step with further
  149. instructions on how to go on. You are now prepared to start using Ceph,
  150. even though you will need to create additional xref:pve_ceph_monitors[monitors],
  151. create some xref:pve_ceph_osds[OSDs] and at least one xref:pve_ceph_pools[pool].
  152. The rest of this chapter will guide you on how to get the most out of
  153. your {pve} based Ceph setup, this will include aforementioned and
  154. more like xref:pveceph_fs[CephFS] which is a very handy addition to your
  155. new Ceph cluster.
  156. [[pve_ceph_install]]
  157. Installation of Ceph Packages
  158. -----------------------------
  159. Use {pve} Ceph installation wizard (recommended) or run the following
  160. command on each node:
  161. [source,bash]
  162. ----
  163. pveceph install
  164. ----
  165. This sets up an `apt` package repository in
  166. `/etc/apt/sources.list.d/ceph.list` and installs the required software.
  167. Create initial Ceph configuration
  168. ---------------------------------
  169. [thumbnail="screenshot/gui-ceph-config.png"]
  170. Use the {pve} Ceph installation wizard (recommended) or run the
  171. following command on one node:
  172. [source,bash]
  173. ----
  174. pveceph init --network 10.10.10.0/24
  175. ----
  176. This creates an initial configuration at `/etc/pve/ceph.conf` with a
  177. dedicated network for ceph. That file is automatically distributed to
  178. all {pve} nodes by using xref:chapter_pmxcfs[pmxcfs]. The command also
  179. creates a symbolic link from `/etc/ceph/ceph.conf` pointing to that file.
  180. So you can simply run Ceph commands without the need to specify a
  181. configuration file.
  182. [[pve_ceph_monitors]]
  183. Ceph Monitor
  184. -----------
  185. The Ceph Monitor (MON)
  186. footnote:[Ceph Monitor https://docs.ceph.com/docs/{ceph_codename}/start/intro/]
  187. maintains a master copy of the cluster map. For high availability you need to
  188. have at least 3 monitors. One monitor will already be installed if you
  189. used the installation wizard. You won't need more than 3 monitors as long
  190. as your cluster is small to midsize, only really large clusters will
  191. need more than that.
  192. [[pveceph_create_mon]]
  193. Create Monitors
  194. ~~~~~~~~~~~~~~~
  195. [thumbnail="screenshot/gui-ceph-monitor.png"]
  196. On each node where you want to place a monitor (three monitors are recommended),
  197. create it by using the 'Ceph -> Monitor' tab in the GUI or run.
  198. [source,bash]
  199. ----
  200. pveceph mon create
  201. ----
  202. [[pveceph_destroy_mon]]
  203. Destroy Monitors
  204. ~~~~~~~~~~~~~~~~
  205. To remove a Ceph Monitor via the GUI first select a node in the tree view and
  206. go to the **Ceph -> Monitor** panel. Select the MON and click the **Destroy**
  207. button.
  208. To remove a Ceph Monitor via the CLI first connect to the node on which the MON
  209. is running. Then execute the following command:
  210. [source,bash]
  211. ----
  212. pveceph mon destroy
  213. ----
  214. NOTE: At least three Monitors are needed for quorum.
  215. [[pve_ceph_manager]]
  216. Ceph Manager
  217. ------------
  218. The Manager daemon runs alongside the monitors. It provides an interface to
  219. monitor the cluster. Since the Ceph luminous release at least one ceph-mgr
  220. footnote:[Ceph Manager https://docs.ceph.com/docs/{ceph_codename}/mgr/] daemon is
  221. required.
  222. [[pveceph_create_mgr]]
  223. Create Manager
  224. ~~~~~~~~~~~~~~
  225. Multiple Managers can be installed, but at any time only one Manager is active.
  226. [source,bash]
  227. ----
  228. pveceph mgr create
  229. ----
  230. NOTE: It is recommended to install the Ceph Manager on the monitor nodes. For
  231. high availability install more then one manager.
  232. [[pveceph_destroy_mgr]]
  233. Destroy Manager
  234. ~~~~~~~~~~~~~~~
  235. To remove a Ceph Manager via the GUI first select a node in the tree view and
  236. go to the **Ceph -> Monitor** panel. Select the Manager and click the
  237. **Destroy** button.
  238. To remove a Ceph Monitor via the CLI first connect to the node on which the
  239. Manager is running. Then execute the following command:
  240. [source,bash]
  241. ----
  242. pveceph mgr destroy
  243. ----
  244. NOTE: A Ceph cluster can function without a Manager, but certain functions like
  245. the cluster status or usage require a running Manager.
  246. [[pve_ceph_osds]]
  247. Ceph OSDs
  248. ---------
  249. Ceph **O**bject **S**torage **D**aemons are storing objects for Ceph over the
  250. network. It is recommended to use one OSD per physical disk.
  251. NOTE: By default an object is 4 MiB in size.
  252. [[pve_ceph_osd_create]]
  253. Create OSDs
  254. ~~~~~~~~~~~
  255. [thumbnail="screenshot/gui-ceph-osd-status.png"]
  256. via GUI or via CLI as follows:
  257. [source,bash]
  258. ----
  259. pveceph osd create /dev/sd[X]
  260. ----
  261. TIP: We recommend a Ceph cluster size, starting with 12 OSDs, distributed
  262. evenly among your, at least three nodes (4 OSDs on each node).
  263. If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot
  264. sector and any OSD leftover the following command should be sufficient.
  265. [source,bash]
  266. ----
  267. ceph-volume lvm zap /dev/sd[X] --destroy
  268. ----
  269. WARNING: The above command will destroy data on the disk!
  270. .Ceph Bluestore
  271. Starting with the Ceph Kraken release, a new Ceph OSD storage type was
  272. introduced, the so called Bluestore
  273. footnote:[Ceph Bluestore https://ceph.com/community/new-luminous-bluestore/].
  274. This is the default when creating OSDs since Ceph Luminous.
  275. [source,bash]
  276. ----
  277. pveceph osd create /dev/sd[X]
  278. ----
  279. .Block.db and block.wal
  280. If you want to use a separate DB/WAL device for your OSDs, you can specify it
  281. through the '-db_dev' and '-wal_dev' options. The WAL is placed with the DB, if
  282. not specified separately.
  283. [source,bash]
  284. ----
  285. pveceph osd create /dev/sd[X] -db_dev /dev/sd[Y] -wal_dev /dev/sd[Z]
  286. ----
  287. You can directly choose the size for those with the '-db_size' and '-wal_size'
  288. paremeters respectively. If they are not given the following values (in order)
  289. will be used:
  290. * bluestore_block_{db,wal}_size from ceph configuration...
  291. ** ... database, section 'osd'
  292. ** ... database, section 'global'
  293. ** ... file, section 'osd'
  294. ** ... file, section 'global'
  295. * 10% (DB)/1% (WAL) of OSD size
  296. NOTE: The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s
  297. internal journal or write-ahead log. It is recommended to use a fast SSD or
  298. NVRAM for better performance.
  299. .Ceph Filestore
  300. Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs.
  301. Starting with Ceph Nautilus, {pve} does not support creating such OSDs with
  302. 'pveceph' anymore. If you still want to create filestore OSDs, use
  303. 'ceph-volume' directly.
  304. [source,bash]
  305. ----
  306. ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
  307. ----
  308. [[pve_ceph_osd_destroy]]
  309. Destroy OSDs
  310. ~~~~~~~~~~~~
  311. To remove an OSD via the GUI first select a {PVE} node in the tree view and go
  312. to the **Ceph -> OSD** panel. Select the OSD to destroy. Next click the **OUT**
  313. button. Once the OSD status changed from `in` to `out` click the **STOP**
  314. button. As soon as the status changed from `up` to `down` select **Destroy**
  315. from the `More` drop-down menu.
  316. To remove an OSD via the CLI run the following commands.
  317. [source,bash]
  318. ----
  319. ceph osd out <ID>
  320. systemctl stop ceph-osd@<ID>.service
  321. ----
  322. NOTE: The first command instructs Ceph not to include the OSD in the data
  323. distribution. The second command stops the OSD service. Until this time, no
  324. data is lost.
  325. The following command destroys the OSD. Specify the '-cleanup' option to
  326. additionally destroy the partition table.
  327. [source,bash]
  328. ----
  329. pveceph osd destroy <ID>
  330. ----
  331. WARNING: The above command will destroy data on the disk!
  332. [[pve_ceph_pools]]
  333. Ceph Pools
  334. ----------
  335. A pool is a logical group for storing objects. It holds **P**lacement
  336. **G**roups (`PG`, `pg_num`), a collection of objects.
  337. Create Pools
  338. ~~~~~~~~~~~~
  339. [thumbnail="screenshot/gui-ceph-pools.png"]
  340. When no options are given, we set a default of **128 PGs**, a **size of 3
  341. replicas** and a **min_size of 2 replicas** for serving objects in a degraded
  342. state.
  343. NOTE: The default number of PGs works for 2-5 disks. Ceph throws a
  344. 'HEALTH_WARNING' if you have too few or too many PGs in your cluster.
  345. It is advised to calculate the PG number depending on your setup, you can find
  346. the formula and the PG calculator footnote:[PG calculator
  347. https://ceph.com/pgcalc/] online. While PGs can be increased later on, they can
  348. never be decreased.
  349. You can create pools through command line or on the GUI on each PVE host under
  350. **Ceph -> Pools**.
  351. [source,bash]
  352. ----
  353. pveceph pool create <name>
  354. ----
  355. If you would like to automatically also get a storage definition for your pool,
  356. mark the checkbox "Add storages" in the GUI or use the command line option
  357. '--add_storages' at pool creation.
  358. Further information on Ceph pool handling can be found in the Ceph pool
  359. operation footnote:[Ceph pool operation
  360. https://docs.ceph.com/docs/{ceph_codename}/rados/operations/pools/]
  361. manual.
  362. Destroy Pools
  363. ~~~~~~~~~~~~~
  364. To destroy a pool via the GUI select a node in the tree view and go to the
  365. **Ceph -> Pools** panel. Select the pool to destroy and click the **Destroy**
  366. button. To confirm the destruction of the pool you need to enter the pool name.
  367. Run the following command to destroy a pool. Specify the '-remove_storages' to
  368. also remove the associated storage.
  369. [source,bash]
  370. ----
  371. pveceph pool destroy <name>
  372. ----
  373. NOTE: Deleting the data of a pool is a background task and can take some time.
  374. You will notice that the data usage in the cluster is decreasing.
  375. [[pve_ceph_device_classes]]
  376. Ceph CRUSH & device classes
  377. ---------------------------
  378. The foundation of Ceph is its algorithm, **C**ontrolled **R**eplication
  379. **U**nder **S**calable **H**ashing
  380. (CRUSH footnote:[CRUSH https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf]).
  381. CRUSH calculates where to store to and retrieve data from, this has the
  382. advantage that no central index service is needed. CRUSH works with a map of
  383. OSDs, buckets (device locations) and rulesets (data replication) for pools.
  384. NOTE: Further information can be found in the Ceph documentation, under the
  385. section CRUSH map footnote:[CRUSH map https://docs.ceph.com/docs/{ceph_codename}/rados/operations/crush-map/].
  386. This map can be altered to reflect different replication hierarchies. The object
  387. replicas can be separated (eg. failure domains), while maintaining the desired
  388. distribution.
  389. A common use case is to use different classes of disks for different Ceph pools.
  390. For this reason, Ceph introduced the device classes with luminous, to
  391. accommodate the need for easy ruleset generation.
  392. The device classes can be seen in the 'ceph osd tree' output. These classes
  393. represent their own root bucket, which can be seen with the below command.
  394. [source, bash]
  395. ----
  396. ceph osd crush tree --show-shadow
  397. ----
  398. Example output form the above command:
  399. [source, bash]
  400. ----
  401. ID CLASS WEIGHT TYPE NAME
  402. -16 nvme 2.18307 root default~nvme
  403. -13 nvme 0.72769 host sumi1~nvme
  404. 12 nvme 0.72769 osd.12
  405. -14 nvme 0.72769 host sumi2~nvme
  406. 13 nvme 0.72769 osd.13
  407. -15 nvme 0.72769 host sumi3~nvme
  408. 14 nvme 0.72769 osd.14
  409. -1 7.70544 root default
  410. -3 2.56848 host sumi1
  411. 12 nvme 0.72769 osd.12
  412. -5 2.56848 host sumi2
  413. 13 nvme 0.72769 osd.13
  414. -7 2.56848 host sumi3
  415. 14 nvme 0.72769 osd.14
  416. ----
  417. To let a pool distribute its objects only on a specific device class, you need
  418. to create a ruleset with the specific class first.
  419. [source, bash]
  420. ----
  421. ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>
  422. ----
  423. [frame="none",grid="none", align="left", cols="30%,70%"]
  424. |===
  425. |<rule-name>|name of the rule, to connect with a pool (seen in GUI & CLI)
  426. |<root>|which crush root it should belong to (default ceph root "default")
  427. |<failure-domain>|at which failure-domain the objects should be distributed (usually host)
  428. |<class>|what type of OSD backing store to use (eg. nvme, ssd, hdd)
  429. |===
  430. Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.
  431. [source, bash]
  432. ----
  433. ceph osd pool set <pool-name> crush_rule <rule-name>
  434. ----
  435. TIP: If the pool already contains objects, all of these have to be moved
  436. accordingly. Depending on your setup this may introduce a big performance hit
  437. on your cluster. As an alternative, you can create a new pool and move disks
  438. separately.
  439. Ceph Client
  440. -----------
  441. [thumbnail="screenshot/gui-ceph-log.png"]
  442. You can then configure {pve} to use such pools to store VM or
  443. Container images. Simply use the GUI too add a new `RBD` storage (see
  444. section xref:ceph_rados_block_devices[Ceph RADOS Block Devices (RBD)]).
  445. You also need to copy the keyring to a predefined location for an external Ceph
  446. cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
  447. done automatically.
  448. NOTE: The file name needs to be `<storage_id> + `.keyring` - `<storage_id>` is
  449. the expression after 'rbd:' in `/etc/pve/storage.cfg` which is
  450. `my-ceph-storage` in the following example:
  451. [source,bash]
  452. ----
  453. mkdir /etc/pve/priv/ceph
  454. cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring
  455. ----
  456. [[pveceph_fs]]
  457. CephFS
  458. ------
  459. Ceph provides also a filesystem running on top of the same object storage as
  460. RADOS block devices do. A **M**eta**d**ata **S**erver (`MDS`) is used to map
  461. the RADOS backed objects to files and directories, allowing to provide a
  462. POSIX-compliant replicated filesystem. This allows one to have a clustered
  463. highly available shared filesystem in an easy way if ceph is already used. Its
  464. Metadata Servers guarantee that files get balanced out over the whole Ceph
  465. cluster, this way even high load will not overload a single host, which can be
  466. an issue with traditional shared filesystem approaches, like `NFS`, for
  467. example.
  468. [thumbnail="screenshot/gui-node-ceph-cephfs-panel.png"]
  469. {pve} supports both, using an existing xref:storage_cephfs[CephFS as storage]
  470. to save backups, ISO files or container templates and creating a
  471. hyper-converged CephFS itself.
  472. [[pveceph_fs_mds]]
  473. Metadata Server (MDS)
  474. ~~~~~~~~~~~~~~~~~~~~~
  475. CephFS needs at least one Metadata Server to be configured and running to be
  476. able to work. One can simply create one through the {pve} web GUI's `Node ->
  477. CephFS` panel or on the command line with:
  478. ----
  479. pveceph mds create
  480. ----
  481. Multiple metadata servers can be created in a cluster. But with the default
  482. settings only one can be active at any time. If an MDS, or its node, becomes
  483. unresponsive (or crashes), another `standby` MDS will get promoted to `active`.
  484. One can speed up the hand-over between the active and a standby MDS up by using
  485. the 'hotstandby' parameter option on create, or if you have already created it
  486. you may set/add:
  487. ----
  488. mds standby replay = true
  489. ----
  490. in the ceph.conf respective MDS section. With this enabled, this specific MDS
  491. will always poll the active one, so that it can take over faster as it is in a
  492. `warm` state. But naturally, the active polling will cause some additional
  493. performance impact on your system and active `MDS`.
  494. .Multiple Active MDS
  495. Since Luminous (12.2.x) you can also have multiple active metadata servers
  496. running, but this is normally only useful for a high count on parallel clients,
  497. as else the `MDS` seldom is the bottleneck. If you want to set this up please
  498. refer to the ceph documentation. footnote:[Configuring multiple active MDS
  499. daemons https://docs.ceph.com/docs/{ceph_codename}/cephfs/multimds/]
  500. [[pveceph_fs_create]]
  501. Create CephFS
  502. ~~~~~~~~~~~~~
  503. With {pve}'s CephFS integration into you can create a CephFS easily over the
  504. Web GUI, the CLI or an external API interface. Some prerequisites are required
  505. for this to work:
  506. .Prerequisites for a successful CephFS setup:
  507. - xref:pve_ceph_install[Install Ceph packages], if this was already done some
  508. time ago you might want to rerun it on an up to date system to ensure that
  509. also all CephFS related packages get installed.
  510. - xref:pve_ceph_monitors[Setup Monitors]
  511. - xref:pve_ceph_monitors[Setup your OSDs]
  512. - xref:pveceph_fs_mds[Setup at least one MDS]
  513. After this got all checked and done you can simply create a CephFS through
  514. either the Web GUI's `Node -> CephFS` panel or the command line tool `pveceph`,
  515. for example with:
  516. ----
  517. pveceph fs create --pg_num 128 --add-storage
  518. ----
  519. This creates a CephFS named `'cephfs'' using a pool for its data named
  520. `'cephfs_data'' with `128` placement groups and a pool for its metadata named
  521. `'cephfs_metadata'' with one quarter of the data pools placement groups (`32`).
  522. Check the xref:pve_ceph_pools[{pve} managed Ceph pool chapter] or visit the
  523. Ceph documentation for more information regarding a fitting placement group
  524. number (`pg_num`) for your setup footnote:[Ceph Placement Groups
  525. https://docs.ceph.com/docs/{ceph_codename}/rados/operations/placement-groups/].
  526. Additionally, the `'--add-storage'' parameter will add the CephFS to the {pve}
  527. storage configuration after it was created successfully.
  528. Destroy CephFS
  529. ~~~~~~~~~~~~~~
  530. WARNING: Destroying a CephFS will render all its data unusable, this cannot be
  531. undone!
  532. If you really want to destroy an existing CephFS you first need to stop, or
  533. destroy, all metadata servers (`M̀DS`). You can destroy them either over the Web
  534. GUI or the command line interface, with:
  535. ----
  536. pveceph mds destroy NAME
  537. ----
  538. on each {pve} node hosting a MDS daemon.
  539. Then, you can remove (destroy) CephFS by issuing a:
  540. ----
  541. ceph fs rm NAME --yes-i-really-mean-it
  542. ----
  543. on a single node hosting Ceph. After this you may want to remove the created
  544. data and metadata pools, this can be done either over the Web GUI or the CLI
  545. with:
  546. ----
  547. pveceph pool destroy NAME
  548. ----
  549. Ceph maintenance
  550. ----------------
  551. Replace OSDs
  552. ~~~~~~~~~~~~
  553. One of the common maintenance tasks in Ceph is to replace a disk of an OSD. If
  554. a disk is already in a failed state, then you can go ahead and run through the
  555. steps in xref:pve_ceph_osd_destroy[Destroy OSDs]. Ceph will recreate those
  556. copies on the remaining OSDs if possible. This rebalancing will start as soon
  557. as an OSD failure is detected or an OSD was actively stopped.
  558. NOTE: With the default size/min_size (3/2) of a pool, recovery only starts when
  559. `size + 1` nodes are available. The reason for this is that the Ceph object
  560. balancer xref:pve_ceph_device_classes[CRUSH] defaults to a full node as
  561. `failure domain'.
  562. To replace a still functioning disk, on the GUI go through the steps in
  563. xref:pve_ceph_osd_destroy[Destroy OSDs]. The only addition is to wait until
  564. the cluster shows 'HEALTH_OK' before stopping the OSD to destroy it.
  565. On the command line use the following commands.
  566. ----
  567. ceph osd out osd.<id>
  568. ----
  569. You can check with the command below if the OSD can be safely removed.
  570. ----
  571. ceph osd safe-to-destroy osd.<id>
  572. ----
  573. Once the above check tells you that it is save to remove the OSD, you can
  574. continue with following commands.
  575. ----
  576. systemctl stop ceph-osd@<id>.service
  577. pveceph osd destroy <id>
  578. ----
  579. Replace the old disk with the new one and use the same procedure as described
  580. in xref:pve_ceph_osd_create[Create OSDs].
  581. Trim/Discard
  582. ~~~~~~~~~~~~
  583. It is a good measure to run 'fstrim' (discard) regularly on VMs or containers.
  584. This releases data blocks that the filesystem isn’t using anymore. It reduces
  585. data usage and resource load. Most modern operating systems issue such discard
  586. commands to their disks regularly. You only need to ensure that the Virtual
  587. Machines enable the xref:qm_hard_disk_discard[disk discard option].
  588. [[pveceph_scrub]]
  589. Scrub & Deep Scrub
  590. ~~~~~~~~~~~~~~~~~~
  591. Ceph ensures data integrity by 'scrubbing' placement groups. Ceph checks every
  592. object in a PG for its health. There are two forms of Scrubbing, daily
  593. cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
  594. the objects and uses checksums to ensure data integrity. If a running scrub
  595. interferes with business (performance) needs, you can adjust the time when
  596. scrubs footnote:[Ceph scrubbing https://docs.ceph.com/docs/{ceph_codename}/rados/configuration/osd-config-ref/#scrubbing]
  597. are executed.
  598. Ceph monitoring and troubleshooting
  599. -----------------------------------
  600. A good start is to continuosly monitor the ceph health from the start of
  601. initial deployment. Either through the ceph tools itself, but also by accessing
  602. the status through the {pve} link:api-viewer/index.html[API].
  603. The following ceph commands below can be used to see if the cluster is healthy
  604. ('HEALTH_OK'), if there are warnings ('HEALTH_WARN'), or even errors
  605. ('HEALTH_ERR'). If the cluster is in an unhealthy state the status commands
  606. below will also give you an overview of the current events and actions to take.
  607. ----
  608. # single time output
  609. pve# ceph -s
  610. # continuously output status changes (press CTRL+C to stop)
  611. pve# ceph -w
  612. ----
  613. To get a more detailed view, every ceph service has a log file under
  614. `/var/log/ceph/` and if there is not enough detail, the log level can be
  615. adjusted footnote:[Ceph log and debugging https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/log-and-debug/].
  616. You can find more information about troubleshooting
  617. footnote:[Ceph troubleshooting https://docs.ceph.com/docs/{ceph_codename}/rados/troubleshooting/]
  618. a Ceph cluster on the official website.
  619. ifdef::manvolnum[]
  620. include::pve-copyright.adoc[]
  621. endif::manvolnum[]