diff options
Diffstat (limited to 'Documentation/md')
| -rw-r--r-- | Documentation/md/index.rst | 12 | ||||
| -rw-r--r-- | Documentation/md/md-cluster.rst (renamed from Documentation/md/md-cluster.txt) | 184 | ||||
| -rw-r--r-- | Documentation/md/raid5-cache.rst (renamed from Documentation/md/raid5-cache.txt) | 28 | ||||
| -rw-r--r-- | Documentation/md/raid5-ppl.rst (renamed from Documentation/md/raid5-ppl.txt) | 2 |
4 files changed, 151 insertions, 75 deletions
diff --git a/Documentation/md/index.rst b/Documentation/md/index.rst new file mode 100644 index 000000000000..c4db34ed327d --- /dev/null +++ b/Documentation/md/index.rst @@ -0,0 +1,12 @@ +:orphan: + +==== +RAID +==== + +.. toctree:: + :maxdepth: 1 + + md-cluster + raid5-cache + raid5-ppl diff --git a/Documentation/md/md-cluster.txt b/Documentation/md/md-cluster.rst index e1055f105cf5..96eb52cec7eb 100644 --- a/Documentation/md/md-cluster.txt +++ b/Documentation/md/md-cluster.rst @@ -1,19 +1,24 @@ +========== +MD Cluster +========== + The cluster MD is a shared-device RAID for a cluster, it supports two levels: raid1 and raid10 (limited support). 1. On-disk format +================= Separate write-intent-bitmaps are used for each cluster node. The bitmaps record all writes that may have been started on that node, -and may not yet have finished. The on-disk layout is: +and may not yet have finished. The on-disk layout is:: -0 4k 8k 12k -------------------------------------------------------------------- -| idle | md super | bm super [0] + bits | -| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] | -| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits | -| bm bits [3, contd] | | | + 0 4k 8k 12k + ------------------------------------------------------------------- + | idle | md super | bm super [0] + bits | + | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] | + | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits | + | bm bits [3, contd] | | | During "normal" functioning we assume the filesystem ensures that only one node writes to any given block at a time, so a write request will @@ -28,10 +33,12 @@ node) is writing. 2. DLM Locks for management +=========================== There are three groups of locks for managing the device: 2.1 Bitmap lock resource (bm_lockres) +------------------------------------- The bm_lockres protects individual node bitmaps. They are named in the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a @@ -48,6 +55,7 @@ There are three groups of locks for managing the device: joins the cluster. 2.2 Message passing locks +------------------------- Each node has to communicate with other nodes when starting or ending resync, and for metadata superblock updates. This communication is @@ -55,116 +63,155 @@ There are three groups of locks for managing the device: with the Lock Value Block (LVB) of one of the "message" lock. 2.3 new-device management +------------------------- A single lock: "no-new-dev" is used to co-ordinate the addition of new devices - this must be synchronized across the array. Normally all nodes hold a concurrent-read lock on this device. 3. Communication +================ Messages can be broadcast to all nodes, and the sender waits for all other nodes to acknowledge the message before proceeding. Only one message can be processed at a time. 3.1 Message Types +----------------- There are six types of messages which are passed: - 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has +3.1.1 METADATA_UPDATED +^^^^^^^^^^^^^^^^^^^^^^ + + informs other nodes that the metadata has been updated, and the node must re-read the md superblock. This is performed synchronously. It is primarily used to signal device failure. - 3.1.2 RESYNCING: informs other nodes that a resync is initiated or +3.1.2 RESYNCING +^^^^^^^^^^^^^^^ + informs other nodes that a resync is initiated or ended so that each node may suspend or resume the region. Each RESYNCING message identifies a range of the devices that the sending node is about to resync. This overrides any previous notification from that node: only one ranged can be resynced at a time per-node. - 3.1.3 NEWDISK: informs other nodes that a device is being added to +3.1.3 NEWDISK +^^^^^^^^^^^^^ + + informs other nodes that a device is being added to the array. Message contains an identifier for that device. See below for further details. - 3.1.4 REMOVE: A failed or spare device is being removed from the +3.1.4 REMOVE +^^^^^^^^^^^^ + + A failed or spare device is being removed from the array. The slot-number of the device is included in the message. - 3.1.5 RE_ADD: A failed device is being re-activated - the assumption + 3.1.5 RE_ADD: + + A failed device is being re-activated - the assumption is that it has been determined to be working again. - 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap + 3.1.6 BITMAP_NEEDS_SYNC: + + If a node is stopped locally but the bitmap isn't clean, then another node is informed to take the ownership of resync. 3.2 Communication mechanism +--------------------------- The DLM LVB is used to communicate within nodes of the cluster. There are three resources used for the purpose: - 3.2.1 token: The resource which protects the entire communication +3.2.1 token +^^^^^^^^^^^ + The resource which protects the entire communication system. The node having the token resource is allowed to communicate. - 3.2.2 message: The lock resource which carries the data to - communicate. +3.2.2 message +^^^^^^^^^^^^^ + The lock resource which carries the data to communicate. - 3.2.3 ack: The resource, acquiring which means the message has been +3.2.3 ack +^^^^^^^^^ + + The resource, acquiring which means the message has been acknowledged by all nodes in the cluster. The BAST of the resource is used to inform the receiving node that a node wants to communicate. The algorithm is: - 1. receive status - all nodes have concurrent-reader lock on "ack". + 1. receive status - all nodes have concurrent-reader lock on "ack":: + + sender receiver receiver + "ack":CR "ack":CR "ack":CR - sender receiver receiver - "ack":CR "ack":CR "ack":CR + 2. sender get EX on "token", + sender get EX on "message":: - 2. sender get EX on "token" - sender get EX on "message" - sender receiver receiver - "token":EX "ack":CR "ack":CR - "message":EX - "ack":CR + sender receiver receiver + "token":EX "ack":CR "ack":CR + "message":EX + "ack":CR Sender checks that it still needs to send a message. Messages received or other events that happened while waiting for the "token" may have made this message inappropriate or redundant. - 3. sender writes LVB. + 3. sender writes LVB + sender down-convert "message" from EX to CW + sender try to get EX of "ack" - [ wait until all receivers have *processed* the "message" ] - [ triggered by bast of "ack" ] - receiver get CR on "message" - receiver read LVB - receiver processes the message - [ wait finish ] - receiver releases "ack" - receiver tries to get PR on "message" + :: + + [ wait until all receivers have *processed* the "message" ] - sender receiver receiver - "token":EX "message":CR "message":CR - "message":CW - "ack":EX + [ triggered by bast of "ack" ] + receiver get CR on "message" + receiver read LVB + receiver processes the message + [ wait finish ] + receiver releases "ack" + receiver tries to get PR on "message" + + sender receiver receiver + "token":EX "message":CR "message":CR + "message":CW + "ack":EX 4. triggered by grant of EX on "ack" (indicating all receivers have processed message) + sender down-converts "ack" from EX to CR + sender releases "message" + sender releases "token" - receiver upconvert to PR on "message" - receiver get CR of "ack" - receiver release "message" - sender receiver receiver - "ack":CR "ack":CR "ack":CR + :: + + receiver upconvert to PR on "message" + receiver get CR of "ack" + receiver release "message" + + sender receiver receiver + "ack":CR "ack":CR "ack":CR 4. Handling Failures +==================== 4.1 Node Failure +---------------- When a node fails, the DLM informs the cluster with the slot number. The node starts a cluster recovery thread. The cluster @@ -177,11 +224,11 @@ The algorithm is: - cleans the bitmap of the failed node - releases bitmap<number> lock of the failed node - initiates resync of the bitmap on the current node - md_check_recovery is invoked within recover_bitmaps, - then md_check_recovery -> metadata_update_start/finish, - it will lock the communication by lock_comm. - Which means when one node is resyncing it blocks all - other nodes from writing anywhere on the array. + md_check_recovery is invoked within recover_bitmaps, + then md_check_recovery -> metadata_update_start/finish, + it will lock the communication by lock_comm. + Which means when one node is resyncing it blocks all + other nodes from writing anywhere on the array. The resync process is the regular md resync. However, in a clustered environment when a resync is performed, it needs to tell other nodes @@ -198,6 +245,7 @@ The algorithm is: particular I/O range should be suspended or not. 4.2 Device Failure +================== Device failures are handled and communicated with the metadata update routine. When a node detects a device failure it does not allow @@ -205,38 +253,41 @@ The algorithm is: acknowledged by all other nodes. 5. Adding a new Device +---------------------- For adding a new device, it is necessary that all nodes "see" the new device to be added. For this, the following algorithm is used: - 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues + 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD) - 2. Node 1 sends a NEWDISK message with uuid and slot number - 3. Other nodes issue kobject_uevent_env with uuid and slot number + 2. Node 1 sends a NEWDISK message with uuid and slot number + 3. Other nodes issue kobject_uevent_env with uuid and slot number (Steps 4,5 could be a udev rule) - 4. In userspace, the node searches for the disk, perhaps + 4. In userspace, the node searches for the disk, perhaps using blkid -t SUB_UUID="" - 5. Other nodes issue either of the following depending on whether + 5. Other nodes issue either of the following depending on whether the disk was found: ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and - disc.number set to slot number) + disc.number set to slot number) ioctl(CLUSTERED_DISK_NACK) - 6. Other nodes drop lock on "no-new-devs" (CR) if device is found - 7. Node 1 attempts EX lock on "no-new-dev" - 8. If node 1 gets the lock, it sends METADATA_UPDATED after + 6. Other nodes drop lock on "no-new-devs" (CR) if device is found + 7. Node 1 attempts EX lock on "no-new-dev" + 8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk as SpareLocal - 9. If not (get "no-new-dev" lock), it fails the operation and sends + 9. If not (get "no-new-dev" lock), it fails the operation and sends METADATA_UPDATED. 10. Other nodes get the information whether a disk is added or not by the following METADATA_UPDATED. -6. Module interface. +6. Module interface +=================== There are 17 call-backs which the md core can make to the cluster module. Understanding these can give a good overview of the whole process. 6.1 join(nodes) and leave() +--------------------------- These are called when an array is started with a clustered bitmap, and when the array is stopped. join() ensures the cluster is @@ -244,11 +295,13 @@ The algorithm is: Only the first 'nodes' nodes in the cluster can use the array. 6.2 slot_number() +----------------- Reports the slot number advised by the cluster infrastructure. Range is from 0 to nodes-1. 6.3 resync_info_update() +------------------------ This updates the resync range that is stored in the bitmap lock. The starting point is updated as the resync progresses. The @@ -256,6 +309,7 @@ The algorithm is: It does *not* send a RESYNCING message. 6.4 resync_start(), resync_finish() +----------------------------------- These are called when resync/recovery/reshape starts or stops. They update the resyncing range in the bitmap lock and also @@ -265,8 +319,8 @@ The algorithm is: resync_finish() also sends a BITMAP_NEEDS_SYNC message which allows some other node to take over. -6.5 metadata_update_start(), metadata_update_finish(), - metadata_update_cancel(). +6.5 metadata_update_start(), metadata_update_finish(), metadata_update_cancel() +------------------------------------------------------------------------------- metadata_update_start is used to get exclusive access to the metadata. If a change is still needed once that access is @@ -275,6 +329,7 @@ The algorithm is: can be used to release the lock. 6.6 area_resyncing() +-------------------- This combines two elements of functionality. @@ -289,6 +344,7 @@ The algorithm is: a node failure. 6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack() +--------------------------------------------------------------- These are used to manage the new-disk protocol described above. When a new device is added, add_new_disk_start() is called before @@ -300,17 +356,20 @@ The algorithm is: new_disk_ack() is called. 6.8 remove_disk() +----------------- This is called when a spare or failed device is removed from the array. It causes a REMOVE message to be send to other nodes. 6.9 gather_bitmaps() +-------------------- This sends a RE_ADD message to all other nodes and then gathers bitmap information from all bitmaps. This combined bitmap is then used to recovery the re-added device. 6.10 lock_all_bitmaps() and unlock_all_bitmaps() +------------------------------------------------ These are called when change bitmap to none. If a node plans to clear the cluster raid's bitmap, it need to make sure no other @@ -319,6 +378,7 @@ The algorithm is: accordingly. 7. Unsupported features +======================= There are somethings which are not supported by cluster MD yet. diff --git a/Documentation/md/raid5-cache.txt b/Documentation/md/raid5-cache.rst index 2b210f295786..d7a15f44a7c3 100644 --- a/Documentation/md/raid5-cache.txt +++ b/Documentation/md/raid5-cache.rst @@ -1,4 +1,6 @@ -RAID5 cache +================ +RAID 4/5/6 cache +================ Raid 4/5/6 could include an extra disk for data cache besides normal RAID disks. The role of RAID disks isn't changed with the cache disk. The cache disk @@ -6,19 +8,19 @@ caches data to the RAID disks. The cache can be in write-through (supported since 4.4) or write-back mode (supported since 4.10). mdadm (supported since 3.4) has a new option '--write-journal' to create array with cache. Please refer to mdadm manual for details. By default (RAID array starts), the cache is -in write-through mode. A user can switch it to write-back mode by: +in write-through mode. A user can switch it to write-back mode by:: -echo "write-back" > /sys/block/md0/md/journal_mode + echo "write-back" > /sys/block/md0/md/journal_mode -And switch it back to write-through mode by: +And switch it back to write-through mode by:: -echo "write-through" > /sys/block/md0/md/journal_mode + echo "write-through" > /sys/block/md0/md/journal_mode In both modes, all writes to the array will hit cache disk first. This means the cache disk must be fast and sustainable. -------------------------------------- -write-through mode: +write-through mode +================== This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an unclean shutdown can cause data in some stripes to not be in consistent state, eg, data @@ -42,8 +44,8 @@ exposed to 'write hole' again. In write-through mode, the cache disk isn't required to be big. Several hundreds megabytes are enough. --------------------------------------- -write-back mode: +write-back mode +=============== write-back mode fixes the 'write hole' issue too, since all write data is cached on cache disk. But the main goal of 'write-back' cache is to speed up @@ -64,16 +66,16 @@ data loss. In write-back mode, MD also caches data in memory. The memory cache includes the same data stored on cache disk, so a power loss doesn't cause data loss. The memory cache size has performance impact for the array. It's recommended -the size is big. A user can configure the size by: +the size is big. A user can configure the size by:: -echo "2048" > /sys/block/md0/md/stripe_cache_size + echo "2048" > /sys/block/md0/md/stripe_cache_size Too small cache disk will make the write aggregation less efficient in this mode depending on the workloads. It's recommended to use a cache disk with at least several gigabytes size in write-back mode. --------------------------------------- -The implementation: +The implementation +================== The write-through and write-back cache use the same disk format. The cache disk is organized as a simple write log. The log consists of 'meta data' and 'data' diff --git a/Documentation/md/raid5-ppl.txt b/Documentation/md/raid5-ppl.rst index bfa092589e00..357e5515bc55 100644 --- a/Documentation/md/raid5-ppl.txt +++ b/Documentation/md/raid5-ppl.rst @@ -1,4 +1,6 @@ +================== Partial Parity Log +================== Partial Parity Log (PPL) is a feature available for RAID5 arrays. The issue addressed by PPL is that after a dirty shutdown, parity of a particular stripe |