WiteWulf Posted September 27, 2022 Share #1 Posted September 27, 2022 (edited) I'm trying to figure out how to make the best use of a relatively small NVMe SSD and larger spinning HDD in the ESXi host I run one of my xpenology systems on. After doing some digging about I think I've got it figured and wanted to share in the hopes it may be useful. All of the following assumes your are not using any hardware RAID on your host system, and are passing the physical or virtual disks straight through to DSM DSM stores a copy of the system files on every HDD (physical or virtual) that the system can see, to ensure there's always a working version of the OS (if not all the user data) if there should be a HDD failure. On the system I've been testing with I have a relatively small (30GB) virtual disk on the SSD that was the only disk present when I built the initial DSM install. Storage Manager reports only 18.2GB of usable space in the volume/storage pool created with it, so there's obviously a relatively large amount of space reserved for system files on there. I subsequently added 100GB and 500GB virtual disks (both on a 1TB SATA HDD in the ESXi host). Storage Manager presents these as 85.4GB and 469.4GB volumes respectively. It would appear that DSM reserves a variable amount of space, but not one that's consistent as a percentage of the disk size. We've got, roughly: 11.8GB of 30GB or 39% 14.4GB of 100GB or 14.4% 30.6GB of 500GB or 6% ...reserved on each disk. I install all my DSM Apps, and keep docker containers and data, on the SSD volume to ensure they're as fast as can be, but how does the system know/choose which copy of the OS to boot from and use from the many disks in most systems? Looking at the output of 'cat /proc/mdstat': dsm2:~$ cat /proc/mdstat Personalities : [raid1] [linear] md2 : active raid1 sata2p3[0] 20732928 blocks super 1.2 [1/1] [U] md4 : active raid1 sata3p3[0] 513563648 blocks super 1.2 [1/1] [U] md3 : active linear sata1p3[0] 94133248 blocks super 1.2 64k rounding [1/1] [U] md1 : active raid1 sata1p2[0] sata3p2[2] sata2p2[1] 2097088 blocks [16/3] [UUU_____________] md0 : active raid1 sata1p1[0] sata3p1[2] sata2p1[1] 8388544 blocks [16/3] [UUU_____________] I can see four RAID1 (mirrored) sets (md0, md1, md2 and md4), where md0 and md1 are on all three disks, and md4 and md5 are on single disk. There's also a linear/JBOD set on a single disk, md3. DSM uses /dev/mapper to map these block devices onto user visible volumes. We can see the this in the output of 'sudo dmsetup deps -o devname': vg1-syno_vg_reserved_area: 1 dependencies : (md2) vg2-syno_vg_reserved_area: 1 dependencies : (md3) cachedev_2: 1 dependencies : (vg1-volume_1) vg3-volume_3: 1 dependencies : (md4) cachedev_1: 1 dependencies : (vg2-volume_2) cachedev_0: 1 dependencies : (vg3-volume_3) vg2-volume_2: 1 dependencies : (md3) vg3-syno_vg_reserved_area: 1 dependencies : (md4) vg1-volume_1: 1 dependencies : (md2) ...and of 'df -h': Filesystem Size Used Avail Use% Mounted on /dev/md0 7.9G 1.7G 6.1G 22% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 124K 2.0G 1% /dev/shm tmpfs 2.0G 20M 1.9G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup tmpfs 2.0G 632K 2.0G 1% /tmp /dev/mapper/cachedev_2 19G 5.0G 14G 28% /volume1 /dev/mapper/cachedev_0 470G 120G 350G 26% /volume3 /dev/mapper/cachedev_1 86G 3.3G 83G 4% /volume2 So it would appear that: - md0 is a 7.9GB RAID1 volume containing the OS that is present on every disk in the system and mounted as / - md1 is a 2GB RAID1 volume again spread across all disks. I think this used for devtmpfs - md2 is a 19GB RAID1 volume with a single disk (on my SSD) visible to the user as /volume1 - md3 is a 86GB linear/JBOD volume with a single disk (on my SATA HDD) visible to the user as /volume2 - md4 is a 470GB RAID1 volume with a single disk (on my SATA HDD) visible to the user as /volume3 I still wasn't clear why the amount of data reserved on each member disk for the md0 RAID1 set seemed inconsistent, or what the logic behind it is. So I looked at the partitions for each disk: sudo fdisk -l /dev/sata1 Password: Disk /dev/sata1: 100 GiB, 107374182400 bytes, 209715200 sectors Disk model: Virtual SATA Hard Drive Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x51a58d33 Device Boot Start End Sectors Size Id Type /dev/sata1p1 8192 16785407 16777216 8G fd Linux raid autodetect /dev/sata1p2 16785408 20979711 4194304 2G fd Linux raid autodetect /dev/sata1p3 21241856 209510399 188268544 89.8G fd Linux raid autodetect garyp@dsm2:/dev$ sudo fdisk -l /dev/sata2 Disk /dev/sata2: 30 GiB, 32212254720 bytes, 62914560 sectors Disk model: Virtual SATA Hard Drive Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xdaa5db65 Device Boot Start End Sectors Size Id Type /dev/sata2p1 8192 16785407 16777216 8G fd Linux raid autodetect /dev/sata2p2 16785408 20979711 4194304 2G fd Linux raid autodetect /dev/sata2p3 21241856 62709759 41467904 19.8G fd Linux raid autodetect garyp@dsm2:/dev$ sudo fdisk -l /dev/sata3 Disk /dev/sata3: 500 GiB, 536870912000 bytes, 1048576000 sectors Disk model: Virtual SATA Hard Drive Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xae6ebcf7 Device Boot Start End Sectors Size Id Type /dev/sata3p1 8192 16785407 16777216 8G fd Linux raid autodetect /dev/sata3p2 16785408 20979711 4194304 2G fd Linux raid autodetect /dev/sata3p3 21241856 1048371199 1027129344 489.8G fd Linux raid autodetect We can now see that each disk has a uniform space allocation! There's an 8GB partition for the OS, a 2GB partition for devtmpfs, and the rest exposed to DSM for use. I think the additional "missing" space must be filesystem overhead. As I'm using btrfs here, that's understandable: https://btrfs.wiki.kernel.org/index.php/FAQ#Why_is_there_so_much_space_overhead.3F It would appear that my efforts to create a fast OS drive on the SSD were in vain, as the OS data is mirrored across disks in the system. As I understand RAID, this would mean reads will benefit from the fastest disk, but writes will be constrained by the slower SATA HDD. So, in summary: - you'll always lose at least 10GB of whatever disk you put in your Xpenology system - you can lose all but one of the disks and the system will still boot - when mixing different disk types, OS (not user data) read speeds are dictated by the fastest device - OS write speeds are dictated by the slowest device - your data is not protected in the same way unless you choose to create redundancy in your storage pools! I hopeful this little exploration proves useful to some of you.... Edited September 27, 2022 by WiteWulf Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.