Jump to content
XPEnology Community

How DSM Uses Disks and Volumes, Trying To Optimise Storage Speed


Recommended Posts

I'm trying to figure out how to make the best use of a relatively small NVMe SSD and larger spinning HDD in the ESXi host I run one of my xpenology systems on. After doing some digging about I think I've got it figured and wanted to share in the hopes it may be useful.

 

All of the following assumes your are not using any hardware RAID on your host system, and are passing the physical or virtual disks straight through to DSM

 

DSM stores a copy of the system files on every HDD (physical or virtual) that the system can see, to ensure there's always a working version of the OS (if not all the user data) if there should be a HDD failure.

 

On the system I've been testing with I have a relatively small (30GB) virtual disk on the SSD that was the only disk present when I built the initial DSM install. Storage Manager reports only 18.2GB of usable space in the volume/storage pool created with it, so there's obviously a relatively large amount of space reserved for system files on there.

 

I subsequently added 100GB and 500GB virtual disks (both on a 1TB SATA HDD in the ESXi host). Storage Manager presents these as 85.4GB and 469.4GB volumes respectively.

 

It would appear that DSM reserves a variable amount of space, but not one that's consistent as a percentage of the disk size. We've got, roughly:
11.8GB of 30GB or 39%

14.4GB of 100GB or 14.4%

30.6GB of 500GB or 6%

...reserved on each disk.

 

I install all my DSM Apps, and keep docker containers and data, on the SSD volume to ensure they're as fast as can be, but how does the system know/choose which copy of the OS to boot from and use from the many disks in most systems?

 

Looking at the output of 'cat /proc/mdstat':
 

dsm2:~$ cat /proc/mdstat
Personalities : [raid1] [linear] 
md2 : active raid1 sata2p3[0]
      20732928 blocks super 1.2 [1/1] [U]
      
md4 : active raid1 sata3p3[0]
      513563648 blocks super 1.2 [1/1] [U]
      
md3 : active linear sata1p3[0]
      94133248 blocks super 1.2 64k rounding [1/1] [U]
      
md1 : active raid1 sata1p2[0] sata3p2[2] sata2p2[1]
      2097088 blocks [16/3] [UUU_____________]
      
md0 : active raid1 sata1p1[0] sata3p1[2] sata2p1[1]
      8388544 blocks [16/3] [UUU_____________]

 

I can see four RAID1 (mirrored) sets (md0, md1, md2 and md4), where md0 and md1 are on all three disks, and md4 and md5 are on single disk.
There's also a linear/JBOD set on a single disk, md3.

 

DSM uses /dev/mapper to map these block devices onto user visible volumes. We can see the this in the output of 'sudo dmsetup deps -o devname':

 

vg1-syno_vg_reserved_area: 1 dependencies       : (md2)
vg2-syno_vg_reserved_area: 1 dependencies       : (md3)
cachedev_2:                1 dependencies      	: (vg1-volume_1)
vg3-volume_3:              1 dependencies  		: (md4)
cachedev_1:                1 dependencies     	: (vg2-volume_2)
cachedev_0:                1 dependencies       : (vg3-volume_3)
vg2-volume_2:              1 dependencies       : (md3)
vg3-syno_vg_reserved_area: 1 dependencies       : (md4)
vg1-volume_1:              1 dependencies       : (md2)

 

...and of 'df -h':
 

Filesystem              Size  Used Avail Use% Mounted on
/dev/md0                7.9G  1.7G  6.1G  22% /
devtmpfs                2.0G     0  2.0G   0% /dev
tmpfs                   2.0G  124K  2.0G   1% /dev/shm
tmpfs                   2.0G   20M  1.9G   1% /run
tmpfs                   2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs                   2.0G  632K  2.0G   1% /tmp
/dev/mapper/cachedev_2   19G  5.0G   14G  28% /volume1
/dev/mapper/cachedev_0  470G  120G  350G  26% /volume3
/dev/mapper/cachedev_1   86G  3.3G   83G   4% /volume2

 

So it would appear that:

- md0 is a 7.9GB RAID1 volume containing the OS that is present on every disk in the system and mounted as /

- md1 is a 2GB RAID1 volume again spread across all disks. I think this used for devtmpfs

- md2 is a 19GB RAID1 volume with a single disk (on my SSD) visible to the user as /volume1

- md3 is a 86GB linear/JBOD volume with a single disk (on my SATA HDD) visible to the user as /volume2

- md4 is a 470GB RAID1 volume with a single disk (on my SATA HDD) visible to the user as /volume3

 

I still wasn't clear why the amount of data reserved on each member disk for the md0 RAID1 set seemed inconsistent, or what the logic behind it is. So I looked at the partitions for each disk:
 

sudo fdisk -l /dev/sata1
Password: 
Disk /dev/sata1: 100 GiB, 107374182400 bytes, 209715200 sectors
Disk model: Virtual SATA Hard Drive 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x51a58d33

Device       Boot    Start       End   Sectors  Size Id Type
/dev/sata1p1          8192  16785407  16777216    8G fd Linux raid autodetect
/dev/sata1p2      16785408  20979711   4194304    2G fd Linux raid autodetect
/dev/sata1p3      21241856 209510399 188268544 89.8G fd Linux raid autodetect

garyp@dsm2:/dev$ sudo fdisk -l /dev/sata2
Disk /dev/sata2: 30 GiB, 32212254720 bytes, 62914560 sectors
Disk model: Virtual SATA Hard Drive 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdaa5db65

Device       Boot    Start      End  Sectors  Size Id Type
/dev/sata2p1          8192 16785407 16777216    8G fd Linux raid autodetect
/dev/sata2p2      16785408 20979711  4194304    2G fd Linux raid autodetect
/dev/sata2p3      21241856 62709759 41467904 19.8G fd Linux raid autodetect

garyp@dsm2:/dev$ sudo fdisk -l /dev/sata3
Disk /dev/sata3: 500 GiB, 536870912000 bytes, 1048576000 sectors
Disk model: Virtual SATA Hard Drive 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xae6ebcf7

Device       Boot    Start        End    Sectors   Size Id Type
/dev/sata3p1          8192   16785407   16777216     8G fd Linux raid autodetect
/dev/sata3p2      16785408   20979711    4194304     2G fd Linux raid autodetect
/dev/sata3p3      21241856 1048371199 1027129344 489.8G fd Linux raid autodetect

 

We can now see that each disk has a uniform space allocation! There's an 8GB partition for the OS, a 2GB partition for devtmpfs, and the rest exposed to DSM for use. I think the additional "missing" space must be filesystem overhead. As I'm using btrfs here, that's understandable:
https://btrfs.wiki.kernel.org/index.php/FAQ#Why_is_there_so_much_space_overhead.3F

 

It would appear that my efforts to create a fast OS drive on the SSD were in vain, as the OS data is mirrored across disks in the system. As I understand RAID, this would mean reads will benefit from the fastest disk, but writes will be constrained by the slower SATA HDD.

 

So, in summary:

- you'll always lose at least 10GB of whatever disk you put in your Xpenology system

- you can lose all but one of the disks and the system will still boot

- when mixing different disk types, OS (not user data) read speeds are dictated by the fastest device

- OS write speeds are dictated by the slowest device

- your data is not protected in the same way unless you choose to create redundancy in your storage pools!

 

I hopeful this little exploration proves useful to some of you....

Edited by WiteWulf
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...