merve04

SHR-2 Migration

Recommended Posts

Posted (edited)

Has anyone ever done this? I decided to convert my volume from SHR-1 to SHR-2 as I'm planning to add more HDD's to the volume and I feel after a while the more hdd's in the array, the greater the risk of failure. its been about 4 days now, storage manager reports 13.4%, mdstat show 26.7% on the first array, but the speed has been slow.. avg in the 2000's and 3000's KBps.

Im using a LSI 9211-8i in it mode and some drives direct to mobo. Mobo is gigabyte B365m-ds3h. Loader 1.04 with latest dsm.

 

There was a time i did a repair while still in shr-1, i was hitting well over 100000KBps sometimes hitting 200000, is changing the raid type just that difficult the compute?

Edited by merve04

Share this post


Link to post
Share on other sites

I guess I’m the first risk taker, 152hours, dsm reporting 21.5%, Mdstat shows 43% on first array of 2. 

Share this post


Link to post
Share on other sites
Posted (edited)

It's a lot of I/O and multiple stages as I recall.  You can speed it up with MDADM tuning, or in DSM 6.2.x Storage Manager | Storage Pool | Configurations | Custom and change max and min to something silly like 2500MB (25000000).  Put it back to Lower Impact when done.

Edited by flyride

Share this post


Link to post
Share on other sites

I will ask this, maybe some knows.. so there is X amount of data which had 1 parity, now changing to having 2 parity.. that's all great as its reading all the files and figuring stuff about, but due to the nature of how long this takes, I'm constantly adding more and more data in the GB to TB range while this is processing.. is that data written as double parity as I'm adding these new files? or is it caching somewhere to be processed once shr2 migration is completed?

Share this post


Link to post
Share on other sites

Just out of curiosity:

How many drives (and what sizes of disk's and  volumes) did you have in total with your SHR-1 ?

How many drives (and what sizes) did you add before starting the conversion to SHR-2?

Have you tried not adding extra data to your Volume(s) during the conversion, to see what happens to the speed?

 

Share this post


Link to post
Share on other sites
Posted (edited)
18 hours ago, merve04 said:

I will ask this, maybe some knows.. so there is X amount of data which had 1 parity, now changing to having 2 parity.. that's all great as its reading all the files and figuring stuff about, but due to the nature of how long this takes, I'm constantly adding more and more data in the GB to TB range while this is processing.. is that data written as double parity as I'm adding these new files? or is it caching somewhere to be processed once shr2 migration is completed?

 

This article by the author of md might help answer your questions. 

http://neil.brown.name/blog/20090817000931

 

This excerpt should help explain why your conversion speed is so slow:

Quote

For a reshape that does not change the number of devices, such as changing chunksize or layout, every write will be over-writing the old layout of that same data so after a crash there will definitely be a range of blocks that we cannot know whether they are in the old layout or the new layout or a bit of both. So we need to always have a backup of the range of blocks that are currently being reshaped.

 

This is the most complex part of the new functionality in mdadm 3.1 [snip]. mdadm monitors the reshape, setting an upper bound of how far it can progress at any time and making sure the area that it is allow to rearrange has writes disabled and has been backed-up.

 

This means that all the data is copied twice, once to the backup and once to the new layout on the array. This clearly means that such a reshape will go very slowly. But that is the price we have to pay for safety. It is like insurance. You might hate having to pay it, but you would hate it much more if you didn't and found that you needed it.

 

The TL;DR version: If you convert from RAID5 to RAID6 while adding a disk, conversion takes one pass.  An existing array set takes two passes.

Edited by flyride

Share this post


Link to post
Share on other sites
Posted (edited)
13 hours ago, bearcat said:

Just out of curiosity:

How many drives (and what sizes of disk's and  volumes) did you have in total with your SHR-1 ?

How many drives (and what sizes) did you add before starting the conversion to SHR-2?

Have you tried not adding extra data to your Volume(s) during the conversion, to see what happens to the speed?

 

When I was still SHR-1, i had 3x 8TB and 4x 4TB, 1 volume just over 29TB

I added a new 8TB to start the processed to SHR-2, once completed will be adding another 4TB and 8TB.

I've added plenty of data to my volume, maybe +500GB since I started the process of changing to SHR-2, my question was, what happens to the new data added to the volume while a migration is happening? Is the new data being written to the volume including double parity?

Forgot to mention, heck ya when I add huge batches of data, it kills the speed of the resync, just checked, downloading a bunch of movies at 30-40MB\s and my resync is down in the 500kbps range, usually if traffic is light its in the high 2000 to low 3000kbps

Edited by merve04
  • Like 1

Share this post


Link to post
Share on other sites

OK, I see :-)

Well, as explained above by flyride, there will be "some" extra reading/writing going on, 

might want to put those movies on hold, untill the conversion is done, to speed things up 🗓️

Share this post


Link to post
Share on other sites

Well 393 hours, first array is complete, now the 2nd and last one is processing, estimated 15-17 days probably before its completed.

Share this post


Link to post
Share on other sites

lol 864 hours 71% , i am shiting my pants i just start a clean 85TB SHR-2, so i guest it will take a year to finish lol

Share this post


Link to post
Share on other sites

A clean install only requires a single write of the array, so it will be the same as a RAID5 or RAID1 using the same size member devices.

Share this post


Link to post
Share on other sites

i got a message The consistency check of storage pool 1 (SHR) on Server has ended , but then the Parity consistency restarted and start all over again, i have no parity check scheduled setup

Share this post


Link to post
Share on other sites

It finally finished at about 1068 hours. It was an extremely slow process avg 1200-1500kbps. 
im now expanding my volume with couple hard drives, first one in and at roughly 25% after 8 hours. 

Edited by merve04
  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)

its not just wd, seagate does that too, they tried to hide the recording type and when you looked closer they invented n new name (tgmr) for it and you had to dig much deep to read that its just smr with some cmr as buffer, afair it was seagate barracuda at 1st (but these are desktop drives by the spec not nas drives) BUT they also sell Iron Wolf drives (nas) with "tgmr" so they are in the same boat as wd and there are a lot nas users in the last two years with such slow drives(?)

https://www.seagate.com/www-content/product-content/ironwolf/en-us/docs/100807039b.pdf

 

edit:

seagate's TGMR stands for "Tunneling Giant Magneto-Resistive" and thechnology for the read/write of the head (used in the hdd for each plate)

it seems it can be used for PMR and SMR so it depends on the hdd modell and how its pre formated and its firmware (PMR only or SMR with PMR zones as cache)

the key word for seagate to look for might be "MTC Technology" or "Multi-Tier Caching Technology"

on the barracuda webpage this tech is explicitly named for 2.5" drives (higher density for recording then 3.5") and "normal" barracuda drives

https://www.seagate.com/internal-hard-drives/hdd/barracuda/

(lower section with 5 types of barracuda compared)

 

"Multi-Tier Caching Technology" itself includes SMR as the lowest tech/tier as shown here

https://www.seagate.com/files/www-content/product-content/barracuda-fam/barracuda-new/files/multi-tier-caching-technology-white-paper-2017.pdf

 

the product manual for the ST4000DM004 does not mention it but it IS a SMR drive so its hard to guess what drives uses SMR

for a 4TB drive it might be 2 plates / 4 heads for SMR and 3 plates / 6 heads for PMR drives

barracuda pro drives might not use smr but as there is no indication of how to determine it can be changed at any time by seagate

the only safe way to determine seem to flood all "caches" by hammering the drive with ~20% of its capacity and have a eagle eye on the data rate, if it plummets significantly below 100MB/s for some time or stops responding for some time (flushing the pmr cache to smr) it must me a smr drive

 

seagate actually stated that there is no SMR used in iron wolf drives

 

its also a good idea to look into the official compatibility chart for DSM from synology (example 918+ for 4TB seagate drives)

https://www.synology.com/en-global/compatibility?search_by=products&model=DS918%2B&category=hdds_no_ssd_trim&filter_brand=Seagate&filter_size=4TB&p=1

 

no ST4000DM004 (2plates/4heads) there but ST4000DM005 (3plates/6heads) is ok

also iron wolf ST4000VN008 (3plates/6heads) is there

 

Edited by IG-88
  • Sad 1

Share this post


Link to post
Share on other sites
On 4/15/2020 at 3:19 PM, bearcat said:

@merve04 I might have missed it, but what type of drives are you using?

I came across this article today, and it made me think twice...

They are all Segate Barracuda's, a mix of 4 and 8 TB drives.

Share this post


Link to post
Share on other sites
On 4/17/2020 at 4:57 AM, merve04 said:

They are all Segate Barracuda's, a mix of 4 and 8 TB drives.

could you list the exact models you have?

(see my edited post above about smr/pmr disks from seagate)

Share this post


Link to post
Share on other sites

https://blocksandfiles.com/2020/04/15/seagate-2-4-and-8tb-barracuda-and-desktop-hdd-smr/

SMR:

Barracuda 4TB – 5,400rpm – SATA 6gig – ST4000DM004

Barracuda 8TB – 5,400rpm – SATA 6gig – ST8000DM004

 

so that#s the reason for your extremely poor performace

you are using SMR disks (all the 8TB disks and one 4TB disk)

  • Sad 1

Share this post


Link to post
Share on other sites

Well considering I had never done it, therefore have nothing to compare against it. When shr2 conversion was done, I expanded the volume twice with a 4 and 8 TB drives. It was a little bit slower than when I was on SHR1 but not a huge deal. Read write speeds on normal operations still maxes out based on saturation of my gigabit connection. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.