1812+ - Expanded SHR-1 system errors - caused system reboot and scan

[ccfc1986]

Hi All!

I have a 1812+ 8-bay system. I recently added another 20TB drive... so my makeup is the following:
3 20TB HDD
5 8TB HDD

After rebuilding the array (SHR-1) for 5 days - i received a notification system errors occurred. The process did finish and it said 'Healthy' but had a 'Warning' that there were some system errors. The recommendation was to reboot and do a system scan. I allowed it to reboot and do this scan.

When this scan was occurring after reboot, it was doing a e2fsck scan. This took about 5/6 hours. And has since finished - based on checking 'top'.

Another process is now running and has been for about 90 minutes. This process is:

/sbin/debugfs -q /.remap.vg1000.lv /dev/vg1000/lv

Should i let this process finish? I assume the answer is yes! Please confirm?

I have done some other system checks/commands, and the outputs are below. It LOOKS to me that the volume is still generally OK... but i do not see if when i do a 'df -h'.

Here is the output:

lvm vgscan

Reading all physical volumes.  This may take a while...
Found volume group "vg1000" using metadata type lvm2

 

lvm lvscan

ACTIVE            '/dev/vg1000/lv' [70.92 TiB] inherit

pvs

  PV         VG     Fmt  Attr PSize  PFree
  /dev/md2   vg1000 lvm2 a--  19.07t    0
  /dev/md3   vg1000 lvm2 a--  19.10t    0
  /dev/md4   vg1000 lvm2 a--  10.92t    0
  /dev/md5   vg1000 lvm2 a--  21.83t    0

 

vgs

  VG     #PV #LV #SN Attr   VSize  VFree
  vg1000   4   1   0 wz--n- 70.92t    0

 

lvs
  LV   VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv   vg1000 -wi-ao---- 70.92t

 

pvdisplay
--- Physical volume ---
  PV Name               /dev/md2
  VG Name               vg1000
  PV Size               19.07 TiB / not usable 2.94 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              4999687
  Free PE               0
  Allocated PE          4999687
PV UUID               39SMYw-LvD9-csns-Ibx0-Xk6L-NcRN-gH5z57

     --- Physical volume ---
  PV Name               /dev/md3
  VG Name               vg1000
  PV Size               19.10 TiB / not usable 1.31 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              5007581
  Free PE               0
  Allocated PE          5007581
PV UUID               zigxlk-rzBz-w71G-XG0c-T5NK-Kdty-2asJUO   

  --- Physical volume ---
  PV Name               /dev/md4
  VG Name               vg1000
  PV Size               10.92 TiB / not usable 2.69 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              2861564
  Free PE               0
  Allocated PE          2861564
PV UUID               gaMsmu-3noo-SQlO-A2Ol-9irc-OeQP-PAsC7H   

  --- Physical volume ---
  PV Name               /dev/md5
  VG Name               vg1000
  PV Size               21.83 TiB / not usable 3.94 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              5721574
  Free PE               0
  Allocated PE          5721574
  PV UUID               cMM5qC-lxe1-ET5H-uUjR-1X0R-9oqd-2mf5pX

vgdisplay
--- Volume group ---
VG Name               vg1000  System ID             
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  57
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               70.92 TiB
  PE Size               4.00 MiB
  Total PE              18590406
  Alloc PE / Size       18590406 / 70.92 TiB
  Free  PE / Size       0 / 0   
VG UUID               R3FyH1-QjrW-UoEu-xDM6-Ihzq-AuQ0-45PC2M

 

lvdisplay
--- Logical volume ---
  LV Path                /dev/vg1000/lv
  LV Name                lv
  VG Name                vg1000
  LV UUID                7dAjuW-c4q8-223V-L2QF-DmFw-dfJv-UR5ZND
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                70.92 TiB
  Current LE             18590406
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
Block device           253:0

 

cat /etc/fstab
none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/vg1000/lv /volume1 ext4 usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,synoacl,relatime 0 0

 

I just would like someone to confirm with me that the debugfs is an EXPECTED process of the reboot/scan following the completion of the e2fsck scan? If so, I will be PATIENT and wait for it to finish and i assume reboot the NAS and allow me to login into the web GUI... if it is NOT expected to run, any recommended next steps?

 

Edited July 13, 2024 by ccfc1986
ease of reading - bolded and underlined commands

[ccfc1986]

Update:

After some patience - the server DID reboot itself. However, it still showed the storage volume the old size (60TB) in the disk area, but did show it sees 70TB in main page. When i went to the drives page it showed a warning, that i needed to complete the SHR process, so i clicked through the pop up, and it started 'Checking consistency' again. Then it immediately shut itself down. Completely. No lights no nothing after about 1 minute. I just restarted it (clicking the power button) - and its loading back up again. 

Very weird behavior. Going to try doing the same step again - but if anyone has any ideas - how to get it to see the full new size (as it seems to have correctly rebuilt the array with the new disk).

Thanks all!

[ccfc1986]

Update 2: 

Drive is currently doing a "Verifying drives in the background - check disk parity consistency 1.01%)".

Its running at about 200k/second. The time to finish is roughly 24 hours. I hope this connects the dots of the process and everything is working roughly this time tomorrow! Will keep you all posted.

In the meantime - if anyone has any suggestions on speeding the process up, or validating things are going smoothly (using cat /proc/mdstat currently to see what its doing).
 

ash-4.3# cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]

md3 : active raid5 sdh6[15] sdc6[10] sdd6[9] sdb6[12] sda6[11] sde6[13] sdf6[8] sdg6[14]

      20511053696 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

      

md4 : active raid5 sdc7[7] sdf7[10] sdh7[6] sdg7[5] sde7[4] sdd7[3] sdb7[9] sda7[8]

      11720969472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

      [>....................]  reshape =  0.8% (16664348/1953494912) finish=1476.6min speed=21860K/sec

      

md5 : active raid5 sdc8[0] sdb8[2] sda8[1]

      23435571712 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

      

md2 : active raid5 sda5[13] sdh5[8] sdg5[9] sdf5[10] sde5[11] sdd5[14] sdc5[15] sdb5[12]

      20478721536 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

      

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3] sde2[4] sdf2[5] sdg2[6] sdh2[7]

      2097088 blocks [8/8] [UUUUUUUU]

      

md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[4] sdf1[5] sdg1[6] sdh1[7]

      2490176 blocks [8/8] [UUUUUUUU]

[ccfc1986]

It just finished and now says "Healthy" but did not expand the volume. Any ideas?

[ccfc1986]

Was able to 'expand'. Now says 'Available Capacity' is 0 bytes under 'Storage Pool'. But i have over 12TB free when in the 'Volume' section. Any ideas why? I want to put in another 20TB disk - but don't want to do this until these numbers make complete sense etc.

 

Thanks all!