Jump to content
XPEnology Community

Help save my 55TB SHR1! Or mount it via Ubuntu :(


Recommended Posts

27 minutes ago, flyride said:

Try this, but I don't really expect it to work:

 

# mount -t btrfs -o ro,nologreplay /dev/vg1/volume_1 /volume1

Nope, doesn't work. 

root@homelab:~# mount -t btrfs -o ro,nologreplay /dev/vg1/volume_1 /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@homelab:~# dmesg | tail
[16079.284996] init: dhcp-client (eth4) main process (16851) killed by TERM signal
[16079.439991] init: nmbd main process (17405) killed by TERM signal
[16083.428550] alx 0000:05:00.0 eth4: NIC Up: 100 Mbps Full
[16084.471931] iSCSI:iscsi_target.c:520:iscsit_add_np CORE[0] - Added Network Portal: 192.168.0.83:3260 on iSCSI/TCP
[16084.472048] iSCSI:iscsi_target.c:520:iscsit_add_np CORE[0] - Added Network Portal: [fe80::d250:99ff:fe26:36a8]:3260 on iSCSI/TCP
[16084.498404] init: dhcp-client (eth4) main process (17245) killed by TERM signal
[27893.370010] usb 3-13: usbfs: USBDEVFS_CONTROL failed cmd blazer_usb rqt 33 rq 9 len 8 ret -110
[33089.867131] bio: create slab <bio-2> at 2
[33485.021943] hfsplus: unable to parse mount options
[33485.026980] UDF-fs: bad mount option "norecovery" or missing value
root@homelab:~#

 

Edited by C-Fu
Link to comment
Share on other sites

Ok, nothing new got added to dmesg.  Because /dev/md2 is first in the vg, this is definitively telling us that /dev/md2 is not good.

 

We are starting to run low on options. If for some reason we have the position of the 10TB drive wrong (slot 10), we can try the other spot (slot 11).  A more likely scenario is that the drive is damaged from corrupt and we are out of luck for this drive. But let's test the wrong slot possibility:

 

# vgchange -an

# mdadm --stop /dev/md2
# mdadm -Cf /dev/md2 -e1.2 -n13 -l5 --verbose --assume-clean /dev/sd[bcdefpqlmn]5 missing /dev/sdo5 /dev/sdk5 -u43699871:217306be:dc16f5e8:dcbe1b0d  
# vgchange -ay
# mount -t btrfs -o ro,nologreplay /dev/vg1/volume_1 /volume1

 

Post results of all commands.

Link to comment
Share on other sites

11 minutes ago, flyride said:

We are starting to run low on options. If for some reason we have the position of the 10TB drive wrong (slot 10), we can try the other spot (slot 11).  A more likely scenario is that the drive is damaged from corrupt and we are out of luck for this drive. But let's test the wrong slot possibility:

 

Would reinserting the "bad" 10TB and switch with the current 10TB, or add to the array help?

 

root@homelab:/# vgchange -an
  0 logical volume(s) in volume group "vg1" now active
root@homelab:/#
root@homelab:/# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
root@homelab:/# mdadm -Cf /dev/md2 -e1.2 -n13 -l5 --verbose --assume-clean /dev/sd[bcdefpqlmn]5 missing /dev/sdo5 /dev/sdk5 -u43699871:217306be:dc16f5e8:dcbe1b0d
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdb5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdc5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdd5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sde5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdf5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdl5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdm5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdn5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdp5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdq5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdo5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: /dev/sdk5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sat Jan 18 10:33:37 2020
mdadm: size set to 2925435456K
Continue creating array? yes
mdadm: array /dev/md2 started.
root@homelab:/# vgchange -ay
  2 logical volume(s) in volume group "vg1" now active
root@homelab:/# mount -t btrfs -o ro,nologreplay /dev/vg1/volume_1 /volume1
mount: wrong fs type, bad option, bad superblock on /dev/vg1/volume_1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@homelab:/# dmesg | tail
[38215.216573]  disk 5, o:1, dev:sdl5
[38215.216574]  disk 6, o:1, dev:sdm5
[38215.216574]  disk 7, o:1, dev:sdn5
[38215.216575]  disk 8, o:1, dev:sdp5
[38215.216575]  disk 9, o:1, dev:sdq5
[38215.216576]  disk 11, o:1, dev:sdo5
[38215.216577]  disk 12, o:1, dev:sdk5
[38215.242566] md2: detected capacity change from 0 to 35947750883328
[38215.245519]  md2: unknown partition table
[38221.515480] bio: create slab <bio-2> at 2

 

 

 

[38215.242566] md2: detected capacity change from 0 to 35947750883328

This is good, right?

Edited by C-Fu
Link to comment
Share on other sites

Unfortunately this is the same result.  We moved the 10TB from slot 10 to slot 11, and the array is still not readable.

 

24 minutes ago, C-Fu said:

Would reinserting the "bad" 10TB and switch with the current 10TB, or add to the array help?

 

We don't want to add to the array.  But replacing the 10TB drive is the next (and probably last) step.  This goes back to this conversation here:

"now you have determined that the "good" 10TB drive is failed and removed it from the system, which leaves the "bad" superblock-zeroed 10TB drive as the only path to recover /dev/md2.  It is important to be correct about that, as it is possible that the "bad" 10TB drive has corrupted data on it due to the failed resync"

 

So here's how this needs to go.

 

# vgchange -an

# mdadm --examine /dev/sd?5

# mdadm --stop /dev/md2

 

Post the result of the examine.

 

Then if at all possible, disconnect the "old" 10TB drive and replace it with the other "new" 10TB without powering down the system.  Please install the "new" drive to the same port.  If you have to power down and power up the system, please post that you did.

 

Assuming the drive is stable and not going up and down, then:

 

# mdadm --examine /dev/sd?5

# cat /proc/mdstat

 

Post the result of both commands.

Link to comment
Share on other sites

5 minutes ago, flyride said:

# vgchange -an

# mdadm --examine /dev/sd?5

# mdadm --stop /dev/md2

root@homelab:~# vgchange -an
  0 logical volume(s) in volume group "vg1" now active
  
  
root@homelab:~# mdadm --examine /dev/sd?5
/dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 3bd62073:a14acceb:9b6772fd:a2bd2c1b

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 5be7b741 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : c24f4ce9:f858ffe5:39e3b5f9:a2431b4f

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : fc7840be - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : de9e7cfd:9e0c3f85:0440908f:597a81b4

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : ab28d703 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : a81570df:dd73a097:3d4f4d84:58ec9c2d

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : d563645 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : e862fe90:222e6333:f541fb52:3830097f

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 7ac17462 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 065068bb:0ec18b72:af02796c:eb6cab77

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : f673f1e1 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 12
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 81ee1c1d:fe670a7d:4c291e84:281abb0f

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 125c0b1f - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 07f750e2:ebd026ac:f5c03557:7de8dd53

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 1de6e292 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 6
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b75ababb:4b982bb0:ebc1e473:a6716e38

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : fc9497c1 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 7
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : aefb3c70:11304796:c4b38dc3:aaaee205

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : b44fff5f - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 11
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b18d599a:f3efc211:24138276:d19eeb29

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 30e5a0c8 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 8
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 843a38a2:eabb6de6:1f5b53a1:a7d08136

    Update Time : Tue Jan 21 05:07:11 2020
       Checksum : 44d69365 - correct
         Events : 1

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 9
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
    
    
root@homelab:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2

 

5 minutes ago, flyride said:

Then if at all possible, disconnect the "old" 10TB drive and replace it with the other "new" 10TB without powering down the system.  Please install the "new" drive to the same port.  If you have to power down and power up the system, please post that you did.

 

So just to be clear, you want me to take out the power and sata cable from the current 10TB, and put them into the "bad" 10TB?

I don't have to power down btw.

Link to comment
Share on other sites

2 minutes ago, C-Fu said:
9 minutes ago, flyride said:

Then if at all possible, disconnect the "old" 10TB drive and replace it with the other "new" 10TB without powering down the system.  Please install the "new" drive to the same port.  If you have to power down and power up the system, please post that you did.

 

So just to be clear, you want me to take out the power and sata cable from the current 10TB, and put them into the "bad" 10TB?

I don't have to power down btw.

 

Yes, exactly that.

Link to comment
Share on other sites

@flyride I wanted to wait 10 mins, but my ssh session just got closed a few seconds after I connect the drive. Still gonna wait 10 mins.

 

fdisk -l shows the drive is at /dev/sdo.

Disk /dev/sdo: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1713E819-3B9A-4CE3-94E8-5A3DBF1D5983

 

# mdadm --examine /dev/sd?5
/dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 3bd62073:a14acceb:9b6772fd:a2bd2c1b

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 5be7c167 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : c24f4ce9:f858ffe5:39e3b5f9:a2431b4f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : fc784ae4 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : de9e7cfd:9e0c3f85:0440908f:597a81b4

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : ab28e129 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : a81570df:dd73a097:3d4f4d84:58ec9c2d

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : d56406b - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : e862fe90:222e6333:f541fb52:3830097f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 7ac17e88 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 065068bb:0ec18b72:af02796c:eb6cab77

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : f673fc07 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 12
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 81ee1c1d:fe670a7d:4c291e84:281abb0f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 125c1545 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 07f750e2:ebd026ac:f5c03557:7de8dd53

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 1de6ecb8 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 6
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b75ababb:4b982bb0:ebc1e473:a6716e38

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : fc94a1e7 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 7
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Sun Sep 22 21:55:03 2019
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 73610f83:fb3cf895:c004147e:b4de2bfe

    Update Time : Sat Jan 18 07:13:53 2020
       Checksum : d1dde2c9 - correct
         Events : 371014

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 11
   Array State : AAAAA.AAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b18d599a:f3efc211:24138276:d19eeb29

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 30e5aaee - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 8
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 843a38a2:eabb6de6:1f5b53a1:a7d08136

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 44d69d8b - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 9
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
root@homelab:~#
root@homelab:~# mdadm --examine /dev/sd?5
/dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 3bd62073:a14acceb:9b6772fd:a2bd2c1b

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 5be7c167 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : c24f4ce9:f858ffe5:39e3b5f9:a2431b4f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : fc784ae4 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : de9e7cfd:9e0c3f85:0440908f:597a81b4

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : ab28e129 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : a81570df:dd73a097:3d4f4d84:58ec9c2d

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : d56406b - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : e862fe90:222e6333:f541fb52:3830097f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 7ac17e88 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 065068bb:0ec18b72:af02796c:eb6cab77

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : f673fc07 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 12
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 81ee1c1d:fe670a7d:4c291e84:281abb0f

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 125c1545 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 07f750e2:ebd026ac:f5c03557:7de8dd53

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 1de6ecb8 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 6
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b75ababb:4b982bb0:ebc1e473:a6716e38

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : fc94a1e7 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 7
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdo5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Sun Sep 22 21:55:03 2019
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 73610f83:fb3cf895:c004147e:b4de2bfe

    Update Time : Sat Jan 18 07:13:53 2020
       Checksum : d1dde2c9 - correct
         Events : 371014

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 11
   Array State : AAAAA.AAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdp5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : b18d599a:f3efc211:24138276:d19eeb29

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 30e5aaee - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 8
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdq5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 43699871:217306be:dc16f5e8:dcbe1b0d
           Name : homelab:2  (local to host homelab)
  Creation Time : Tue Jan 21 05:07:10 2020
     Raid Level : raid5
   Raid Devices : 13

 Avail Dev Size : 5850870912 (2789.91 GiB 2995.65 GB)
     Array Size : 35105225472 (33478.95 GiB 35947.75 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=0 sectors
          State : clean
    Device UUID : 843a38a2:eabb6de6:1f5b53a1:a7d08136

    Update Time : Tue Jan 21 05:50:28 2020
       Checksum : 44d69d8b - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 9
   Array State : AAAAAAAAAA.AA ('A' == active, '.' == missing, 'R' == replacing)

 

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md4 : active raid5 sdl6[0] sdn6[2] sdm6[1]
      11720987648 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/3] [UUU__]

md5 : active raid1 super 1.2 [2/0] [__]

md1 : active raid1 sdb2[0] sdc2[1] sdd2[2] sde2[3] sdf2[4] sdk2[5] sdl2[6] sdm2[7] sdn2[8] sdp2[24](F) sdq2[11]
      2097088 blocks [24/10] [UUUUUUUUU__U____________]

md0 : active raid1 sdb1[1] sdc1[2] sdd1[3] sdf1[5]
      2490176 blocks [12/4] [_UUU_U______]

unused devices: <none>

 

Edited by C-Fu
Link to comment
Share on other sites

1 minute ago, flyride said:

# fdisk -l /dev/sd?

# fdisk -l /dev/sd?
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x696935dc

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1        2048 468857024 468854977 223.6G fd Linux raid autodetect
Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 43C8C355-AE0A-42DC-97CC-508B0FB4EF37

Device       Start        End    Sectors  Size Type
/dev/sdb1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdb2  4982528    9176831    4194304    2G Linux RAID
/dev/sdb5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0600DFFC-A576-4242-976A-3ACAE5284C4C

Device       Start        End    Sectors  Size Type
/dev/sdc1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdc2  4982528    9176831    4194304    2G Linux RAID
/dev/sdc5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 58B43CB1-1F03-41D3-A734-014F59DE34E8

Device       Start        End    Sectors  Size Type
/dev/sdd1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdd2  4982528    9176831    4194304    2G Linux RAID
/dev/sdd5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sde: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E5FD9CDA-FE14-4F95-B776-B176E7130DEA

Device       Start        End    Sectors  Size Type
/dev/sde1     2048    4982527    4980480  2.4G Linux RAID
/dev/sde2  4982528    9176831    4194304    2G Linux RAID
/dev/sde5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdf: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 48A13430-10A1-4050-BA78-723DB398CE87

Device       Start        End    Sectors  Size Type
/dev/sdf1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdf2  4982528    9176831    4194304    2G Linux RAID
/dev/sdf5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdk: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A3E39D34-4297-4BE9-B4FD-3A21EFC38071

Device       Start        End    Sectors  Size Type
/dev/sdk1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdk2  4982528    9176831    4194304    2G Linux RAID
/dev/sdk5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdl: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 849E02B2-2734-496B-AB52-A572DF8FE63F

Device          Start         End    Sectors  Size Type
/dev/sdl1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdl2     4982528     9176831    4194304    2G Linux RAID
/dev/sdl5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdl6  5860342336 11720838239 5860495904  2.7T Linux RAID
Disk /dev/sdm: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 423D33B4-90CE-4E34-9C40-6E06D1F50C0C

Device          Start         End    Sectors  Size Type
/dev/sdm1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdm2     4982528     9176831    4194304    2G Linux RAID
/dev/sdm5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdm6  5860342336 11720838239 5860495904  2.7T Linux RAID
Disk /dev/sdn: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 09CB7303-C2E7-46F8-ADA0-D4853F25CB00

Device          Start         End    Sectors  Size Type
/dev/sdn1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdn2     4982528     9176831    4194304    2G Linux RAID
/dev/sdn5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdn6  5860342336 11720838239 5860495904  2.7T Linux RAID
Disk /dev/sdo: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1713E819-3B9A-4CE3-94E8-5A3DBF1D5983

Device           Start         End    Sectors  Size Type
/dev/sdo1         2048     4982527    4980480  2.4G Linux RAID
/dev/sdo2      4982528     9176831    4194304    2G Linux RAID
/dev/sdo5      9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdo6   5860342336 11720838239 5860495904  2.7T Linux RAID
/dev/sdo7  11720854336 19532653311 7811798976  3.7T Linux RAID
Disk /dev/sdp: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1D5B8B09-8D4A-4729-B089-442620D3D507

Device       Start        End    Sectors  Size Type
/dev/sdp1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdp2  4982528    9176831    4194304    2G Linux RAID
/dev/sdp5  9453280 5860326239 5850872960  2.7T Linux RAID
Disk /dev/sdq: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 54D81C51-AB85-4DE2-AA16-263DF1C6BB8A

Device       Start        End    Sectors  Size Type
/dev/sdq1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdq2  4982528    9176831    4194304    2G Linux RAID
/dev/sdq5  9453280 5860326239 5850872960  2.7T Linux RAID

 

Link to comment
Share on other sites

Sorry, but please run the fdisk -l as individual commands.

 

# fdisk -l /dev/sdb

# fdisk -l /dev/sdc

# fdisk -l /dev/sdd

# fdisk -l /dev/sde

# fdisk -l /dev/sdf

# fdisk -l /dev/sdk

# fdisk -l /dev/sdl

# fdisk -l /dev/sdm

# fdisk -l /dev/sdn

# fdisk -l /dev/sdo

# fdisk -l /dev/sdp

# fdisk -l /dev/sdq

 

Also:

 

# fgrep "hotswap" /var/log/disk.log

# date

 

Edited by flyride
Link to comment
Share on other sites

# fdisk -l /dev/sdb
Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 43C8C355-AE0A-42DC-97CC-508B0FB4EF37

Device       Start        End    Sectors  Size Type
/dev/sdb1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdb2  4982528    9176831    4194304    2G Linux RAID
/dev/sdb5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdc
Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0600DFFC-A576-4242-976A-3ACAE5284C4C

Device       Start        End    Sectors  Size Type
/dev/sdc1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdc2  4982528    9176831    4194304    2G Linux RAID
/dev/sdc5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdd
Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 58B43CB1-1F03-41D3-A734-014F59DE34E8

Device       Start        End    Sectors  Size Type
/dev/sdd1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdd2  4982528    9176831    4194304    2G Linux RAID
/dev/sdd5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sde
Disk /dev/sde: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E5FD9CDA-FE14-4F95-B776-B176E7130DEA

Device       Start        End    Sectors  Size Type
/dev/sde1     2048    4982527    4980480  2.4G Linux RAID
/dev/sde2  4982528    9176831    4194304    2G Linux RAID
/dev/sde5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdf
Disk /dev/sdf: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 48A13430-10A1-4050-BA78-723DB398CE87

Device       Start        End    Sectors  Size Type
/dev/sdf1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdf2  4982528    9176831    4194304    2G Linux RAID
/dev/sdf5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdk
Disk /dev/sdk: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A3E39D34-4297-4BE9-B4FD-3A21EFC38071

Device       Start        End    Sectors  Size Type
/dev/sdk1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdk2  4982528    9176831    4194304    2G Linux RAID
/dev/sdk5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdl
Disk /dev/sdl: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 849E02B2-2734-496B-AB52-A572DF8FE63F

Device          Start         End    Sectors  Size Type
/dev/sdl1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdl2     4982528     9176831    4194304    2G Linux RAID
/dev/sdl5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdl6  5860342336 11720838239 5860495904  2.7T Linux RAID
# fdisk -l /dev/sdm
Disk /dev/sdm: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 423D33B4-90CE-4E34-9C40-6E06D1F50C0C

Device          Start         End    Sectors  Size Type
/dev/sdm1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdm2     4982528     9176831    4194304    2G Linux RAID
/dev/sdm5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdm6  5860342336 11720838239 5860495904  2.7T Linux RAID
# fdisk -l /dev/sdn
Disk /dev/sdn: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 09CB7303-C2E7-46F8-ADA0-D4853F25CB00

Device          Start         End    Sectors  Size Type
/dev/sdn1        2048     4982527    4980480  2.4G Linux RAID
/dev/sdn2     4982528     9176831    4194304    2G Linux RAID
/dev/sdn5     9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdn6  5860342336 11720838239 5860495904  2.7T Linux RAID
# fdisk -l /dev/sdo
Disk /dev/sdo: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1713E819-3B9A-4CE3-94E8-5A3DBF1D5983

Device           Start         End    Sectors  Size Type
/dev/sdo1         2048     4982527    4980480  2.4G Linux RAID
/dev/sdo2      4982528     9176831    4194304    2G Linux RAID
/dev/sdo5      9453280  5860326239 5850872960  2.7T Linux RAID
/dev/sdo6   5860342336 11720838239 5860495904  2.7T Linux RAID
/dev/sdo7  11720854336 19532653311 7811798976  3.7T Linux RAID
# fdisk -l /dev/sdp
Disk /dev/sdp: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1D5B8B09-8D4A-4729-B089-442620D3D507

Device       Start        End    Sectors  Size Type
/dev/sdp1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdp2  4982528    9176831    4194304    2G Linux RAID
/dev/sdp5  9453280 5860326239 5850872960  2.7T Linux RAID
# fdisk -l /dev/sdq
Disk /dev/sdq: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 54D81C51-AB85-4DE2-AA16-263DF1C6BB8A

Device       Start        End    Sectors  Size Type
/dev/sdq1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdq2  4982528    9176831    4194304    2G Linux RAID
/dev/sdq5  9453280 5860326239 5850872960  2.7T Linux RAID

 

 

 

Uh what's a table error?

# fgrep "hotswap" /var/log/disk.log
2020-01-18T10:21:23+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-18T10:21:23+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-18T10:28:46+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-20T04:16:45+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T04:16:48+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T04:24:00+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T04:24:02+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T14:18:35+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-21T05:58:00+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [remove] ====
2020-01-21T05:58:20+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====

 

4 hours ago, flyride said:

date

# date
Tue Jan 21 12:24:08 CST 2020

 

Link to comment
Share on other sites

I was referring to the array table I made not matching up to one of the mdstats.  This appears to suggest continuing hotplug operations (drives going up and down) at the moment you keyed the first array creation command, which might have made the first array test invalid.

 

image.thumb.png.4bf0aaf71bede21e4d95d4035c3f6b49.png

 

Continuing our current trajectory: First, repeat fgrep "hotplug" /var/log/disk.log and make sure there are no new entries in the log since the report above. If there are new entries, your system still isn't stable.  If it's the same as before, move forward with:

 

# mdadm -Cf /dev/md2 -e1.2 -n13 -l5 --verbose --assume-clean /dev/sd[bcdefpqlmn]5 missing /dev/sdo5 /dev/sdk5  -u43699871:217306be:dc16f5e8:dcbe1b0d

# cat /proc/mdstat

Link to comment
Share on other sites

10 hours ago, flyride said:

Continuing our current trajectory: First, repeat fgrep "hotplug" /var/log/disk.log and make sure there are no new entries in the log since the report above. If there are new entries, your system still isn't stable.  If it's the same as before, move forward with:

ok I did a fgrep hotswap and compare both with notepad++ and seems like there are a few additions:

 

2020-01-21T18:19:12+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-21T18:19:14+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-21T20:39:20+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-21T20:39:22+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====

I thought sda is the ssd?

# fdisk -l /dev/sda
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x696935dc

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1        2048 468857024 468854977 223.6G fd Linux raid autodetect

 

The fgrep hotplug is in the pastebin link, pasting the whole log here seems to break the board. https://pastebin.com/8fydgapn

 

Link to comment
Share on other sites

To compare, we would have to

 

fgrep "hotswap" /var/log/disk.log

 

not "hotplug"

 

But looking at your pastebin it appears that the only drive to hotplug out/in is sda, which doesn't affect us.  But why that happened is still concerning.

 

Please run it again (with hotswap) and make sure there are no array drives changing after 2020-01-21T05:58:20+08:00

Link to comment
Share on other sites

2 minutes ago, flyride said:

Please run it again (with hotswap) and make sure there are no array drives changing after 2020-01-21T05:58:20+08:00

 

I can confirm, no array drives changing since 9 hours ago. The one I just ran still has the same 4 "new" entries of add & remove [sda] ssd drive compared to Tuesday's post #113.

 

# fgrep "hotswap" /var/log/disk.log
2020-01-18T10:21:23+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-18T10:21:23+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-18T10:21:24+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-18T10:28:46+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-18T10:28:47+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-20T04:16:45+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T04:16:48+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T04:24:00+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T04:24:02+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T14:18:35+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdk] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdl] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdm] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdn] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdp] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdq] hotswap [add] ====
2020-01-20T18:31:33+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdb] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdc] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdd] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sde] hotswap [add] ====
2020-01-20T18:31:34+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdf] hotswap [add] ====
2020-01-21T05:58:00+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [remove] ====
2020-01-21T05:58:20+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sdo] hotswap [add] ====
2020-01-21T18:19:12+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-21T18:19:14+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====
2020-01-21T20:39:20+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [remove] ====
2020-01-21T20:39:22+08:00 homelab hotplugd: hotplugd.c:1451 ==== SATA disk [sda] hotswap [add] ====

 

Link to comment
Share on other sites

36 minutes ago, flyride said:

Ok then, lets complete the last task then, try to incorporate your second 10TB drive into the array:

 

Great! :D

 

root@homelab:~# mdadm -Cf /dev/md2 -e1.2 -n13 -l5 --verbose --assume-clean /dev/sd[bcdefpqlmn]5 missing /dev/sdo5 /dev/sdk5  -u43699871:217306be:dc16f5e8:dcbe1b0d
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdb5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdc5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdd5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sde5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdf5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdl5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdm5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdn5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdp5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdq5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: /dev/sdo5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Sun Sep 22 21:55:03 2019
mdadm: /dev/sdk5 appears to be part of a raid array:
       level=raid5 devices=13 ctime=Tue Jan 21 05:07:10 2020
mdadm: size set to 2925435456K
Continue creating array? yes             
mdadm: array /dev/md2 started.
root@homelab:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1] 
md2 : active raid5 sdk5[12] sdo5[11] sdq5[9] sdp5[8] sdn5[7] sdm5[6] sdl5[5] sdf5[4] sde5[3] sdd5[2] sdc5[1] sdb5[0]
      35105225472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [13/12] [UUUUUUUUUU_UU]
      
md4 : active raid5 sdl6[0] sdn6[2] sdm6[1]
      11720987648 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/3] [UUU__]
      
md5 : active raid1 super 1.2 [2/0] [__]
      
md1 : active raid1 sdb2[0] sdc2[1] sdd2[2] sde2[3] sdf2[4] sdk2[5] sdl2[6] sdm2[7] sdn2[8] sdp2[24](F) sdq2[11]
      2097088 blocks [24/10] [UUUUUUUUU__U____________]
      
md0 : active raid1 sdb1[1] sdc1[2] sdd1[3] sdf1[5]
      2490176 blocks [12/4] [_UUU_U______]
      

 

@flyride Should I unplug the sata and power cable for sda? I just realised this 1600W psu has no more sata power plug, and I ran out of sata power splitter plug. Since we probably are going to plug in the other 10TB (I think). It's 1.34am right now, and I can only go to a pc shop to get a splitter tomorrow. Please advice.

Edited by C-Fu
Link to comment
Share on other sites

1 minute ago, flyride said:

I don't have a plan to plug the disconnected 10TB at this time.

 

Cool, no worries.

 

# mdadm --detail /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Tue Sep 24 19:36:08 2019
     Raid Level : raid1
     Array Size : 3905898432 (3724.96 GiB 3999.64 GB)
  Used Dev Size : 3905898432 (3724.96 GiB 3999.64 GB)
   Raid Devices : 2
  Total Devices : 0
    Persistence : Superblock is persistent

    Update Time : Tue Jan 21 05:58:00 2020
          State : clean, FAILED 
 Active Devices : 0
 Failed Devices : 0
  Spare Devices : 0

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
# mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
  Creation Time : Sun Sep 22 21:55:04 2019
     Raid Level : raid5
     Array Size : 11720987648 (11178.00 GiB 12002.29 GB)
  Used Dev Size : 2930246912 (2794.50 GiB 3000.57 GB)
   Raid Devices : 5
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Tue Jan 21 05:58:00 2020
          State : clean, FAILED 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : homelab:4  (local to host homelab)
           UUID : 648fc239:67ee3f00:fa9d25fe:ef2f8cb0
         Events : 7217

    Number   Major   Minor   RaidDevice State
       0       8      182        0      active sync   /dev/sdl6
       1       8      198        1      active sync   /dev/sdm6
       2       8      214        2      active sync   /dev/sdn6
       -       0        0        3      removed
       -       0        0        4      removed

 

Link to comment
Share on other sites

# mdadm --stop /dev/md4

# mdadm --stop /dev/md5

# mdadm --assemble /dev/md4 -u648fc239:67ee3f00:fa9d25fe:ef2f8cb0

# mdadm --assemble /dev/md5 -uae55eeff:e6a5cc66:2609f5e0:2e2ef747 

  

The first one will probably error out complaining that /dev/sdo6 is not current.  We'll be able to fix that.

Link to comment
Share on other sites

2 minutes ago, flyride said:

The first one will probably error out complaining that /dev/sdo6 is not current.  We'll be able to fix that.

Two errors:

 

root@homelab:~# mdadm --stop /dev/md4
mdadm: stopped /dev/md4
root@homelab:~# mdadm --stop /dev/md5
mdadm: stopped /dev/md5
root@homelab:~# mdadm --assemble /dev/md4 -u648fc239:67ee3f00:fa9d25fe:ef2f8cb0
mdadm: /dev/md4 assembled from 3 drives - not enough to start the array.
root@homelab:~# mdadm --assemble /dev/md5 -uae55eeff:e6a5cc66:2609f5e0:2e2ef747
mdadm: /dev/md5 assembled from 0 drives and 1 rebuilding - not enough to start the array.

What do you mean by not current btw?

Link to comment
Share on other sites

There was another powercut. Dammit!

 

Now I can't even do cat /proc/mdstat.

 

I'll wait a while just to see if it will work or not. Sorry! So frustrating :evil:

I connected the pc to an UPS btw. So not really sure why powercuts can still affect the whole system.

 

edit: ok cat /proc/mdstat works but slow. Should I continue? with mdadm assemble md4 md5? or maybe take out sda SSD and see if it fixes the slow mdstat? Please advice.

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1] 
md4 : active raid5 sdl6[0] sdq6[2] sdm6[1]
      11720987648 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/3] [UUU__]
      
md2 : active raid5 sdb5[0] sdk5[12] sdn5[11] sdp5[9] sdo5[8] sdq5[7] sdm5[6] sdl5[5] sdf5[4] sde5[3] sdd5[2] sdc5[1]
      35105225472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [13/12] [UUUUUUUUUU_UU]
      
md5 : active raid1 sdn7[3]
      3905898432 blocks super 1.2 [2/0] [__]
      
md1 : active raid1 sdb2[0] sdc2[1] sdd2[2] sde2[3] sdf2[4] sdk2[5] sdl2[6] sdm2[7] sdn2[8] sdo2[9] sdp2[10] sdq2[11]
      2097088 blocks [24/12] [UUUUUUUUUUUU____________]
      
md0 : active raid1 sdb1[1] sdc1[2] sdd1[3] sdf1[5]
      2490176 blocks [12/4] [_UUU_U______]
      
unused devices: <none>

 

Edited by C-Fu
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...