LSI SAS2008 - Make disk order follow enclosure places


Recommended Posts

Hello,

 

After searching a lot, and testing for hours, i've found something that doesn't seems to in this forum (if it is; sorry i haven't seen it)

 

I've found 2 PERC H310 inside the bin, and saw the fabulous REDPILL LOADER, so i decide to give both a try.

Got some usb stick and free disks, i first follow the classic H310 to 9211-8i IT firmware flash (google for it, it's pretty easy to do)

 

Tryed the redpill loader with my new HBA SAS2008 : worked !!!

BUT, the drives never show up in the correct order. Worse : the order changed uppon reboot.

 

I found this great litterature to help me understand :

S11209_v1.0_SAS-2EnclosureMapping.pdf

 

There is a solution to switch those nice HBA from SAS address, to enclosure/slot mapping mode.

 

Please note that my enclosure is a supermicro with a BPN-SAS-825TQ backplane, with SFF8084/sata breakout cables pluged between the backplane and the LSI SAS2008 card

 

2 counterbacks :

- The order of the disks is reversed. Cable #1 for port A goes to disk 7, and cable #4 for port B goes to disk 0

- If there is a missing disk at boot, the card will bypass it, and it will allows report the disks in continuous order. ie. if disk 3 of 8 is missing at boot, then you will see that disk 8 is missing in DSM.

 

Here is what i have done for this result :

 

- Create my usb stick with redpill, and DSM

- Plugged only the first disk in the enclosure, at the first position (important !) and then install DSM.

 

If your NVRAM is not blank from a fresh flash, make a freedos USB stick with sas2flsh and launch sas2flsh -o -e 3 (this will erase the info from previous drives, and more config, becarefull)

 

- Launch DSM7, got to ssh and :

wget https://github.com/thomaslovell/LSIUtil/raw/master/Binaries/LSIutil_1.71_binaries/Linux/lsiutil.x86_64

chmod +x and sudo run ./lsiutil.x86_64

go into option 9, pagetype 1, pagenumber 8

modify offset 0x0c to 02 (bit 1 = Enclosure/Slot mapping mode)

 

reboot and enjoy.

 

Note that you can see the disk order with the LSIUTIL tool, using option 16

 

 

Fell free to make any comments or ask any questions

 

LSI SAS2008 correct order.png

Edited by exodius
added option 16 info
  • Like 1
  • Thanks 2
Link to post
Share on other sites

Should this work with just SAS to SATA cables too? (no backplane?)

Then to label the cables accordingly (ie: 0>1:5>2:2>3:3>4:4>5:6>6:1>7:7>8)

 

I wish I could find an hba card <- jealous

 

Great Job! Thanks given

Link to post
Share on other sites

My backplane is not a real SAS backplane, my SAS2008 card is plugged in with breakout SATA cables on the backplane.

 

I just use those : Sas-Sata-Cable-Mini-SAS-SFF-8087-to-4-SA

So i think yes that should work without a backplane too

  • Thanks 1
Link to post
Share on other sites
Citation

label the cables accordingly (ie: 0>1:5>2:2>3:3>4:4>5:6>6:1>7:7>8)

 

You don't have to

 

The screenshot can be misleading, as you don't have see previous screens (those i got for hours)

 

Before the tweak, the sas card is following the PhyNum, and address the first responding disk at init time (which change from time to time at boot)

 

After the tweak, the card keep the Handles in the alphanum order, so i just gess it make it's magic but swapping the Phy numbers. BUT i never changed the cables from where there are.

 

Link to post
Share on other sites
16 minutes ago, exodius said:

 

You don't have to

 

The screenshot can be misleading, as you don't have see previous screens (those i got for hours)

 

Before the tweak, the sas card is following the PhyNum, and address the first responding disk at init time (which change from time to time at boot)

 

After the tweak, the card keep the Handles in the alphanum order, so i just gess it make it's magic but swapping the Phy numbers. BUT i never changed the cables from where there are.

 

So the cable in port A is always drive 1-4 and the cable in port B is 5-8 (after you did lsiutil) correctly?

and port A and B got reversed after it mode flash correct? I read after it mode flash port A is on top (so port A and B, top to bottom) otherwise its the otherway around. ports B on top and A bottom

I dont have an hba card yet, I am working on it to build a giant nas to backup everything in my homelab... so 12 to 16 disks maybe.

Edited by phone guy
Link to post
Share on other sites
Posted (edited)

My card has clear label A and B right next to the port so...

 

When i say the order of the disks is reversed, this is like that :

 

In a perfect world, you plug a breakout cable in port A, then disk 1 on sata conn #1, disk 2 on sata conn #2, disk 3 on sata conn #3, disk 4 on sata conn #4

you plug a breakout cable in port B, then disk 5 on sata conn #1, disk 6 on sata conn #2, disk 7 on sata conn #3, disk 8 on sata conn #4

 

In real life, the card find:

 

disk 1 on port B cable #4

disk 2 on port B cable #3

disk 3 on port B cable #2

disk 4 on port B cable #1

disk 5 on port A cable #4

disk 6 on port A cable #3

disk 7 on port A cable #2

disk 8 on port A cable #1

Edited by exodius
Link to post
Share on other sites

Yes I edited to reflect that, I had seen so many with ports labels 0 and 1, A and B, horizontal, vertical.... But I did read with original firmware ports are as labelled on the board, after p20 it mode fw ports gets reversed...? no?

 

Bottomline, you have yours working as above, port a -> disks p1 p2 p3 p4, and port b -> p1=5 p2=6 p3=7 p4=8

(at least all the cables I've seen always say p1-p4 on both cables)

Link to post
Share on other sites

After IT flash, SAS port 0 and SAS port 1 are not swapped. The problem is that the disks are in random positions in DSM, because of the native SAS addressing, which change after each reboot, even if persistence is enable in the card.

 

After changing the card to  Eclosure/slot mode, ALL the ports & cables marking are reversed/mirrored, like i wrote above

 

  • Thanks 1
Link to post
Share on other sites
On 4/17/2022 at 10:50 PM, exodius said:

Hello,

 

After searching a lot, and testing for hours, i've found something that doesn't seems to in this forum (if it is; sorry i haven't seen it)

 

I've found 2 PERC H310 inside the bin, and saw the fabulous REDPILL LOADER, so i decide to give both a try.

Got some usb stick and free disks, i first follow the classic H310 to 9211-8i IT firmware flash (google for it, it's pretty easy to do)

 

Tryed the redpill loader with my new HBA SAS2008 : worked !!!

BUT, the drives never show up in the correct order. Worse : the order changed uppon reboot.

 

I found this great litterature to help me understand :

S11209_v1.0_SAS-2EnclosureMapping.pdf

 

There is a solution to switch those nice HBA from SAS address, to enclosure/slot mapping mode.

 

Please note that my enclosure is a supermicro with a BPN-SAS-825TQ backplane, with SFF8084/sata breakout cables pluged between the backplane and the LSI SAS2008 card

 

2 counterbacks :

- The order of the disks is reversed. Cable #1 for port A goes to disk 7, and cable #4 for port B goes to disk 0

- If there is a missing disk at boot, the card will bypass it, and it will allows report the disks in continuous order. ie. if disk 3 of 8 is missing at boot, then you will see that disk 8 is missing in DSM.

 

Here is what i have done for this result :

 

- Create my usb stick with redpill, and DSM

- Plugged only the first disk in the enclosure, at the first position (important !) and then install DSM.

 

If your NVRAM is not blank from a fresh flash, make a freedos USB stick with sas2flsh and launch sas2flsh -o -e 3 (this will erase the info from previous drives, and more config, becarefull)

 

- Launch DSM7, got to ssh and :

wget https://github.com/thomaslovell/LSIUtil/raw/master/Binaries/LSIutil_1.71_binaries/Linux/lsiutil.x86_64

chmod +x and sudo run ./lsiutil.x86_64

go into option 9, pagetype 1, pagenumber 8

modify offset 0x0c to 02 (bit 1 = Enclosure/Slot mapping mode)

 

reboot and enjoy.

 

Note that you can see the disk order with the LSIUTIL tool, using option 16

 

 

Fell free to make any comments or ask any questions

 

LSI SAS2008 correct order.png

Hi, 

 

Thank you for your contribution and explanation but I am not quite sure if I am doing this right on my controller. 

I was wondering why my disks are in a different order than they are connected to the cable, (I am use the same cable as you).
When I saw your guide, I was sure that I need try to play with my controller. 

I installed DSM 7.1 3622xsp. Only one SSD is connected to P1.

If you could help me with correct setting would be great. 

I am here and doesn't know what to change,  

 

 

 

001.PNG

002.PNG

003.PNG

004.PNG

Link to post
Share on other sites

I was reading some old posts from @IG-88 saying even if the disk order gets mixed or randomized it ok because dsm knows where they are, even if you dont. the only time its important is in the event of drive failure and replacement.... but of course you should want them in the correct sequential order :D just saying it will function either way.

 

complete side note, I just won a H200 on ebay for $20 ;) so I'm pretty excited about that.

Link to post
Share on other sites

 

Il y a 12 heures, MajkelP a dit :

003.PNG

 

 

 

You should edit the NVRAM.

 

Be sure that before, you have run sas2flsh -o -e 3 or it will not work

 

 

Il y a 11 heures, phone guy a dit :

I was reading some old posts from @IG-88 saying even if the disk order gets mixed or randomized it ok because dsm knows where they are, even if you dont. the only time its important is in the event of drive failure and replacement.... but of course you should want them in the correct sequential order :D just saying it will function either way.

 

They real problem is that they don't always appear in the same order in DSM. How do you tell DSM : hey, on mondays, do a SMART test on disk 1, then on tuesday on disk 2, when the disk order is changing ?

In my case, this XPEnology box is ON 1 hour a day, only for backup purposes.

 

Link to post
Share on other sites
9 hours ago, exodius said:

They real problem is that they don't always appear in the same order in DSM. How do you tell DSM : hey, on mondays, do a SMART test on disk 1, then on tuesday on disk 2, when the disk order is changing ?

 

no its not random, if nothing on the disks changes they will keep positions, i a disks fails for some reason, the following disks will fill the gap and if you add a replacement disk on the same cable on the controller the new disk will be last in order seen in dsm

can be problematic when doing disk recovery and disk are not on the same positions as before after a reboot

as long as the disk SN is shown no its possible to deal with it but with 6.2 there was a problem with jun's original driver and the one made from synology's kernel source did not show SN or smart data and in that situation it can be dangerous

lots of people want to have fixed slots in the same way a original units shows and that was a problem with the lsi sas controllers (at least in 6.2)

might be different with dsm7 and redpill loader as the loader might do things differently, i have not tested rp loader and 7.x with lsi sas if its still the same, might be different

 

Link to post
Share on other sites
On 4/19/2022 at 12:25 PM, exodius said:

 

 

You should edit the NVRAM.

 

Be sure that before, you have run sas2flsh -o -e 3 or it will not work

 

 

 

 

Thank you for your answer,

What I did:

1. Downloaded https://github.com/thomaslovell/LSIUtil/tree/master/Binaries/LSIutil_1.71_binaries/efi LSIutil_1.71_binaries for EFI, because my motherboard has got UEFI.

2. Also I had bootable USB pendrive with all required files and tools to flash my  Dell H310 Perc SAS controller to flash to SAS9211-8i IT mode.

3. I could not use sas2flsh -o -e 3 in MSDOS mode but in UEFI, (in MSDOS I had PAL error) so I used sas2flash.efi -o -e 3 and above command did work. 

4. I followed the instruction and I do not know how to modify offset 0x0c to 02 (bit 1 = Enclosure/Slot mapping mode). 

5. On the last picture I have:

Enter offset of value to change: [0004-0014 or Return to quit]

I am not quite sure what or how I can put here above offset.

Any thoughts would be great. 

To the controller I do not have any hdd/ssd connected.   

If I do above steps from SSH to my DSM I have exactly the same results. 

 

DSC00002.jpg

DSC00003.jpg

DSC00006.jpg

Link to post
Share on other sites
Il y a 21 heures, IG-88 a dit :

 

no its not random, if nothing on the disks changes they will keep positions, i a disks fails for some reason, the following disks will fill the gap and if you add a replacement disk on the same cable on the controller the new disk will be last in order seen in dsm

can be problematic when doing disk recovery and disk are not on the same positions as before after a reboot

as long as the disk SN is shown no its possible to deal with it but with 6.2 there was a problem with jun's original driver and the one made from synology's kernel source did not show SN or smart data and in that situation it can be dangerous

lots of people want to have fixed slots in the same way a original units shows and that was a problem with the lsi sas controllers (at least in 6.2)

might be different with dsm7 and redpill loader as the loader might do things differently, i have not tested rp loader and 7.x with lsi sas if its still the same, might be different

 

 

On my multiples tests, the order i got after the 1st boot, is not the same that the disk order i got in the 2nd boot. Disk 1 is only used for DSM, no data. Disks 2 to 8 are just blank without any data or partition. It really just pick the first responding drive, and give it the first sdX device at the kernel boot.

Those are my results after 2-3 days of trying to find a solution for this, wich i expose on the first topic. All tests was done with redpill and DSM7.1, i did not try any other loaders or DSM versions.

 

Link to post
Share on other sites

Hi. From yesterday I am trying in Proxmox to get it work as 918+ virtual machine. I had the same PERC card flashed with LSI IT firmware and it was working well on DSM6 with 5 drives in SHR (no important data on it). In previous DSM I just needed to add line  " set sata_args='SataPortMap=1 DiskIdxMap=1000' " in grub.cfg and then just pass it to Xpenology VM. Got my 918+ with quad GbE card but SAS card is another problem. Could You guide me what drivers, etc should I add to make it work ( I didnt tried to build my own redpill on usb/docker yet, I am just testing image downloaded from internet)

Link to post
Share on other sites
10 hours ago, ruffpl said:

Hi. From yesterday I am trying in Proxmox to get it work as 918+ virtual machine. I had the same PERC card flashed with LSI IT firmware and it was working well on DSM6 with 5 drives in SHR (no important data on it). In previous DSM I just needed to add line  " set sata_args='SataPortMap=1 DiskIdxMap=1000' " in grub.cfg and then just pass it to Xpenology VM. Got my 918+ with quad GbE card but SAS card is another problem. Could You guide me what drivers, etc should I add to make it work ( I didnt tried to build my own redpill on usb/docker yet, I am just testing image downloaded from internet)

for rp loader and dsm 7.x.x you will need mpt3sas available from pocopico github for the sas controller. I believe the correct building process for use with sas card is not to use any satamap commands, there is a tutorial for building 7.1 baremetal, which may help. In that thread there is a script written to help build a vm loader.

 

The script to autobuild rp loader for vm can be done in tc with this command (must have internet access of course)

 

wget https://github.com/PeterSuh-Q3/tinycore-redpill/raw/main/myv.sh.gz; tar -zxvf myv.sh.gz;

 

taken from this tutorial

 

Link to post
Share on other sites
9 hours ago, phone guy said:

for rp loader and dsm 7.x.x you will need mpt3sas available from pocopico github for the sas controller. I believe the correct building process for use with sas card is not to use any satamap commands, there is a tutorial for building 7.1 baremetal, which may help. In that thread there is a script written to help build a vm loader.

 

The script to autobuild rp loader for vm can be done in tc with this command (must have internet access of course)

 

wget https://github.com/PeterSuh-Q3/tinycore-redpill/raw/main/myv.sh.gz; tar -zxvf myv.sh.gz;

 

taken from this tutorial

 

 

 

You have precautions when using ds918+ and HBA.

I haven't checked further in the VM environment
In my bare metal, there is still a problem that disks mounted on HBAs do not recognize the original serial of the disk.

Although ds918+ is marked SN, it is showing an incorrect value.
The fact that SN is not recognized accurately can cause serious problems in RAID.

If disk repair is required, this SN is used to locate the slot on the disk that needs to be replaced.

Edited by Peter Suh
Link to post
Share on other sites
14 hours ago, Peter Suh said:

 

 

You have precautions when using ds918+ and HBA.

I haven't checked further in the VM environment
In my bare metal, there is still a problem that disks mounted on HBAs do not recognize the original serial of the disk.

Although ds918+ is marked SN, it is showing an incorrect value.
The fact that SN is not recognized accurately can cause serious problems in RAID.

If disk repair is required, this SN is used to locate the slot on the disk that needs to be replaced.

 

Ah, so if using an HBA sas card its better to build on 3622 platform. Good to know

Link to post
Share on other sites

Long story short, I bought a couple sas cards on ebay but I think I got the wrong ones.

 

Dell H200 sas, I know that one is ok to flash and can be used as 8 sata ports without issue in it mode

 

IBM M5015 unknown if it has an it mode firmware flash? I got confused with the M1015 🤪

 

Supermicro AOC-SAS2LP-H8IR also unknown? I dont know if an it mode firmware flash is available for this either?

 

Does anyone have any details about these 2 cards? (AOC-SAS2LP-H8IR  and  M5015) I believe they are both raid only cards? But I did see a couple posts on google saying they can be crossed flashed, however the posts are ancient and download links are all dead.... then replies to posts say thats incorrect they can not be flashed...

Obviously, I am a little confused... So any help and guidance would be most appreciated! Maybe some links to the software and firmware and sbr files?  Many Thanks!

Link to post
Share on other sites
2 hours ago, phone guy said:

Long story short, I bought a couple sas cards on ebay but I think I got the wrong ones.

 

Dell H200 sas, I know that one is ok to flash and can be used as 8 sata ports without issue in it mode

 

IBM M5015 unknown if it has an it mode firmware flash? I got confused with the M1015 🤪

 

Supermicro AOC-SAS2LP-H8IR also unknown? I dont know if an it mode firmware flash is available for this either?

 

Does anyone have any details about these 2 cards? (AOC-SAS2LP-H8IR  and  M5015) I believe they are both raid only cards? But I did see a couple posts on google saying they can be crossed flashed, however the posts are ancient and download links are all dead.... then replies to posts say thats incorrect they can not be flashed...

Obviously, I am a little confused... So any help and guidance would be most appreciated! Maybe some links to the software and firmware and sbr files?  Many Thanks!


just out of curiosity, why they should be flashed in IT mode ? Can’t you just use the JBOD function ? I’ve got a couple of old MPTSAS HBAs in JBOD and they work fine with 3622

Edited by pocopico
Link to post
Share on other sites
7 hours ago, pocopico said:

just out of curiosity, why they should be flashed in IT mode ? Can’t you just use the JBOD function ? I’ve got a couple of old MPTSAS HBAs in JBOD and they work fine with 3622

JBOD is a raid mode where you create a raid volume from a bunch of disk (size does not matter) and they are lined up in that volume as sequence

as a result you would have one volume visible to dsm and when it comes to the "older" lsi sas like sas2008 the it also means that the controller uses IR (R like in raid) mode and need a differen driver, not mpt2sas (or mpt3sas) it would be "megaraid" as driver

IT mode (initiator target mode) or HBA mode (host bus adapter)  is when you see every disk as a single disk and there is no raid at all

 

in older lsi sas like the sas2008 you needed a different firmware for IR mode and IT mode to flash, newer chip versions (12G sas?) where able to switch that in the controller bios/firmware so there was no it/it firmware

controller with the chip sas2008 was sold under different names depending if it had a ir or it firmware, ir controller where sold as MegaRAID controllers

 

something similar goes for hp smart array 4xx series 400 had no hba mode at all (at least for linux/windows), 410 was switchable by a software tool and since 420 it can be switched in the controller bios - not that i would suggest to use a hp sa controller with dsm, we never had much luck with them (and you might also have trouble using them in somewhere else as a hpe server)

 

 

9 hours ago, phone guy said:

Obviously, I am a little confused... So any help and guidance would be most appreciated! Maybe some links to the software and firmware and sbr files?  Many

start with that link

https://www.servethehome.com/lsi-sas-2008-raid-controller-hba-information/

and maybe that one

https://www.ikus-soft.com/en/blog/2021-01-20-flash-lsi-controller-for-zfs/

 

if you want one already in the right mode look for one thats labeled to have IT mode firmware

like this https://www.ebay.de/itm/194584436310

 

edit: don't even think about a cheap controller with a older 1068E chip, they have a hardware limit at 2.3TB so you cant use any bigger disks and there is no way around this, no bios, no driver

Edited by IG-88
  • Like 2
  • Thanks 1
Link to post
Share on other sites
On 4/23/2022 at 11:21 AM, phone guy said:

Long story short, I bought a couple sas cards on ebay but I think I got the wrong ones.

 

 

From my experience You can flash all of those cards with LSI 9211 IT mode:

lsi 9211 8i 
Fujitsu 9211 
Dell Perc H200 
IBM ServeRAID M1015
Intel RS2WC080

Dell Perc H310

Link to post
Share on other sites

Ps. I managed to get SAS card to work but in DSM i have disk7 as virtual drive 20GB and 4 ( from 5 attached) disks as disk 13,14,15,16. Tomorrow I will try to finish it/ I need to read what has to be changed.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.