SM 12 3.5" bay SES2 chassis - Does drive numbering work ok?


Recommended Posts

Hi,

 

I've currently got two LSI 9211-8i's which act as my HBAs for my sRAID setup (windows based software raid, equivalent to MDADM on Linux but under development by the author of Flexraid), the issue that caused me to switch was the out of sequence drive ordering that DSM used because of the LSI cards and the way they enumerate the drives; e.g. when booting they where shown in the correct order but once booted in to DSM, the first drive on the list could be assigned the fourth slot in Storage Manager.

 

Now, I'm getting in to the financial position to import a used Supermicro chassis with 12 hot swap SAS2 bays (so 6gbps SAS/SATA) and when connected to my SM X8DTE-F should allow the HBAs to report the drive bay number, let alone the drives unique identification. I'm hoping this will solve the issue with the drives being placed in Storage Manager different to the actual physical slot but before I do this I was hoping that someone else with a fully SES2(SAS2) compliant chassis (e.g. Norco, Dell R510 etc.) could let me know if this will work, i.e. Drive bay 1 = Storage Manager Drive Slot 1, or am I still going to have my drives reported all over the place making it difficult to determine which drive is which when I need to replace it?

 

Thanks in advance,

 

Chris

Link to post
Share on other sites

I can test it in the middle of the next week, because i'm on business travel rigth now...

 

But I have an idea:

If you have a failed drive, with a cli command you get blinking the LED of the failed drive. (MegaCli or something else)

Edited by Guest
Link to post
Share on other sites

That would be fantastic, as I'd rather not have to shift by to SATA3 expansion cards when I already have the HBAs.

 

At the moment my setup doesn't have a SES compatible chassis, I'm using 4in3 hot swap SATA drives so I don't think that would work. Ideally I want the loaded drive order ot be the same as the Storage Manager order but using the SATA hot swap units that I have now doesn't work properly.

Link to post
Share on other sites

In my 16 drive configured XPE box, I've used a supermicro SAS-LP-MV8 adapter with SAS to SATA leads. I found that ports 0-3 report in Storage Manager as drives 12-9 and ports 4-7 report as drives 16-13. (there are some other controllers for drives 1-8)

 

I went for a 'manual' solution and used Storage Manager as the reference and simply cabled the drives by its order to the drive bays, and I labelled the drive cables too. :smile:

Link to post
Share on other sites
In my 16 drive configured XPE box, I've used a supermicro SAS-LP-MV8 adapter with SAS to SATA leads. I found that ports 0-3 report in Storage Manager as drives 12-9 and ports 4-7 report as drives 16-13. (there are some other controllers for drives 1-8)

 

I went for a 'manual' solution and used Storage Manager as the reference and simply cabled the drives by its order to the drive bays, and I labelled the drive cables too. :smile:

my custom chassis does something similar, is yours a supermicro chassis with SM motherboard? I just wondered whether using the backplane to motherboard cabling it would report back to the HBa and therefore Xpenology the correct drive order. If it doesn't solve it, then I'll probably use ESXi on the motherboard and use RDMs to control the drive positions.

Link to post
Share on other sites

my setup is not that sophisticated, its a Gigabyte board with a few PCIx slots and a 'diy' drive array using some Zalman drive bays I labelled. To 'document' my setup and the difference between the drives on the controller vs DSM, I just videoed the marvell booting so I had a snapshot of which disks where where, compared to DSM. I worked on the basis that DSM will tell me that 'disk N' has failed so if that happens I just go to my labelled 'disk N' bay.

Link to post
Share on other sites

I had to something similar but instead my LSI HBAs would reverse drive 0 and 3 around so I had to swap the cables so on boot they where in the wrong order but then as long as they where loaded one at a time starting at drive bay 0 they would be in the right order. As I'm fairly sure, with the correct link to the motherboard from a backplane, that the HBA will no what drive bays there in I'm hoping that would fix the problem as I would then invest in a proper 12 bay 2u chassis for my supermicro xeon motherboard; I can then utilise the 4in3 bays for my other vSphere hosts for hot swap purposes.

Link to post
Share on other sites
I can test it in the middle of the next week, because i'm on business travel rigth now...

 

But I have an idea:

If you have a failed drive, with a cli command you get blinking the LED of the failed drive. (MegaCli or something else)

Hi sfu420

 

Did you manage to have a look at this? If I can't get the LSI cards to play nice with drive numbering then I might need to invest in a new 12 port HBA but I'd rather not if I don't have to.

Link to post
Share on other sites

Thank you, I do appreciate it.

 

I've got three LSI 2008 cards, two of which are currently feeding my 12 bays but I'd be happy to get a different brand if it solves the drive numbering problem or new chassis as appropriate.

 

I'm currently in the states on business so when I get home I can try my Dell R310 with its LSI 2008 based card that is connected to a four bay backplane and see what that does... I'd be all over a 12 bay Dell R510 but they're rare in New Zealand if it works.

Link to post
Share on other sites

I've tested and i have the same issue with drive numbering.

In the BIOS of the LSI card the drive order is correct, but after boot in the OS the drive order is wrong:

Slot 1 --> Drive 1

Slot 2 --> Drive 4

Slot 3 --> Drive 3

Slot 4 --> Drive 2

 

I thought it's just because of the very old version of mpt3sas driver in 5644.4

Then i've tried the latest release of today 5644.5. It has updated mpt3sas driver (the latest version P11). But unfortunately same issue. :sad:

 

I've tested it on Debian as well, and same issue. So it's not an XPE problem.

 

 

After that i've tried it with my Adaptec HBA 1000-16i card, and drive numbering is correct!!! :smile:

But no S.M.A.R.T support in the DSM GUI.

 

Anyway if i type in console:

 

smartctl -a /dev/sda

 

i get all of the infos from disk.

So, S.M.A.R.T working as well, just the GUI can't handle that. :cry:

I've tested driver v1.2.0 and latest v1.2.1 as well.

 

I hope this help you.

Link to post
Share on other sites

Yes thank you, that is fantastic! So, basically I need to look towards a new HBA that supports the correct numbering as well as SMART.. I really appreciate the testing, it was doing my head and I new that the LSI cards did this but I wasn't sure if it would be fixed by using a backplane.

 

So the LSI issue manifests itself in Linux too? Interesting issue with the WWAN used by LSI then I suspect.

Link to post
Share on other sites

SFU420, is your Adaptec a SAS12G card? I've asked another member who has an LSI 12G card to see how drive numbering works there. I'm hoping it does.

 

I'm also wondering how the top Synology Rackmount SAS unit does it as it has a sas card installed with a sas expander, but doesn't use a backplane connector so it must number fine.

 

If we could find out what card that is, that would work as a base to start from!

Link to post
Share on other sites

Hi SFU420, can you confirm that the SAS card is connected by SAS cables? I spoke to Synology about their rackmount sas system and they confirmed that it was using an LSI2008 based HBA and it does drive numbering ok... Now they may have a custom based HBA that solves the typical LSI SAS issue but the more we can refine it, the better it would be at advising those that rely on SAS for drive expansion.

 

Chris

Link to post
Share on other sites
I've tested and i have the same issue with drive numbering.

In the BIOS of the LSI card the drive order is correct, but after boot in the OS the drive order is wrong:

Slot 1 --> Drive 1

Slot 2 --> Drive 4

Slot 3 --> Drive 3

Slot 4 --> Drive 2

 

I thought it's just because of the very old version of mpt3sas driver in 5644.4

Then i've tried the latest release of today 5644.5. It has updated mpt3sas driver (the latest version P11). But unfortunately same issue. :sad:

 

I've tested it on Debian as well, and same issue. So it's not an XPE problem.

 

 

After that i've tried it with my Adaptec HBA 1000-16i card, and drive numbering is correct!!! :smile:

But no S.M.A.R.T support in the DSM GUI.

 

Anyway if i type in console:

 

smartctl -a /dev/sda

 

i get all of the infos from disk.

So, S.M.A.R.T working as well, just the GUI can't handle that. :cry:

I've tested driver v1.2.0 and latest v1.2.1 as well.

 

I hope this help you.

I've been doing a lot of reading on the LSI cards again. Your SFF-8087 cables, are they sideband enabled; i.e. 36 pins enabled versus 28? From what I can read, a sideband enabled cable, card and backplane allows the card to communicate with the backplane and get a load of diagnostics from the backplane. If they're not, I wonder if the adaptec card doesn't need the backplane for port numbering where as the LSI probably does.

 

If the Synology SAS Rackmount is simply using a HBA SAS <> Chassis backplane with only two cables (it does from what the you tube videos have shown) then their LSI card must be communicating with the backplane itself.

Link to post
Share on other sites

Yes, my Adaptec is a SAS 12Gb card. I'm using SFF-8643 to SFF-8643 cables, because my backplane have 3x SFF-8643 connectors and not direct attached SATA. As i know this cables have sideband integrated.

I've tested some cables, that maybe they have different pinouts, but not. Same problem.

LSI00409

LSI00413

Intel AXXCBL380HDHD

Adaptec 2282500-R

 

After that, i found this:

https://docs.oracle.com/cd/E19045-01/blade.6000disk/820-4922-16/z40000191018657.html

 

PS: Synology Rachmount has not 2 but 3 SAS cables to backplane connected.

Link to post
Share on other sites

Thanks. So your adaptec card reports drive numbers correctly in DSM, but not smart data and the LSI cards do not but will display SMART. I presume with the LSI the drives are listed randomly despite the cables?

 

I need to get a new enclosure anyway so I might as well get a SES2 enabled 12 bay one from Ebay (unless I can find a reasonably priced Dell R510 in NZ!) and then I will see.

 

I get back to New Zealand next week so I might try and boot up my Dell R310 with the 4 port SAS2 backplane with afully connected SAS 6/IR and see what that does with DSM; as the sas card is a LSI based chip I suspect it will do the normal thing of screwing around.

 

How did you know the Synology machine was 3 cables? Is it a 4+4+4 configuration rather than expander? Wonder what SAS card it is they've got.

Link to post
Share on other sites
So your adaptec card reports drive numbers correctly in DSM, but not smart data and the LSI cards do not but will display SMART.

Yes.

 

I presume with the LSI the drives are listed randomly despite the cables?

Exactly not randomly, but always the same order:

Slot 1 --> Drive 1

Slot 2 --> Drive 4

Slot 3 --> Drive 3

Slot 4 --> Drive 2

 

How did you know the Synology machine was 3 cables? Is it a 4+4+4 configuration rather than expander?

 

Yes, they are a 4+4+4 config. An expander would be waste of money and unnecessary.

 

Check these videos. It can be see the 3 SFF-8087 cables

RS18016XS+

RS3614xs

Edited by Guest
Link to post
Share on other sites

Thanks, really appreciate the attention to detail.. I read how LSI do their drive mappings and as suspected the WWAN is used. Now LSI store this in persistent nonvolatile memory so that if a drive is moved, it still has the same address & location. There is an LSI utility out there that can firstly delete the mappings so they're redone, it also allows you to set the port to use enclosure data and finally in the hidden options you can disable the persistent mapping. I'm not sure whether this would, when combined with direct connection or SES2 management, force the LSI card to recognise the drives based on enclosure position and therefore list them as port 0 = sda, port 1 = sdb etc.

 

The utility was shown to be working on a slightly older chip I think, but it might work on the newer ones too. I'll have a go on my Dell R310 which I'm about to bring out of retirement so will be a good test.

Link to post
Share on other sites

Apperantly its the LSIUTIL, download links can be found in this thread: http://www.dzhang.com/blog/2013/03/22/w ... ad-lsiutil but I do not know if they still work yet. Within this, there are special commands that affect the LSI cards... AGain, do not know which ones as I'm away but will test when I get back... Some more information in these two threads;

 

https://www.mail-archive.com/zfs-discus ... 13239.html (more stuff at the bottom of the thread, including the option 15 for persistance)

http://broken.net/uncategorized/making- ... ontroller/

Link to post
Share on other sites

So I tried out the LSIUtil option. The hidden option 15 is there but I can't get it to do anything. A Sun/Oracle PDF I saw showed it doing something including using those options... my copy refused to do anything at all... I've dropped a line to Avago for some help but I suspect they can't do anything.

 

If it continues like this then I will most likely look towards an Adaptec card to get this working, as it not only affects DSM but also any windows or Linux based raid approach.

 

Have you tried asking Trantor why the adaptec cards refuse to give SMART data?

Link to post
Share on other sites

Sorry, also wanted to ask about the Adaptec, was this attached via SES2/3 or direct cables to SATA? Its practically been confirmed by LSI that you cannot sort out the numbering, I'm just waiting to see if its all LSI hardware or the SAS2008 chip. If its the entire line, then I'll be switching controller I feel, but it would be useful if the adaptec card works with SATA breakout cables as well as SES2/3 connections for drive numbering.

Link to post
Share on other sites
  • 2 months later...

As I've wrote on other post I have a dell H310 2008 chip card flashed to LSI 2118IT and have the drives ordering problem, but if I use the card with original firmware from dell H310 it works ok, disks are in correct order on DSM and card Bios, only don't have SMART status in DSM!

 

So could be firmware related the ordering issue?

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.