Need a hint on what to try next - Mellanox MNPH29D-XTR (Connect-X2 with dual SFP+)


Recommended Posts

Hi folks - I've done quite a bit of searching but just can't quite seem to find the answer I'm looking for. 

 

TL;DR - need a hint on what loader or extras are needed for Connect-X2 10gbe NIC support.

 

Let me get the basics out of the way:

 

1 - TYAN SS512 motherboard, E3-1230V2 CPU, 16GB ECC, integrated LSI 2008 SAS, integrated 2 x Intel i350 NICs.

 

2 - I am a current Synology hardware owner (DS918+) so I know my way around DSM.  I'm also an IT professional and know my way around hardware and OS's pretty well.

 

3 - A week ago, I installed Xpenology via a lot of reading here (thank you all!).  Used 1.03b loader for DS3617xs which is practically a direct match for my hardware, and the latest 6.2.3.xxx version of DSM.  Absolutely zero issues.  Installed right away and ran perfectly for a solid week.

 

Then my Mellanox Connect-X2 cards came in from Ebay.  I installed one into the rig above and Xpenology/DSM would no longer boot.  Removed the card and it boots into DSM just fine again.

 

OK, I think - let me first make sure the card is good, along with the DAC cable, the upstream switch etc.  I put the same card back in and installed a fresh copy of Windows 10 on the same exact hardware - works perfectly gets 400 MB/sec of traffic with no tweaks.  In short, the card and the upstream infrastructure are fine.

 

This lead me to believe that the 3617xs image and/or Jun's 1.03b loader for 3617 might be missing the driver for the Mellanox.  So I did some more reading and saw that the 3615xs loader is perhaps a bit more lenient on hardware or has more imbedded drivers.  So I went through the process of building a 3615xs 1.03b, and that version unfortunately won't go any further.  It will boot into the loader screen, but I never see link light come up on the Mellanox.  I also went into BIOS and disabled the 2 x existing Intel i350 NICs on the motherboard just to make sure they weren't conflicting.

 

This tells me it is driver related as the card itself is proven good in the same hardware chassis.  So my questions:

 

1 - Definitively, does the Mellanox Connect-X2 have a driver in one of Jun's prebuilt loaders, or do I need to go hunt up something in an extras.lzma?  I see answers that vary wildly on this, which is why I'm trying to get a definitive answer for myself and others who may be searching in the future.

2 - If I don't need any extra drivers, what loader do I need that contains the Mellanox Connect-X2 driver natively?  I don't mind going back to DSM 6.1 if needed.  I just want to use my extremely standard 10 gig cards but I suppose I can return them and swap for Intel DA-520 versions if needed.

 

Thank you so much for any help you can offer.  I've tried quite a bit on my own but failing.  If someone can drop me a hint I'd really appreciate it.

 

Panoz

Edited by Panozguy
Link to post
Share on other sites

I have never had problems with Mellanox cards.  However I have ALWAYS included a 1 GBe Ethernet port active in the system (even if I don't have anything plugged into the port).  If you have 10Gbe only this may be an issue.

 

To be specific I have personally tested the following combinations:

 

Baremetal DS3615xs 6.1.7 Mellanox Connect-X 2 dual

Baremetal DS3615xs 6.1.7 Mellanox Connect-X 3 dual

ESXi passthrough DS3615xs 6.1.7 Connect-X 3 dual

ESXi passthrough DS3615xs 6.2.x Connect-X 3 dual

ESXi passthrough DS3617xs 6.2.x Connect-X 3 dual

Link to post
Share on other sites
8 hours ago, Panozguy said:

So I did some more reading and saw that the 3615xs loader is perhaps a bit more lenient on hardware or has more imbedded drivers

not really, from the synology default drivers the 3617 gets more attention and has newer drivers in some places (like mpt3sas)

the mellanox drivers are the same on both

 

when it comes to default drivers in rd.gz (on the loader together with the kernel) then the 3615 has igb.ko (suiting for i350) and the 3617 is missing that one and has ixgbe.ko instead so when just booting the loader and not having additional drivers the 3617 might not recognise a i350, but with a extra.lzma present - and thats the normal case with 1.03b loader (as it has jun's extra.lzma)  - at least thats what i thought, but when going through juns extra.lzma there is no igb.ko present, the rc.modules does try to load that one but if its not present  and not in the rd.gz ...

its present in dsm so the igb.ko will work after installing, the same goes for the mellanox drivers, not in rd.gz and not in jun's extra.lzma

edit: i overlooked the igb.ko in the rd.gz of the 3617, its there so it should work ootb with a i350

(to late for today, i will go to bed)

 

so the thing flyride mentions is about not beeing able to do a 1st installation with just a mellanox and the default loader

but as i provide a newer igb.ko with my extra.lzma the i350 should at least work when using my extra.lzma 11.2 for 3617

https://xpenology.com/forum/topic/28321-driver-extension-jun-103b104b-for-dsm623-for-918-3615xs-3617xs/

 

a added mellanox should not crash a already installed 3617 system

if that happens then remove the card, boot again and have a look in dmesg, there should be something about that failed boot attempt, if that does not work its possible to use a null-modem-cable (com1) and a serial console to monitor the boot process (dsm does switch to it early and you dont see anything on a monitor)

 

Edited by IG-88
Link to post
Share on other sites
2 hours ago, IG-88 said:

imho connect-x2 should work both 3615 and 3617

if there are problems with3617 send me the dmesg log an i will have a look whats going on

Yes, it actually worked fine on the 3617xs as well.  Guess I missed the part somewhere where you have to actually make the primary connection with a normal NIC.  All good, I learned something.  Thanks for the offer!

Link to post
Share on other sites
16 hours ago, Panozguy said:

I got it worked out, but I’m glad that if someone asks in the future it will be here.  I just didn’t want to be the first noob guy who hadn’t even attempted some research.  Thank you guys.

 

16 hours ago, Panozguy said:

Guess I missed the part somewhere where you have to actually make the primary connection with a normal NIC

afaik that in not mentioned anywhere in the howto's, maybe its suggested to change nic's when having problems or connect all nic's

that problem can also hit people with 918+ from a different direction as it only supports 2 nic's ootb and when adding a 2/4port nic some of the ports will not be usable ootb (the patch in the loader "only" extends the disks form 4 to 16 but does not extend nic's to anything near the default of 8 from the 3615/17, its still 2 nic's after installing and >2 needs manually tweaking atm)

 

the mellanox drivers might be the only nic drivers not working directly with the loader (only after installing dsm) as there are recent enough drivers in dsm itself so they did not make it into the extra.lzma (yet) that beside kernel/rd.gz is also loaded at 1st boot when installing, synology does not support them as to install for a new system, thats only supported with the onboard solutions (obviously or they would be part of rd.gz like igb.ko)

that only applies to 3615/17, as 918+ does not have generic support in dsm the mellanox drivers, they are added be the extra.lzma and will work on 1st boot for installing dsm

 

its possible to add them to the 3615/17 extra.lzma but its would bring extra.lzma from 5MB to 20MB, making it impossible to use extra.lzma for the MBR loader version that comes with 20MB partition size (for kernel and extra, jun's defaul size is 30MB), only adding mlx4 (and all lower hardware version) would not have that much impact, there will be only very few to try installing with just a connect-x5 in the system

(i'm aware of the 20MB limit as i use old HP desktops for testing that need MBR version of the loader)

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.