spyrule

Cannot find file on xpenology.trantor.be/beta/ for 10gig NIC

Recommended Posts

Hello,

 

I've got a Mellanox ConnectX-2 EN 10Gb NIC, and I cannot get the card to be seen by Synology.

 

I've read here :

 

viewtopic.php?p=11054#p11054

 

That its possible to get the card working, however I cannot find the http://xpenology.trantor.be/beta/qlcnic.ko and the http://xpenology.trantor.be/beta/mellanox.7z files that are specified as required to make this card work.

 

I'm currently running DSM 5.0-4528 Update 2

 

If anybody has any suggestions/pointers/thread references on where to get these files and/or if they will work for DSM 5.0, it would be greatly appreciated.

 

Thanks in advance!

Share this post


Link to post
Share on other sites

You're in luck! I happen to have two Mellanox ConnectX-2 EN cards myself.

 

First, ConnectX-2 EN kernel modules are included with the XPEnoboot 5.1-5022.3 bootloader img. So, if you use it with DSM5.1, it's already built in.

 

However, if you must have it for DSM5.0, I've built this from source.

Mellanox ConnetX Kernel Modules.zip

 

This zip contains the two kernel modules, mlx4_core.ko and mlx4_en.ko, needed to enable Mellanox EN cards.

 

1) Extract the contents of the above zip, and you'll find mlx4_core.ko and mlx4_en.ko

 

2) Copy mlx4_core.ko and mlx4_en.ko into /lib/modules/

 

3) Edit both /etc/rc and /etc.defaults/rc with the following change...

 

Find this around line 327

NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx"

and modify it to

NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx4_core mlx4_en"

 

4) Reboot your system.

 

If done correctly, you should see your Mellanox ConnectX-2 show up as an extra Network Interface.

If your ConnectX-2 card is connected to another card or 10Gbit switch, you can look at the Network Interfaces to see which interface is 10000 Mbps. Alternatively, you use ifconfig and look at the MAC address to determine which one is your Mellanox card.

 

NOTE: I've only tested this with Nanoboot and DSM5.0

 

NOTE 2: While I've been able to achieve 9.89 Gbits/sec using iperf between two systems, it's pretty difficult to achieve even close to this speed via rsync and smb/cifs unless you have fast CPU and enough disks in RAID configuration on both servers. It's really taxing on the CPU to process all this data. The fastest I've been able to achieve so far with my hardware via copying over smb/cifs is around 350MB/sec. I may be able to increase this if I upgrade my second server's CPU.

Server 1: Xeon E3-1275v3, 10 disks in RAID5

Server 2: Celeron G550, 12 disks in RAID5 <-- currently CPU bottlenecked with the Celeron G550. Need to upgrade to a dual or quad-core CPU with very fast single-threaded performance since rsync is single-threaded

 

 

 

Good Luck!

Share this post


Link to post
Share on other sites

Thanks for the info/links Kanedo,

 

Sadly it still not coming up in ifconfig.

 

I actually upgraded to 5.1-5022.3 last night, but it seems as though neither of the mlx files were loading.

 

I added the files to the rc files, rebooted, but it still seems as though it's not getting picked up by the OS.

 

Now, my mellanox card is one of the MNPA19-XTR HP 10GB like this:

 

http://www.ebay.ca/itm/350983607688

 

I then actually tried replacing the .ko files that came with 5.1, with the files you provided, and again no change (no errors on boot though either).

 

I'm not too worried about peak speed, but my primary reason is for low latency for 3 ESXi boxes.

Share this post


Link to post
Share on other sites

NVM...

 

Figured it out.

 

I was running just ifconfig... stupid me forgot to check ifconfig -a to see the interfaces that were not up. Doh! ><

 

Its now online and good to go!

Share this post


Link to post
Share on other sites

Glad it's working out for you.

 

For future reference, you don't need to modify /etc/rc if it's included in the bootloader image. It should just work as is.

Share this post


Link to post
Share on other sites

Honestly, no clue man. I don't have an IB card, so I have zero experience with it.

Share this post


Link to post
Share on other sites

No worries it was worth a shot to ask. Thanks for trying to help.

Share this post


Link to post
Share on other sites

i installed the connectx-2 card but got error code during boot up, any idea whats worng?

 

i tested in both 5.1 and 5.2 xpenology with rc file modified under ESXI VM with the card passthru. Both failed, nothing can be seen in ifconfig -a

Share this post


Link to post
Share on other sites

What's the error, and I'm assuming if your doing passthrough, your using ESXi ?

 

Does your hardware properly support passthrough ?

Share this post


Link to post
Share on other sites

yes, I am using ESXI with the ConnectX-2 card passthrough to VM running DSM 5.1. Both mlx4_core.ko and mlx4_en.ko not loaded even I edited the rc file.

 

I can see the card present in the VM through lspci command

 

the mlx4_core.ko and mlx4_en.ko could be loaded with xpenology 5.2 boot image

 

here is the log extracted in 5.2 boot image

 

lspci

 

02:00.0 Class 0200: Device 8086:100f (rev 01)

03:00.0 Class 0200: Device 15b3:6750 (rev b0)

 

dmesg|grep mlx4

 

[ 30.892904] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)

[ 30.892906] mlx4_core: Initializing 0000:03:00.0

[ 33.201898] mlx4_core 0000:03:00.0: irq 72 for MSI/MSI-X

[ 33.201933] mlx4_core 0000:03:00.0: irq 73 for MSI/MSI-X

[ 33.201983] mlx4_core 0000:03:00.0: irq 74 for MSI/MSI-X

[ 33.202002] mlx4_core 0000:03:00.0: irq 75 for MSI/MSI-X

[ 33.202021] mlx4_core 0000:03:00.0: irq 76 for MSI/MSI-X

[ 33.202040] mlx4_core 0000:03:00.0: irq 77 for MSI/MSI-X

[ 33.313248] mlx4_core 0000:03:00.0: NOP command failed to generate MSI-X interrupt IRQ 77).

[ 33.313252] mlx4_core 0000:03:00.0: Trying again without MSI-X.

[ 33.426408] mlx4_core 0000:03:00.0: NOP command failed to generate interrupt (IRQ 73), aborting.

[ 33.426410] mlx4_core 0000:03:00.0: BIOS or ACPI interrupt routing problem?

[ 34.437841] mlx4_core: probe of 0000:03:00.0 failed with error -16

 

ifconfig -a

 

eth0 Link encap:Ethernet HWaddr 00:0C:29:64:F4:8E

inet addr:192.168.0.124 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe64:f48e/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:779 errors:0 dropped:0 overruns:0 frame:0

TX packets:362 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:233280 (227.8 KiB) TX bytes:57723 (56.3 KiB)

 

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:26 errors:0 dropped:0 overruns:0 frame:0

TX packets:26 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:3023 (2.9 KiB) TX bytes:3023 (2.9 KiB)

 

sit0 Link encap:IPv6-in-IPv4

NOARP MTU:1480 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

But when I boot in bare metal, everything works well. The ConnectX-2 recognized in DSM and can communicate with Win7 PC.

Share this post


Link to post
Share on other sites
yes, I am using ESXI with the ConnectX-2 card passthrough to VM running DSM 5.1. Both mlx4_core.ko and mlx4_en.ko not loaded even I edited the rc file.

 

I can see the card present in the VM through lspci command

 

the mlx4_core.ko and mlx4_en.ko could be loaded with xpenology 5.2 boot image

 

But when I boot in bare metal, everything works well. The ConnectX-2 recognized in DSM and can communicate with Win7 PC.

 

Did you tried to manually insmod mlx4_core.ko and mlx4_en.ko in your VM ?

You talking about 5.1 and 5.2, did both works with bare metal and both failed with ESXi ?

Share this post


Link to post
Share on other sites

Hi Trantor,

 

I did try manually insmod the two ko files in 5.1. No error for executing the insmod commands, but error in dmesg as follows,

 

[ 309.891940] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.891941] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.891942] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.891943] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.891944] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.891945] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000

[ 309.899328] mlx4_core 0000:03:00.0: NOP command failed to generate interrupt (IRQ 77), aborting.

[ 309.899330] mlx4_core 0000:03:00.0: BIOS or ACPI interrupt routing problem?

[ 309.908571] mlx4_core 0000:03:00.0: PCI INT A disabled

[ 309.908581] mlx4_core: probe of 0000:03:00.0 failed with error -16

 

there were a lot of "mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000" message which I just cut a few lines to show here.

 

I tried both 5.1 & 5.2 VM failed, bare-metal only tried 5.1 successful but not tried 5.2 as I think it should be ok.

Edited by Guest

Share this post


Link to post
Share on other sites

Honestly, it sounds like your systems isn't supporting Passthrough properly. It looks like it's not passing through IRQ requests to the card and back to the VM. Have you set any special settings in your VM, besides allowing passthrough (make sure you reboot your ESX box after adding the card to passthrough), then add the card as a device within the VM ?

Share this post


Link to post
Share on other sites

My system supports passthrough, I tried passthrough for dual port network card, display card successfully to VM before. I am using HP Microserver Gen 8 with E3-1265Lv2 CPU. I rebooted after setting passthrough and in vSphere screen showed passthrough had been in effect. I did not add any special setting, just add the PCI device for the Mellanox ConnectX-2 to the VM for DSM 5.1 or DSM 5.2.

 

As you can see from the output of lspci, it shows, 03:00.0 Class 0200: Device 15b3:6750 (rev b0)

 

I am thinking if it was due to the firmware of the ConnectX-2 is not upgraded to latest (2.9.1200), current on card is 2.9.1000. Since it is an HP OEM card from Mellanox which I am still searching where to download the required firmware from HP. Please let me know if any hint on where to download the required firmware. It stated in VMware website that the required firmware for ConnectX-2 should be 2.9.1200 for ESXI 5.5Ux.

Share this post


Link to post
Share on other sites

Hi Spyrule,

 

Thanks for the information. I finally flashed the firmware to 2.9.1200 successfully. Referenced your thread information and other thread, I did not use the switch of --ALLOW_PSID_CHANGE. I modified the generic firmware INI file so that the HP PSID (HP_0F60000010) is kept.

 

After flashing the new firmware, it now can passthrough the Mellanox ConenctX-2 card to VM running DSM 5.1 & 5.2 under ESXI 6.0. There is no more error message, and IP address can be configured manually in DSM. Both bare metal and VM are tested in 5.1 & 5.2 version, both are OK.

 

The file copying operation is not that impressive, around 150MB - 250MB per second.

 

Thanks again for all you guys' help.

Share this post


Link to post
Share on other sites

Did you flash the boot rom as well?

If not, you could be running a mismatched Rom and Firmware, which could cause unforeseen future issues. There is no benefit of keeping the HP ID, unless you want to struggle to apply future updates. (assuming there is any... which is doubtful).

 

As for performance on these cards, chances are your hard drives/controller are your bottleneck and not the NIC anymore.

Share this post


Link to post
Share on other sites

I didn't flash the rom, I just follow the Mellanox manual (MFT) to customize the firmware for OEM card by editing the INI and use the MLX file for flashing.

Share this post


Link to post
Share on other sites