spyrule Posted March 10, 2015 #1 Posted March 10, 2015 Hello, I've got a Mellanox ConnectX-2 EN 10Gb NIC, and I cannot get the card to be seen by Synology. I've read here : viewtopic.php?p=11054#p11054 That its possible to get the card working, however I cannot find the http://xpenology.trantor.be/beta/qlcnic.ko and the http://xpenology.trantor.be/beta/mellanox.7z files that are specified as required to make this card work. I'm currently running DSM 5.0-4528 Update 2 If anybody has any suggestions/pointers/thread references on where to get these files and/or if they will work for DSM 5.0, it would be greatly appreciated. Thanks in advance!
Kanedo Posted March 10, 2015 #3 Posted March 10, 2015 You're in luck! I happen to have two Mellanox ConnectX-2 EN cards myself. First, ConnectX-2 EN kernel modules are included with the XPEnoboot 5.1-5022.3 bootloader img. So, if you use it with DSM5.1, it's already built in. However, if you must have it for DSM5.0, I've built this from source. Mellanox ConnetX Kernel Modules.zip This zip contains the two kernel modules, mlx4_core.ko and mlx4_en.ko, needed to enable Mellanox EN cards. 1) Extract the contents of the above zip, and you'll find mlx4_core.ko and mlx4_en.ko 2) Copy mlx4_core.ko and mlx4_en.ko into /lib/modules/ 3) Edit both /etc/rc and /etc.defaults/rc with the following change... Find this around line 327 NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx" and modify it to NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx4_core mlx4_en" 4) Reboot your system. If done correctly, you should see your Mellanox ConnectX-2 show up as an extra Network Interface. If your ConnectX-2 card is connected to another card or 10Gbit switch, you can look at the Network Interfaces to see which interface is 10000 Mbps. Alternatively, you use ifconfig and look at the MAC address to determine which one is your Mellanox card. NOTE: I've only tested this with Nanoboot and DSM5.0 NOTE 2: While I've been able to achieve 9.89 Gbits/sec using iperf between two systems, it's pretty difficult to achieve even close to this speed via rsync and smb/cifs unless you have fast CPU and enough disks in RAID configuration on both servers. It's really taxing on the CPU to process all this data. The fastest I've been able to achieve so far with my hardware via copying over smb/cifs is around 350MB/sec. I may be able to increase this if I upgrade my second server's CPU. Server 1: Xeon E3-1275v3, 10 disks in RAID5 Server 2: Celeron G550, 12 disks in RAID5 <-- currently CPU bottlenecked with the Celeron G550. Need to upgrade to a dual or quad-core CPU with very fast single-threaded performance since rsync is single-threaded Good Luck!
spyrule Posted March 10, 2015 Author #4 Posted March 10, 2015 Thanks for the info/links Kanedo, Sadly it still not coming up in ifconfig. I actually upgraded to 5.1-5022.3 last night, but it seems as though neither of the mlx files were loading. I added the files to the rc files, rebooted, but it still seems as though it's not getting picked up by the OS. Now, my mellanox card is one of the MNPA19-XTR HP 10GB like this: http://www.ebay.ca/itm/350983607688 I then actually tried replacing the .ko files that came with 5.1, with the files you provided, and again no change (no errors on boot though either). I'm not too worried about peak speed, but my primary reason is for low latency for 3 ESXi boxes.
spyrule Posted March 10, 2015 Author #5 Posted March 10, 2015 NVM... Figured it out. I was running just ifconfig... stupid me forgot to check ifconfig -a to see the interfaces that were not up. Doh! >< Its now online and good to go!
Kanedo Posted March 10, 2015 #6 Posted March 10, 2015 Glad it's working out for you. For future reference, you don't need to modify /etc/rc if it's included in the bootloader image. It should just work as is.
PnoT Posted May 7, 2015 #7 Posted May 7, 2015 Not to Hijack but was curious if either one of you knew how to get Infiniband working. Here's a link to the post http://xpenology.com/forum/viewtopic.php?f=2&t=5847 Thanks
spyrule Posted May 7, 2015 Author #8 Posted May 7, 2015 Honestly, no clue man. I don't have an IB card, so I have zero experience with it.
PnoT Posted May 7, 2015 #9 Posted May 7, 2015 No worries it was worth a shot to ask. Thanks for trying to help.
falconhk2007 Posted December 5, 2015 #10 Posted December 5, 2015 i installed the connectx-2 card but got error code during boot up, any idea whats worng? i tested in both 5.1 and 5.2 xpenology with rc file modified under ESXI VM with the card passthru. Both failed, nothing can be seen in ifconfig -a
spyrule Posted December 5, 2015 Author #11 Posted December 5, 2015 What's the error, and I'm assuming if your doing passthrough, your using ESXi ? Does your hardware properly support passthrough ?
falconhk2007 Posted December 6, 2015 #12 Posted December 6, 2015 yes, I am using ESXI with the ConnectX-2 card passthrough to VM running DSM 5.1. Both mlx4_core.ko and mlx4_en.ko not loaded even I edited the rc file. I can see the card present in the VM through lspci command the mlx4_core.ko and mlx4_en.ko could be loaded with xpenology 5.2 boot image here is the log extracted in 5.2 boot image lspci 02:00.0 Class 0200: Device 8086:100f (rev 01) 03:00.0 Class 0200: Device 15b3:6750 (rev b0) dmesg|grep mlx4 [ 30.892904] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [ 30.892906] mlx4_core: Initializing 0000:03:00.0 [ 33.201898] mlx4_core 0000:03:00.0: irq 72 for MSI/MSI-X [ 33.201933] mlx4_core 0000:03:00.0: irq 73 for MSI/MSI-X [ 33.201983] mlx4_core 0000:03:00.0: irq 74 for MSI/MSI-X [ 33.202002] mlx4_core 0000:03:00.0: irq 75 for MSI/MSI-X [ 33.202021] mlx4_core 0000:03:00.0: irq 76 for MSI/MSI-X [ 33.202040] mlx4_core 0000:03:00.0: irq 77 for MSI/MSI-X [ 33.313248] mlx4_core 0000:03:00.0: NOP command failed to generate MSI-X interrupt IRQ 77). [ 33.313252] mlx4_core 0000:03:00.0: Trying again without MSI-X. [ 33.426408] mlx4_core 0000:03:00.0: NOP command failed to generate interrupt (IRQ 73), aborting. [ 33.426410] mlx4_core 0000:03:00.0: BIOS or ACPI interrupt routing problem? [ 34.437841] mlx4_core: probe of 0000:03:00.0 failed with error -16 ifconfig -a eth0 Link encap:Ethernet HWaddr 00:0C:29:64:F4:8E inet addr:192.168.0.124 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe64:f48e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:779 errors:0 dropped:0 overruns:0 frame:0 TX packets:362 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:233280 (227.8 KiB) TX bytes:57723 (56.3 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:26 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3023 (2.9 KiB) TX bytes:3023 (2.9 KiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) But when I boot in bare metal, everything works well. The ConnectX-2 recognized in DSM and can communicate with Win7 PC.
Trantor Posted December 6, 2015 #13 Posted December 6, 2015 yes, I am using ESXI with the ConnectX-2 card passthrough to VM running DSM 5.1. Both mlx4_core.ko and mlx4_en.ko not loaded even I edited the rc file. I can see the card present in the VM through lspci command the mlx4_core.ko and mlx4_en.ko could be loaded with xpenology 5.2 boot image But when I boot in bare metal, everything works well. The ConnectX-2 recognized in DSM and can communicate with Win7 PC. Did you tried to manually insmod mlx4_core.ko and mlx4_en.ko in your VM ? You talking about 5.1 and 5.2, did both works with bare metal and both failed with ESXi ?
falconhk2007 Posted December 7, 2015 #14 Posted December 7, 2015 (edited) Hi Trantor, I did try manually insmod the two ko files in 5.1. No error for executing the insmod commands, but error in dmesg as follows, [ 309.891940] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.891941] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.891942] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.891943] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.891944] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.891945] mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000 [ 309.899328] mlx4_core 0000:03:00.0: NOP command failed to generate interrupt (IRQ 77), aborting. [ 309.899330] mlx4_core 0000:03:00.0: BIOS or ACPI interrupt routing problem? [ 309.908571] mlx4_core 0000:03:00.0: PCI INT A disabled [ 309.908581] mlx4_core: probe of 0000:03:00.0 failed with error -16 there were a lot of "mlx4_core 0000:03:00.0: Completion event for bogus CQ 00000000" message which I just cut a few lines to show here. I tried both 5.1 & 5.2 VM failed, bare-metal only tried 5.1 successful but not tried 5.2 as I think it should be ok. Edited December 7, 2015 by Guest
spyrule Posted December 7, 2015 Author #15 Posted December 7, 2015 Honestly, it sounds like your systems isn't supporting Passthrough properly. It looks like it's not passing through IRQ requests to the card and back to the VM. Have you set any special settings in your VM, besides allowing passthrough (make sure you reboot your ESX box after adding the card to passthrough), then add the card as a device within the VM ?
falconhk2007 Posted December 7, 2015 #16 Posted December 7, 2015 My system supports passthrough, I tried passthrough for dual port network card, display card successfully to VM before. I am using HP Microserver Gen 8 with E3-1265Lv2 CPU. I rebooted after setting passthrough and in vSphere screen showed passthrough had been in effect. I did not add any special setting, just add the PCI device for the Mellanox ConnectX-2 to the VM for DSM 5.1 or DSM 5.2. As you can see from the output of lspci, it shows, 03:00.0 Class 0200: Device 15b3:6750 (rev b0) I am thinking if it was due to the firmware of the ConnectX-2 is not upgraded to latest (2.9.1200), current on card is 2.9.1000. Since it is an HP OEM card from Mellanox which I am still searching where to download the required firmware from HP. Please let me know if any hint on where to download the required firmware. It stated in VMware website that the required firmware for ConnectX-2 should be 2.9.1200 for ESXI 5.5Ux.
spyrule Posted December 7, 2015 Author #18 Posted December 7, 2015 I have the exact same card... (mods, feel free to remove this if this breaks a rule to link to another forum). Here is my solution to the FW update: https://forums.servethehome.com/index.p ... tx-2.7560/
falconhk2007 Posted December 8, 2015 #19 Posted December 8, 2015 Hi Spyrule, Thanks for the information. I finally flashed the firmware to 2.9.1200 successfully. Referenced your thread information and other thread, I did not use the switch of --ALLOW_PSID_CHANGE. I modified the generic firmware INI file so that the HP PSID (HP_0F60000010) is kept. After flashing the new firmware, it now can passthrough the Mellanox ConenctX-2 card to VM running DSM 5.1 & 5.2 under ESXI 6.0. There is no more error message, and IP address can be configured manually in DSM. Both bare metal and VM are tested in 5.1 & 5.2 version, both are OK. The file copying operation is not that impressive, around 150MB - 250MB per second. Thanks again for all you guys' help.
spyrule Posted December 8, 2015 Author #20 Posted December 8, 2015 Did you flash the boot rom as well? If not, you could be running a mismatched Rom and Firmware, which could cause unforeseen future issues. There is no benefit of keeping the HP ID, unless you want to struggle to apply future updates. (assuming there is any... which is doubtful). As for performance on these cards, chances are your hard drives/controller are your bottleneck and not the NIC anymore.
falconhk2007 Posted December 9, 2015 #21 Posted December 9, 2015 I didn't flash the rom, I just follow the Mellanox manual (MFT) to customize the firmware for OEM card by editing the INI and use the MLX file for flashing.
sorinbidian Posted January 1, 2016 #22 Posted January 1, 2016 Can you post de custom firmware you built? I have the same card. Thanks
falconhk2007 Posted January 8, 2016 #23 Posted January 8, 2016 cannot upload attachment here. quota reached.
falconhk2007 Posted January 10, 2016 #25 Posted January 10, 2016 Warning: use at your own risk https://www.dropbox.com/s/0g7t16qconftb ... 2.bin?dl=0
Recommended Posts