PnoT Posted May 22, 2015 Share #1 Posted May 22, 2015 First of all I'd like to thank Trantor for compiling the modules here: You've seriously made my week... thank you! Here are some screenies of it in action and my system specs: Supermicro X8SIL-F with an Intel® Xeon® CPU X3450 @ 2.67GHz with 16GB of RAM First up is me querying the switch and it stating 4x10GB detected and enabled for "MT25408" which is apparently what the drivers is reporting the name to be. In reality this should be the hostname but I'm not picky. This is my initial test from a Windows Server 2012 R2 server with 4x256GB SSD in RAID 10 to the XPEnology box. Yes sir 1GB! I should be able to get close to 3-4GB/sec once I can get a RAM drive setup on my XPEnology. I'm having some difficulty getting it done so if anyone has some tips let me know. Here are a few graphs of the interface and what it's showing. As you can see my writes to disk are the bottleneck. If that wasn't enough here is my system utilization when all of this is going on which is pretty low. Next steps would be to create a RAM drive on the windows machine and xpenology and test again to see what the upper limits are on this setup. Quote Link to comment Share on other sites More sharing options...
Trantor Posted May 23, 2015 Share #2 Posted May 23, 2015 Woaw, Amazing speed ! I'm glad to see my driver works Quote Link to comment Share on other sites More sharing options...
epicurean Posted July 1, 2015 Share #3 Posted July 1, 2015 Hi, I just purchased this Mellanox card http://cgi.ebay.com/ws/eBayISAPI.dll?Vi ... 1643534014 would very much like to know step by step, exactly how you enabled the drivers to allow the full function of this card. Much thanks!! Quote Link to comment Share on other sites More sharing options...
PnoT Posted July 1, 2015 Author Share #4 Posted July 1, 2015 Hi,I just purchased this Mellanox card http://cgi.ebay.com/ws/eBayISAPI.dll?Vi ... 1643534014 would very much like to know step by step, exactly how you enabled the drivers to allow the full function of this card. Much thanks!! Look at the README file in his zip as it tells you exactly what to do in order to get things working. I will say that upon upgrading to the latest build these modules don't seem to work anymore as my card is not being detected. When trying to insert the module manually: insmod /lib/modules/mlx4_ib.ko insmod: can't insert '/lib/modules/mlx4_ib.ko': invalid module format Quote Link to comment Share on other sites More sharing options...
hnamanh Posted July 1, 2015 Share #5 Posted July 1, 2015 What's the build number? Quote Link to comment Share on other sites More sharing options...
PnoT Posted July 3, 2015 Author Share #6 Posted July 3, 2015 What's the build number? I'm on the latest XPEnoboot 5.2-5565.2 (07/06/2015) Quote Link to comment Share on other sites More sharing options...
linenoise Posted July 3, 2015 Share #7 Posted July 3, 2015 Any update? Have you got it working on the current build? Which version did you have it working on? I have 2 Mellanox MHGA28-XTC I would like to get working. Quote Link to comment Share on other sites More sharing options...
Trantor Posted July 4, 2015 Share #8 Posted July 4, 2015 DSM 5.2 and DSM 5.1 have different kernel. I recompile the infiniband/mellanox drivers for DSM 5.2 (kernel 3.10.35) https://download.xpenology.fr/drivers/i ... nd-5.2.zip Enjoy Quote Link to comment Share on other sites More sharing options...
PnoT Posted July 4, 2015 Author Share #9 Posted July 4, 2015 DSM 5.2 and DSM 5.1 have different kernel. I recompile the infiniband/mellanox drivers for DSM 5.2 (kernel 3.10.35) https://download.xpenology.fr/drivers/i ... nd-5.2.zip Enjoy Thank you! Quote Link to comment Share on other sites More sharing options...
epicurean Posted July 5, 2015 Share #10 Posted July 5, 2015 DSM 5.2 and DSM 5.1 have different kernel. I recompile the infiniband/mellanox drivers for DSM 5.2 (kernel 3.10.35) https://download.xpenology.fr/drivers/i ... nd-5.2.zip Enjoy Thanks Trantor! Quote Link to comment Share on other sites More sharing options...
PnoT Posted July 6, 2015 Author Share #11 Posted July 6, 2015 I can get the InfiniBand portion to work properly but it's not recognizing 10GB Ethernet on the other port. The device is showing up from the command line but not in the GUI, which it was before, and the protocol remains "UNSPEC" instead of Ethernet. Any ideas? Quote Link to comment Share on other sites More sharing options...
Trantor Posted July 11, 2015 Share #12 Posted July 11, 2015 I can get the InfiniBand portion to work properly but it's not recognizing 10GB Ethernet on the other port. The device is showing up from the command line but not in the GUI, which it was before, and the protocol remains "UNSPEC" instead of Ethernet. Any ideas? Check with lsmod if eveyone modules are properly loaded. Quote Link to comment Share on other sites More sharing options...
Neohunter Posted July 14, 2015 Share #13 Posted July 14, 2015 Hello My question is not quite InfiniBand - but about working Mellanox MHGH29-XTC (mode Ethernet 10GbE) As I understand it for running Ethernet 10GbE in XPEnoBoot 5.2-5565.2 all drivers (mlx4_core mlx4_en) are built, BUT! On boot I see (in dmesg) adapter say - command 0xc failed: fw status = 0x40 This problem what driver non initialize adapter as Ethernet 10GbE (see https://community.mellanox.com/thread/1092) It is necessary to initialize the driver stage is when loading DSM pass parameters - mlx4_core port_type_array = "2,2" The question is how to do it? Now I have added (on the advice of a reputable Trantor) in the / etc / rc and /etc.default/rc - NET_DRIVERS = "dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx4_core mlx4_en" But how to convey DSM boot option port_type_array = "2,2" - the driver mlx4_core? Please tell me how to do it? p.s. By the way, I checked the adapter job - after the download DSM If unload the driver - rmmod mlx4_en mlx4_core and then upload insmod mlx4_core.ko port_type_array = "2,2" insmod mlx4_en.ko So on both adapters, the green LED However, the network interface and the console DSM, and does not appear in the GUI I think that if the driver is properly initialized at boot the DSM interface should appear in the console and GUI Quote Link to comment Share on other sites More sharing options...
Trantor Posted July 14, 2015 Share #14 Posted July 14, 2015 @Neohunter : Did you try to swith mode to eth like siszlai said ? you can also flip from IB to Eth by setting this module parameter, write directly in procfs echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2 echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1 If I understand correctly, the IB stack of the mellanox driver is useless in this setup ? PnoT and you only use the ethernet part of your adapter. I compile the latest mlx4 driver form mellanox website (v3.0 Feb 2014 instead of the old v2.0 Dec 2011 currently embedded) without ib part. mlx4 driver : https://download.xpenology.fr/drivers/mlx_v3.0.7z This driver depend on mlx_compat that is not compatible with compat driver. You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers) Then copy *.ko in /lib/modules/ and add : mlx_compat mlx4_core_new mlx4_en_new to /etc/rc to load these drivers at boot. I'm forced to add _new to the name of mlx4_core and mlx4_en otherwise the bootloader replace them with the old version Hope you can get your card to works. Quote Link to comment Share on other sites More sharing options...
Neohunter Posted July 14, 2015 Share #15 Posted July 14, 2015 @Neohunter : Did you try to swith mode to eth like siszlai said ? you can also flip from IB to Eth by setting this module parameter, write directly in procfs echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2 echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1 No - because as i understand this command switch only in current session, after reboot its reset reinitialization by driver If I understand correctly, the IB stack of the mellanox driver is useless in this setup ? PnoT and you only use the ethernet part of your adapter. Yes - in my case i need only Ethernet part This driver depend on mlx_compat that is not compatible with compat driver. You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers) You mean what change string in syslinux.cfg like this LABEL xpenology MENU LABEL XPEnology DSM 5.2-5565 KERNEL /zImage APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat correct ? Quote Link to comment Share on other sites More sharing options...
Neohunter Posted July 14, 2015 Share #16 Posted July 14, 2015 @Neohunter : Did you try to swith mode to eth like siszlai said ? you can also flip from IB to Eth by setting this module parameter, write directly in procfs echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2 echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1 No - because as i understand this command switch only in current session, after reboot its reset reinitialization by driver If I understand correctly, the IB stack of the mellanox driver is useless in this setup ? PnoT and you only use the ethernet part of your adapter. Yes - in my case i need only Ethernet part This driver depend on mlx_compat that is not compatible with compat driver. You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers) You mean what change string in syslinux.cfg like this LABEL xpenology MENU LABEL XPEnology DSM 5.2-5565 KERNEL /zImage APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat correct ? Quote Link to comment Share on other sites More sharing options...
Trantor Posted July 14, 2015 Share #17 Posted July 14, 2015 You mean what change string in syslinux.cfg like this LABEL xpenology MENU LABEL XPEnology DSM 5.2-5565 KERNEL /zImage APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat correct ? Yes exactly to prevent compat module to load at boot Quote Link to comment Share on other sites More sharing options...
Neohunter Posted July 14, 2015 Share #18 Posted July 14, 2015 You mean what change string in syslinux.cfg like this LABEL xpenology MENU LABEL XPEnology DSM 5.2-5565 KERNEL /zImage APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat correct ? Yes exactly to prevent compat module to load at boot Hm its very strange result - but step by step what i do I copy mlx_compat mlx4_core_new mlx4_en_new in /lib/modules/ change /etc/rc & /etc.defaults/rc NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx_compat mlx4_core_new mlx4_en_new" add APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat in syslinux.cfg and reboot 1 on DSM boot in console i see many failed on load some drivers 2 on the end of boot i see 2 string updated mlx4_core - ok updated mlx4_en - ok and load on boot old mlx4_core & mlx4_en but not mlx4_core_new mlx4_en_new - why? in dmg the same error [ 7.137757] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40 [ 7.137887] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40 [ 7.150187] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011) After that i delete old mlx4_core & mlx4_en in lib/modules and rename mlx4_core_new mlx4_en_new to mlx4_core & mlx4_en change /etc/rc & /etc.defaults/rc NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx_compat mlx4_core mlx4_en" and reboot but loaded the same old mlx4_core & mlx4_en - why ? i delete old driver where loader find it ? ps update info about old version mlx4_core & mlx4_en if i use it on boot and after that in console i enter rmmod mlx4_en mlx4_core insmod mlx4_core.ko port_type_array="2,2" insmod mlx4_en.ko and than /etc/rc.network restart i see 2 new ethernet interface (in console and in GUI) So may be we still use old version driver - because its work but not automatically in boot one problem how add port_type_array="2,2" parameter on boot DSM ? Quote Link to comment Share on other sites More sharing options...
epicurean Posted August 11, 2015 Share #19 Posted August 11, 2015 Does the infiniband 2 drivers work with the latest xpenoboot ? Any special procedure to get them working? Much thanks Quote Link to comment Share on other sites More sharing options...
Trantor Posted September 6, 2015 Share #20 Posted September 6, 2015 Does the infiniband 2 drivers work with the latest xpenoboot ?Any special procedure to get them working? Much thanks I realise that the DSM 5.1 bootloader load mlx4_core and mlx4_en but not the DSM 5.2 bootloader. I think this is why PnoT and Neohunter can't get thet ethernet part of the card working. The link for the mellanox infiniband/ethernet drivers for DSM 5.2 is the same : https://download.xpenology.fr/drivers/i ... nd-5.2.zip but you have to follow this: - Copy *.ko files to /lib/modules/ - Add these modules to the NET_DRIVERS variable (line 310) in /etc/rc and /etc.default/rc for automatic load Modules: ib_core ib_mad ib_mthca ib_addr ib_sa ib_cm ib_ipoib iw_cm rdma_cm xprtrdma svcrdma mlx4_core mlx4_ib mlx4_en Example: NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx ib_core ib_mad ib_mthca ib_addr ib_sa ib_cm ib_ipoib iw_cm rdma_cm xprtrdma svcrdma mlx4_core mlx4_ib mlx4_en" After reboot you can check if everything is loaded: #> lsmod | grep mlx mlx4_en 67584 0 mlx4_ib 104330 0 mlx4_core 169852 2 mlx4_en,mlx4_ib ib_sa 19338 4 mlx4_ib,rdma_cm,ib_ipoib,ib_cm ib_mad 34788 4 mlx4_ib,ib_cm,ib_sa,ib_mthca ib_core 47092 10 mlx4_ib,svcrdma,xprtrdma,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_mthca,ib_mad Quote Link to comment Share on other sites More sharing options...
epicurean Posted September 7, 2015 Share #21 Posted September 7, 2015 Hi Trantor I must have done something wrong because now I cannot even get my ethernet network ports going. I changed the 2 rc files as stated only the mlx_en and mlx_ib showed up when I did the "lsmod | grep mlx " command. How do I rectify this? anyone has a spare rc files to help me? Quote Link to comment Share on other sites More sharing options...
epicurean Posted September 7, 2015 Share #22 Posted September 7, 2015 Help, am desperate. I cannot get my standard network ports enabled again after the amendments made to the 2 rc files Quote Link to comment Share on other sites More sharing options...
Trantor Posted September 9, 2015 Share #23 Posted September 9, 2015 Help, am desperate. I cannot get my standard network ports enabled again after the amendments made to the 2 rc files When you say standard network you mean classic intel/realtek/broacom or your mellanox adapter ? Can't provide you original file right away but the default line 310 is : NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx" Hope this will fix your issue. Quote Link to comment Share on other sites More sharing options...
epicurean Posted September 12, 2015 Share #24 Posted September 12, 2015 Hi Trantor, It was my onboard intel drivers for the classic NIC, not the mellanox I managed to revert back to the default line.. but it simply would not recognise the PCI Intel 2 port NIC card (which it used to). Is there a way to restart the recognition of all network ports again? This is even after i restarted the xpenology box a few times. Also almost all my drives in the volumes have the "system partition error" warning. How do I get rid of them? Did not expect so much grief from simply trying to enable 40GB in the xpenology..but I will persevere. Quote Link to comment Share on other sites More sharing options...
Trantor Posted September 14, 2015 Share #25 Posted September 14, 2015 Hi Trantor, It was my onboard intel drivers for the classic NIC, not the mellanox I managed to revert back to the default line.. but it simply would not recognise the PCI Intel 2 port NIC card (which it used to). Is there a way to restart the recognition of all network ports again? This is even after i restarted the xpenology box a few times. Also almost all my drives in the volumes have the "system partition error" warning. How do I get rid of them? Did not expect so much grief from simply trying to enable 40GB in the xpenology..but I will persevere. Strange because intel drivers (e1000e and igb) are loaded at boot by the bootloader. Use lsmod to confirm they are loaded and if not use insmod to manually load them. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.