PnoT

40GB InfiniBand Support at work on my Xpenology box!

Recommended Posts

First of all I'd like to thank Trantor for compiling the modules here:

:ugeek: You've seriously made my week... thank you!

 

Here are some screenies of it in action and my system specs: Supermicro X8SIL-F with an Intel® Xeon® CPU X3450 @ 2.67GHz with 16GB of RAM

 

First up is me querying the switch and it stating 4x10GB detected and enabled for "MT25408" which is apparently what the drivers is reporting the name to be. In reality this should be the hostname but I'm not picky.

 

MXYNxeS.jpg

 

This is my initial test from a Windows Server 2012 R2 server with 4x256GB SSD in RAID 10 to the XPEnology box.

 

bqjWJCt.jpg

 

Yes sir 1GB! I should be able to get close to 3-4GB/sec once I can get a RAM drive setup on my XPEnology. I'm having some difficulty getting it done so if anyone has some tips let me know.

 

Here are a few graphs of the interface and what it's showing. As you can see my writes to disk are the bottleneck.

 

aaBGSKR.jpg

z5bzuCM.jpg

 

If that wasn't enough here is my system utilization when all of this is going on which is pretty low.

 

wCB2x3l.jpg

 

Next steps would be to create a RAM drive on the windows machine and xpenology and test again to see what the upper limits are on this setup.

Share this post


Link to post
Share on other sites
Hi,

I just purchased this Mellanox card

http://cgi.ebay.com/ws/eBayISAPI.dll?Vi ... 1643534014

 

would very much like to know step by step, exactly how you enabled the drivers to allow the full function of this card. Much thanks!!

 

Look at the README file in his zip as it tells you exactly what to do in order to get things working.

 

I will say that upon upgrading to the latest build these modules don't seem to work anymore as my card is not being detected.

 

When trying to insert the module manually:

 

insmod /lib/modules/mlx4_ib.ko

insmod: can't insert '/lib/modules/mlx4_ib.ko': invalid module format

Share this post


Link to post
Share on other sites
What's the build number?

 

I'm on the latest XPEnoboot 5.2-5565.2 (07/06/2015)

Share this post


Link to post
Share on other sites

Any update? Have you got it working on the current build? Which version did you have it working on? I have 2 Mellanox MHGA28-XTC I would like to get working.

Share this post


Link to post
Share on other sites

I can get the InfiniBand portion to work properly but it's not recognizing 10GB Ethernet on the other port. The device is showing up from the command line but not in the GUI, which it was before, and the protocol remains "UNSPEC" instead of Ethernet.

 

Any ideas?

Share this post


Link to post
Share on other sites
I can get the InfiniBand portion to work properly but it's not recognizing 10GB Ethernet on the other port. The device is showing up from the command line but not in the GUI, which it was before, and the protocol remains "UNSPEC" instead of Ethernet.

 

Any ideas?

 

Check with lsmod if eveyone modules are properly loaded.

Share this post


Link to post
Share on other sites

Hello

My question is not quite InfiniBand - but about working Mellanox MHGH29-XTC (mode Ethernet 10GbE) As I understand it for running Ethernet 10GbE in XPEnoBoot 5.2-5565.2 all drivers (mlx4_core mlx4_en) are built, BUT!

On boot I see (in dmesg) adapter say - command 0xc failed: fw status = 0x40

This problem what driver non initialize adapter as Ethernet 10GbE (see https://community.mellanox.com/thread/1092)

It is necessary to initialize the driver stage is when loading DSM pass parameters - mlx4_core port_type_array = "2,2"

The question is how to do it?

 

Now I have added (on the advice of a reputable Trantor) in the / etc / rc and /etc.default/rc -

NET_DRIVERS = "dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx4_core mlx4_en"

But how to convey DSM boot option port_type_array = "2,2" - the driver

mlx4_core?

 

Please tell me how to do it?

 

p.s.

By the way, I checked the adapter job - after the download DSM

If unload the driver - rmmod mlx4_en mlx4_core

and then upload

insmod mlx4_core.ko port_type_array = "2,2"

insmod mlx4_en.ko

So on both adapters, the green LED

However, the network interface and the console DSM, and does not appear in the GUI

I think that if the driver is properly initialized at boot the DSM interface should appear in the console and GUI

Share this post


Link to post
Share on other sites

@Neohunter : Did you try to swith mode to eth like siszlai said ?

 

you can also flip from IB to Eth by setting this module parameter, write directly in procfs

 

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1

 

If I understand correctly, the IB stack of the mellanox driver is useless in this setup ?

PnoT and you only use the ethernet part of your adapter.

 

I compile the latest mlx4 driver form mellanox website (v3.0 Feb 2014 instead of the old v2.0 Dec 2011 currently embedded) without ib part.

 

mlx4 driver : https://download.xpenology.fr/drivers/mlx_v3.0.7z

 

This driver depend on mlx_compat that is not compatible with compat driver.

You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers)

Then copy *.ko in /lib/modules/ and add : mlx_compat mlx4_core_new mlx4_en_new to /etc/rc to load these drivers at boot.

 

I'm forced to add _new to the name of mlx4_core and mlx4_en otherwise the bootloader replace them with the old version :sad:

 

Hope you can get your card to works.

Share this post


Link to post
Share on other sites
@Neohunter : Did you try to swith mode to eth like siszlai said ?

 

you can also flip from IB to Eth by setting this module parameter, write directly in procfs

 

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1

No - because as i understand this command switch only in current session, after reboot its reset reinitialization by driver

If I understand correctly, the IB stack of the mellanox driver is useless in this setup ?

PnoT and you only use the ethernet part of your adapter.

Yes - in my case i need only Ethernet part

 

This driver depend on mlx_compat that is not compatible with compat driver.

You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers)

You mean what change string in syslinux.cfg like this

LABEL xpenology

MENU LABEL XPEnology DSM 5.2-5565

KERNEL /zImage

APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat

correct ?

Share this post


Link to post
Share on other sites
@Neohunter : Did you try to swith mode to eth like siszlai said ?

 

you can also flip from IB to Eth by setting this module parameter, write directly in procfs

 

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2

echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1

No - because as i understand this command switch only in current session, after reboot its reset reinitialization by driver

If I understand correctly, the IB stack of the mellanox driver is useless in this setup ?

PnoT and you only use the ethernet part of your adapter.

Yes - in my case i need only Ethernet part

 

This driver depend on mlx_compat that is not compatible with compat driver.

You need to add rmmod=compat in your syslinux.cfg in order to prevent XPEnoboot to load it (no other driver use it. it's maybe for some wifi drivers)

You mean what change string in syslinux.cfg like this

LABEL xpenology

MENU LABEL XPEnology DSM 5.2-5565

KERNEL /zImage

APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat

correct ?

Share this post


Link to post
Share on other sites

You mean what change string in syslinux.cfg like this

LABEL xpenology

MENU LABEL XPEnology DSM 5.2-5565

KERNEL /zImage

APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat

correct ?

 

Yes exactly to prevent compat module to load at boot

Share this post


Link to post
Share on other sites

You mean what change string in syslinux.cfg like this

LABEL xpenology

MENU LABEL XPEnology DSM 5.2-5565

KERNEL /zImage

APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat

correct ?

 

Yes exactly to prevent compat module to load at boot

 

Hm its very strange result - but step by step what i do

 

I copy mlx_compat mlx4_core_new mlx4_en_new in /lib/modules/

change /etc/rc & /etc.defaults/rc

NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx_compat mlx4_core_new mlx4_en_new"

add APPEND root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3615xs sn=B3J4N01003 vid=0x0EA0 pid=0x2168 loglevel=0 vga=0x305 rmmod=compat in syslinux.cfg

and reboot

1 on DSM boot in console i see many failed on load some drivers

2 on the end of boot i see 2 string

updated mlx4_core - ok

updated mlx4_en - ok

and load on boot old mlx4_core & mlx4_en but not mlx4_core_new mlx4_en_new - why?

in dmg the same error

[ 7.137757] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40

[ 7.137887] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40

[ 7.150187] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011)

After that i delete old mlx4_core & mlx4_en in lib/modules and rename mlx4_core_new mlx4_en_new to mlx4_core & mlx4_en

change /etc/rc & /etc.defaults/rc

NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx mlx_compat mlx4_core mlx4_en"

and reboot

but loaded the same old mlx4_core & mlx4_en - why ? i delete old driver where loader find it ?

 

ps

update info about old version mlx4_core & mlx4_en

if i use it on boot and after that in console i enter

rmmod mlx4_en mlx4_core

insmod mlx4_core.ko port_type_array="2,2"

insmod mlx4_en.ko

and than /etc/rc.network restart

i see 2 new ethernet interface (in console and in GUI)

So may be we still use old version driver - because its work but not automatically in boot

one problem how add port_type_array="2,2" parameter on boot DSM ?

Share this post


Link to post
Share on other sites

Does the infiniband 2 drivers work with the latest xpenoboot ?

Any special procedure to get them working?

Much thanks

Share this post


Link to post
Share on other sites
Does the infiniband 2 drivers work with the latest xpenoboot ?

Any special procedure to get them working?

Much thanks

 

I realise that the DSM 5.1 bootloader load mlx4_core and mlx4_en but not the DSM 5.2 bootloader.

I think this is why PnoT and Neohunter can't get thet ethernet part of the card working.

 

The link for the mellanox infiniband/ethernet drivers for DSM 5.2 is the same : https://download.xpenology.fr/drivers/i ... nd-5.2.zip

but you have to follow this:

 

- Copy *.ko files to /lib/modules/

- Add these modules to the NET_DRIVERS variable (line 310) in /etc/rc and /etc.default/rc for automatic load

 

Modules:

ib_core ib_mad ib_mthca ib_addr ib_sa ib_cm ib_ipoib iw_cm rdma_cm xprtrdma svcrdma mlx4_core mlx4_ib mlx4_en

 

Example:

NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx ib_core ib_mad ib_mthca ib_addr ib_sa ib_cm ib_ipoib iw_cm rdma_cm xprtrdma svcrdma mlx4_core mlx4_ib mlx4_en"

 

After reboot you can check if everything is loaded:

#> lsmod | grep mlx
mlx4_en                67584  0
mlx4_ib               104330  0
mlx4_core             169852  2 mlx4_en,mlx4_ib
ib_sa                  19338  4 mlx4_ib,rdma_cm,ib_ipoib,ib_cm
ib_mad                 34788  4 mlx4_ib,ib_cm,ib_sa,ib_mthca
ib_core                47092 10 mlx4_ib,svcrdma,xprtrdma,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_mthca,ib_mad

Share this post


Link to post
Share on other sites

Hi Trantor

I must have done something wrong because now I cannot even get my ethernet network ports going.

I changed the 2 rc files as stated

 

only the mlx_en and mlx_ib showed up when I did the "lsmod | grep mlx " command.

 

How do I rectify this?

anyone has a spare rc files to help me?

Share this post


Link to post
Share on other sites
Help, am desperate. I cannot get my standard network ports enabled again after the amendments made to the 2 rc files

 

When you say standard network you mean classic intel/realtek/broacom or your mellanox adapter ?

Can't provide you original file right away but the default line 310 is : NET_DRIVERS="dca e1000e i2c-algo-bit igb be2net ixgbe tn40xx"

Hope this will fix your issue.

Share this post


Link to post
Share on other sites

Hi Trantor,

It was my onboard intel drivers for the classic NIC, not the mellanox

I managed to revert back to the default line.. but it simply would not recognise the PCI Intel 2 port NIC card (which it used to). Is there a way to restart the recognition of all network ports again?

This is even after i restarted the xpenology box a few times. Also almost all my drives in the volumes have the "system partition error" warning. How do I get rid of them?

 

Did not expect so much grief from simply trying to enable 40GB in the xpenology..but I will persevere.:smile:

Share this post


Link to post
Share on other sites
Hi Trantor,

It was my onboard intel drivers for the classic NIC, not the mellanox

I managed to revert back to the default line.. but it simply would not recognise the PCI Intel 2 port NIC card (which it used to). Is there a way to restart the recognition of all network ports again?

This is even after i restarted the xpenology box a few times. Also almost all my drives in the volumes have the "system partition error" warning. How do I get rid of them?

 

Did not expect so much grief from simply trying to enable 40GB in the xpenology..but I will persevere.:smile:

 

Strange because intel drivers (e1000e and igb) are loaded at boot by the bootloader.

Use lsmod to confirm they are loaded and if not use insmod to manually load them.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now