Jump to content
XPEnology Community

Mellanox MHQH29B-XTR


RacerX
 Share

Recommended Posts

Yesterday I upgraded my N54L to DSM 5.2-5644 Update 5.

It works fine, thanks....

 

I had a little time to test out my Mellanox MHQH29B-XTR.

 

lscpi

00:00.0 Class 0600: Device 1022:9601

00:01.0 Class 0604: Device 103c:9602

00:02.0 Class 0604: Device 1022:9603

00:06.0 Class 0604: Device 1022:9606

00:11.0 Class 0106: Device 1002:4391 (rev 40)

00:12.0 Class 0c03: Device 1002:4397

00:12.2 Class 0c03: Device 1002:4396

00:13.0 Class 0c03: Device 1002:4397

00:13.2 Class 0c03: Device 1002:4396

00:14.0 Class 0c05: Device 1002:4385 (rev 42)

00:14.3 Class 0601: Device 1002:439d (rev 40)

00:14.4 Class 0604: Device 1002:4384 (rev 40)

00:16.0 Class 0c03: Device 1002:4397

00:16.2 Class 0c03: Device 1002:4396

00:18.0 Class 0600: Device 1022:1200

00:18.1 Class 0600: Device 1022:1201

00:18.2 Class 0600: Device 1022:1202

00:18.3 Class 0600: Device 1022:1203

00:18.4 Class 0600: Device 1022:1204

01:05.0 Class 0300: Device 1002:9712

02:00.0 Class 0c06: Device 15b3:673c (rev b0)

03:00.0 Class 0200: Device 14e4:165b (rev 10)

 

dmesg|grep mlx4

[ 4.576776] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)

[ 4.576781] mlx4_core: Initializing 0000:02:00.0

[ 6.849435] mlx4_core 0000:02:00.0: irq 43 for MSI/MSI-X

[ 6.849445] mlx4_core 0000:02:00.0: irq 44 for MSI/MSI-X

[ 6.849451] mlx4_core 0000:02:00.0: irq 45 for MSI/MSI-X

[ 6.849457] mlx4_core 0000:02:00.0: irq 46 for MSI/MSI-X

[ 6.849463] mlx4_core 0000:02:00.0: irq 47 for MSI/MSI-X

[ 6.849470] mlx4_core 0000:02:00.0: irq 48 for MSI/MSI-X

[ 6.849475] mlx4_core 0000:02:00.0: irq 49 for MSI/MSI-X

[ 6.849481] mlx4_core 0000:02:00.0: irq 50 for MSI/MSI-X

[ 6.849487] mlx4_core 0000:02:00.0: irq 51 for MSI/MSI-X

[ 6.849492] mlx4_core 0000:02:00.0: irq 52 for MSI/MSI-X

[ 6.872762] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40

[ 6.872931] mlx4_core 0000:02:00.0: command 0xc failed: fw status = 0x40

[ 6.882538] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011)

[ 6.882679] mlx4_en 0000:02:00.0: UDP RSS is not supported on this device.

 

lsmod

Module Size Used by Tainted: P

cifs 247268 0

udf 78298 0

isofs 31935 0

loop 17688 0

nf_conntrack_ipv6 6267 0

nf_defrag_ipv6 22249 1 nf_conntrack_ipv6

ip6table_filter 1236 1

ip6_tables 16272 1 ip6table_filter

xt_geoip 2934 0

xt_recent 8132 0

xt_iprange 1448 0

xt_limit 1721 0

xt_state 1135 0

xt_multiport 1630 0

xt_LOG 12099 0

nf_conntrack_ipv4 11115 0

nf_defrag_ipv4 1147 1 nf_conntrack_ipv4

iptable_filter 1296 1

ip_tables 15570 1 iptable_filter

xt_length 1124 4

xt_tcpudp 2311 4

nf_conntrack 52846 3 nf_conntrack_ipv6,xt_state,nf_conntrack_ipv4

x_tables 15527 13 ip6table_filter,ip6_tables,xt_geoip,xt_recent,xt _iprange,xt_limit,xt_state,xt_multiport,xt_LOG,iptable_filter,ip_tables,xt_length,xt_tcpudp

hid_generic 1057 0

usbhid 26686 0

hid 82504 2 hid_generic,usbhid

usblp 10674 0

usb_storage 46383 1

oxu210hp_hcd 24469 0

bromolow_synobios 42769 0

btrfs 788682 0

synoacl_vfs 17043 1

zlib_deflate 20180 1 btrfs

hfsplus 91979 0

md4 3337 0

hmac 2793 0

mlx_compat 5376 0

tn40xx 24739 0

fuse 74004 0

vfat 10009 1

fat 50784 1 vfat

glue_helper 3914 0

lrw 3309 0

gf128mul 5346 1 lrw

ablk_helper 1684 0

arc4 1847 0

rng_core 3520 0

cpufreq_conservative 6240 0

cpufreq_powersave 862 0

cpufreq_performance 866 2

cpufreq_ondemand 8039 0

acpi_cpufreq 6982 0

mperf 1107 1 acpi_cpufreq

processor 26471 1 acpi_cpufreq

cpufreq_stats 2985 0

freq_table 2380 3 cpufreq_ondemand,acpi_cpufreq,cpufreq_stats

dm_snapshot 26708 0

crc_itu_t 1235 1 udf

quota_v2 3783 2

quota_tree 7970 1 quota_v2

psnap 1717 0

p8022 979 0

llc 3441 2 psnap,p8022

sit 14446 0

tunnel4 2061 1 sit

ip_tunnel 11572 1 sit

zram 8191 2

r8152 49404 0

asix 20370 0

ax88179_178a 16864 0

usbnet 18137 2 asix,ax88179_178a

etxhci_hcd 84833 0

xhci_hcd 84493 0

ehci_pci 3504 0

ehci_hcd 39963 1 ehci_pci

uhci_hcd 22668 0

ohci_hcd 21168 0

usbcore 176728 14 usbhid,usblp,usb_storage,oxu210hp_hcd,r8152,asix ,ax88179_178a,usbnet,etxhci_hcd,xhci_hcd,ehci_pci,ehci_hcd,uhci_hcd,ohci_hcd

usb_common 1488 1 usbcore

hptiop 15828 0

3w_sas 21786 0

3w_9xxx 33873 0

mvsas 51605 0

hpsa 50480 0

arcmsr 28228 0

pm80xx 134800 0

aic94xx 72946 0

megaraid_sas 138070 0

megaraid_mbox 29835 0

megaraid_mm 7816 1 megaraid_mbox

nvme 40947 0

mpt3sas 220677 0

mptsas 39249 0

mptspi 13047 0

mptscsih 18838 2 mptsas,mptspi

mptbase 62504 3 mptsas,mptspi,mptscsih

scsi_transport_spi 19551 1 mptspi

sg 25017 0

ata_piix 24664 0

sata_uli 3004 0

sata_svw 4453 0

sata_qstor 5612 0

sata_sis 3884 0

pata_sis 10858 1 sata_sis

stex 15005 0

sata_sx4 9284 0

sata_promise 10799 0

sata_nv 20623 0

sata_via 7803 0

sata_sil 7783 0

pdc_adma 5868 0

pata_via 8827 0

iscsi_tcp 8897 0

libiscsi_tcp 12850 1 iscsi_tcp

libiscsi 35195 2 iscsi_tcp,libiscsi_tcp

enic 54982 0

qlge 79283 0

qlcnic 214986 0

qla3xxx 36934 0

netxen_nic 98970 0

mlx4_en 67584 0

mlx4_core 169852 1 mlx4_en

cxgb4 115680 0

cxgb3 134726 0

cnic 71038 0

ipv6 303225 49 nf_conntrack_ipv6,nf_defrag_ipv6,sit

bnx2x 1430214 0

bna 122380 0

be2net 89072 0

sis900 20811 0

sis190 17297 0

jme 34677 0

atl1e 28011 0

atl1c 33596 0

atl2 23124 0

atl1 30315 0

alx 26373 0

sky2 47242 0

skge 38482 0

via_velocity 30007 0

crc_ccitt 1235 1 via_velocity

via_rhine 21014 0

r8101 122784 0

r8168 318247 0

r8169 30519 0

8139too 18685 0

8139cp 20694 0

tg3 171348 0

broadcom 7174 0

b44 26983 0

bnx2 189638 0

ssb 38587 1 b44

uio 7592 1 cnic

forcedeth 55647 0

i40e 249005 0

ixgbe 246171 0

ixgb 37937 0

igb 178198 0

ioatdma 44390 0

e1000e 168235 0

e1000 100754 0

e100 29844 0

dca 4600 3 ixgbe,igb,ioatdma

pcnet32 31251 0

amd8111e 16602 0

mdio 3365 3 cxgb3,bnx2x,alx

mii 3803 15 r8152,asix,ax88179_178a,usbnet,sis900,sis190,jme ,atl1,via_rhine,8139too,8139cp,b44,e100,pcnet32,amd8111e

evdev 9156 0

button 4320 0

thermal_sys 18275 1 processor

compat 4529 1 mlx_compat

cryptd 7040 1 ablk_helper

ecryptfs 75063 0

sha512_generic 4976 0

sha256_generic 9884 0

sha1_generic 2206 0

ecb 1849 0

aes_x86_64 7239 0

authenc 6600 0

des_generic 15915 0

libcrc32c 906 1 bnx2x

ansi_cprng 3445 0

cts 3968 0

md5 2153 0

cbc 2448 0

 

At XPEnoboot boot

Loading Module Megaraid_SAS fails it's red

All other Modules load correctly in green

 

The Mellanox MHQH29B-XTR is the only card in the N54L.

Starting to think that the Mac Address for the Mellanox MHQH29B-XTR needs to be written into the firmware.

Link to comment
Share on other sites

A little more detail

 

Library> lspci -v

00:00.0 Class 0600: Device 1022:9601

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 0

Capabilities: [c4] HyperTransport: Slave or Primary Interface

Capabilities: [54] HyperTransport: UnitID Clumping

Capabilities: [40] HyperTransport: Retry Mode

Capabilities: [9c] HyperTransport: #1a

Capabilities: [f8] HyperTransport: #1c

 

00:01.0 Class 0604: Device 103c:9602

Flags: bus master, 66MHz, medium devsel, latency 64

Bus: primary=00, secondary=01, subordinate=01, sec-latency=64

I/O behind bridge: 0000e000-0000efff

Memory behind bridge: fe600000-fe7fffff

Prefetchable memory behind bridge: 00000000f0000000-00000000f7ffffff

Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+

Capabilities: [b0] Subsystem: Device 103c:1609

 

00:02.0 Class 0604: Device 1022:9603

Flags: bus master, fast devsel, latency 0

Bus: primary=00, secondary=02, subordinate=02, sec-latency=0

Memory behind bridge: fe800000-fe8fffff

Prefetchable memory behind bridge: 00000000fd800000-00000000fdffffff

Capabilities: [50] Power Management version 3

Capabilities: [58] Express Root Port (Slot+), MSI 00

Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit-

Capabilities: [b0] Subsystem: Device 103c:1609

Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+

Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

Capabilities: [110] Virtual Channel

Kernel driver in use: pcieport

 

00:06.0 Class 0604: Device 1022:9606

Flags: bus master, fast devsel, latency 0

Bus: primary=00, secondary=03, subordinate=03, sec-latency=0

Memory behind bridge: fe900000-fe9fffff

Capabilities: [50] Power Management version 3

Capabilities: [58] Express Root Port (Slot-), MSI 00

Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit-

Capabilities: [b0] Subsystem: Device 103c:1609

Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+

Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

Capabilities: [110] Virtual Channel

Kernel driver in use: pcieport

 

00:11.0 Class 0106: Device 1002:4391 (rev 40) (prog-if 01)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 42

I/O ports at d000

I/O ports at c000

I/O ports at b000

I/O ports at a000

I/O ports at 9000

Memory at fe5ffc00 (32-bit, non-prefetchable)

Capabilities: [50] MSI: Enable+ Count=1/8 Maskable- 64bit+

Capabilities: [70] SATA HBA v1.0

Capabilities: [a4] PCI Advanced Features

Kernel driver in use: ahci

 

00:12.0 Class 0c03: Device 1002:4397 (prog-if 10)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18

Memory at fe5fe000 (32-bit, non-prefetchable)

Kernel driver in use: ohci_hcd

 

00:12.2 Class 0c03: Device 1002:4396 (prog-if 20)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17

Memory at fe5ff800 (32-bit, non-prefetchable)

Capabilities: [c0] Power Management version 2

Capabilities: [e4] Debug port: BAR=1 offset=00e0

Kernel driver in use: ehci-pci

 

00:13.0 Class 0c03: Device 1002:4397 (prog-if 10)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18

Memory at fe5fd000 (32-bit, non-prefetchable)

Kernel driver in use: ohci_hcd

 

00:13.2 Class 0c03: Device 1002:4396 (prog-if 20)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17

Memory at fe5ff400 (32-bit, non-prefetchable)

Capabilities: [c0] Power Management version 2

Capabilities: [e4] Debug port: BAR=1 offset=00e0

Kernel driver in use: ehci-pci

 

00:14.0 Class 0c05: Device 1002:4385 (rev 42)

Flags: 66MHz, medium devsel

 

00:14.3 Class 0601: Device 1002:439d (rev 40)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 0

 

00:14.4 Class 0604: Device 1002:4384 (rev 40) (prog-if 01)

Flags: bus master, 66MHz, medium devsel, latency 64

Bus: primary=00, secondary=04, subordinate=04, sec-latency=64

 

00:16.0 Class 0c03: Device 1002:4397 (prog-if 10)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18

Memory at fe5fc000 (32-bit, non-prefetchable)

Kernel driver in use: ohci_hcd

 

00:16.2 Class 0c03: Device 1002:4396 (prog-if 20)

Subsystem: Device 103c:1609

Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17

Memory at fe5ff000 (32-bit, non-prefetchable)

Capabilities: [c0] Power Management version 2

Capabilities: [e4] Debug port: BAR=1 offset=00e0

Kernel driver in use: ehci-pci

 

00:18.0 Class 0600: Device 1022:1200

Flags: fast devsel

Capabilities: [80] HyperTransport: Host or Secondary Interface

 

00:18.1 Class 0600: Device 1022:1201

Flags: fast devsel

 

00:18.2 Class 0600: Device 1022:1202

Flags: fast devsel

 

00:18.3 Class 0600: Device 1022:1203

Flags: fast devsel

Capabilities: [f0] Secure device <?>

 

00:18.4 Class 0600: Device 1022:1204

Flags: fast devsel

 

01:05.0 Class 0300: Device 1002:9712

Subsystem: Device 103c:1609

Flags: bus master, fast devsel, latency 0, IRQ 10

Memory at f0000000 (32-bit, prefetchable)

I/O ports at e000

Memory at fe7f0000 (32-bit, non-prefetchable)

Memory at fe600000 (32-bit, non-prefetchable)

Expansion ROM at [disabled]

Capabilities: [50] Power Management version 3

Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+

 

02:00.0 Class 0c06: Device 15b3:673c (rev b0)

Subsystem: Device 15b3:0021

Flags: bus master, fast devsel, latency 0, IRQ 18

Memory at fe800000 (64-bit, non-prefetchable)

Memory at fd800000 (64-bit, prefetchable)

Capabilities: [40] Power Management version 3

Capabilities: [48] Vital Product Data

Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-

Capabilities: [60] Express Endpoint, MSI 00

Capabilities: [100] Alternative Routing-ID Interpretation (ARI)

Kernel driver in use: mlx4_core

 

03:00.0 Class 0200: Device 14e4:165b (rev 10)

Subsystem: Device 103c:705d

Flags: bus master, fast devsel, latency 0, IRQ 53

Memory at fe9f0000 (64-bit, non-prefetchable)

Capabilities: [48] Power Management version 3

Capabilities: [40] Vital Product Data

Capabilities: [60] Vendor Specific Information: Len=6c <?>

Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+

Capabilities: [cc] Express Endpoint, MSI 00

Capabilities: [100] Advanced Error Reporting

Capabilities: [13c] Virtual Channel

Capabilities: [160] Device Serial Number 9c-b6-54-ff-fe-0b-e6-32

Capabilities: [16c] Power Budgeting <?>

Kernel driver in use: tg3

 

Any thoughts would be helpful

Link to comment
Share on other sites

Some more Information. I removed the Mellanox MHQH29B-XTR from my N54L. To get a little more info without messing up my existing xpnology setup. I installed the Mellanox MHQH29B-XTR in a WIndows 10 64bit PC. The operating system found the card right out of the box in the device manager. There are two tools that need to downloaded

 

MLNX_VPI_WinOF-5_10_All_win2012R2_x64 and WinMFT_x64_3_8_0_56 for connectX 2 cards.

 

c:\Program Files\Mellanox\WinMFT>mst status
mt26428_pci_cr0
mt26428_pciconf0

 

mt26428_pci_cr0 is the name of the device you need this to update the firmware.

 

C:\Program Files\Mellanox\WinMFT>vstat.exe
hca_idx=0
       uplink={BUS=PCI_E Gen2, SPEED=5.0 Gbps, WIDTH=x8, CAPS=5.0*x8}
       MSI-X={ENABLED=1, SUPPORTED=256, GRANTED=4, ALL_MASKED=N}
       vendor_id=0x02c9
       vendor_part_id=26428
       hw_ver=0xb0
       fw_ver=2.07.0000
       PSID=HP_0180000009
       node_guid=0002:c903:0009:5eac
       num_phys_ports=2
               port=1
               port_guid=0002:c903:0009:5ead
               port_state=PORT_DOWN (1)
               link_speed=NA
               link_width=NA
               rate=NA
               port_phys_state=POLLING (2)
               active_speed=2.50 Gbps
               sm_lid=0x0000
               port_lid=0x0000
               port_lmc=0x0
               transport=IB
               max_mtu=4096 (5)
               active_mtu=4096 (5)
               GID[0]=fe80:0000:0000:0000:0002:c903:0009:5ead

               port=2
               port_guid=0002:c903:0009:5eae
               port_state=PORT_DOWN (1)
               link_speed=NA
               link_width=NA
               rate=NA
               port_phys_state=POLLING (2)
               active_speed=2.50 Gbps
               sm_lid=0x0000
               port_lid=0x0000
               port_lmc=0x0
               transport=IB
               max_mtu=4096 (5)
               active_mtu=4096 (5)
               GID[0]=fe80:0000:0000:0000:0002:c903:0009:5eae

 

Notice fw_ver=2.07.0000 this means the firmware is old. The current one is fw_ver=2.09.1000

also you can see the mac address for each port.

 

c:\Program Files\Mellanox\WinMFT>ibstat
CA 'ibv_device0'
       CA type:
       Number of ports: 2
       Firmware version: 2.7.0
       Hardware version: 0xb0
       Node GUID: 0x0002c90300095eac
       System image GUID: 0x0002c90300095eaf
   Port 1:
       State: Down
       Physical state: Polling
       Rate: 10
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c90300095ead
       Link layer: IB
       Transport: IB
   Port 2:
       State: Down
       Physical state: Polling
       Rate: 10
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c90300095eae
       Link layer: IB
       Transport: IB

 

This page helped me figure out how to flash the firmware on the card

 

ftp://ftp.supermicro.com/utility/.../Firmware/InfiniBand/How%20to%20Flash%20the%20IB%20firmware%20&%20Check%20FW%20Rev%20&%20Status-03.06.2015.pdf

 

The firmware file for Mellanox MHQH29B-XTR is fw-ConnectX2-rel-2_9_1000-MHQH29B-XTR_A1.bin

 

I tested the file before I flashed the Mellanox MHQH29B-XTR.

 

flint -d mt26428_pci_cr0 fw-ConnectX2-rel-2_9_1000-MHQH29B-XTR_A1.bin v

Then I flashed the Mellanox MHQH29B-XTR

flint -d mt26428_pci_cr0 -i fw-ConnectX2-rel-2_9_1000-MHQH29B-XTR_A1.bin -allow_psid_change -y b

reboot

 

C:\Program Files\Mellanox\WinMFT>vstat.exe

       hca_idx=0
       uplink={BUS=PCI_E Gen2, SPEED=5.0 Gbps, WIDTH=x8, CAPS=5.0*x8}
       MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=4, ALL_MASKED=N}
       vendor_id=0x02c9
       vendor_part_id=26428
       hw_ver=0xb0
       fw_ver=2.09.1000
       PSID=MT_0D80110009
       node_guid=0002:c903:0009:5eac
       num_phys_ports=2
               port=1
               port_guid=0002:c903:0009:5ead
               port_state=PORT_DOWN (1)
               link_speed=NA
               link_width=NA
               rate=NA
               port_phys_state=POLLING (2)
               active_speed=0.00 Gbps
               sm_lid=0x0000
               port_lid=0x0000
               port_lmc=0x0
               transport=IB
               max_mtu=4096 (5)
               active_mtu=4096 (5)
               GID[0]=fe80:0000:0000:0000:0002:c903:0009:5ead

               port=2
               port_guid=0002:c903:0009:5eae
               port_state=PORT_DOWN (1)
               link_speed=NA
               link_width=NA
               rate=NA
               port_phys_state=POLLING (2)
               active_speed=0.00 Gbps
               sm_lid=0x0000
               port_lid=0x0000
               port_lmc=0x0
               transport=IB
               max_mtu=4096 (5)
               active_mtu=4096 (5)
               GID[0]=fe80:0000:0000:0000:0002:c903:0009:5eae

 

The Mellanox MHQH29B-XTR is now has fw_ver=2.09.1000

 

These tools are available for Ubuntu, SLES, RHEL/CENTOS, OEL, FEDORA, DEBIAN, and CITRIX XENSERVER HOST

Link to comment
Share on other sites

I had some more time today to work on the Mellanox MHQH29B-XTR. I put the card into the N54L and tested some more

 

Lspci -v
02:00.0 Class 0c06: Device 15b3:673c (rev b0)
       Subsystem: Device 15b3:0048
       Flags: bus master, fast devsel, latency 0, IRQ 18
       Memory at fe800000 (64-bit, non-prefetchable) [size=1M]
       Memory at fd800000 (64-bit, prefetchable) [size=8M]
       Capabilities: [40] Power Management version 3
       Capabilities: [48] Vital Product Data
       Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
       Capabilities: [60] Express Endpoint, MSI 00
       Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
       Capabilities: [148] Device Serial Number 00-02-c9-03-00-09-5e-ac
       Kernel driver in use: mlx4_core

 

I installed installed the infiniband-5.2 ko files and changed /etc/rc and /etc.default/rc with vi

Then rebooted, I was hoping that this would straighten it out.....

 

lsmod | grep mlx
mlx_compat              5376  0
mlx4_ib               104330  0
ib_sa                  19338  4 mlx4_ib,rdma_cm,ib_ipoib,ib_cm
ib_mad                 34788  4 mlx4_ib,ib_cm,ib_sa,ib_mthca
ib_core                47092 10 mlx4_ib,svcrdma,xprtrdma,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_mthca,ib_mad
mlx4_en                67584  0
mlx4_core             169852  2 mlx4_ib,mlx4_en
compat                  4529  1 mlx_compat

 

Library> ifconfig
eth0      Link encap:Ethernet  HWaddr 9C:B6:54:0B:E6:32
         inet addr:192.168.1.11  Bcast:192.168.1.255  Mask:255.255.255.0
         inet6 addr: fe80::9eb6:54ff:fe0b:e632/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:7672 errors:0 dropped:0 overruns:0 frame:0
         TX packets:8736 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:2477146 (2.3 MiB)  TX bytes:5347852 (5.0 MiB)
         Interrupt:18

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:65536  Metric:1
         RX packets:13942 errors:0 dropped:0 overruns:0 frame:0
         TX packets:13942 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:1909832 (1.8 MiB)  TX bytes:1909832 (1.8 MiB)

 

Library> lsmod

Module                  Size  Used by    Tainted: P
cifs                  247268  0
udf                    78298  0
isofs                  31935  0
loop                   17688  0
nf_conntrack_ipv6       6267  0
nf_defrag_ipv6         22249  1 nf_conntrack_ipv6
ip6table_filter         1236  1
ip6_tables             16272  1 ip6table_filter
xt_geoip                2934  0
xt_recent               8132  0
xt_iprange              1448  0
xt_limit                1721  0
xt_state                1135  0
xt_multiport            1630  0
xt_LOG                 12099  0
nf_conntrack_ipv4      11115  0
nf_defrag_ipv4          1147  1 nf_conntrack_ipv4
iptable_filter          1296  1
ip_tables              15570  1 iptable_filter
hid_generic             1057  0
usbhid                 26686  0
hid                    82504  2 hid_generic,usbhid
usblp                  10674  0
usb_storage            46383  1
oxu210hp_hcd           24469  0
xt_length               1124  4
xt_tcpudp               2311  4
nf_conntrack           52846  3 nf_conntrack_ipv6,xt_state,nf_conntrack_ipv4
x_tables               15527 13 ip6table_filter,ip6_tables,xt_geoip,xt_recent,xt_iprange,xt_limit,xt_state,xt_multiport,xt_LOG,iptable_filter,ip_tables,xt_length,xt_tcpudp
bromolow_synobios      42769  0
btrfs                 788682  0
synoacl_vfs            17043  1
zlib_deflate           20180  1 btrfs
hfsplus                91979  0
md4                     3337  0
hmac                    2793  0
mlx_compat              5376  0
tn40xx                 24739  0
mlx4_ib               104330  0
svcrdma                24891  0
xprtrdma               23349  0
rdma_cm                29859  2 svcrdma,xprtrdma
iw_cm                   6702  1 rdma_cm
ib_ipoib               50417  0
ib_cm                  29507  1 rdma_cm
ib_sa                  19338  4 mlx4_ib,rdma_cm,ib_ipoib,ib_cm
ib_addr                 4498  1 rdma_cm
ib_mthca              116252  0
ib_mad                 34788  4 mlx4_ib,ib_cm,ib_sa,ib_mthca
ib_core                47092 10 mlx4_ib,svcrdma,xprtrdma,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_mthca,ib_mad
fuse                   74004  0
vfat                   10009  1
fat                    50784  1 vfat
glue_helper             3914  0
lrw                     3309  0
gf128mul                5346  1 lrw
ablk_helper             1684  0
arc4                    1847  0
rng_core                3520  0
cpufreq_conservative     6240  0
cpufreq_powersave        862  0
cpufreq_performance      866  2
cpufreq_ondemand        8039  0
acpi_cpufreq            6982  0
mperf                   1107  1 acpi_cpufreq
processor              26471  1 acpi_cpufreq
cpufreq_stats           2985  0
freq_table              2380  3 cpufreq_ondemand,acpi_cpufreq,cpufreq_stats
dm_snapshot            26708  0
crc_itu_t               1235  1 udf
quota_v2                3783  2
quota_tree              7970  1 quota_v2
psnap                   1717  0
p8022                    979  0
llc                     3441  2 psnap,p8022
sit                    14446  0
tunnel4                 2061  1 sit
ip_tunnel              11572  1 sit
zram                    8191  2
r8152                  49404  0
asix                   20370  0
ax88179_178a           16864  0
usbnet                 18137  2 asix,ax88179_178a
etxhci_hcd             84833  0
xhci_hcd               84493  0
ehci_pci                3504  0
ehci_hcd               39963  1 ehci_pci
uhci_hcd               22668  0
ohci_hcd               21168  0
usbcore               176728 14 usbhid,usblp,usb_storage,oxu210hp_hcd,r8152,asix,ax88179_178a,usbnet,etxhci_hcd,xhci_hcd,ehci_pci,ehci_hcd,uhci_hcd,ohci_hcd
usb_common              1488  1 usbcore
hptiop                 15828  0
3w_sas                 21786  0
3w_9xxx                33873  0
mvsas                  51605  0
hpsa                   50480  0
arcmsr                 28228  0
pm80xx                134800  0
aic94xx                72946  0
megaraid_sas          138070  0
megaraid_mbox          29835  0
megaraid_mm             7816  1 megaraid_mbox
nvme                   40947  0
mpt3sas               220677  0
mptsas                 39249  0
mptspi                 13047  0
mptscsih               18838  2 mptsas,mptspi
mptbase                62504  3 mptsas,mptspi,mptscsih
scsi_transport_spi     19551  1 mptspi
sg                     25017  0
ata_piix               24664  0
sata_uli                3004  0
sata_svw                4453  0
sata_qstor              5612  0
sata_sis                3884  0
pata_sis               10858  1 sata_sis
stex                   15005  0
sata_sx4                9284  0
sata_promise           10799  0
sata_nv                20623  0
sata_via                7803  0
sata_sil                7783  0
pdc_adma                5868  0
pata_via                8827  0
iscsi_tcp               8897  0
libiscsi_tcp           12850  1 iscsi_tcp
libiscsi               35195  2 iscsi_tcp,libiscsi_tcp
enic                   54982  0
qlge                   79283  0
qlcnic                214986  0
qla3xxx                36934  0
netxen_nic             98970  0
mlx4_en                67584  0
mlx4_core             169852  2 mlx4_ib,mlx4_en
cxgb4                 115680  0
cxgb3                 134726  0
cnic                   71038  0
ipv6                  303225 58 nf_conntrack_ipv6,nf_defrag_ipv6,ib_addr,sit
bnx2x                1430214  0
bna                   122380  0
be2net                 89072  0
sis900                 20811  0
sis190                 17297  0
jme                    34677  0
atl1e                  28011  0
atl1c                  33596  0
atl2                   23124  0
atl1                   30315  0
alx                    26373  0
sky2                   47242  0
skge                   38482  0
via_velocity           30007  0
crc_ccitt               1235  1 via_velocity
via_rhine              21014  0
r8101                 122784  0
r8168                 318247  0
r8169                  30519  0
8139too                18685  0
8139cp                 20694  0
tg3                   171348  0
broadcom                7174  0
b44                    26983  0
bnx2                  189638  0
ssb                    38587  1 b44
uio                     7592  1 cnic
forcedeth              55647  0
i40e                  249005  0
ixgbe                 246171  0
ixgb                   37937  0
igb                   178198  0
ioatdma                44390  0
e1000e                168235  0
e1000                 100754  0
e100                   29844  0
dca                     4600  3 ixgbe,igb,ioatdma
pcnet32                31251  0
amd8111e               16602  0
mdio                    3365  3 cxgb3,bnx2x,alx
mii                     3803 15 r8152,asix,ax88179_178a,usbnet,sis900,sis190,jme,atl1,via_rhine,8139too,8139cp,b44,e100,pcnet32,amd8111e
evdev                   9156  0
button                  4320  0
thermal_sys            18275  1 processor
compat                  4529  1 mlx_compat
cryptd                  7040  1 ablk_helper
ecryptfs               75063  0
sha512_generic          4976  0
sha256_generic          9884  0
sha1_generic            2206  0
ecb                     1849  0
aes_x86_64              7239  0
authenc                 6600  0
des_generic            15915  0
libcrc32c                906  1 bnx2x
ansi_cprng              3445  0
cts                     3968  0
md5                     2153  0
cbc                     2448  0

 

 

I added a second card into my Windows 10 PC to try and make a direct connection to the N54L.

C:\Program Files\Mellanox>ibstat
CA 'ibv_device0'
       CA type:
       Number of ports: 2
       Firmware version: 2.9.1000
       Hardware version: 0xb0
       Node GUID: 0x0002c903000972dc
       System image GUID: 0x0002c903000972df
   Port 1:
       State: Initializing
       Physical state: LinkUp
       Rate: 40
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972dd
       Link layer: IB
       Transport: IB
   Port 2:
       State: Down
       Physical state: Polling
       Rate: 70
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972de
       Link layer: IB
       Transport: IB

 

It says I'm at 40GB over IB. I now have a green light lit on the back end of each card. But Windows Networking says the ports are unplugged.

 

So with the N54L DSM 5.2-5644 Update 5

control panel

network

network interface

the Mellanox MHQH29B-XTR ports are not available. Any thoughts.....

Link to comment
Share on other sites

I had a little more time to test

 

On my Windows 10 PC

 

C:\WINDOWS\system32>ibstat
CA 'ibv_device0'
       CA type:
       Number of ports: 2
       Firmware version: 2.9.1000
       Hardware version: 0xb0
       Node GUID: 0x0002c903000972dc
       System image GUID: 0x0002c903000972df
   Port 1:
       State: Initializing
       Physical state: LinkUp
       Rate: 40
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972dd
       Link layer: IB
       Transport: IB
   Port 2:
       State: Down
       Physical state: Polling
       Rate: 70
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972de
       Link layer: IB
       Transport: IB

 

C:\WINDOWS\system32>ibstat -p
0x0002c903000972dd
0x0002c903000972de

 

Launched the subnet manager

C:\WINDOWS\system32>opensm
-------------------------------------------------
OpenSM 3.3.11 UMAD
Command Line Arguments:
Log File: %windir%\temp\osm.log
-------------------------------------------------
OpenSM 3.3.11 UMAD

Entering DISCOVERING state

Using default GUID 0x2c903000972dd
Entering MASTER state

SUBNET UP

 

Ibstat again

C:\WINDOWS\system32>ibstat
CA 'ibv_device0'
       CA type:
       Number of ports: 2
       Firmware version: 2.9.1000
       Hardware version: 0xb0
       Node GUID: 0x0002c903000972dc
       System image GUID: 0x0002c903000972df
   Port 1:
       State: Active
       Physical state: LinkUp
       Rate: 40
       Real rate: 32.00 (QDR)
       Base lid: 1
       LMC: 0
       SM lid: 1
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972dd
       Link layer: IB
       Transport: IB
   Port 2:
       State: Down
       Physical state: Polling
       Rate: 70
       Base lid: 0
       LMC: 0
       SM lid: 0
       Capability mask: 0x90580000
       Port GUID: 0x0002c903000972de
       Link layer: IB
       Transport: IB

 

Running "ibnetdiscover" gives you list of all nodes connected to your IB network and their "lid" numbers.

C:\WINDOWS\system32>ibnetdiscover
#
# Topology file: generated on Sun Feb 28 13:23:12 2016
#
# Initiated from node 0002c903000972dc port 0002c903000972dd

vendid=0x2c9
devid=0x673c
sysimgguid=0x2c90300095eaf
caguid=0x2c90300095eac
Ca      2 "H-0002c90300095eac"          # "MT25408 ConnectX Mellanox Technologies"
[2](2c90300095eae)      "H-0002c903000972dc"[1] (2c903000972dd)                 # lid 2 lmc 0 "LOFT" lid 1 4xQDR

vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000972df
caguid=0x2c903000972dc
Ca      2 "H-0002c903000972dc"          # "LOFT"
[1](2c903000972dd)      "H-0002c90300095eac"[2] (2c90300095eae)                 # lid 1 lmc 0 "MT25408 ConnectX Mellanox Technologies" lid 2 4xQDR

 

With the extra "-d" (debug) and "-v" (verbose) flags, you should see extra info on both sides for each IB ping.

C:\WINDOWS\system32>ibping -d -v 11
ibdebug: [6052] ibping: Ping..
ibwarn: [6052] ib_vendor_call_via: route Lid 11 data 000000256ECFFC50
ibwarn: [6052] ib_vendor_call_via: class 0x132 method 0x1 attr 0x0 mod 0x0 datasz 216 off 40 res_ex 1
ibwarn: [6052] mad_rpc_rmpp: rmpp 0000000000000000 data 000000256ECFFC50
ibwarn: [6052] _do_madrpc: recv failed: m
ibwarn: [6052] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 11)
ibdebug: [6052] main: ibping to Lid 11 failed
ibdebug: [6052] ibping: Ping..
ibwarn: [6052] ib_vendor_call_via: route Lid 11 data 000000256ECFFC50
ibwarn: [6052] ib_vendor_call_via: class 0x132 method 0x1 attr 0x0 mod 0x0 datasz 216 off 40 res_ex 1
ibwarn: [6052] mad_rpc_rmpp: rmpp 0000000000000000 data 000000256ECFFC50
ibwarn: [6052] _do_madrpc: recv failed: m
ibwarn: [6052] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 11)
ibdebug: [6052] main: ibping to Lid 11 failed
ibdebug: [6052] ibping: Ping..
ibwarn: [6052] ib_vendor_call_via: route Lid 11 data 000000256ECFFC50
ibwarn: [6052] ib_vendor_call_via: class 0x132 method 0x1 attr 0x0 mod 0x0 datasz 216 off 40 res_ex 1
ibwarn: [6052] mad_rpc_rmpp: rmpp 0000000000000000 data 000000256ECFFC50
ibwarn: [6052] _do_madrpc: recv failed: m
ibwarn: [6052] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 11)
ibdebug: [6052] main: ibping to Lid 11 failed
ibdebug: [6052] report: out due signal 2

---  (Lid 11) ibping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 5650 ms
rtt min/avg/max = 0.000/0.000/0.000 ms

 

That's it for now

Link to comment
Share on other sites

  • 4 months later...

Hi, question is how did you get the card to be recognised in xpenology. I am running 5.2.5644, and can see the drivers are already configured in the rc files. And the modules(mlx_compat.ko mlx4_en.ko and mlx4_core.ko) are in the /lib/modules/ folder. But card does not show up in GUI or in the verbose when booting up. If I do ifconfig -a it is not there. If I do lsmod they show up there but don't think mlx4_en.ko is getting loaded correctly.

Any help would be appreciated. I have the card in the topic.

Link to comment
Share on other sites

I added the infininband 5.2 drivers to the /lib/modules/ folder and edited the rc accordingly. And now have lights on card and subnet manager on Windows pc shows it is up. I can also see the card using ifconfig -a in putty, all drivers show loaded when doing lsmod. But I still cant get it to show in the xpenology network gui, so I can set the ip address. I am installed bare metal. Any help or advice would be appreciated.

Link to comment
Share on other sites

I have gotten both the infiniband and ethernet to kinda work on this card. The ethernet shows up in xpenology network control panel, when I have the port set to infiniband it does not. So still need to know how to make that happen. I can manually assign the infiniband port a ip address and connect to the xpenology gui with that address, but does not show up in network gui.

Link to comment
Share on other sites

I've been looking at rolling out infiniband across my network as a cheap way to get to 10G for my vsphere servers. I recall reading that DSM appears to be hardcoded for certain things, such as drives starting with sdx which leads to cards that don't come up as that not showing drives.... Maybe this is the same sort of thing, DSM doesn't recognise any adapters/connections that do not start with eth as I think the infiniband adapters will start with ib?

 

You should be able to roll the infiniband cards in to vsphere and run it on the hypervisor as you can then use the emulated 10g Ethernet VMX3 driver in Xpenology and get the full speed.

Link to comment
Share on other sites

 Share

×
×
  • Create New...