s3ntro

Odd network issue w/Intel x540-T2

Recommended Posts

I'm attempting to get 10G running and I'm experiencing a strange issue. I'm on 6.0.2-8451 update 8. The network configuration (and Windows 10 machines on the other end) shows the links as up and full 10G speed but no packets traverse the connection. I assigned manual IP addresses to both interfaces (10.0.1.2 and 10.0.2.2) and those correspond with 1.1 and 2.1 addresses on two computers directly connected to the box. I have other connections configured the same way with no problems so I don't think it's an addressing issue. arp -a on the Windows machines doesn't return a mac address for those connections on the Xpenology box.

 

Oddly, LAN 2 (eth1) and LAN 3 (eth2) both show the same MAC address even though I assigned them in the grub.conf file. dmesg | grep "eth" doesn't show a mac address for the two 10G connections.

 

I thought this thread viewtopic.php?f=2&t=24412 might help but I'm a stranger to Linux so I'm lost without some further guidance.

 

Ideas?

 

Here's the grep output:

 

[ 10.662906] ixgbe 0000:02:00.0 eth0: MAC: 3, PHY: 3, PBA No: G93560-001

[ 10.662916] ixgbe 0000:02:00.0 eth0: Enabled Features: RxQ: 8 TxQ: 8 FdirHash

[ 10.685012] ixgbe 0000:02:00.0 eth0: Intel® 10 Gigabit Network Connection

[ 11.474412] ixgbe 0000:02:00.1 eth1: MAC: 3, PHY: 3, PBA No: G93560-001

[ 11.474434] ixgbe 0000:02:00.1 eth1: Enabled Features: RxQ: 8 TxQ: 8 FdirHash

[ 11.496531] ixgbe 0000:02:00.1 eth1: Intel® 10 Gigabit Network Connection

[ 11.616981] jme: JMicron JMC2XX ethernet driver version 1.0.8

[ 12.438362] ixgbe 0000:02:00.0: registered PHC device on eth0

[ 12.658470] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

[ 12.658515] ixgbe 0000:02:00.0 eth0: changing MTU from 1500 to 9000

[ 15.390737] ixgbe 0000:02:00.1: registered PHC device on eth1

[ 15.611884] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready

[ 15.612184] ixgbe 0000:02:00.1 eth1: changing MTU from 1500 to 9000

[ 18.373936] eth2: 0xffffc900084ec000, a0:36:9f:4b:6f:d2, IRQ 78

[ 18.413620] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready

[ 19.436898] eth3: 0xffffc900084ee000, a0:36:9f:4b:6f:d3, IRQ 79

[ 19.477740] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready

[ 20.170881] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX?TX

[ 20.171035] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[ 20.196470] ixgbe 0000:02:00.1 eth1: NIC Link is Down

[ 22.423280] r8168: eth2: link up

[ 22.423288] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[ 23.481639] init: dhcp-client (eth2) main process (11982) killed by TERM signal

[ 24.547546] init: dhcp-client (eth2) main process (14796) killed by TERM signal

[ 24.592616] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 25.814107] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

[ 25.814236] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[ 27.525634] r8168: eth2: link up

[ 28.597426] init: dhcp-client (eth2) main process (15045) killed by TERM sign

 

eth2 is my DHCP motherboard connection. eth3 is the secondary motherboard connection but I don't use it.

Share this post


Link to post
Share on other sites

Hi i had same issue, it seems a problem SRV-IO in my case, try this.

edit /etc.defaults/modules.conf and you can see something like

module_pfe_args_comcerto2k="lro_mode=1 tx_qos=0 burnin_mode=0"
module_ixgbe_args_bromolow="max_vfs=8,8,8,8"
module_ixgbe_args_broadwell="max_vfs=8,8,8,8,8,8"
module_ixgbe_args_grantley="max_vfs=8,8,8,8,8,8"

as you can see ixgbe on bromolow has max_vfs param , in my case it seems a problem ( Gen8 baremetal DSM 6.0 U9 )

so i deleted row and my new /etc.defaults/modules.conf it's' like this

 

module_pfe_args_comcerto2k="lro_mode=1 tx_qos=0 burnin_mode=0"

 

Same in /etc/modules.conf, make backup of original files and try.

But, it seems it's not enough, to workaround i had to unload and reload ixgbe and i had to restart network services running /etc/rc.network restart.

After, i could use intel 10Gb card.

I'm figuring how to found a more elegant solution, i have to try new ixgbe driver compiled from intel sources.

Try my work around and see what happens

Hope it helps.

Share this post


Link to post
Share on other sites

Thank you Aigor.

 

I'd be happy to try this but I need some further guidance on how to edit /etc.defaults/modules.conf. I followed forum directions for editing grub.conf but don't the modules.conf must be somewhere else. I appreciate your help and I understand if guiding a linux noob through this isn't high on your priority list. Still, if you (or anyone) could point me in the right direction for learning more, I'd appreciate it.

Share this post


Link to post
Share on other sites
Thank you Aigor.

 

I'd be happy to try this but I need some further guidance on how to edit /etc.defaults/modules.conf. I followed forum directions for editing grub.conf but don't the modules.conf must be somewhere else. I appreciate your help and I understand if guiding a linux noob through this isn't high on your priority list. Still, if you (or anyone) could point me in the right direction for learning more, I'd appreciate it.

 

Hi, you need to learn some linux skills :grin: it's' not so hard.

You need to logon in your NAS via ssh, so first of all install putty ( i assume you are using windows )

Enable ssh in your nas via webinterface under control panel, into terminal snmp

 

Search on youtube VIM tutorial vim it's' standard editor in almost every unix/linux systems

 

in Terminal you can use "cd" to change folder, ls to list folder contents, pwd to show where you are in filesystem, mv to make backup of one file, cp to copy.

Pay attention where you are editing, you act as root and you have power to screw up your system

Remember, with great power, comes great responsibilities :grin:

Share this post


Link to post
Share on other sites
Thank you Aigor.

 

I'd be happy to try this but I need some further guidance on how to edit /etc.defaults/modules.conf. I followed forum directions for editing grub.conf but don't the modules.conf must be somewhere else. I appreciate your help and I understand if guiding a linux noob through this isn't high on your priority list. Still, if you (or anyone) could point me in the right direction for learning more, I'd appreciate it.

 

Never mind. Some sleuthing on linux CLI stuff is filling in the missing pieces.

Share this post


Link to post
Share on other sites
Thank you Aigor.

 

I'd be happy to try this but I need some further guidance on how to edit /etc.defaults/modules.conf. I followed forum directions for editing grub.conf but don't the modules.conf must be somewhere else. I appreciate your help and I understand if guiding a linux noob through this isn't high on your priority list. Still, if you (or anyone) could point me in the right direction for learning more, I'd appreciate it.

 

Hi, you need to learn some linux skills :grin: it's' not so hard.

You need to logon in your NAS via ssh, so first of all install putty ( i assume you are using windows )

Enable ssh in your nas via webinterface under control panel, into terminal snmp

 

Search on youtube VIM tutorial vim it's' standard editor in almost every unix/linux systems

 

in Terminal you can use "cd" to change folder, ls to list folder contents, pwd to show where you are in filesystem, mv to make backup of one file, cp to copy.

Pay attention where you are editing, you act as root and you have power to screw up your system

Remember, with great power, comes great responsibilities :grin:

 

Thank you!

Share this post


Link to post
Share on other sites
Hi i had same issue, it seems a problem SRV-IO in my case, try this.

edit /etc.defaults/modules.conf and you can see something like

module_pfe_args_comcerto2k="lro_mode=1 tx_qos=0 burnin_mode=0"
module_ixgbe_args_bromolow="max_vfs=8,8,8,8"
module_ixgbe_args_broadwell="max_vfs=8,8,8,8,8,8"
module_ixgbe_args_grantley="max_vfs=8,8,8,8,8,8"

as you can see ixgbe on bromolow has max_vfs param , in my case it seems a problem ( Gen8 baremetal DSM 6.0 U9 )

so i deleted row and my new /etc.defaults/modules.conf it's' like this

 

module_pfe_args_comcerto2k="lro_mode=1 tx_qos=0 burnin_mode=0"

 

Same in /etc/modules.conf, make backup of original files and try.

But, it seems it's not enough, to workaround i had to unload and reload ixgbe and i had to restart network services running /etc/rc.network restart.

After, i could use intel 10Gb card.

I'm figuring how to found a more elegant solution, i have to try new ixgbe driver compiled from intel sources.

Try my work around and see what happens

Hope it helps.

 

Okay, got the files edited with vi. Now I need to unload and reload ixgbe before restarting network services. Not sure how that works. Any pointers?

Share this post


Link to post
Share on other sites

In my case work.

 

in terminal go into /lib/modules ( cd /lib/modules )

unload ixgbe driver ( rmmod ixgbe ) take a little time

reload module ( insmod ixgbe.ko)

Look into dmesg ( simply run dmesg command )

restart network services ( /etc/rc.network restart )

 

You must be root or you need to use sudo to act like root

Share this post


Link to post
Share on other sites
In my case work.

 

in terminal go into /lib/modules ( cd /lib/modules )

unload ixgbe driver ( rmmod ixgbe ) take a little time

reload module ( insmod ixgbe.ko)

Look into dmesg ( simply run dmesg command )

restart network services ( /etc/rc.network restart )

 

You must be root or you need to use sudo to act like root

 

That's exactly what I needed. Thank you so much! I'm still getting a weird mac address clone on eth0 and eth2 but I think I can fix that. Will test more later. I really, *really* appreciate your help.

Share this post


Link to post
Share on other sites
In my case work.

 

in terminal go into /lib/modules ( cd /lib/modules )

unload ixgbe driver ( rmmod ixgbe ) take a little time

reload module ( insmod ixgbe.ko)

Look into dmesg ( simply run dmesg command )

restart network services ( /etc/rc.network restart )

 

You must be root or you need to use sudo to act like root

 

That's exactly what I needed. Thank you so much! I'm still getting a weird mac address clone on eth0 and eth2 but I think I can fix that. Will test more later. I really, *really* appreciate your help.

 

let me know if works

Share this post


Link to post
Share on other sites

Weird, I thought I posted one saying it worked.

 

Yes - the link went active and packets can now traverse the link. Thank you SO MUCH for your help and guidance.

 

Unfortunately, the network speed doesn't appear to do much better than gigabit :sad: I have files going from SSD to SSD so I should be able to hit at least 300MB but no. I still have to put an SSD cache in and perhaps that will help.

Share this post


Link to post
Share on other sites

That's another problem, i use synology like datastore for my lab Esxi via iSCSI i think it's iSCSI subsystem cause of bottleneck.

have you set mtu to 9000 on both interfaces? ( in synology and in other computer )

Which kind of sharing protocol are you using? SMB iSCSI NFS or what?

If you are using SMB may you have to setup DSM to use smb Version 3 protocol to do that go into control panel, file services, under win/smb click into advanced options, and set smb3 enabled

When i will have spare time i would like to try to add Fibre Channel support on my NAS i have all the stuff needed ( cards and optical cables )

Share this post


Link to post
Share on other sites

The box seems to be unstable after resetting ixgbe and reloading network services. Stays up for about 10 minutes.

Share this post


Link to post
Share on other sites
The box seems to be unstable after resetting ixgbe and reloading network services. Stays up for about 10 minutes.

Thtat's weird , which hardware are you using?

Share this post


Link to post
Share on other sites
The box seems to be unstable after resetting ixgbe and reloading network services. Stays up for about 10 minutes.

Thtat's weird , which hardware are you using?

 

I'm wondering if the power supply is finally dying. I have one showing up later to test.

 

Do you have this ixgbe issue with all versions of xpenology? 5.2 as well?

Share this post


Link to post
Share on other sites

Hi, i'm start to use 10Gb only in 6.0 , so i don't know if there was same issues with old 5.2

Share this post


Link to post
Share on other sites

I do some test on my rig, i hit 400 MByte of transfer from my pc to iSCSI lun on synology, mechanical disk to mechanical disk

Share this post


Link to post
Share on other sites

Hi,

 

I need some help if someone is willing to give me the time :smile:.

 

Currently I get to the point with the bootloader (Jun 1.01) that my system is waiting for me to log in with sinology assistant. I cannot find my system though. Looking in my router (Asus AC68) I see no connection in my client list.

 

I'm not certain what is going on.

 

I am doing a bare metal install and have monitor keyboard connected. Is there a log or file I can post up here to get some assistance? I have some Linux experience (minimal) and mainly a windows / dos background sorry....

 

 

 

I'm using a Xeon-D SOC:

 

CPU and Chipset Drivers:

Broadwell based SOC

Intel® Xeon® processor D-1521

FCBGA 1667

 

 

NIC Driver Request:

Intel® D-1500 SoC 10GbE Controller

SoC 10GbE X552/X557-AT 10GBASE-T

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now