10GBit Bandwidth ~30% slower after upgrade to 7.2.1-69057-5

[crurer]

Hello friends,

 

I've spent some time to play around upgrading/migrating my Synology Boxes, which are Socket 1150 with Xeon CPU and DDR3 RAM. I'm using DS3615XS since many years as SMB share and iSCSI target.

The idea was to use the latest DSM rev. 7.2.1-69057-5 which is available for DS3617XS or DS2622XS+. I successfully migrated the existing Systems from DS3615XS to DS2622XS+ or DS3617XS. There was no other hardware change, only used another 4GB USB 2.0 stick with RPTC 0.10.0.0 instead of 0.9.3.0.

9G

Now the Problem: With newer DSM rev. >=7.2.1 the 10GBit transfer rate decreased drastic from ~9GBits/sec to ~6MBit/sec measured with iperf3. This is also recognized when transferring data on iSCSI or SMB level, which is much slower than before.

Here is a screen with DSM 7.1.1 on DS3615XS with expected speed on older FSC TX1330 M2 with Synology E10G18-T1 10GBit LAN card connected to PCIe x8 gen3 slot.

 

 

I didn't take a screen from the box upgraded to DS3617XS DSM rev. 7.2.1-69057-5....but you can trust it's much slower as mentioned above.

 

The unexpected behavior is recreated on 2 different hardware (boxes) using also different 10GBit LAN cards. The other Box which is also lowering the bandwidth is using a INTEL Dual LAN X540 T2. This confirming it's not an issue of card driver for Synology LAN card.

Meanwhile, I've reverted my systems back to DS3615XS on latest available DSM revision.

 

Question: Is this somehow/somewhere reported as known issue? If yes, are there some tweeks available to have expected 10GBit bandwidth available on DSM rev. 7.2.1-69057-5

 

cheers Chris

Edited April 23 by crurer
.

[DSfuchs]

Probably only @IG-88 can help.

[IG-88]

On 4/23/2024 at 4:58 PM, crurer said:

~9GBits/sec to ~6MBit/sec

6MBit like 1000 times less? thats like not working at all

i have my main with 7.1.1 and backup system with 7.2.1, the later one does 250 MByte/s with a bunch of older disks and hyperbackup rsync (not tested wit iperf) so from the 7.1.1 to 7.2.1 is no problem

the main 7.1.1 from a win11 system makes 1G Byte/s as long as it fills up ram and then drops to 650MB/s, cant ask for more (good nvme ssd on win11 and 20GB single file)

just tested it for 7.2.1 from win11 and its the same as with 7.1.1, 1GByte/s as long as its ram and then ~600MByte/s (no iperf neede here if it looks that good already)

(win11 and 7.1 is a sfp+ mellanox and 7.2 a rj-45 Tehuti TN9210 based (might be similar to you syno card)

 

there are some base differences even with systems having the same dsm version, 3615 is kernel 3.10 but 3617 and 3622 are both 4.4 so looks no like its not related to that as 3615 and 3617/22 seem to perform both badly

but anyway try arc loader arc-c (sa6400) with 7.2.1 that's kernel 5.x and might be different

any switch involved there might be differences when connecting directly or through a switch as of how speed is negotiated

what speed does the system show (ethtool eth0 | grep Speed) when its only 6Mbit/s

 

its interesting but if you want to save time consider just keep it with 7.1, as its a LTS version and afair will get the same support as 7.2, if there are no features you need from 7.2 you dont miss anything when just using 7.1

Edited April 26 by IG-88

[Peter Suh]

The Intel 10G ixgbe modules of DS3617xs(broadwell) and DS3622xs+(broadwellnk) operate somewhat specially.
Both models must use the vanilla ixgbe that was originally built into the original model.

I don't know if the TCRP of pocopico you used has adjustments for this vanilla module.

rr and my mshell have this vanilla ixgbe adjustment.
ixgbe compiled separately from redpill should only be used on other than these two platforms broadwell / broadwellnk.

These two files are vanilla module files that are imported directly from the original module in Synology.
The module pack is managed by overwriting this file on top of the ixgbe compiled by redpill.

https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwell-4.4.302/ixgbe.ko
https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwellnk-4.4.302/ixgbe.ko

I hope you try it with rr or mshell.

[IG-88]

as my 7.1 long term informations where older i looked it up, looks like there is a catch to that (at least now)

only older units (presumably kernel 3.10, mostly 2015 units and older) will get one more year up to 6/2025, namely the ds3615xs, the other commonly used types like 3617 or 3622 will loose update support 6/2024 and will have to be updated to 7.2 (if security updates are important)

https://kb.synology.com/en-global/WP/Synology_Security_White_Paper/3

 

looks like i will have to upgrade my dva1622 in the next 3 month to 7.2

 

 

18 hours ago, Peter Suh said:

The Intel 10G ixgbe modules of DS3617xs(broadwell) and DS3622xs+(broadwellnk) operate somewhat specially.
Both models must use the vanilla ixgbe that was originally built into the original model.

I don't know if the TCRP of pocopico you used has adjustments for this vanilla module.

 

 

there was also a "Synology E10G18-T1" with that performance problem so it might stretch to more nic's or its something different

but if exchanging the driver on the already running system clears the speed problem then it would be clearly that issue

 

edit: i looked up the syno nic, its a "newer" model, not tehuti bsed, its marvell aquantia (aka AQtion Aquantia AQC) and the driver module to look for is "atlantic.ko", sadly "modinfo" is not part of dsm but you would see driver realted information when the driver is loaded ( dmesg |grep atlantic)

also a sign of a original driver would be if its signed by synology, you could check this by looking in raw/hex at the end of the driver

"xxd /lib/modules/8021q.ko" for one that usually is still original and "xxd /lib/modules/atlantic.ko"

you will see "Module signature appended" at the very end of the driver when its a driver provided by synology

 

forgot to mention it but my test system was using arc loader, as most loaders have there own driver set its worth checking the driver version and comparing to the original driver from synology (if there is one), in some cases it might not work using synology's drivers as there might not cover other oem versions of nic's, seen that with ixgbe drivers on 6.2 (but also tehuti and other intel drivers and realtek 2.5G nic driver)

so it might not work to "downgrade" the driver to syno's original in some cases, that might result in a non working nic, it might find the device but if the phy chip is not supported the it will not show any connection or the driver might not load at all

so in case of experiments like that its handy to keep a 2nd nic inside that can be used if the 10G nic fails to work after changing the driver

Edited April 27 by IG-88

[crurer]

On 4/26/2024 at 4:10 PM, IG-88 said:

6MBit like 1000 times less? thats like not working at all

....

what speed does the system show (ethtool eth0 | grep Speed) when its only 6Mbit/s

 

sorry...6MBit/s is a typo......my fault. IT should be ~6GBit/s instead as mentioned in the title.

Edited Sunday at 08:19 AM by crurer

[crurer]

On 4/26/2024 at 4:29 PM, Peter Suh said:

I hope you try it with rr or mshell.

thank you for your reply and shared thoughts. I'll not spend more time here with trial and error for evaluating the best setup to get expected bandwidth between my Windows Server 2022 and the Synology box. I also will not use anymore the Intel X540 T2 NIC - it's even slow when used on another Windows server System (X9DRD-iF) during my own tests performed. My Synology (configured as 3615xs DSM 7.1.1-42962 Update 6) is currently using an Emulex OCE14102-NT 2x 10GB which is running fine as expected.