10GBit Bandwidth ~30% slower after upgrade to 7.2.1-69057-5

[crurer]

Hello friends,

 

I've spent some time to play around upgrading/migrating my Synology Boxes, which are Socket 1150 with Xeon CPU and DDR3 RAM. I'm using DS3615XS since many years as SMB share and iSCSI target.

The idea was to use the latest DSM rev. 7.2.1-69057-5 which is available for DS3617XS or DS2622XS+. I successfully migrated the existing Systems from DS3615XS to DS2622XS+ or DS3617XS. There was no other hardware change, only used another 4GB USB 2.0 stick with RPTC 0.10.0.0 instead of 0.9.3.0.

9G

Now the Problem: With newer DSM rev. >=7.2.1 the 10GBit transfer rate decreased drastic from ~9GBits/sec to ~6MBit/sec measured with iperf3. This is also recognized when transferring data on iSCSI or SMB level, which is much slower than before.

Here is a screen with DSM 7.1.1 on DS3615XS with expected speed on older FSC TX1330 M2 with Synology E10G18-T1 10GBit LAN card connected to PCIe x8 gen3 slot.

 

 

I didn't take a screen from the box upgraded to DS3617XS DSM rev. 7.2.1-69057-5....but you can trust it's much slower as mentioned above.

 

The unexpected behavior is recreated on 2 different hardware (boxes) using also different 10GBit LAN cards. The other Box which is also lowering the bandwidth is using a INTEL Dual LAN X540 T2. This confirming it's not an issue of card driver for Synology LAN card.

Meanwhile, I've reverted my systems back to DS3615XS on latest available DSM revision.

 

Question: Is this somehow/somewhere reported as known issue? If yes, are there some tweeks available to have expected 10GBit bandwidth available on DSM rev. 7.2.1-69057-5

 

cheers Chris

Edited April 23, 2024 by crurer
.

[DSfuchs]

Probably only @IG-88 can help.

[IG-88]

On 4/23/2024 at 4:58 PM, crurer said:

~9GBits/sec to ~6MBit/sec

6MBit like 1000 times less? thats like not working at all

i have my main with 7.1.1 and backup system with 7.2.1, the later one does 250 MByte/s with a bunch of older disks and hyperbackup rsync (not tested wit iperf) so from the 7.1.1 to 7.2.1 is no problem

the main 7.1.1 from a win11 system makes 1G Byte/s as long as it fills up ram and then drops to 650MB/s, cant ask for more (good nvme ssd on win11 and 20GB single file)

just tested it for 7.2.1 from win11 and its the same as with 7.1.1, 1GByte/s as long as its ram and then ~600MByte/s (no iperf neede here if it looks that good already)

(win11 and 7.1 is a sfp+ mellanox and 7.2 a rj-45 Tehuti TN9210 based (might be similar to you syno card)

 

there are some base differences even with systems having the same dsm version, 3615 is kernel 3.10 but 3617 and 3622 are both 4.4 so looks no like its not related to that as 3615 and 3617/22 seem to perform both badly

but anyway try arc loader arc-c (sa6400) with 7.2.1 that's kernel 5.x and might be different

any switch involved there might be differences when connecting directly or through a switch as of how speed is negotiated

what speed does the system show (ethtool eth0 | grep Speed) when its only 6Mbit/s

 

its interesting but if you want to save time consider just keep it with 7.1, as its a LTS version and afair will get the same support as 7.2, if there are no features you need from 7.2 you dont miss anything when just using 7.1

Edited April 26, 2024 by IG-88

[Peter Suh]

The Intel 10G ixgbe modules of DS3617xs(broadwell) and DS3622xs+(broadwellnk) operate somewhat specially.
Both models must use the vanilla ixgbe that was originally built into the original model.

I don't know if the TCRP of pocopico you used has adjustments for this vanilla module.

rr and my mshell have this vanilla ixgbe adjustment.
ixgbe compiled separately from redpill should only be used on other than these two platforms broadwell / broadwellnk.

These two files are vanilla module files that are imported directly from the original module in Synology.
The module pack is managed by overwriting this file on top of the ixgbe compiled by redpill.

https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwell-4.4.302/ixgbe.ko
https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwellnk-4.4.302/ixgbe.ko

I hope you try it with rr or mshell.

[IG-88]

as my 7.1 long term informations where older i looked it up, looks like there is a catch to that (at least now)

only older units (presumably kernel 3.10, mostly 2015 units and older) will get one more year up to 6/2025, namely the ds3615xs, the other commonly used types like 3617 or 3622 will loose update support 6/2024 and will have to be updated to 7.2 (if security updates are important)

https://kb.synology.com/en-global/WP/Synology_Security_White_Paper/3

 

looks like i will have to upgrade my dva1622 in the next 3 month to 7.2

 

 

18 hours ago, Peter Suh said:

The Intel 10G ixgbe modules of DS3617xs(broadwell) and DS3622xs+(broadwellnk) operate somewhat specially.
Both models must use the vanilla ixgbe that was originally built into the original model.

I don't know if the TCRP of pocopico you used has adjustments for this vanilla module.

 

 

there was also a "Synology E10G18-T1" with that performance problem so it might stretch to more nic's or its something different

but if exchanging the driver on the already running system clears the speed problem then it would be clearly that issue

 

edit: i looked up the syno nic, its a "newer" model, not tehuti bsed, its marvell aquantia (aka AQtion Aquantia AQC) and the driver module to look for is "atlantic.ko", sadly "modinfo" is not part of dsm but you would see driver realted information when the driver is loaded ( dmesg |grep atlantic)

also a sign of a original driver would be if its signed by synology, you could check this by looking in raw/hex at the end of the driver

"xxd /lib/modules/8021q.ko" for one that usually is still original and "xxd /lib/modules/atlantic.ko"

you will see "Module signature appended" at the very end of the driver when its a driver provided by synology

 

forgot to mention it but my test system was using arc loader, as most loaders have there own driver set its worth checking the driver version and comparing to the original driver from synology (if there is one), in some cases it might not work using synology's drivers as there might not cover other oem versions of nic's, seen that with ixgbe drivers on 6.2 (but also tehuti and other intel drivers and realtek 2.5G nic driver)

so it might not work to "downgrade" the driver to syno's original in some cases, that might result in a non working nic, it might find the device but if the phy chip is not supported the it will not show any connection or the driver might not load at all

so in case of experiments like that its handy to keep a 2nd nic inside that can be used if the 10G nic fails to work after changing the driver

Edited April 27, 2024 by IG-88

[crurer]

On 4/26/2024 at 4:10 PM, IG-88 said:

6MBit like 1000 times less? thats like not working at all

....

what speed does the system show (ethtool eth0 | grep Speed) when its only 6Mbit/s

 

sorry...6MBit/s is a typo......my fault. IT should be ~6GBit/s instead as mentioned in the title.

Edited April 28, 2024 by crurer

[crurer]

On 4/26/2024 at 4:29 PM, Peter Suh said:

I hope you try it with rr or mshell.

thank you for your reply and shared thoughts. I'll not spend more time here with trial and error for evaluating the best setup to get expected bandwidth between my Windows Server 2022 and the Synology box. I also will not use anymore the Intel X540 T2 NIC - it's even slow when used on another Windows server System (X9DRD-iF) during my own tests performed. My Synology (configured as 3615xs DSM 7.1.1-42962 Update 6) is currently using an Emulex OCE14102-NT 2x 10GB which is running fine as expected.

 

 

[smileyworld]

Can't complain about the Intel X540-T2.

The screenshot below is form a SMB share (RAID0 with a bunch of SSDs on my NAS).

 

I use it in my DS3622xs+

Edited September 8, 2024 by smileyworld

[Captainfingerbang]

plz forgive resurrecting old thread but i needed both you heavy hitters (peter and IG-88) here, and i have the same type questions.

Every year i increase my performance parts and with each build it seems my x540's slow down.

 

HORRIBLY SLOW X540's now!  Around 200BM/s  when before getting 1G

Box 1:

Xpenology SA6400 Arc 1.1.22 DSM 7.2.2-72806 Update 2. X570 Taichi 5900x 32gb Ran @ 2999   Raid 10 (all connected to 9300-16i 1st pcie x16) -  12 2tb higher end ssd's ((crucial mx500, micron 1100, 870 evo)  Then the ssd caches... 1 x 1tb wd Sn850x pcie 4.0 nvme ssd read cache, and 1 x 1tb Sk Hynix pcie 4.0 p41  write cache.

1-Supermicro Branded X540-t2 Nic

 

Box 2:

Also have a Dva3221 with 1 ssd volume  and bunch of spins. Xpenology Arc 1.1.22 DSM 7.2.2-72806 Update 2

1-Supermicro Branded X540-t2 Nic

 

Ive been doing xpen at least 5-7 years im guessing, so i know some stuff, but not like you two gurus. but i've  never loaded drivers. I've loaded addons but no drivers. 

I finally reached this page and have hope i can do something for my 10gbe. 

 

These nics have always been strong.

BTW i direct connect them to each other when backups are needed. 

Also have a windows pc next to them with a 3rd x540.

 

But ive never configured these. Never updated fw. Never downloaded or loaded drivers on purpose...

 

Some quick advice please guys?

 

 

 

 

 

Edited December 8, 2024 by Captainfingerbang

[Captainfingerbang]

On 4/26/2024 at 10:10 AM, IG-88 said:

6MBit like 1000 times less? thats like not working at all

i have my main with 7.1.1 and backup system with 7.2.1, the later one does 250 MByte/s with a bunch of older disks and hyperbackup rsync (not tested wit iperf) so from the 7.1.1 to 7.2.1 is no problem

the main 7.1.1 from a win11 system makes 1G Byte/s as long as it fills up ram and then drops to 650MB/s, cant ask for more (good nvme ssd on win11 and 20GB single file)

just tested it for 7.2.1 from win11 and its the same as with 7.1.1, 1GByte/s as long as its ram and then ~600MByte/s (no iperf neede here if it looks that good already)

(win11 and 7.1 is a sfp+ mellanox and 7.2 a rj-45 Tehuti TN9210 based (might be similar to you syno card)

 

there are some base differences even with systems having the same dsm version, 3615 is kernel 3.10 but 3617 and 3622 are both 4.4 so looks no like its not related to that as 3615 and 3617/22 seem to perform both badly

but anyway try arc loader arc-c (sa6400) with 7.2.1 that's kernel 5.x and might be different

any switch involved there might be differences when connecting directly or through a switch as of how speed is negotiated

what speed does the system show (ethtool eth0 | grep Speed) when its only 6Mbit/s

 

its interesting but if you want to save time consider just keep it with 7.1, as its a LTS version and afair will get the same support as 7.2, if there are no features you need from 7.2 you dont miss anything when just using 7.1

 

On 4/26/2024 at 10:29 AM, Peter Suh said:

The Intel 10G ixgbe modules of DS3617xs(broadwell) and DS3622xs+(broadwellnk) operate somewhat specially.
Both models must use the vanilla ixgbe that was originally built into the original model.

I don't know if the TCRP of pocopico you used has adjustments for this vanilla module.

rr and my mshell have this vanilla ixgbe adjustment.
ixgbe compiled separately from redpill should only be used on other than these two platforms broadwell / broadwellnk.

These two files are vanilla module files that are imported directly from the original module in Synology.
The module pack is managed by overwriting this file on top of the ixgbe compiled by redpill.

https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwell-4.4.302/ixgbe.ko
https://github.com/PeterSuh-Q3/arpl-modules/blob/main/vanilla/broadwellnk-4.4.302/ixgbe.ko

I hope you try it with rr or mshell.

 

 

 

Sorry! The above post was supposed to be quoted to you IG-88 and you Peter Suh!!!!