stanza

Guide to faster network transfers

Recommended Posts

OK here is a quick guide to speed up your network transfers, either between multiple Xpenologies or between your Xpenology NAS and say a Linux Server setup for Virtual Machine Hosting

 

Requirements

 

Each computer needs multiple lan interfaces, how you get there doesn't matter....

Either multiple single interfaces (onboard Motherboard plus add in single port cards)

Or adding in Multi port cards

 

Seems at present (not confirmed) you are limited to a maximum of 8 interfaces using DS3612xs version... tho depending on your final outcome requirements / configuration that might be enough....your mileage may vary

 

The hardware I am using for the below guide is

Intel Socket 775 Serverboard Model S3200SH

4Gb Ram

Dell H310 SAS HBA

7 a 450G Drives in a SHR1 Config

Intel Pro 1000 PCI Ethernet Adapter in a PCI Slot

HP NC360T Dual Port Ethernet Adapter in a x4 PCI-E slot

For a total of 4 x 1Gb lan ports

 

So first

 

Lets configure Xpenology properly, with the correct number of interfaces and the corresponding correct matching MAC addresses

 

How do we do that?

 

Easiest way I have found is as follows

 

SSH into your Xpenology box

 

Dump the dmesg output and search it for anything that looks like eth (as eth0 eth1 etc etc will be your lan interfaces if using standard network ports......for infiniband / 10g networking it will be something else)

 

so lets run the above and dump the output into a file for browsing through

 

> dmesg | grep “eth” > /tmp/dmesg.txt

 

Which saves the output in our tmp directory into a file called dmesg.txt

 

Now we can open this file and read / scroll through it to find out our “REAL” MAC addresses of the lan interfaces

 

>vi /tmp/dmesg.txt

 

in my case the contents show

 

[ 0.232970] ACPI Error: Method parse/execution failed [\_SB_._OSC] (Node ffff88011f04f4c0), AE_AML_BUFFER_LIMIT (20110623/psparse-536) 
[    2.993119] e1000 0000:04:01.0: eth0: (PCI:33MHz:32-bit) 00:0e:0c:82:a8:f9 
[    2.993226] e1000 0000:04:01.0: eth0: Intel(R) PRO/1000 Network Connection 
[    3.191111] e1000 0000:04:02.0: eth1: (PCI:33MHz:32-bit) 00:15:17:28:06:58 
[    3.191216] e1000 0000:04:02.0: eth1: Intel(R) PRO/1000 Network Connection 
[    3.191343] jme: JMicron JMC2XX ethernet driver version 1.0.8 
[    4.934800] e1000e 0000:02:00.0: eth2: (PCI Express:2.5GT/s:Width x4) 00:1f:29:55:21:60 
[    4.934934] e1000e 0000:02:00.0: eth2: Intel(R) PRO/1000 Network Connection 
[    4.935120] e1000e 0000:02:00.0: eth2: MAC: 0, PHY: 4, PBA No: D51930-004 
[    5.117775] e1000e 0000:02:00.1: eth3: (PCI Express:2.5GT/s:Width x4) 00:1f:29:55:21:61 
[    5.117909] e1000e 0000:02:00.1: eth3: Intel(R) PRO/1000 Network Connection 
[    5.118095] e1000e 0000:02:00.1: eth3: MAC: 0, PHY: 4, PBA No: D51930-004 
[   19.427330] usbcore: registered new interface driver ethub 
[   20.217839] ADDRCONF(NETDEV_UP): eth0: link is not ready 
[   21.234865] ADDRCONF(NETDEV_UP): eth1: link is not ready 
[   22.488838] ADDRCONF(NETDEV_UP): eth2: link is not ready 
[   23.502336] ADDRCONF(NETDEV_UP): eth3: link is not ready 
[   23.519820] e1000: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX 
[   23.520646] ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready 

 

(you might have to scroll right to see the full lines of output....depending on screen resolution)

 

So now we can find anything that looks like a MAC Address and either copy paste them somewhere or write them down.

 

In the above example these 4 lines look interesting

 

[    2.993119] e1000 0000:04:01.0: eth0: (PCI:33MHz:32-bit) 00:0e:0c:82:a8:f9 
[    3.191111] e1000 0000:04:02.0: eth1: (PCI:33MHz:32-bit) 00:15:17:28:06:58 
[    4.934800] e1000e 0000:02:00.0: eth2: (PCI Express:2.5GT/s:Width x4) 00:1f:29:55:21:60 
[    5.117775] e1000e 0000:02:00.1: eth3: (PCI Express:2.5GT/s:Width x4) 00:1f:29:55:21:61 

 

So now we have our 4 MAC Addresses we can edit the grub.conf file of the USB stick so we are using correct and REAL MAC Addresses of our hardware

 

Though the numbering of the interfaces get shuffled due to the file /etc.defaults/synoinfo.conf

 

Which has the line

netif_seq=”0 2 1 3”

 

Which reorders things and does your head in a little trying to figure out which is which

 

Anyway in my case it ended up as follows

 

Onboard > eth3 > Lan 4

NC360T Top Port > eth0 > Lan 1

PCI Card > eth1 > Lan 2

NC360T Bottom Port > eth2 > Lan 3

:roll:

 

Your best bet is to plug in one cable at a time and see which interface connects then edit the grub.conf file with vi and change it to match

 

> vi /volumeUSB1/usbshare/boot/grub/grub.conf

 

#serial --unit=0 --speed=115200
#terminal serial
default 0
timeout 3
fallback 0
title XPEnology DSM 4.3-3827 v1.1 Beta7 - HBA (SCSI/SAS)
root (hd0,0)
#        vender /vender show
#        hw_model
kernel /zImage root=/dev/md0 ihd_num=0 netif_num=4 syno_hw_version=DS3612xs mac1=001F29552160 mac2=000E0C82A8F9 mac3=001F29552161 mac4=001517280658 sn=B3J4N666333
initrd /rd.gz 

 

Save the file and reboot

 

Your IP Address might change here, depending on if it's set to DHCP or manual

 

Log back into your Xpenology web interface and open system information and Control-Panel Network to check your work

bond1.png

bond2.png

 

Now we can make a Bonded network interface to try and speed up things a little

 

To do this, open control-panel, go to network, then network interface tab.

 

Select Create button and then next (you can leave it on IEEE 802ad Dynamic Link Aggregation) as we will manually change this after by editing some files

bond3.png

 

Choose which interfaces you want to bond together, which in my case was Lan 1 and Lan 3

bond4.png

 

Click Next and set a manual IP Address

bond5.png

 

My normal network addresses for home use are in the 10.0.0 range.... so for this separate network I made the IP Address 192.168.0.1 with a subnet of 255.255.255.0

Jumbo Frames I left as default for now.

 

Click Apply and your bond is created, tho will show errors as we have not setup any Network Switch with an 802ad pair of ports.

bond6.png

 

Safely ignore this

 

Here is where you choose how you would like to connect / cable the computers together.

 

If you have just 2 Xpenology boxes, the simplest way would be direct connections....

that is connect a cable from your bonded ports to each other Modern lan interfaces shouldn't need cross over cables to do this)

 

So in my test example I am connecting an Xpenology box to a Linux (Ubuntu 12.04 Box) which both have multi port lan cards

I simply connect port 0 to port 0 and port1 to port 1 with short cables

 

If you are looking to connect multiple machines together, then for a basic setup you need 2 separate switches

 

Connect all devices port 0 to one switch and all devices port 1 to the second switch. (it will not work using only one switch unless you perform some vlan magic....but that's another advanced configuration for later)

 

Now we need to go and manually edit a config file to change the setup from “IEEE 802ad Dynamic Link Aggregation Mode” to use instead the hidden “Balance Round Robin Mode”

 

so if we look in the directory

 

/etc/sysconfig/network-scripts/

 

We see there are config files for up to 8 eth ports and also a newly created ifcfg-bond0 config file

 

all we need to change is the ifcfg-bond0 file

 

from

 

DEVICE=bond0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.0.1
NETMASK=255.255.255.0
BONDING_OPTS="mode=4 use_carrier=1 miimon=100 updelay=100 lacp_rate=fast"
USERCTL=no

 

to

 

DEVICE=bond0
ONBOOT=yes
BONDING_OPTS=”mode=0 use_carrier=1 miimon=100 updelay=100″
USERCTL=no
BOOTPROTO=static
IPADDR=192.168.0.1
NETMASK=255.255.255.0

 

Save the file and reboot the Xpenology box

 

How to configure your Linux Box is up to you......as each and every version of Linux does it slightly differently.

 

If your joining 2 Xpenologies together.... simply do as above on the second box... but use a different IP Address naturally.

 

Eg

Xpenology1 - 192.168.0.1 255.255.255.0

Xpenology2 -192.168.0.2 255.255.255.0

 

No for some tests to see if we improved things

 

 

In this test example, I created a shared folder, and shared it out with NFS.

On the Linux Box I mounted this NFS share and will use DD to write a 20Gig file to the share and then after waiting for a flush will read it back again.

 

1st test I will do a baseline so will pull out 1 lan cable to simulate a normal everyday connection to see how things perform

 

Results

 

Write Test with 1 Cable Connected MTU = 1500

P55-UD3R:~$ sudo time dd if=/dev/zero of=/VMachines/ddtest2 bs=1024000 count=20000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 176.601 s, 116 MB/s

0.01user 12.70system 2:57.29elapsed 7%CPU (0avgtext+0avgdata 7728maxresident)k

0inputs+40000000outputs (0major+533minor)pagefaults 0swaps

 

Read Test with 1 Cable Connected MTU = 1500

P55-UD3R:~$ sudo time dd if=/VMachines/ddtest2 of=/dev/null bs=1024000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 178.783 s, 115 MB/s

0.01user 16.17system 2:58.78elapsed 9%CPU (0avgtext+0avgdata 7712maxresident)k

40000000inputs+0outputs (0major+532minor)pagefaults 0swaps

 

Not bad, we are getting a nice 100+ Mb/s thruput in each direction with a standard MTU of 1500

 

 

Write Test with 2 Cables Connected MTU = 1500

P55-UD3R:~$ sudo time dd if=/dev/zero of=/VMachines/ddtest2 bs=1024000 count=20000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 118.625 s, 173 MB/s

0.02user 12.88system 1:58.62elapsed 10%CPU (0avgtext+0avgdata 7712maxresident)k

0inputs+40000000outputs (0major+532minor)pagefaults 0swaps

 

Read Test 2 with Cables Connected MTU = 1500

P55-UD3R:~$ sudo time dd if=/VMachines/ddtest2 of=/dev/null bs=1024000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 120.116 s, 171 MB/s

0.02user 15.05system 2:00.11elapsed 12%CPU (0avgtext+0avgdata 7712maxresident)k

40000024inputs+0outputs (1major+531minor)pagefaults 0swaps

 

Nice, we improved the speed to nice 170+ Mb/s thruput in each direction with a standard MTU of 1500.....CPU load jumps up to around 50%

It's not a double in throughput as we thought....so

Now lets try again 2 x cables but bumping the MTU to 9000 (enabling Jumbo Frames)

 

Write Test with 2 Cables Connected MTU = 9000

P55-UD3R:~$ sudo time dd if=/dev/zero of=/VMachines/ddtest2 bs=1024000 count=20000

[sudo] password for antonyplatt:

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 113.281 s, 181 MB/s

0.00user 12.28system 1:53.28elapsed 10%CPU (0avgtext+0avgdata 7728maxresident)k

0inputs+40000000outputs (0major+533minor)pagefaults 0swaps

 

Read Test 2 with Cables Connected MTU = 9000

P55-UD3R:~$ sudo time dd if=/VMachines/ddtest2 of=/dev/null bs=1024000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 115.037 s, 178 MB/s

0.04user 13.80system 1:55.03elapsed 12%CPU (0avgtext+0avgdata 7712maxresident)k

40000000inputs+0outputs (0major+533minor)pagefaults 0swaps

 

Similar, though slightly improved throughput.....but our CPU has lessened down to around 35% which helps.

 

 

And just for fun, I added a 3rd lan port into the bond (tho it's only a PCI card) to see if things improve any more (squeeze the last little bit extra)

 

 

Write Test with 3 Cables Connected MTU = 9000

P55-UD3R:~$ sudo time dd if=/dev/zero of=/VMachines/ddtest2 bs=1024000 count=20000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 71.1939 s, 288 MB/s

0.00user 13.27system 1:11.19elapsed 18%CPU (0avgtext+0avgdata 7712maxresident)k

0inputs+40000000outputs (0major+532minor)pagefaults 0swaps

 

Read Test with 3 Cables Connected MTU = 9000

P55-UD3R:~$ sudo time dd if=/VMachines/ddtest2 of=/dev/null bs=1024000

20000+0 records in

20000+0 records out

20480000000 bytes (20 GB) copied, 82.8826 s, 247 MB/s

0.03user 11.24system 1:22.88elapsed 13%CPU (0avgtext+0avgdata 7728maxresident)k

40000024inputs+0outputs (1major+532minor)pagefaults 0swaps

 

Spikes from 245 up to 341Mb/s is nice.....

But I think my Xpenology box has hit it's limit with the current HDD's installed and hardware

 

Please also NOTE, only use an MTU of 9000 if your switch supports it, or you are direct connecting.

 

So if you had a setup with 4 x decent PCI-E based lan interfaces / jumbo frames and a good CPU plus some driver tweaking should get roughly 450Mb/s I would guesstimate.

 

I have uploaded a video here for those who like to see pretty graphs, showing a write then a read to the Xpenology box from the Linux Box.

 

You will note also how wildly inaccurate the Synology Resource monitor is.

 

Video 30.2Mb MKV File

http://www.stanzabike.net/images/xpenology/test_bond2-MTU9000.mkv

 

Have fun :mrgreen:

Edited by Guest

Share this post


Link to post
Share on other sites

Very-very interesting... I have only one question after reading but before experiments on real hardware. You wrote:

If you are looking to connect multiple machines together, then for a basic setup you need 2 separate switches

 

I separated my home environment into two zones: one with nas-es, media players and so on. And the second one with workstations, internet routers and wifi. There are 10 meters between them. Reconfiguration is very comlex and even not possible at all (for example - I have optic line from one of the ISP used, its router is absolutely immovable).

 

So now it is connected via a pair of Netgear GS108T (manageable switches). Switches interconnected with two cables of those 10 meters length. Both nas-es (two of three) and one workstation have 2-3 nic-s onboard. All of them now configured for LACP both on switches and on linuxes. I want to try your method to increase network throughoutput before trying 10G (and really it is a home environment - so I do not need fault tolerance, but I need doubling/tripling of nfs transfer rate to speedup FF/Rewind functions of mplayer). :smile:

 

So... Did I understand your idea correctly - I need two/three/four/so_on pairs of switches between my two zones/rooms to obtain doubling/so_on of throughoutput (with the corresponding number of cables between any pair of switches)? And if I am right - I believe that in this case I do not need manageable switches, but any cheap 1G switches, isn't it? Any other issues to think on (for example - are there any opportunity to use high force of Netgear GS108T instead of extensive doubling of switches?)

Share this post


Link to post
Share on other sites

No you only need dumb switches

 

The reason you need a switch per port.... Is ..... Think of it this way

Each port of the BONDED end.... Actually gets the same MAC Address...which is what confuses a switch when you try to plug both ports into it.

 

Packet one goes out port one.... Destined for xyz MAC Address.... But on the switch.... It doesn't know which port to send the packet down.... As it see's two ports with the same destination MAC address.

 

So to get around that, if you use two switches.... Each switch only see's one MAC address per port.

 

Eg in your case you would only need one extra dumb switch...as you already have one switch connected to each computer / NAS ....

 

There is another way around the problem using vlans.... I have not tried that yet.... But presume it goes something like

 

Computer One

Port one = vlan20

Port two = vlan30

 

Then bond both vlan's together

 

Computer Two

Port one = vlan20

Port two = vlan30

 

Then bond both vlans together....

 

Switch

Port 1 = vlan20 - connected to computer ones port 1

Port 2 = vlan20 - connected to computer twos port 1

Port 3 = vlan30 - connected to computer ones port 2

Port 4 = vlan30 - connected to computer twos port 2

 

Though then you would have to be carefull about the ordering of bringing the interfaces up etc

 

The reason vlans work is that to the switch, your "virtually segmenting" it up into smaller "virtual switches" as as such the duplicate MAC address problem is avoided.

 

This is what I will be trying to figure out next, to get around the need of an extra switch.... Though cheap 8 ports dumb switches are everywhere.

 

.

Share this post


Link to post
Share on other sites

dont think vlan's are working...

the switch stores in his fdb (forwading database) the port & mac association.

vlan capable switches also associate the vlan information there..

 

from normal networking perspective i would not recommend this setup..but if you want to interconnect several storages it will be of help at a cheap cost....(you only need the extra nics and a spare switch)

Share this post


Link to post
Share on other sites

In fact, I have extra nics and spare dump giga switch, but I don't think it's a good idea to interconnect nas/ servers or whatever you call it. I acutally cannot set up network cables around my house like this, from 2nd floor to basement, from 1st floor to basement, ect.

Share this post


Link to post
Share on other sites
In fact, I have extra nics and spare dump giga switch, but I don't think it's a good idea to interconnect nas/ servers or whatever you call it. I acutally cannot set up network cables around my house like this, from 2nd floor to basement, from 1st floor to basement, ect.

 

stanza's suggestion is for situations where you want to transfer data between 2 or more devices with a higher speed without spending too much money on new equipment...

so stanza's setup is NOT working for normal clients, as the most OS can't handle "packet based round robin"...so no need to change all your infrastructure/network

 

I for myself have my main client and storages upgraded to 10g ethernet...for "long" distances i use fiber-cables ( om4 mmf), for "short" distances i use DAC (direct attatched cables)

but such a solution requires a good amount of money (at least 500€, unless you live in the US)

 

I also recommend everyone, who's currently building, buying or renovating a hous/appartment to think and install some cabling-ways....it will save money and trouble later....

of course you can use wireless....take the new 802.11ac gbit capable standard.....

first you need to find some fast NIC's and then the fast AP's....and then.....omg....wireless is a shared medium....All clients sharing the same bandwith, which means your single client bandwith will decrease... and i haven't started talking about the impact of walls, furniture or other wireless networks...

so wireless is a nice thinh, but no network....thats why i tell always " networks use cables, air is for breathing"

Share this post


Link to post
Share on other sites

"Seems at present (not confirmed) you are limited to a maximum of 8 interfaces using DS3612xs version... tho depending on your final outcome requirements / configuration that might be enough....your mileage may vary"

 

 

How can I add more as 8 LAN interfaces at this model?

Share this post


Link to post
Share on other sites