10gbe + sata expansion - limited to 1 pcie slot

[SnowDrifter]

Admittedly, I've been getting more out of my hardware than I thought I would... There are 2 things I would like to do. SSD cache, and 10gbe nic. Issue is, I don't have enough sata connectors on my board, and I only have a single pcie slot.

Regarding the SSD cache, intent is a read cache. But perhaps read/write is more appropriate for working over network? Seems the crashing issue is related to NVME drives from what I can find. If anyone wants to chime in on the issue, please do. Always welcome more info.


Hardware details:

Asrock Rack c246 WSI - 8x sata
Intel i3-9100
Silverstone DS380 - 8x 3.5" + 4x 2.5"


Details on board:

Has an m.2 2242 slot. However the user manual indicates that sata 1 will be disabled if the NVME slot is set to sata. Documentation is a bit lacking, so I'm unsure if this allows for all 8 sata ports to be active with a pcie drive, or if it's with any storage drive in there

Board has 4 native sata ports, with an additional 4 added via oculink




Brain dump of thoughts:

- Board has a single pcie lane, but case has 2 slots. Possible to use a splitter + half height cards? Tried to find something to fit this but was unsuccessful. Would need assistance locating compatible hardware - both compatible 10gbe nic and breakout board of some variety to give 4x sata.

- Possible to use a m.2 to sata adapter card? Found some adapter cards like this https://www.amazon.com/Syba-SD-ADA40118-Socket-Adapter-Converter/dp/B01N5O4UNB
However I have no experience with them and I'm uncertain if they play nice with respect to speeds / reliability or compatibility with the software

- Not opposed to purchasing a new motherboard. But as of yet, I'm unable to locate a mini itx board that supports 9th gen intel processors (coffee lake refresh), with 10gbe, and 8+ sata ports, that supports ECC memory


Anyone have any ideas here? It's hard for me to justify 10gbe without an SSD cache, or vice-versa, as either option would leave a lot of performance on the table

Edited April 20, 2021 by SnowDrifter

[flyride]

10Gbe is always worth it.  SSD cache is often not worth it.  Cache wears out an SSD quickly, and the write option has many instances of corruption.  It has nothing to do with NVMe vs SATA.  I always advocate for more RAM (which is automatically used for cache) than SSD cache.  Or just run SSD's period (line rate 10Gbe read/write sustained indefinitely with SSD RAIDF1 and no cache).

 

Most M.2 dual slots will take one port from the SATA controller when in SATA mode, but in NVMe mode they are just a PCIe device and no impact to SATA.

 

You haven't said what disks you are using, but you have 8 disks online now?  That should be more than enough to saturate 10GBe WITHOUT cache. Try the 10Gbe card first.  Mellanox and Intel are super cheap on eBay and you can move to new system or resell as needed.  Then if you still want to add more disks, you will need to sort out how to get more ports. Many ITX boards will bifurcate an x16 or x8 slot into two x8 or x4 slots.  That should be a configuration in your BIOS. You'll need to figure out a riser cable and how to mount two cards elsewhere in the case.

 

ITX motherboards are hard to find with the specs you want, particularly if you want to keep using your CPU (although the E3C246D4I-2T might meet your needs).  I went with a case that accommodated mATX for that reason.

[IG-88]

there are a lot of nice looking components that will things go south, you need to look very careful for specs and chips on cards

1 hour ago, SnowDrifter said:

https://www.amazon.com/Syba-SD-ADA40118-Socket-Adapter-Converter/dp/B01N5O4UNB

like this one, hyper duo ? sounds like marvell, might only have one at best two pcie lanes and can only do pcie 2.0

i looked it up and its a 88SE9230 so its 2 lanes pcie 2.0 resulting in max 1000MB/s for the 4 drives, so not optimal

 

the M.2 is pcie 3.0 x4 and there are M.2 to pcie 4x slot adapters with flex cable, so the m.2 can become a pcie 4x slot

https://www.amazon.com/ADT-LINK-Extension-Express-Extender-R42SF/dp/B08DRDT47K

(i bought two, the shorter one above works so far but i did not test much as its in my backup system with 10 disks and kind of crammed)

 

the pcie 16 slot looks like as it has 16 lanes and is pcie 3.0

maybe a jmb585 based card that supports pcie 3.0 (even with just 2 lanes its way better then the old marvell cards with just pcie 2.0

 

read this thread about some options

https://xpenology.com/forum/topic/35882-new-sataahci-cards-with-more-then-4-ports-and-no-sata-multiplexer/

 

i fully agree with @flyride ssd cache might not be worth the risk and cost, 16GB or 32GB ram do some full 10G speed buffering too

might be better to have a ssd volume for special purpose like vm/docker

using the 16x slot for a two or four m.2 ssd seems not a option here as the cheap adapters need bifurication to work and the manual does not mention that

as flyride suggested try 10G nic first and see how it performs and then decide how to go further

ssd cache makes no sense with 1G nic you won't see any difference your 8 disks should easily saturate that nic

 

i also use a matx but one the has lower depth just 10mm more the m-itx ()but its wider and has 4 pcie slots), getting a new housing might be cheaper and easier the a m-itx with 10G onboard and enough sata options to have 12 or more ports (i know that as my system has 12 ports and 10G and i had m-itx before)

[SnowDrifter]

Thanks for the info

I like that m.2 riser cable


Since I have 32g in my machine... Let's tackle one thing at a time

Sanity check: These should be compatible w/ 918+ / 6.2.3 U3 / 1.04b / extended driver lzma, yeah? Just need to get the mac address off the things and update in grub + set the number of ethernet adapters to 4

https://www.ebay.com/itm/293859944087

Edited April 21, 2021 by SnowDrifter