ferno

Don't fill all 12 Bays!

Recommended Posts

OK, I thought it to be a good idea to share my experience (and frustration) here to help others and prevent them form loosing time etc.

 

I have been running Xpenology for a while now, first native on my HP proliant micro server G7 and then moved to a HP proliant xenon based server with ESXI 5.5 and now 6.0 with vmdk files.

All was running super stable on 5.2-5592.X until I decided to upgrade to the latest version and a new xenoboot, then the problems started.

First I ran into issues with the upgrade and after I was able to fix them I started having storage issues.

 

My first assumptions was that something got corrupted during the first upgrade that went wrong so I started over after a backup and this time I decided to fill up all bays to use all space possible, I never had done that, alway had some bays left for future expandability, but this time I decided to use them all.

 

And after the install etc. all problems started, one drive kept crashing, reboot and drive was normal again but had to repair the volume etc. and during the repair the same drive kept crashing.

Shut down of the VM removing the vodka and recreating it did not solve the issue, same error.

So at this time I am pulling out hair out of my head and decided to blame the 5644.X for the problems which was weird since my backup test penology station was running fine on 5644.X, but you probably guessed it already, with 4 bays empty and not filled up like the production one I was trying to setup.

 

So I went back to the latest version I had been using for more then a year without problems and the only difference this time was a filled drive bay and once again after a while one drive crashed etc.

 

So I started eliminating options,

 

vmware tools, with and without = same issue

Eager zeroed drive vs lazy zeroed = same issue

Copying data in smaller amount = same issue

etc.

 

So at a certain point it dawned upon me, the only big difference was the filled up drive bays.

 

So I started again with all the options I needed but with one bay free and BINGO!!

Xpenology running fine for more then 2 weeks now, no problem what so ever.

 

So my conclusion is that there is a issue with xenoboot when you use all bays of the DS3615xs.

 

Anybody having these issues? I could reproduce them on my test environment so I am pretty sure this is an issue but would like to hear form other people.

Nobody using all the 12 bays? I mean the 12 bays for storage and the usb xenoboot drive hidden as an ice drive.

 

Would like to hear from people so we can report this as an real issue.

 

Thank for your patience if you red though the whole thing. :wink:

Share this post


Link to post
Share on other sites

I have seen build with 20 HDD's so i doubt that 12 bay's will is a problem (bare metal).

Running bare metal here, so i can only guess what went wrong.

Did you try with another hypervisor? Will try later this week a vbox with 12 disks and report back.

 

 

Posted via Xpenology.us

Share this post


Link to post
Share on other sites

I'm running 2 *12 bay units and one 16 bay unit (edited synoinfo.conf) all bare metal, .4 XPE

The biggest problem I had was with the 16 bay unit, which suffered from drive disconnects and crashed volumes. I tracked it to the 500w PSU, which was not powerful enough to run all the drives + hardware, (even though the current ratings said it should). I got a 550w PSU and its stable now.

Share this post


Link to post
Share on other sites

I'm running 2 *12 bay units and one 16 bay unit (edited synoinfo.conf) all bare metal, .4 XPE

The biggest problem I had was with the 16 bay unit, which suffered from drive disconnects and crashed volumes. I tracked it to the 500w PSU, which was not powerful enough to run all the drives hardware, (even though the current ratings said it should). I got a 550w PSU and its stable now.

What brand did you use?

 

 

Posted via Xpenology.us

Share this post


Link to post
Share on other sites

Hi Brantje, What do you mean by brand?? Harddisks?

 

Well the hardware is a HP Proliant microserver G8 with a xeon E3-1230 v2 processor and 4 3TB WD red drives and 16GB of ram.

But whats is more relevant (I think) is the hypervisor, that is ESXi 6.0 an the VM is setup as such:

 

NIC = E1000

SCSI = LSI Logic paralel

Mem = 4GB

CPU = 8 cores

HARDDIKS = VMDK lazy zeroed (tried eager zeroed with same result)

 

But when I do not fill up all slots it works like a charm.

Share this post


Link to post
Share on other sites
I have seen build with 20 HDD's so i doubt that 12 bay's will is a problem (bare metal).

Running bare metal here, so i can only guess what went wrong.

Did you try with another hypervisor? Will try later this week a vbox with 12 disks and report back.

 

 

Posted via Xpenology.us

 

Hi Thanx for trying, really want to end out what si causing this.

Also maybe it is related to using large drive and all bays filled so if possible try with 1 TB drives.

 

How did you get 20 Bays? The DS3615XS only supports 12, did you manage to get a drive expansion bay virtualised? And if so how??

Share this post


Link to post
Share on other sites

I spoke to soon, when DSM finished expainding the volume, the entire volume crashed, so i restarted DSM.

After the reboot was complete, i was able to see all 12 drives, and sintall software.

Tested with virtualbox 5.0.14

 

 

 

Posted via Xpenology.us

Share this post


Link to post
Share on other sites

Hi,

 

exactly, was what I experienced, first it looks good but when you expand or try to copy large amounts of data etc. volume crashes and on disk keeps failing.

After a reboot drive is green again but then the repair fails after 32%

Share this post


Link to post
Share on other sites

Didn't buy a top branded PSU, cheapest one I could find on ebuyer,

http://www.ebuyer.com/566862-ace-black- ... suace550br

didnt want to waste money in case it was another problem, but so far the system is stable.

This is my backup target, 16 drives of various sizes and brands and ages (one of them has 44000 hours in SMART) and while I want it to be stable, if I lose a disk its not critical.

Share this post


Link to post
Share on other sites

PSU is the most important in a NAS, if it dies, it can take your whole nas with it.

I prefer to spend a bit more on the PSU, then playing russian roullete with my data :razz:.

 

 

Posted via Xpenology.us

Share this post


Link to post
Share on other sites

Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU.

Share this post


Link to post
Share on other sites

Hi Guys,

 

I feel we are getting offtrack here, I can almost certainly rule out PSU problems.

My Nas is running on a HP Proliant server with top notch PSU and since the problems affect the VM and not the host I think it is safe to rule that out as the cause.

 

One thing I have to rule out is if I start with the 12 drives right away it causes the same issues, until now I filled all available slots afterwards.

During the rebuild to expand the volume problems start or when I start copying large amounts of data to it.

 

Also, I will have to test if the size of the VMDK files can affect the outcome, this second part is harder to test as I do not have much space left on my datastores.

Share this post


Link to post
Share on other sites
Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU.

 

 

How much power does this badboy draw?? Power is one of the things I pay attention to as it runs 24x7.

 

My Proliant server now with 4 3Tb wd Red drives and 16GB ram only draws 59 Watt running ESXi 6, xpenology (with loads of services on it) VM, Vcenter VM and one windows 2012 R2 server core edition.

 

I think that is pretty sweet since my original DS1512+ draw 50 Watt with 5 drives but did not have the punch to transcode several streams in plex and no Hypervisor options.

Share this post


Link to post
Share on other sites

Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks?

Share this post


Link to post
Share on other sites
Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks?

 

Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive.

Share this post


Link to post
Share on other sites
Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU.

 

 

How much power does this badboy draw?? Power is one of the things I pay attention to as it runs 24x7.

 

My system idles around 100W and goes up to 155W under load. System consists of Xeon 1275v3, 2x8GB DDR3 ECC, LSI 9211-8i SAS2 card, Intel SAS2 Expander, and a Mellanox 10Gb ConnectX-2 EN

Share this post


Link to post
Share on other sites
Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks?

 

Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive.

 

AFAIK, DSM has no awareness of which drives vmdks physically reside on. So, with multiple vmdks per physical drive, I think you're at risk of

SHR reporting that it's resilient, when in fact it isn't (because data and parity reside in separate vmdks on the same physical drive)

 

I think that, in this scenario, you need to mirror physical drive topology in the logical topology and have a single vmdk per physical drive.

Share this post


Link to post
Share on other sites
Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks?

 

Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive.

 

AFAIK, DSM has no awareness of which drives vmdks physically reside on. So, with multiple vmdks per physical drive, I think you're at risk of

SHR reporting that it's resilient, when in fact it isn't (because data and parity reside in separate vmdks on the same physical drive)

 

I think that, in this scenario, you need to mirror physical drive topology in the logical topology and have a single vmdk per physical drive.

 

Hi,

 

yes, I am aware of that, I just use the RAID to create a larger volume, and even knowing that since I do not have real redundancy I should opt for no spare drive (raid 0) I thought (and my thought proved right a couple of times) this way ik I somehow mess with one of the vodka's I won't be screwed right away.

Share this post


Link to post
Share on other sites

OK,

 

I have tried the expansion to 20 drives, At first I did not manage to get it running, but after some filling I got it running.

But same issue, when I use the 20 drives volume crashes and when I leave one drive empty everything works fine.

 

Even though I can live with this situation it seems like there is a real issue with xpenology, at least when using ESX and vmdk's

Share this post


Link to post
Share on other sites

G'day

 

I don't have a solution but I'm also experiencing the same issue as the OP, my setup as follows;

 

ESXi v6.0

15 various sized physical drives mapped to VM

(No power supply issues)

 

1. Originally had 2 separate DSM's inside ESXi, both less than 12 drives

2. This worked no issues whatsoever for a couple of years

3. Ran out of space, decided to combine the two DSM's

4. Modified the synoinfo.conf, this worked no worries - 45 total slots

5. Added 1 more drive (still less than 12), crashes

6. Realized its because i didn't format the other extra drive

7. Formatted, added, works good

8. Added rest of the drives total 15, works when drives are blank

9. Expanded the volume, all still working

10. after reset, it crashes, the extra (12+) drives disappear and the synoinfo.config resets

11. Every time i attempt to modify the synoinfo.config, it resets after reboot

 

Conclusion

 

-More than 12 drives is ok if they're blank

-As soon as there is a synology partition on the extra drives, it crashes

-From what everyone else is saying this doesn't happen on bare metal or if you passthrough the whole PCI storage adaptor

 

Seems DSM sees the drives as foreign or in a different order when in a virtual environment, I don't know?

 

Question is, where is the original synoinfo.config file coming from, can we modify the original? Is it on the boot image?

 

reclaime.com finds the raid and shows all the data still there, i'm going to try a PCIe passthrough same as Kanedo

 

Hope this info was somewhat usefull to someone

Share this post


Link to post
Share on other sites

Ben,

 

what you are describing is the synoinfo.config losing the changes, it has nothing to do with the problem being a 12 drives limitation.

 

How are you editing and saving the changes to synoinfo.config, when the machine is running? or when it's powered off ?

 

I edit it on the USB drive, on another machine linux/windows then I bring it back to the server, and it remains unchanged unless I make new changes to it.

Share this post


Link to post
Share on other sites

AllGamer,

 

I'll try that and post results soon

 

BTW i'm using XPEnoboot_DS3615xs_5.2-5644.4.vmdk

 

Cheers

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.