ferno Posted February 28, 2016 #1 Posted February 28, 2016 OK, I thought it to be a good idea to share my experience (and frustration) here to help others and prevent them form loosing time etc. I have been running Xpenology for a while now, first native on my HP proliant micro server G7 and then moved to a HP proliant xenon based server with ESXI 5.5 and now 6.0 with vmdk files. All was running super stable on 5.2-5592.X until I decided to upgrade to the latest version and a new xenoboot, then the problems started. First I ran into issues with the upgrade and after I was able to fix them I started having storage issues. My first assumptions was that something got corrupted during the first upgrade that went wrong so I started over after a backup and this time I decided to fill up all bays to use all space possible, I never had done that, alway had some bays left for future expandability, but this time I decided to use them all. And after the install etc. all problems started, one drive kept crashing, reboot and drive was normal again but had to repair the volume etc. and during the repair the same drive kept crashing. Shut down of the VM removing the vodka and recreating it did not solve the issue, same error. So at this time I am pulling out hair out of my head and decided to blame the 5644.X for the problems which was weird since my backup test penology station was running fine on 5644.X, but you probably guessed it already, with 4 bays empty and not filled up like the production one I was trying to setup. So I went back to the latest version I had been using for more then a year without problems and the only difference this time was a filled drive bay and once again after a while one drive crashed etc. So I started eliminating options, vmware tools, with and without = same issue Eager zeroed drive vs lazy zeroed = same issue Copying data in smaller amount = same issue etc. So at a certain point it dawned upon me, the only big difference was the filled up drive bays. So I started again with all the options I needed but with one bay free and BINGO!! Xpenology running fine for more then 2 weeks now, no problem what so ever. So my conclusion is that there is a issue with xenoboot when you use all bays of the DS3615xs. Anybody having these issues? I could reproduce them on my test environment so I am pretty sure this is an issue but would like to hear form other people. Nobody using all the 12 bays? I mean the 12 bays for storage and the usb xenoboot drive hidden as an ice drive. Would like to hear from people so we can report this as an real issue. Thank for your patience if you red though the whole thing. Quote
brantje Posted February 28, 2016 #2 Posted February 28, 2016 I have seen build with 20 HDD's so i doubt that 12 bay's will is a problem (bare metal). Running bare metal here, so i can only guess what went wrong. Did you try with another hypervisor? Will try later this week a vbox with 12 disks and report back. Posted via Xpenology.us Quote
sbv3000 Posted February 28, 2016 #3 Posted February 28, 2016 I'm running 2 *12 bay units and one 16 bay unit (edited synoinfo.conf) all bare metal, .4 XPE The biggest problem I had was with the 16 bay unit, which suffered from drive disconnects and crashed volumes. I tracked it to the 500w PSU, which was not powerful enough to run all the drives + hardware, (even though the current ratings said it should). I got a 550w PSU and its stable now. Quote
brantje Posted February 28, 2016 #4 Posted February 28, 2016 I'm running 2 *12 bay units and one 16 bay unit (edited synoinfo.conf) all bare metal, .4 XPE The biggest problem I had was with the 16 bay unit, which suffered from drive disconnects and crashed volumes. I tracked it to the 500w PSU, which was not powerful enough to run all the drives hardware, (even though the current ratings said it should). I got a 550w PSU and its stable now. What brand did you use? Posted via Xpenology.us Quote
ferno Posted February 29, 2016 Author #5 Posted February 29, 2016 Hi Brantje, What do you mean by brand?? Harddisks? Well the hardware is a HP Proliant microserver G8 with a xeon E3-1230 v2 processor and 4 3TB WD red drives and 16GB of ram. But whats is more relevant (I think) is the hypervisor, that is ESXi 6.0 an the VM is setup as such: NIC = E1000 SCSI = LSI Logic paralel Mem = 4GB CPU = 8 cores HARDDIKS = VMDK lazy zeroed (tried eager zeroed with same result) But when I do not fill up all slots it works like a charm. Quote
ferno Posted February 29, 2016 Author #6 Posted February 29, 2016 I have seen build with 20 HDD's so i doubt that 12 bay's will is a problem (bare metal).Running bare metal here, so i can only guess what went wrong. Did you try with another hypervisor? Will try later this week a vbox with 12 disks and report back. Posted via Xpenology.us Hi Thanx for trying, really want to end out what si causing this. Also maybe it is related to using large drive and all bays filled so if possible try with 1 TB drives. How did you get 20 Bays? The DS3615XS only supports 12, did you manage to get a drive expansion bay virtualised? And if so how?? Quote
pastrychef Posted February 29, 2016 #7 Posted February 29, 2016 I think the problem is isolated to your set up. I've seen quite a few reports of people with 12 or more drives running. Quote
brantje Posted February 29, 2016 #8 Posted February 29, 2016 Ferno, in case yiu want more than 12 drives: https://xpenology.us/tutorials/miscellaneous/more-12-drives Currently firing up a VM with 12 drives, will keep you posted. Posted via Xpenology.us Quote
brantje Posted February 29, 2016 #9 Posted February 29, 2016 Seems to be working, will install some software after it finishes expanding. Posted via Xpenology.us Quote
brantje Posted February 29, 2016 #10 Posted February 29, 2016 I spoke to soon, when DSM finished expainding the volume, the entire volume crashed, so i restarted DSM. After the reboot was complete, i was able to see all 12 drives, and sintall software. Tested with virtualbox 5.0.14 Posted via Xpenology.us Quote
ferno Posted February 29, 2016 Author #11 Posted February 29, 2016 Hi, exactly, was what I experienced, first it looks good but when you expand or try to copy large amounts of data etc. volume crashes and on disk keeps failing. After a reboot drive is green again but then the repair fails after 32% Quote
sbv3000 Posted February 29, 2016 #12 Posted February 29, 2016 Didn't buy a top branded PSU, cheapest one I could find on ebuyer, http://www.ebuyer.com/566862-ace-black- ... suace550br didnt want to waste money in case it was another problem, but so far the system is stable. This is my backup target, 16 drives of various sizes and brands and ages (one of them has 44000 hours in SMART) and while I want it to be stable, if I lose a disk its not critical. Quote
brantje Posted March 1, 2016 #13 Posted March 1, 2016 PSU is the most important in a NAS, if it dies, it can take your whole nas with it. I prefer to spend a bit more on the PSU, then playing russian roullete with my data . Posted via Xpenology.us Quote
Kanedo Posted March 1, 2016 #14 Posted March 1, 2016 Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU. Quote
ferno Posted March 1, 2016 Author #15 Posted March 1, 2016 Hi Guys, I feel we are getting offtrack here, I can almost certainly rule out PSU problems. My Nas is running on a HP Proliant server with top notch PSU and since the problems affect the VM and not the host I think it is safe to rule that out as the cause. One thing I have to rule out is if I start with the 12 drives right away it causes the same issues, until now I filled all available slots afterwards. During the rebuild to expand the volume problems start or when I start copying large amounts of data to it. Also, I will have to test if the size of the VMDK files can affect the outcome, this second part is harder to test as I do not have much space left on my datastores. Quote
ferno Posted March 1, 2016 Author #16 Posted March 1, 2016 Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU. How much power does this badboy draw?? Power is one of the things I pay attention to as it runs 24x7. My Proliant server now with 4 3Tb wd Red drives and 16GB ram only draws 59 Watt running ESXi 6, xpenology (with loads of services on it) VM, Vcenter VM and one windows 2012 R2 server core edition. I think that is pretty sweet since my original DS1512+ draw 50 Watt with 5 drives but did not have the punch to transcode several streams in plex and no Hypervisor options. Quote
berwhale Posted March 1, 2016 #17 Posted March 1, 2016 Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks? Quote
ferno Posted March 1, 2016 Author #18 Posted March 1, 2016 Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks? Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive. Quote
Kanedo Posted March 1, 2016 #19 Posted March 1, 2016 Running 13 drives here with ESXi 6.0 and all 13 drives connected to the Xpenology VM guest using VT-d PCIe passthrough with a LSI 9211-8i SAS card in IT mode + SAS2 Expander. Not seeing any issues here. Oh, and I'm only using a Corsair CS450M PSU. How much power does this badboy draw?? Power is one of the things I pay attention to as it runs 24x7. My system idles around 100W and goes up to 155W under load. System consists of Xeon 1275v3, 2x8GB DDR3 ECC, LSI 9211-8i SAS2 card, Intel SAS2 Expander, and a Mellanox 10Gb ConnectX-2 EN Quote
berwhale Posted March 2, 2016 #20 Posted March 2, 2016 Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks? Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive. AFAIK, DSM has no awareness of which drives vmdks physically reside on. So, with multiple vmdks per physical drive, I think you're at risk of SHR reporting that it's resilient, when in fact it isn't (because data and parity reside in separate vmdks on the same physical drive) I think that, in this scenario, you need to mirror physical drive topology in the logical topology and have a single vmdk per physical drive. Quote
ferno Posted March 4, 2016 Author #21 Posted March 4, 2016 Ferno, I'm not sure I understand your setup. Are you creating multiple vmdks on each 3TB drive and presenting these to your Xpenology VM? If so, why not just create fewer, larger vmdks? Hi Good Point, Since I still want to have (SHR)RAID in synology I opted for a smaller size VMDK so I won't loose too much space because of the parity drive. AFAIK, DSM has no awareness of which drives vmdks physically reside on. So, with multiple vmdks per physical drive, I think you're at risk of SHR reporting that it's resilient, when in fact it isn't (because data and parity reside in separate vmdks on the same physical drive) I think that, in this scenario, you need to mirror physical drive topology in the logical topology and have a single vmdk per physical drive. Hi, yes, I am aware of that, I just use the RAID to create a larger volume, and even knowing that since I do not have real redundancy I should opt for no spare drive (raid 0) I thought (and my thought proved right a couple of times) this way ik I somehow mess with one of the vodka's I won't be screwed right away. Quote
ferno Posted March 4, 2016 Author #22 Posted March 4, 2016 OK, I have tried the expansion to 20 drives, At first I did not manage to get it running, but after some filling I got it running. But same issue, when I use the 20 drives volume crashes and when I leave one drive empty everything works fine. Even though I can live with this situation it seems like there is a real issue with xpenology, at least when using ESX and vmdk's Quote
Ben0 Posted June 3, 2016 #23 Posted June 3, 2016 G'day I don't have a solution but I'm also experiencing the same issue as the OP, my setup as follows; ESXi v6.0 15 various sized physical drives mapped to VM (No power supply issues) 1. Originally had 2 separate DSM's inside ESXi, both less than 12 drives 2. This worked no issues whatsoever for a couple of years 3. Ran out of space, decided to combine the two DSM's 4. Modified the synoinfo.conf, this worked no worries - 45 total slots 5. Added 1 more drive (still less than 12), crashes 6. Realized its because i didn't format the other extra drive 7. Formatted, added, works good 8. Added rest of the drives total 15, works when drives are blank 9. Expanded the volume, all still working 10. after reset, it crashes, the extra (12+) drives disappear and the synoinfo.config resets 11. Every time i attempt to modify the synoinfo.config, it resets after reboot Conclusion -More than 12 drives is ok if they're blank -As soon as there is a synology partition on the extra drives, it crashes -From what everyone else is saying this doesn't happen on bare metal or if you passthrough the whole PCI storage adaptor Seems DSM sees the drives as foreign or in a different order when in a virtual environment, I don't know? Question is, where is the original synoinfo.config file coming from, can we modify the original? Is it on the boot image? reclaime.com finds the raid and shows all the data still there, i'm going to try a PCIe passthrough same as Kanedo Hope this info was somewhat usefull to someone Quote
AllGamer Posted June 3, 2016 #24 Posted June 3, 2016 Ben, what you are describing is the synoinfo.config losing the changes, it has nothing to do with the problem being a 12 drives limitation. How are you editing and saving the changes to synoinfo.config, when the machine is running? or when it's powered off ? I edit it on the USB drive, on another machine linux/windows then I bring it back to the server, and it remains unchanged unless I make new changes to it. Quote
Ben0 Posted June 3, 2016 #25 Posted June 3, 2016 AllGamer, I'll try that and post results soon BTW i'm using XPEnoboot_DS3615xs_5.2-5644.4.vmdk Cheers Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.