LeeBear

[GUIDE] Nanoboot ESXi 5.5.0 Perfect Install

Recommended Posts

I've been lurking in this forum for awhile and have learned quite a bit and figured it's time to contribute back and maybe help someone else. I'd like to thank Trantor, sancome, Diverge, Poechi, and anyone else I've missed with sharing there knowledge that made this guide possible.

 

This guide will provide step by step instructions to create the "Perfect Install" of Nanoboot on your ESXi host. I am calling this the "Perfect Install" because when you are done there won't be any extra "unused" disk showing up in DSM, nor will your disks start in the 3rd or 4th slot, also your Nanoboot boot drive won't get overwritten during install. You are left with something like this...

Final.jpg

Nice, Clean, Perfect!

 

Now Let's get started!

 

Requirements: Nanoboot IMG file, StarWind V2V Converter, WinImage, Synology DSM 5.0-4482 .pat file

 

Overview of steps:

1. Create the VM

2. Modify IMG file to prevent boot drive from showing up in DSM

3. Convert Nanoboot IMG to VMDK and upload to Datastore

4. Adding Hard Drives to your VM

5. Installing Synology DSM 5.0-4482

 

1. Create the VM

- Use VSphere Client and Create new Custom VM on your ESXi host.

- Name it what you want. [i used DiskStation]

- Store VM where you want. [I used a fast SSD Drive]

- Choose Virtual Machine Version 8.

- Choose any 64 bit Linux as Guest Operating System. [I used Ubuntu Linux 64-bit]

- Configure CPU/RAM according to what you have. [I used 8 core, 4GB RAM]

- Choose “E1000” as network adapter.

- Choose “VMware Paravirtual” as SCSI Controller. [Other may work as well]

- Choose Do Not Create Disk

- Check “Edit before completion of VM”

- Remove CD and Floppy Drive from VM configuration. [Not necessary but I don’t like unnecessary devices]

 

2. Modify IMG file to prevent boot drive from showing up in DSM

- Start up WinImage

- File -> Open, Select “NanoBoot-5.0.2.4-fat.img”

- Browse to \boot\syslinux right click -> Extract on syslinux.cfg

- Open syslinux.cfg that you just extract with Notepad

- Add the “rmmod=ata_piix” (without quotes) to the end of the lines that start with “kernel /ZImage”…

*Should be 5 lines, but the only 2 really required are the ones labeled “MENU LABEL Synology DSM 5.0” and “MENU LABEL Synology DSM 5.0-4482”

- Save this modified syslinux.cfg file and “Inject” it back to the img file using WinImage (overwrite the file when asked)

- Save Current Image before you exit WinImage

 

3. Convert Nanoboot IMG to VMDK and upload to Datastore

- Start up Starwind V2V converter

- Choose the NanoBoot-5.0.2.4-fat.img file you want to convert

- Choose “VMware pre-allocated image”, Choose “IDE” type

*Choosing "IDE" type is very important

- This will created 2 vmdk files, NanoBoot-5.0.2.4-fat.vmdk and NanoBoot-5.0.2.4-fat-flat.vmdk.

- Upload File to ESX Host Datastore (configuration tab -> Storage -> right click drive -> Browse Data Store)

- Upload both vmdk file to the folder of your VM Name created in previous step, they will merge to a single vmdk file automatically.

 

4. Adding Hard Drives to your VM

- Edit your virtual machine setting

- Add a hard drive

- Choose “Use an existing virtual disk”

- Choose the vmdk you uploaded in the previous step.

- Make sure drive is set as IDE (0:0) and check “Independent -> non-persistent”

*IDE (0:0) is important because this is the boot disk that starts up Nanoboot*

*non-persistent is important because it prevents the non-booting situation after you do the DSM install. Technically explanation is during DSM install all hard drives gets repartition including the nanoboot drive, the non-persistent setting makes these changes temporary and after reboot the original nanoboot boot partition is returned.

 

[Optional but recommended step]

- Add another hard drive

- Choose “Create a new virtual disk”

- Choose size 8+ GB, Thick Provision Lazy Zeroed. [If you will be using Plex transcoding I suggest 32GB as Plex uses alot of temp space during transcoding]

- Choose SCSI ID (0:0), check “independent -> persistent”

*This drive will become Volume1 of your NAS later on. It is the default location where Synology Apps are installed so having them on a virtual disk [sSD] makes it faster to launch apps. If you omit this step your mechanical (data) drives you add later on will become Volume1 and Apps will install on the same volume as your data. Having a separate drive also allows you to make snapshotting (backup) easier (only 8-32GB instead of TB’s).*

 

Adding your Data Drives

-Add Drives to your VM like you’ve previously done. [use Raw Device Mapping if you can for best performance] If your system doesn’t support it you must create virtual RDM first. Make sure they are all using SCSI ID's.

*IDE drive types will not show up in DSM with this install only SCSI types. SCSI (0:0) will be the drive in first slot of, SCSI (0:1) the second slot, etc.

Your VM should look something like this when you are done.

VM_config.jpg

 

5. Installing Synology DSM 5.0-4482

- Right Click select “Open Console” so you can view the VM

- Start the VM, you will see Nanoboot screen, then the menu… choose “Upgrade/Degrade”

- On Next Menu choose the DSM version you want to install. [In our case 5.0-4482]

- Once fully booted use a web browser and goto the IP address of the DiskStation. You can use Synology Assistant to find the IP.

- Choose “Install file from my computer or installation disc” select your .pat file

*Uncheck “Create a Synology Hybrid (SHR)” volume after installation”

- Wait a few minutes and you should see the DSM Login screen.

- Log in using admin account.

- Skip the Quick Connect setup [You can’t use Quick Connect because our DiskStation doesn’t have a real serial number in Synology Database]

- Goto Storage Manager -> Volume and create your Volumes

DONE!

 

UNAS_running.jpgESXi_host.jpg

Edited by Guest

Share this post


Link to post
Share on other sites

Short question,

How are you using all those HDDs in ESX? You have a separate partition where you installed ESX or you created a raid matrix and presented it to ESX?

Share this post


Link to post
Share on other sites

Hm probably some stupid mistake but its not working for me -

"Partition layout is not Diskstation style"

 

Edit: Ok, its due to the LSI adapter (and its set of old installation disks) - without i was able to install properly, attaching afterwards works

Edited by Guest

Share this post


Link to post
Share on other sites
Short question,

How are you using all those HDDs in ESX? You have a separate partition where you installed ESX or you created a raid matrix and presented it to ESX?

 

I created RDM mappings for the physical drive using this guide. (ignore the last step about the SCSI IDs). Make sure the physical drives aren't part of your "Datastore" and you will be able do this. Mine looks like this:

 

v_RDM.jpg

 

Don't worry about the 4TB size, it's not actually using up 4TB on your Datastore drive. The benefits of doing it this way (instead of a regular Virtual Disk) is faster speed, and in theory you should be able to physically remove those drives later on and put them in a real Synology Diskstation as the raw data/layout will be DSM's RAID/SHR (I used SHR) style.

 

The only drive I have visible to the ESXi host is my 256GB SSD, which houses the Datastore. The four phyiscal 4TB drives are essentially passed to the VM running DSM. The Raid matrix is created and handled by DSM (just like a real Synology DiskStation), no third party controller, etc.

 

My Setup is like this:

32GB USB Stick - VMware ESXi is installed onto this drive and boots from it

256GB SSD - Added to Datastore on ESXi host, stores VMs, and Virtual Disk

4 x 4TB - Mapped Raw LUN to the VM running Diskstation

Share this post


Link to post
Share on other sites
Hm probably some stupid mistake but its not working for me -

"Partition layout is not Diskstation style"

 

You need to provide more information. Having the "Partition layout is not Diskstation style" is normal. What step are you stuck on?

Share this post


Link to post
Share on other sites

May I ask if the procedure for changing serial key for NANOBOOT the same as GNOBOOT?

Share this post


Link to post
Share on other sites

Excellent guide, an additional resource to the other guides presented by other board members.

 

The IMG to VMDK thing is fantastic, wondered how to do that myself.

 

Thanks

Share this post


Link to post
Share on other sites

@LeeBear : Thanks a lot for that great guide.

I have one question. At the moment im running xpenology 4.3 on a hp n54l (installed on ESX 5.5). 4 hdd are presented to the vm as RDM disk. Everything is running fine but I would like to upgrade to nanoboot + dsm 5.0-4482.

Would it be possible to upgrade directly into dsm 5 without loosing any data ?

Thanks.

Share this post


Link to post
Share on other sites
@LeeBear : Thanks a lot for that great guide.

I have one question. At the moment im running xpenology 4.3 on a hp n54l (installed on ESX 5.5). 4 hdd are presented to the vm as RDM disk. Everything is running fine but I would like to upgrade to nanoboot + dsm 5.0-4482.

Would it be possible to upgrade directly into dsm 5 without loosing any data ?

Thanks.

 

There shouldn't be any problems upgrading to DSM 5 as long as you carry over the entire set of disk (RDM and any virtual disk you have). You should try to also keep the disk order the same (although I've never had any problems out of order disks). What you are doing is essential "Migrating" (taking a set of disk and moving it to another) from one DiskStation to another and should take the proper steps to backup your configuration files before doing it.

 

I don't have an HP N54L, but I have migrated from gnoboot 5.0-4458 to nanoboot 5.0-4482 without any problems. What I did was turn off the VM running gnoboot, create a new VM running Nanoboot using my guide and just attached the old drives (from gnoboot) and during the install step when you use Synology Assistant to find your DiskStation... it showed as Migrateable. If it says that then you are good (it's seeing your drives with DSM partition) and when you do the install you won't get that popup that says "Disk X, Y, Z will be deleted". If it doesn't say Migrateable then I'd come back and ask for more assistance. Goodluck.

Share this post


Link to post
Share on other sites
@LeeBear : Thanks a lot for that great guide.

I have one question. At the moment im running xpenology 4.3 on a hp n54l (installed on ESX 5.5). 4 hdd are presented to the vm as RDM disk. Everything is running fine but I would like to upgrade to nanoboot + dsm 5.0-4482.

Would it be possible to upgrade directly into dsm 5 without loosing any data ?

Thanks.

 

There shouldn't be any problems upgrading to DSM 5 as long as you carry over the entire set of disk (RDM and any virtual disk you have). You should try to also keep the disk order the same (although I've never had any problems out of order disks). What you are doing is essential "Migrating" (taking a set of disk and moving it to another) from one DiskStation to another and should take the proper steps to backup your configuration files before doing it.

 

I don't have an HP N54L, but I have migrated from gnoboot 5.0-4458 to nanoboot 5.0-4482 without any problems. What I did was turn off the VM running gnoboot, create a new VM running Nanoboot using my guide and just attached the old drives (from gnoboot) and during the install step when you use Synology Assistant to find your DiskStation... it showed as Migrateable. If it says that then you are good (it's seeing your drives with DSM partition) and when you do the install you won't get that popup that says "Disk X, Y, Z will be deleted". If it doesn't say Migrateable then I'd come back and ask for more assistance. Goodluck.

 

Thanks for your reply, I will follow your advise and give a try when I will find some free time. I will post my feedback here (hopefully soon !) .

 

EDIT :

Upgrade successful ! No issue except shared folders permission lost, 2min in order to fix it. Thanks again.

Share this post


Link to post
Share on other sites

Thanks for your reply, I will follow your advise and give a try when I will find some free time. I will post my feedback here (hopefully soon !) .

 

EDIT :

Upgrade successful ! No issue except shared folders permission lost, 2min in order to fix it. Thanks again.

Awesome to hear that the upgrade worked for you. In the future you can avoid losing user/groups/shared folders permission by backing up your configuration file then restoring it after. Even though it only took you 2 minutes to fix this time it may not always be the case in the future.

 

I guess the important thing to understand is when we use gnoboot or nanoboot we can't do a normal "upgrade" like on a real DiskStation because our setup doesn't have an internal boot flash that can be updated. Everytime we move from one DSM version to another we are actually doing a "Migration" from one DiskStation to another, while we can retain the data on the disks we lose some configuration settings (like your folder permissions, etc). It's why backing up your configuration is always a good idea.

Share this post


Link to post
Share on other sites

I have followed your guide but when I try to install it thinks my machine is a DS214Play. Any work around for that? Or did I screw something up? Yep I used the wrong image file without the rmmod in it. Sorry!!!

Thanks!!

Share this post


Link to post
Share on other sites

I have install DSM on ESX 5.1. Installation went without any problems.

I now have the problem that we DSM works only with the local network.

DSM does not have access to the Internet.

Does anyone have any idea what could be wrong?

Share this post


Link to post
Share on other sites
I have install DSM on ESX 5.1. Installation went without any problems.

I now have the problem that we DSM works only with the local network.

DSM does not have access to the Internet.

Does anyone have any idea what could be wrong?

 

We need to know how your system is setup, especially the Physical Network Port. If you setup the port to only allow internal connections that will explain why you don't have internet access.

Share this post


Link to post
Share on other sites

Many thanks for your guide.

I'm curious about your READ/Write speed using that new avoton board with esxi 5.5.

 

Regards

Rob

Share this post


Link to post
Share on other sites
Many thanks for your guide.

I'm curious about your READ/Write speed using that new avoton board with esxi 5.5.

 

Regards

Rob

I wasn't getting great speed's with ESXi 5.5, it was erratic usually fluctuating between 20-80 MB/s using virtual RDM and E1000 network. I have a feeling it's the network drivers or some settings in it that wasn't correct. Anyways to test my theory I decided to duplicate the setup on Hyper-V Server 2012 R2... this was easy to do as I had my ESXi booted off a USB stick (but Datastore on an SSD) and Hyper-V booting off an SSD. Since I didn't use virtual drives for my data the only drive I had to convert to VHD format was my 32GB virtual drive for application. Once I created the Hyper-V VM and mounted the drives, DSM started up fine, no reinstall of anything. I did some copying over the network to see if the speed was better and sure enough it is... this is the results I'm getting in Hyper-V while copying approximately 100GB of mostly 50MB files:

 

Hyper_V_Rate.jpg

 

As you can see the speeds way better then ESXi. This is a 4 drive (5400 rpm) in SHR configuration over 1 network port. I know DSM usually shows higher rates cause there's 1 drive of parity (so approximately 33% higher rate in a 4 drive configuration), Windows shows transfer rate between 90-110 MB/s very consistent. I had to disable VMQ (Virtual Machine Queuing) setting in the Network controller to get this results. With VMQ on the speed was erratic like in ESXi. I will move back to ESXi when I have time and try to figure out if there's an equivalent setting that needs to be disabled to get the same performance as Hyper-V. If there is then it's going to be a tough choice... ESXi is more widely supported and very simple to manage, while Hyper-V Server is ridiculously hard to manage (no gui, can be remote managed only by Win 8.1 machine, usually requires a domain), has good performance and lower power consumption (approx 43W vs 50W).

Share this post


Link to post
Share on other sites

LeeBear Your a Champ :smile:

I'm also abit confused in wich way to go but your information is really helpfull since Speed is a priority in a

hacked synology system.

 

I hope more people post their speed and solutions so we can all benefit this crucial information.

 

Cheers and Thanks again.

Share this post


Link to post
Share on other sites

Hi,

 

I think it's worth mentioning another aspect of HyperV vs ESXi:

 

HyperV allows you to pass-through just a single disk to a VM with access to the disk's SMART as opposed to ESXi ability to only pass-through entire disk controller. If you do RDM with ESXi, then DSM's SMART monitoring doesn't work.

 

Cheers,

Vladimir

Share this post


Link to post
Share on other sites

I registered my account just to thank you for the help.

 

I have successfully installed DSM on my ESXi 5.5 home server. I'm just using old second hand consumer components so I hope I don't get any long term stability issues, especially with overclocking. The specs are i7 920 @ 4GHz OC + gigabyte ex58-ud5 with 24GBs of RAM. I'm using Crucial m500 240GB for ESXi and the DSM VM, as well as 6 other 5400rpm hard disks for the data storage. This will host other VMs in the future as well for software development and testing purposes.

 

I think that the hardware is overkill and storing my data using virtual RDM is not a good idea. I assume that if I lose my hardware I can just redo the installation on another machine and data on my existing disks will still work?

 

How do I install this on the hardware itself instead of installing on ESXi? I assume it would involve a USB stick but is there anyway I can just install Nanoboot on the SSD itself? I hope you can also do a short guide on how to do a direct installation.

Share this post


Link to post
Share on other sites
I registered my account just to thank you for the help.

 

I have successfully installed DSM on my ESXi 5.5 home server. I'm just using old second hand consumer components so I hope I don't get any long term stability issues, especially with overclocking. The specs are i7 920 @ 4GHz OC + gigabyte ex58-ud5 with 24GBs of RAM. I'm using Crucial m500 240GB for ESXi and the DSM VM, as well as 6 other 5400rpm hard disks for the data storage. This will host other VMs in the future as well for software development and testing purposes.

 

I think that the hardware is overkill and storing my data using virtual RDM is not a good idea. I assume that if I lose my hardware I can just redo the installation on another machine and data on my existing disks will still work?

 

How do I install this on the hardware itself instead of installing on ESXi? I assume it would involve a USB stick but is there anyway I can just install Nanoboot on the SSD itself? I hope you can also do a short guide on how to do a direct installation.

I'll try to answer your questions the best I can, since I am not an expert. Your first question regarding using virtual RDM and why I suggest using it instead of a virtual disk (vmdk). Besides getting better speed you get portability of your data. This means you can take your hard drives and put it in a real Synology DSM and your data will still work, or you can put the drives in a Hyper-V environment and your data will still be there. I have personally tested Hyper-V so I know it works. The reason it works is because if you use RDM you are letting the DiskStation software have direct access to the disk drive to create the DSM partitions. If we compare what is physically stored on the disk drive if you took the hard drive out and put it in a non ESXi environment you will get this (I will use a 1TB drive as an example):

 

virtual RDM drive: 2GB Partition (DSM software version), 2.5GB Partition (Volume information), 965 GB Partition (Data)

virtual (vmdk) drive: 1TB Partition with disk.vmdk file

 

As you can see if you put a virtual drive instead of an RDM drive into a real Synology DiskStation it won't see the data or DSM partition and it will tell you to format the drive (you lose all your data). With the RDM drive it will see the DSM partition and know it came from a DiskStation and mount the volume with data in tact.

 

Regarding your question of virtual RDM not being a good idea I don't think that's correct. We have to use the virtual RDM "trick" because VMware by default doesn't let you use a local drive as an RDM... it only allows you to use a SAN or other storage. I don't believe there's any risk in using virtual RDM because the Synology software is actually handling the RAID, if there was some error DSM would alert you.

 

Now for your second question of installing Nanoboot directly to your system. Yes you can do that, you will need to use a USB or CD to boot off of, you can't boot off an SSD because DSM will detect the SSD drive and format it during install. Remember in a real DiskStation there is a flash drive inside that it boots off of we "fake" this using Nanoboot on a USB or CD. I will not write a short guide on doing a direct install because there are similar guides already (Look at install guides for HP N40L/N54L). The reason for this is because every system has different hardware, unlike ESXi where it doesn't matter what hardware you have the Virtual Machine will be the same. Also keep in mine that Nanoboot has to have drivers for your hardware if you want to do a direct install unless you know how to add your own drivers and compile Nanoboot yourself. You can always unplug your drives and make a USB stick with Nanoboot on it and see if your hardware is support then decide if you want to do a direct install.

Share this post


Link to post
Share on other sites