ESXi SMART - Extract data and present as website


Recommended Posts

Good afternoon all, and a happy new year to you all.

 

So I use VMWare ESXi to host my Synology 6.2 - and though I know there's a plugin to retrieve SMART data, for me it doesn't work. Presumably because my server is a mish mash of random hardware; that or I'm too lazy to bugfix.

Something I've wanted to do for as long as I can remember is to extract SMART data from ESXi and add to a database, then present it as a website that's super easy to read.

 

What the site does

Well, as you can see from the image below, I have ESXi SMART data presented in an easy to read format for each of my drives, as well as other important information such as RDM location, what drive number the hard drive is on the vm host, if it's an SSD, etc, etc. It also presents you with 5 hours worth of temperature changes, just in case like me you panic every time the sun comes out.


 Capture.thumb.PNG.9cc897a388018c03b1d51e217825836b.PNG

 

How does it work?

OK so I'm a PowerShell developer. The site was built as a learning curve using PHP - it is not my natural environment. Bash is like a totally foreign language to me and one I'm not comfortable with, so I acknowledge this is a long ass way around doing this. Nevertheless, it works and I know every single aspect of what it's doing.

A PowerShell script runs as a scheduled task on a Windows machine, rips data straight out of ESXi and imports it into a MariaDB database. 

In my case I have a Windows 10 device on my ESXi host running said script, with my Xpenology OS hosting both the database and website. 

 

How do I install it?

You want a Windows device running 24/7, since this device will be ripping SMART data every thirty minutes (or whatever schedule you give it). With that in mind you're best off using your ESXi host for your Windows machine.

 

  • This website explains how to download Windows 10 without a key (this is totally legit. Downloading without a key is different to installing without a key which I do not recommend!)
  • Once you have your windows device up and running, open PowerShell as an administrator and type: 
    Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Confirm:$false

     

  • Next, you'll want to hit my BitBucket repo here and download both repos. The one specific to ripping data out is called ESXiSMART 
  • Extract the contents of the zip file and check out the README.md file for complete instructions. This will explain how to set up a scheduled task.

 

NOTE: There are a few steps you need to take before the scheduled task will work.

  • Create your scheduled task .ps1 file as per the README.md file, then wherever you save it, open PowerShell in the same folder and run the .ps1 file from the PowerShell console. This is important as it'll download the VMWare plugins and if MySQL dll file is missing, give you the link to download and install it. This information is contained within the README.md file but the script will prompt you for action if you can't be bothered to read it.

 

OK I have a scheduled task running, and I can see the database on myphpadmin, now what?

Well now you want the website. Which can be found at the same BitBucket url.

There's instructions there on how to host. This has been tested on both XAMPP and Synology. 

 

It's working! But the temp graph is just showing the same temp every time!

Yeah it shows the last 10 entries, so just be patient.

 

It's not working! What should I do?

Post here obviously, with your bugs, and I'll try resolve it for you.

 

This is nice, but it could be better!

Oh totally. Well, feel free to make your own changes. Totally open source. Or, if you have suggestions and no coding skills, just let me know what you'd like.
I'd like to do a version 2 of the website which includes alerting. I can do this in PowerShell fairly easily but as I'm learning PHP would sooner take my time and do it through here.

 

You know there's an easier way to get smart data out of VMWare without a Windows machine and PowerCLI right? 

Yeah I know. But bash is alien to me. With PowerShell and PHP I know what each command does and the risks involved. With bash I'm just typing out stuff totally unknown to me and hoping very bad things aren't happening under the hood. So I totally welcome any easier approach to this. If you know bash and want to help out, I'm very keen to have you on board. In fact, I encourage it. There's a guide here on how to get extended SMART data from ESXi which includes Total Bytes Written on SSD's. But it means installing stuff on ESXi and I'm not comfortable doing it. If you want to help build a tool that installs smartctl and runs it as a scheduled task, please do. Not having to use a Windows box to get SMART data out is the perfect scenario.

 

Are there any licenses I should be aware of?

Nope. Well. A windows license, but (and this hasn't been tested as mine is totally legit) I believe you could run a scheduled task without activating Windows. If someone can confirm I'll update this post. MySQL is free, PowerCLI is free, and the images used on the website are from Pixabay under open source license agreements.  

 

  • Like 1
Link to post
Share on other sites
  • 3 months later...
Posted (edited)

Good afternoon all.

 

I've finally got round to updating both the website and PowerShell scripts. Both code improvements and new features.

 

Website Update and Improvements.

 

    The website now hosts a 'SETTINGS' page, in which certain aspects can be controlled.

  • Hard Drive order can now be set from small to large, large to small, Raw Device Mapping Naming convention, Controller Location, and Hard Drive Number.
  • An option to hide Removed Drives (great for when you upgrade drives)
  • Alter how many plot points there are on your temperature graph.
  • Each drive can have individually set temperature threshold points.

   The Index page will alert you to any drive temperature alerts.

 

 

PowerShell SMART Data Improvements.

  • On the first run, if an email address and plain text password are provided an encrypted credentials file is generated in the temp directory.
  • From there on out, if just an email address is provided, then any irregularities found in the SMART data will trigger an email alert.

 

Email Alerts.

  • As mentioned above, email alerts are a thing now. I've set the script up to send email from GMail, Yahoo and Microsoft based providers. If you're using your own mail server you'll need to provide details of said provider as both a parameter on the main Start-SmartDataScript and in the Send-EmailAlerts script (see image below for information).
  • If you have MFA / 2-Step authentication switched on (and if you don't you really should!) you'll need to provide an app password rather than your standard password. More information on this is provided in the README.MD file within the PowerShell Module.

 

 

If anyone uses it and finds any bugs, please do let me know. I know of two already but neither effect normal usability and will be fixed in version 3.0.0 - which will include Windows Event logging.

 

Images from Version 2.0.0
The first is a comparison of the old index vs the new. I've replaced a drive whilst building version 2.0.0, this is no longer present on the index. Also note the graph is plotted over a longer time period.


In the second image, you can see the new Settings page. Note the SSD drive has different default Temperature thresholds than spinning disks.

 

The third image is an email I got when I removed a drive in favour of a larger one. 

 

The fourth is the PowerShell function Send-EmailAlert.psm1. You will need to edit this if your Email provider is not one of the big three.
 

The last image is a work in progress on version 3.0.0. This is a means of building a scheduled task without having to use plain text passwords.

 

Download Information

My BitBucket Repo is here . Please note, whilst writing this forum post I noticed a small bug on ESXISmartWebsite. This has been posted to Master, but I have no idea how to update Release 2.0.0  so make sure to grab from Master.

 

 

Updating Existing Files

In both instances, the new repos can be placed directly over the existing ones. Both the PowerShell module and website will attempt to update the existing Database.

 

Scheduled Task Updates

Unfortunately, although the update process is easy, there is one extra thing you'll need to do, and that's to update the Scheduled Task you have running. 

Before that, open a PowerShell window from within the ESXiSMART folder and type:

Import-Module EsxiSmart.psd1 -Force -Verbose

If you get an error message about vis then just re-run. This is because your scheduled task is already using VmWare resources.

Next you'll need to provide your Email address and Email password to build an encrypted file.

 

Start-SmartDataScript -Email youremail@yourdomain.com -EmailPass yourEmailPassword

 

You only need to do this once. From here amend your Scheduled Task script with just the extra parameter

-Email youremmail@yourdomain.com

 

 

Version 3.0.0

Version 3.0.0 is already in the making. Most of the work will be within PowerShell. I intend to add Event Logging so you have visibility of the what's happening in the script whenever it's ran. 

Also, I'm currently building a small application that encrypts your ESXi and MySQL passwords so they're not needed in plain text for the scheduled task to run. 

 

 

If anyone has any suggestions, notices any bugs, wants to improve the code, or whatever. I'm all ears. Please do let me know.

comparison.PNG

Settings.PNG

emailalert.png

 

SendEmailAlerts.PNG

version3.PNG

Edited by elmuziko
  • Like 1
Link to post
Share on other sites
  • 4 weeks later...
  • 2 weeks later...
On 5/2/2021 at 11:55 PM, vbap said:

This is nice. Any chance of docker?


One would assume you want Docker for the scheduled task aspect yes? 

Windows on Docker doesn't work on Synology, but I'll look at altering the PowerShell script to run on Linux and thus solve the problem

Link to post
Share on other sites

Actually want docker for the ease of deployment (and re-deployment). I could deploy it using docker on Synology/Xpenology rather than having separate 24x7 windows VM. I also have a linux VM on ESxi where I run all my docker apps that I don't want to run inside my NAS. 

  • Like 1
Link to post
Share on other sites
  • 4 weeks later...

Hi elmuziko,

i really love your project. thanks for development.
i also use Docker in some case. so it could be easier to deploy your solution.

 

i will try first v3 on windows and give you my feedback.

if i can test anything, let me know.

 

cheers

Link to post
Share on other sites
3 minutes ago, sepult said:

Hi elmuziko,

i really love your project. thanks for development.
i also use Docker in some case. so it could be easier to deploy your solution.

 

i will try first v3 on windows and give you my feedback.

if i can test anything, let me know.

 

cheers


Thank you. I've found a few bugs in the PowerShell code so have been concentrating on that.
I'm also translating this into Linux PowerShell so I can create a Linux Docker image. 

Sadly work commitments are slowing me down but will have a lot more free time next month to complete the next version.

 

I will absolutely message when I'm at a testing stage.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.