Jump to content
XPEnology Community

ESXi SMART - Extract data and present as website


elmuziko
 Share

Recommended Posts

Good afternoon all, and a happy new year to you all.

 

So I use VMWare ESXi to host my Synology 6.2 - and though I know there's a plugin to retrieve SMART data, for me it doesn't work. Presumably because my server is a mish mash of random hardware; that or I'm too lazy to bugfix.

Something I've wanted to do for as long as I can remember is to extract SMART data from ESXi and add to a database, then present it as a website that's super easy to read.

 

What the site does

Well, as you can see from the image below, I have ESXi SMART data presented in an easy to read format for each of my drives, as well as other important information such as RDM location, what drive number the hard drive is on the vm host, if it's an SSD, etc, etc. It also presents you with 5 hours worth of temperature changes, just in case like me you panic every time the sun comes out.


 Capture.thumb.PNG.9cc897a388018c03b1d51e217825836b.PNG

 

How does it work?

OK so I'm a PowerShell developer. The site was built as a learning curve using PHP - it is not my natural environment. Bash is like a totally foreign language to me and one I'm not comfortable with, so I acknowledge this is a long ass way around doing this. Nevertheless, it works and I know every single aspect of what it's doing.

A PowerShell script runs as a scheduled task on a Windows machine, rips data straight out of ESXi and imports it into a MariaDB database. 

In my case I have a Windows 10 device on my ESXi host running said script, with my Xpenology OS hosting both the database and website. 

 

How do I install it?

You want a Windows device running 24/7, since this device will be ripping SMART data every thirty minutes (or whatever schedule you give it). With that in mind you're best off using your ESXi host for your Windows machine.

 

  • This website explains how to download Windows 10 without a key (this is totally legit. Downloading without a key is different to installing without a key which I do not recommend!)
  • Once you have your windows device up and running, open PowerShell as an administrator and type: 
    Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Confirm:$false

     

  • Next, you'll want to hit my BitBucket repo here and download both repos. The one specific to ripping data out is called ESXiSMART 
  • Extract the contents of the zip file and check out the README.md file for complete instructions. This will explain how to set up a scheduled task.

 

NOTE: There are a few steps you need to take before the scheduled task will work.

  • Create your scheduled task .ps1 file as per the README.md file, then wherever you save it, open PowerShell in the same folder and run the .ps1 file from the PowerShell console. This is important as it'll download the VMWare plugins and if MySQL dll file is missing, give you the link to download and install it. This information is contained within the README.md file but the script will prompt you for action if you can't be bothered to read it.

 

OK I have a scheduled task running, and I can see the database on myphpadmin, now what?

Well now you want the website. Which can be found at the same BitBucket url.

There's instructions there on how to host. This has been tested on both XAMPP and Synology. 

 

It's working! But the temp graph is just showing the same temp every time!

Yeah it shows the last 10 entries, so just be patient.

 

It's not working! What should I do?

Post here obviously, with your bugs, and I'll try resolve it for you.

 

This is nice, but it could be better!

Oh totally. Well, feel free to make your own changes. Totally open source. Or, if you have suggestions and no coding skills, just let me know what you'd like.
I'd like to do a version 2 of the website which includes alerting. I can do this in PowerShell fairly easily but as I'm learning PHP would sooner take my time and do it through here.

 

You know there's an easier way to get smart data out of VMWare without a Windows machine and PowerCLI right? 

Yeah I know. But bash is alien to me. With PowerShell and PHP I know what each command does and the risks involved. With bash I'm just typing out stuff totally unknown to me and hoping very bad things aren't happening under the hood. So I totally welcome any easier approach to this. If you know bash and want to help out, I'm very keen to have you on board. In fact, I encourage it. There's a guide here on how to get extended SMART data from ESXi which includes Total Bytes Written on SSD's. But it means installing stuff on ESXi and I'm not comfortable doing it. If you want to help build a tool that installs smartctl and runs it as a scheduled task, please do. Not having to use a Windows box to get SMART data out is the perfect scenario.

 

Are there any licenses I should be aware of?

Nope. Well. A windows license, but (and this hasn't been tested as mine is totally legit) I believe you could run a scheduled task without activating Windows. If someone can confirm I'll update this post. MySQL is free, PowerCLI is free, and the images used on the website are from Pixabay under open source license agreements.  

 

  • Like 1
Link to comment
Share on other sites

  • 3 months later...

Good afternoon all.

 

I've finally got round to updating both the website and PowerShell scripts. Both code improvements and new features.

 

Website Update and Improvements.

 

    The website now hosts a 'SETTINGS' page, in which certain aspects can be controlled.

  • Hard Drive order can now be set from small to large, large to small, Raw Device Mapping Naming convention, Controller Location, and Hard Drive Number.
  • An option to hide Removed Drives (great for when you upgrade drives)
  • Alter how many plot points there are on your temperature graph.
  • Each drive can have individually set temperature threshold points.

   The Index page will alert you to any drive temperature alerts.

 

 

PowerShell SMART Data Improvements.

  • On the first run, if an email address and plain text password are provided an encrypted credentials file is generated in the temp directory.
  • From there on out, if just an email address is provided, then any irregularities found in the SMART data will trigger an email alert.

 

Email Alerts.

  • As mentioned above, email alerts are a thing now. I've set the script up to send email from GMail, Yahoo and Microsoft based providers. If you're using your own mail server you'll need to provide details of said provider as both a parameter on the main Start-SmartDataScript and in the Send-EmailAlerts script (see image below for information).
  • If you have MFA / 2-Step authentication switched on (and if you don't you really should!) you'll need to provide an app password rather than your standard password. More information on this is provided in the README.MD file within the PowerShell Module.

 

 

If anyone uses it and finds any bugs, please do let me know. I know of two already but neither effect normal usability and will be fixed in version 3.0.0 - which will include Windows Event logging.

 

Images from Version 2.0.0
The first is a comparison of the old index vs the new. I've replaced a drive whilst building version 2.0.0, this is no longer present on the index. Also note the graph is plotted over a longer time period.


In the second image, you can see the new Settings page. Note the SSD drive has different default Temperature thresholds than spinning disks.

 

The third image is an email I got when I removed a drive in favour of a larger one. 

 

The fourth is the PowerShell function Send-EmailAlert.psm1. You will need to edit this if your Email provider is not one of the big three.
 

The last image is a work in progress on version 3.0.0. This is a means of building a scheduled task without having to use plain text passwords.

 

Download Information

My BitBucket Repo is here . Please note, whilst writing this forum post I noticed a small bug on ESXISmartWebsite. This has been posted to Master, but I have no idea how to update Release 2.0.0  so make sure to grab from Master.

 

 

Updating Existing Files

In both instances, the new repos can be placed directly over the existing ones. Both the PowerShell module and website will attempt to update the existing Database.

 

Scheduled Task Updates

Unfortunately, although the update process is easy, there is one extra thing you'll need to do, and that's to update the Scheduled Task you have running. 

Before that, open a PowerShell window from within the ESXiSMART folder and type:

Import-Module EsxiSmart.psd1 -Force -Verbose

If you get an error message about vis then just re-run. This is because your scheduled task is already using VmWare resources.

Next you'll need to provide your Email address and Email password to build an encrypted file.

 

Start-SmartDataScript -Email youremail@yourdomain.com -EmailPass yourEmailPassword

 

You only need to do this once. From here amend your Scheduled Task script with just the extra parameter

-Email youremmail@yourdomain.com

 

 

Version 3.0.0

Version 3.0.0 is already in the making. Most of the work will be within PowerShell. I intend to add Event Logging so you have visibility of the what's happening in the script whenever it's ran. 

Also, I'm currently building a small application that encrypts your ESXi and MySQL passwords so they're not needed in plain text for the scheduled task to run. 

 

 

If anyone has any suggestions, notices any bugs, wants to improve the code, or whatever. I'm all ears. Please do let me know.

comparison.PNG

Settings.PNG

emailalert.png

 

SendEmailAlerts.PNG

version3.PNG

Edited by elmuziko
  • Like 1
Link to comment
Share on other sites

  • 4 weeks later...
  • 2 weeks later...
On 5/2/2021 at 11:55 PM, vbap said:

This is nice. Any chance of docker?


One would assume you want Docker for the scheduled task aspect yes? 

Windows on Docker doesn't work on Synology, but I'll look at altering the PowerShell script to run on Linux and thus solve the problem

Link to comment
Share on other sites

Actually want docker for the ease of deployment (and re-deployment). I could deploy it using docker on Synology/Xpenology rather than having separate 24x7 windows VM. I also have a linux VM on ESxi where I run all my docker apps that I don't want to run inside my NAS. 

  • Like 1
Link to comment
Share on other sites

  • 4 weeks later...

Hi elmuziko,

i really love your project. thanks for development.
i also use Docker in some case. so it could be easier to deploy your solution.

 

i will try first v3 on windows and give you my feedback.

if i can test anything, let me know.

 

cheers

Link to comment
Share on other sites

3 minutes ago, sepult said:

Hi elmuziko,

i really love your project. thanks for development.
i also use Docker in some case. so it could be easier to deploy your solution.

 

i will try first v3 on windows and give you my feedback.

if i can test anything, let me know.

 

cheers


Thank you. I've found a few bugs in the PowerShell code so have been concentrating on that.
I'm also translating this into Linux PowerShell so I can create a Linux Docker image. 

Sadly work commitments are slowing me down but will have a lot more free time next month to complete the next version.

 

I will absolutely message when I'm at a testing stage.

Link to comment
Share on other sites

  • 1 month later...

VERSION 3 RELEASE

 

 

Evening all.

 

Version 3 is now ready on my BitBucket repo here

 

WEBSITE CHANGES

 

  • Support for NVMe drives.
  • Colour coded drive types.
  • Temperature graph is now a modal, it now displays 48 entries (a days worth of entries when the PowerShell script is running in its default setup).
  • ESXi CPU, Memory and Datastore monitors.
  • New 'Sort-By' mode added to Settings (sort by Drive Type).
  • A ton of bugfixes and code improvement.

image.thumb.png.02ff24b9f5eb6c0e1328608977c70ada.png

 

 

POWERSHELL CHANGES

 

  • Support for Linux!!! (Tested on Ubuntu).
  • Removed pre-requisite 'Oracle MySQL plugin'. Script now uses SimplySQL from the PowerShell gallery. 
  • Will build a scheduled task for you if using a Windows device. Script is now truly a "run once and forget" thing.
  • More bugfixes than I care to count. 

 

379179283_version3onlinux.thumb.png.084595972272811246dba3b74e698a62.png

 

POWERSHELL FIRST USE

In Windows, open up PowerShell_ISE and use :

 

$firstRun = @{

	# If you're using Synology to store the DB, change SQLUser from root, and SQLPort to 3307. If using XAMPP, root and 3306 will work.

	ESXiHost = "IP OR DNS NAME OF ESXI HOST"
	ESXiPass = ( "YOURPASSWORDHERE" | ConvertTo-SecureString -AsplainText -Force )
	SQLUser  = "CHANGEME"
	SQLPass  = ( "YOURPASSWORDHERE" | ConvertTo-SecureString -AsplainText -Force )
	SQLPort  = 3307
	SQLHost  = "IP OR DNS NAME OF SYNOLOGY HERE"
	Email    = "youremailaddresshere"
	EmailPass = "YOUR EMAIL APP PASSWORD HERE, OR IF YOU'RE MAD ENOUGH TO NOT HAVE TWO STEP, JUST PLAIN PASSWORD"
}

Import-Module ESXiSMART.psd1 -Force -Global -Verbose
Start-SmartDataScript @firstRun -Verbose

 

If you're using Linux, Install PowerShell, then use :

 

Import-Module ESXiSMART.psd1 -Force -Global -Verbose

Start-SmartDataScript -SQLUser "CHANGESQLUSER" -SQLHOST "CHANGESQLHOST" -Email "CHANGEEMAILADDRESS" -EmailPass "plainTextpasswordforemail" -Verbose

 

This will then prompt you for the missing essential items.
 

 

NEXT STEPS

 

I would very much appreciate support for Linux. I would very much appreciate it if someone could help me build a Scheduled Task for the script to run every 30 minutes or so. I did look into this, using something called CronTab but this getting into the realm of doing stuff I have no knowledge of. Which is a dangerous road to go down.

 

I also want to look at building a container for this. Now the PowerShell supports Linux it can actually go on ESXi. I would imagine the Scheduled Task completion thing needs doing first? Another unknown, but one I feel more comfortable tinkering with until we have it.

 

If anyone can offer help on either a Linux scheduled task or container, I would really appreciate it.

And if anyone spots any bugs or has any recommendations for improvements/new features, please do let me know.

 

 

 

  • Thanks 1
Link to comment
Share on other sites

  • 7 months later...

Hi elmuziko,

 

this is a great script, thank you. I want to give it a try and I have some further ideas:

 

1. Option to only send email alerts

2. Website with IIS

3. Database on SQL express

 

How can I change your script to only send email alerts of the SMART data? I'm not a experienced user of PowerShell. But I'm good in programming Visual Basic scripts.

Link to comment
Share on other sites

Posted (edited)

Hi elmuziko,

 

without your scripts I never found a way to read SMART data from ESXi by using PowerShell. I was searching a very long time for a solution to do this. Thank you for giving me an idea how to do this.

 

I created a new script based on another example I found later on further investigations. With my script I can now read all available SMART data of each device from ESXi, write all data and the data with warnings to text files and send a warning mail if there are issues with values that are below the tresholds. It took a long time to adapt this example script to meet my requirements and get it running but now it's working great. For this script there's no need to use a SQL database and a web server.


If someone is interested in my script I can give you a copy. Keep in mind that my script can't get the TBW for a better monitoring of SSDs because ESXi resets this data when it has been rebooted.

 

Kind Regards

 

TheExpert

Edited by TheExpert
  • Like 1
Link to comment
Share on other sites

8 hours ago, TheExpert said:

Hi elmuziko,

 

without your scripts I never found a way to read SMART data from ESXi by using PowerShell. I was searching a very long time for a solution to do this. Thank you for giving me an idea how to do this.

 

I created a new script based on another example I found later on further investigations. With my script I can now read all available SMART data of each device from ESXi, write all data and the data with warnings to text files and send a warning mail if there are issues with values that are below the tresholds. It took a long time to adapt this example script to meet my requirements and get it running but now it's working great. For this script there's no need to use a SQL database and a web server.


If someone is interested in my script I can give you a copy. Keep in mind that my script can't get the TBW for a better monitoring of SSDs because ESXi resets this data when it has been rebooted.

 

Kind Regards

 

TheExpert

 

Ah that's brilliant! Love it, and a genius idea. 

Apologies for my tardy reply, I'm without internet at home at the moment, having to use work computers.

 

 

Link to comment
Share on other sites

  • 4 weeks later...
Posted (edited)

Hi elmuziko,

 

the script itself works still fine but you have to trust to the SMART values ESXi is reading. And I found out that these values aren't reliable when read by ESXi!

 

I use enterprise storage disks from WD and here ESXi doesn't read the values right. The values of some important SMART parameters are 200 if your disk is OK, i. e. Reallocated_Sector_Ct. But ESXi reads 0. If the disk is OK the value is greater than the threshold which is at 140. Because of reading the value as 0 you get an error message. You see warnings in the syslog of ESXi

 

2022-04-04T17:58:55.592Z smartd[526738]: [warn] t10.ATA_____WDC_WD3000F9YZ2D09N20L1_______________________WD2DWCC13DRRJZJP: REALLOCATED SECTOR CT below threshold (0 < 140)
2022-04-04T17:58:55.680Z smartd[526738]: [warn] t10.ATA_____WDC_WD3000F9YZ2D09N20L1_______________________WD2DWCC136FZX9CU: REALLOCATED SECTOR CT below threshold (0 < 140)
2022-04-04T17:58:55.927Z smartd[526738]: [warn] t10.ATA_____WDC_WD3000F9YZ2D09N20L1_______________________WD2DWCC130NF3EEZ: REALLOCATED SECTOR CT below threshold (0 < 140)
2022-04-04T17:58:56.016Z smartd[526738]: [warn] t10.ATA_____WDC_WD3000F9YZ2D09N20L1_______________________WD2DWMC130E1ZMKA: REALLOCATED SECTOR CT below threshold (0 < 140)

 

and my script always sends an error notification. So I bought new disks believing that they are getting old and can cause disk errors and data corruption in the near future which isn't the case :-(. By the way, the new WD disks are seen as faulty by ESXi, too.

 

When reading the SMART values of these disks under Windows with CrystalDiskInfo or under ESXi with smartmontools - there's an ESXi version of an older version - the correct value 200 is read.

 

And then I found out that this issue seems to happen with all WD disks, i. e. WD Red etc. I saw a lot of examples with this issue. With disks from other manufactorer the values are read by ESXi right.

 

WD doesn't see an issue regarding using the disks with VMware ESXi and getting wrong SMART values. They won't help to solve the issue.

 

Now I try to get a solution for reading the SMART values by using the smartmontools and getting error notifications when there are really errors.

 

Kind Regards

Edited by TheExpert
Link to comment
Share on other sites

@TheExpert

Hey buddy,

 

I gotta be honest with you. I've abandoned the project. There was little interest on here for it for a year or so, and I got fed up of all the frustrating bugs/quirks/limitations of Xpenology. The certificate expiry thing late last year with no meaningful update was the last straw for me so I migrated to Unraid. And doing so I found the same thing as you did. My WD Reds - 6 year old WD Reds - were failing horrifically and ESXi was giving me false information. They probably would have died without me getting any real alerts from them, thankfully I was able to replace them before losing my RAID.

I would imagine the problem lies with no actual SMART tests beings ran. Whereas a native OS can run a full scan, ESXi doesn't appear to.

When I was building the project I was looking to see if I could add an option to force a full SMART scan. I went down the rabbit hole and found someone had ported smartmontools. But it was all bash code, and that's an area I know so little about, I didn't want to risk borking my live system. 

You can find more on it here.

https://www.virten.net/2016/05/determine-tbw-from-ssds-with-s-m-a-r-t-values-in-esxi-smartctl/?msclkid=da4034e4b4b611ec96b92388e10508c5

 

If you can get it working, fantastic. Otherwise it seems this entire project gives false hope/readings... thank you very much VMWare. 

Link to comment
Share on other sites

Hi elmuziko,

 

yes, this is the smartmontools version for ESXi I mentioned. It's running even with ESXi 7.0 U3d. But unfortunatelly there's no newer build available for ESXi. Newer builds of smartmontools can create json formatted output which can be read by other programs or scripts much easier.

 

I created now a little bash script that reads out the important SMART parameters which are known for pre-fail. And if there are errors the bash script writes them into the output. The next step will be to create a VBScript that gets the output of plink running the bash script and parses the output for error messages to send a mail of the disk health. It's frustrating that we have to do such komplex scriptings in the 2020s :-(.

 

I thought there are better solutions for my homelab, i. e. Proxmox. But Proxmox doesn't fit better in my opinion. I tried to recover one of my Windows VMs with Veeam agent but the VM won't boot. So I think it's much easier to stay at VMware ESXi and create a monitoring tool that reads out the SMART infromation of the hard disks correctly and sends me a mail if theres something wrong with the disks. I also tried to use RDMs for passing through the SATA disks to my Windows VM. Under Windows I can read the SMART information with CrystalDiskInfo, which a is really perfect tool. But I wasn't able to do so. The disks won't be passed through as physical disks and so the software isn't able to read the SMART information.

 

Kind Regards

Link to comment
Share on other sites

Hi elmuziko,

 

I finished now the step to create a VBScript that gets the output of plink running the bash script and parses the output for error messages to send a mail of the disk health. The most difficult thing was to define the mail priority in VBScript (with PowerShell this is so easy).

 

Tonight there will be the first run as scheduled task which is planned for running once a day.

 

Kind Regards

  • Like 1
Link to comment
Share on other sites

Hi elmuziko,

 

because of running a bash script on the ESXi host it should be possible to start full SMART scans. My VBScript may not be able to parse the output when it's not the same as running

 

/opt/smartmontools/smartctl -d sat --all /dev/disks/<drive>

 

which is the command I've running every night now. So at the moment I do not scan the the drives. The bash script is only reading the overall health status, checking the Pre-failure values against the thresholds and writing the status "PASSED" or "FAILED" to each Pre-failure value of each installed disk in each ESXi host. The VBScript is then parsing the output of the bash script for "FAILED" to send me a mail message with high priority/importance. It's enough to have one "FAILED" status for this. When all is "PASSED" the script sends me a mail with normal priority/importance so I can see that the task of checking the disks is running.

 

After knowing now how to handle this it's no problem to do more healthcare regarding the hard disks with smartctl on ESXi and getting an email message of the output with a VBScript running on a Windows system.

 

I never thought about doing regular scans of my hard drives on the ESXi host. Maybe this is done by the smartd of ESXi? Or should I do this with smartctl, too?

 

Kind Regards

Link to comment
Share on other sites

  • 4 weeks later...
Posted (edited)

Thank you for sharing such an amazing project with forum users! Considering how quickly platforms change, I think a program for exporting data to a device is an actual thing. You could create a website to distribute your project and possibly monetize it.  The main thing is not to forget to constantly update the site using creative.onl so that people can see that you are doing constant work and not abandoning the project. If you do create it, throw it into this topic. I will be happy to read it and leave some comments.

Edited by Martcostan
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...