Djey Posted November 12, 2014 #1 Posted November 12, 2014 (edited) Hi, despite all help I could get in this amazing forum, I was never able to display SMART and perform self tests in DSM, because of a lack of feature in the drivers for VMWare ParaVirtual/LSI SCSI controller. Whereas my main concern was just to get SMART data about hard drives in my xpenology set-up as a VM (in a VMWare ESXi hypervisor), 4 hards disks in Raw Device Mapping - Hardware is a HP G7 N54L - Given the fact ESXi already has what it gets to fetch a lot of information from plugged in disks, I began thinking of providing a set of services. Those relying on ESXi 5.5's integrated tools. And so pi-control project was born. To be installed on an always-on device (Raspberry-pi like), pi-control can be used to monitor hypervisor state and perform basic actions. All detailed in API: https://github.com/djey47/pi-control/wiki/API-reference hypervisor ON/OFF switch + schedule + status virtual machine list virtual machine ON/OFF switch + status hard disk list SMART details of particular disk(s) global logging to watch all received service requests ('big brother'). pi-control only requires ruby interpreter and a few common tools to be installed onto Raspberry-pi located on same network as hypervisor. Technically, it accesses ESXi system through ssh to invoke well known tools as esxcli or vim-cmd and throw results back as JSON. JSON gives a convenient way to use those results in a higher level application (script, webapp, ...) You can find latest release hosted on Github: https://github.com/djey47/pi-control/tr ... er/release (select current release archive then click View Raw link to start downloading). New! First docker images are available. pi-control images do not support all features for now: e.g ESXi ON pi-control (for rPi) : https://github.com/djey47/pi-control/wi ... install%3F (instructions) and https://hub.docker.com/r/djey/pi-control-raspbian/ (hub) pi-control (for x64) : https://github.com/djey47/pi-control/wi ... install%3F (instructions) and https://hub.docker.com/r/djey/pi-control-linux-x64/ (hub) smartX (for rPi): https://github.com/djey47/smartX/wiki/Installing (instructions) and https://hub.docker.com/r/djey/smartx-raspbian/ (hub) smartX (for x64): https://github.com/djey47/smartX/wiki/Installing (instructions) and https://hub.docker.com/r/djey/smartx-linux-x64/ (hub) Most of the information is there: https://github.com/djey47/pi-control/wiki. So, enjoy Edited July 29, 2016 by Guest
elmuziko Posted December 18, 2014 #2 Posted December 18, 2014 Does this work for 5.1 ESXi? Using 5.1 as there's more driver support than 5.5 I'm away from home at the moment so can't try anyway; but just wondered as by the time I'm back my NAS will have been on 6 months and nervous it's ready to pop so was prepared to go back to bare bones.
mcdull Posted December 18, 2014 #3 Posted December 18, 2014 what do you want to do with SMART under DSM (with RDM)? SMART has been always work under DSM with RDM, it was just that the DSM interface wont pull the SMART data. You can always obtain the smart detail with smartctl -a -d sat /dev/sdc. If you want notification, get smartd install optware. I just received an email of warning last month that one of my drive got high ATA command failure and noticed I got a bad SATA connection. So I have to switch the bay for temporary fix. It is not difficult to schedule short or long smart test with smartd as well. But now I changed my approach to passthrough the entire LSI controller and the smart status under DSM works like a charm. The email alert is attached for your info. ==================== This email was generated by the smartd daemon running on: host name: XXXXXXXX DNS domain: [unknown] NIS domain: (none) The following warning/error was logged by the smartd daemon: Device: /dev/sdf [sAT], ATA error count increased from 460 to 471 For details see host's SYSLOG (default: /var/log/messages). You can also use the smartctl utility for further investigation. No additional email messages about this problem will be sent.
Djey Posted December 18, 2014 Author #4 Posted December 18, 2014 @elmuziko: it should work with ESXi 5.1, as it does not rely on tools which are 5.5-specific. @McDull: No, SMART support has not always been available with smartctl/smartd, depending on xpenology version and installed bootloader. It did work with Trantor's 4.3. When I migrated to 5.0, gnoBoot did not support it; that's why I had to find a workaround. Now, nanoboot seems to have it fixed. So what will be next ? I can't definely rely on the bootloader... A major advantage is I can monitor my datastore hdd to ESxi the same way:) Anyway you're right, if your hardware support VT-d, adding a 3rd part disk controller and using passthrough mode is the way to go to. That's not an option with N54L.
zilia Posted April 1, 2016 #5 Posted April 1, 2016 Hi, @Djey. Thanks for your great tool. 1) Is it possible to release it as Synology package to install it one click from public repo or just .spk file on the nas itself? 2) I have something strange on my ESXi + Xpenology installation. smartctl on synology console (via telnet of ssh or even the "real" con) shows relevant information about RDM drives, but DSM web interface does not. So may be you can consider future update of your smartX app to get the info not only from remote JSON source but also from DSM local smartclt output. It would be great and very useful plugin for synology.
Djey Posted April 2, 2016 Author #6 Posted April 2, 2016 Hi, pi-control services do actually run on raspberryPi (or similar device). It's not so easy to install (or even create a such a spk file) on the NAS - due to lack of software support on DSM. smartX could be however, as it only need nginx. Some of the reasons why I decided to set everything on a rPi: - the need to control the NAS even when ESXi is OFF (e.g. to start it) - the need to check hypervisor status with all Vms plugged in, even when xpenology is not running. and yes, smartctl seems to work finally on xpenology. The fact is, smartX is just a webapp and can't directly get data from the system, it must use a data source (like JSON sent by pi-control) to get values. A solution would be to enhance pi-control to provide info from smartctl as well ... By the way, as a first step I was planning to use ESXI smartd daemon to get latest events. https://github.com/djey47/pi-control/issues/10 I will take note of your idea, can't guarantee to get time to put my hands on it, but who knows ... Thanks for your interest in this project!
Djey Posted July 17, 2016 Author #7 Posted July 17, 2016 Finally! I was able to create Docker images for both pi-control and smartX projects! Those first images are made for devices running raspbian:jessie. pi-control: https://github.com/djey47/pi-control/wi ... install%3F (instructions) and https://hub.docker.com/r/djey/pi-control-raspbian/ (hub) smartX: https://github.com/djey47/smartX/blob/master/README.md (instructions) and https://hub.docker.com/r/djey/smartx-raspbian/ (hub)
venturis Posted July 21, 2016 #8 Posted July 21, 2016 Hi, I thought I would give the Docker containers a go myself but found I was getting the following error when I tried running the docker container for Pi-control "System Error: exec format error" I'm new to Docker but have followed the instructions in the Wiki and put all my ESXi config into the command string correctly. Any ideas? I've tried several times without success. Im running ESXi 5.5 on a Microserver G8 with an XPENOLOGY 5.2 virtual machine with RDM drives.
Djey Posted July 21, 2016 Author #9 Posted July 21, 2016 Hi, that happens because the current image is based on Debian for ARM (raspbian:jessie) as it's running on a raspberry PI. To make it run on x64 architecture, I have to create a new image. I will do eventually.
venturis Posted July 21, 2016 #10 Posted July 21, 2016 Thanks. I thought that the containers were created to run on Docker in an Xpenology VM on ESXi. Ill keep a look out for the x64 version.
Djey Posted July 21, 2016 Author #11 Posted July 21, 2016 Yeah, my fault I'm afraid. I did not provide details about image compatibility. A x64-based image (from alpine linux I guess) should work well on syno docker.
Djey Posted July 24, 2016 Author #12 Posted July 24, 2016 Docker image for x64 Linux kernel is ready. It should run fine within xpenology. https://hub.docker.com/r/djey/pi-control-linux-x64/ Please note that some limitations do remain, due to the base image I've been using (Alpine). Will fix it in the future.
venturis Posted July 25, 2016 #13 Posted July 25, 2016 That's terrific.....I'll give it a try today. Thanks.
venturis Posted July 25, 2016 #14 Posted July 25, 2016 I keep getting an "operation failed" error message when trying to add the Pi-control image to docker using the GUI. I've not tried the command line. I pulled down the first two images again as a test and they pulled down without error. UPDATE: I just tried to pull the pi-control-linux-x64 image using the command line....I get a "repository not found" error.
Djey Posted July 26, 2016 Author #15 Posted July 26, 2016 That's weird, I could pull the image from PC at work, but DSM GUI couldn't for the same reasons as you. All other images can be retrieved without a hitch. Maybe I should recreate the repository and push again ...
haydibe Posted July 26, 2016 #16 Posted July 26, 2016 Did you try from the command line? There is a known bug in the Docker API that causes that the latest images can not be found. See: https://github.com/docker/hub-feedback/issues/687
venturis Posted July 26, 2016 #17 Posted July 26, 2016 I tried the command line and get a "repository not found" fatal error. Reading through the bug report at the link provided seems to be a different issue than having here. I can find the repository with the Docker GUI but it fails to download. Command line pull also fails. Will wait until Djey has recreated the repository and will try again.
haydibe Posted July 27, 2016 #18 Posted July 27, 2016 Without version number: dsm> docker pull djey/pi-control-linux-x64 Pulling repository djey/pi-control-linux-x64 FATA[0003] Repository not found -> no latest tag! with version number: dsm1> docker pull djey/pi-control-linux-x64:0.1 0.1: Pulling from djey/pi-control-linux-x64 9b7301678506: Pull complete 06984fada629: Pull complete cbd9326b4af3: Pull complete b4c9f3479366: Pull complete 8e9b6908e7de: Pull complete b3137ee5cdae: Pull complete c66d3cde113e: Pull complete fb77b7df8877: Pull complete c4d65ff623c9: Pull complete b064f42ff405: Pull complete 36c2eee769b2: Pull complete 26bf9b3819fd: Pull complete Digest: sha256:7bb9da819495b9a7f04c878adce83ea1bf0f36f45c4575a47226f70088f5f834 Status: Downloaded newer image for djey/pi-control-linux-x64:0.1
Djey Posted July 27, 2016 Author #19 Posted July 27, 2016 Thanks haydibe, looks like DSM GUI always wanna get 'latest' tag. So latest tag is working for some images, some other won't (might be due to different docker version - ARM images are pushed from my raspberry PI directly, x64 ones from virtualized Xubuntu 14.04 ) I guess that manually (command line) pulled images do appear into DSM as well? Sorry I haven't time to try this till tomorrow.
haydibe Posted July 27, 2016 #20 Posted July 27, 2016 I guess that manually (command line) pulled images do appear into DSM as well? yes, they do
Djey Posted July 28, 2016 Author #21 Posted July 28, 2016 Tested and it works as well as containerized on rPi Sadly, DSM GUI did not let me provide all required arguments to run image into a container; I had to type from root shell: docker run -d -p 4600:4600 -e ESXI_HOST='x.x.x.x' -e ESXI_USER='root' -e ESXI_MAC_ADDR='xx:xx:xx:xx:xx:xx' -e LAN_BROADCAST_ADDR='x.x.x.x' -v /root/.ssh:/root/.ssh --restart=always --name pi-control djey/pi-control-linux-x64:0.1 Running container does appear into DSM GUI afterwards, so that's alright finally. - I'm with DSM 5.2-5967 U1 - Edit: x64 Linux image available for smartX web interface. But need to be tested against SynoDocker, still.
venturis Posted August 10, 2016 #22 Posted August 10, 2016 Any updates on the x64 Linux image for SmartX? I tried the published version on my Xpenology v5644.5 running under ESXi 5.5 but it bombs out when starting the container.
Djey Posted August 22, 2016 Author #23 Posted August 22, 2016 Sorry for late reply, I just tested smartX x64 image against SynoDocker. Working here. From root shell: docker run -d -p 8080:80 -e PI_CONTROL_ADDRESS=':4600' --restart=always --name smartx djey/smartx-linux-x64:0.1 , assuming that pi-control container running under host port 4600. Thus smartX web-ui will be available through http://:8080/smartx.html
venturis Posted August 29, 2016 #24 Posted August 29, 2016 I'm close to throwing in the towel here. I have to admit I'm a complete novice at Linux and Docker and I've only made the progress I've made with information in the posts above. I have both containers running and can access the SmartX web page but I don't believe I've properly set up the encryption key that allows pi-control to access the ESXi host without password authentication. The SmartX web page simply displays "updating". Looking at the SmartX container log shows repeated errors every 15 seconds stating "connection refused while connecting to upstream client...." I found some information on creating a public key but not sure were to put the key to make it accessible to the pi-control container. If anyone can help with step-by-step instruction for setting up the pi-control to ESXi host authentication it would be much appreciated.
Djey Posted August 29, 2016 Author #25 Posted August 29, 2016 (edited) Indeed, authentication has to be correctly set-up before starting pi-control container. Here is a starter link : https://blogs.vmware.com/vsphere/2012/0 ... xi-50.html Here comes some explanation on article steps: To generate a private/public key pair onto dsm, you just log onto DSM shell then run: ssh-keygen and follow instructions. It will create keys in directory : /root/.ssh/ ( or if you already have keys, copy them into /root/.ssh/ directory) Then, connect to ESXl from DSM: ssh root@ And confirm to add key to host cache In ESXI, edit /etc/ssh/keys-/authorized_keys file and copy contents of previously generated id_rsa.pub (public key) - that's what ssh-copy-id command does. Exit ESXI shell and try connecting again. If properly configured, next time connection won't require password. To run the container, you just check volume parameter to point your DSM keys directory, check -v parameter : -v /root/.ssh:/root/.ssh Also, after running pi-control container, execute command and post results here : docker logs pi-control Should be better after that. Please let me know Thanks Edited August 29, 2016 by Guest
Recommended Posts