NEW: Virtual DSM in Docker

[JohnyByk]

Great Job :).
When i run from docker run all works great but all settings are default:

docker run -it --rm -p 5000:5000 --device=/dev/kvm --cap-add NET_ADMIN --stop-timeout 60 kroese/virtual-dsm:latest

When i tried run from docker compose (docker compose up) i see this error:

PS D:\Docker\Synology> docker compose up
[+] Running 7/7
 ✔ dsm 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                    7.3s
   ✔ faef57eae888 Pull complete                                                                                    3.8s
   ✔ e527efa5da78 Pull complete                                                                                    4.9s
   ✔ 74472c98d88b Pull complete                                                                                    4.9s
   ✔ 5319b1149e9f Pull complete                                                                                    4.9s
   ✔ f6a43b34e6fa Pull complete                                                                                    5.0s
   ✔ 795699eb0689 Pull complete                                                                                    5.0s
[+] Running 2/2
 ✔ Network synology_default  Created                                                                               0.0s
 ✔ Container dsm             Created                                                                               0.1s
Attaching to dsm
dsm  | ❯ Starting Virtual DSM for Docker v4.12...
dsm  | char device redirected to /dev/pts/0 (label charserial0)
dsm  | qemu-system-x86_64: -drive file=/storage/DSM_VirtualDSM_64570.boot.img,if=none,id=drive-synoboot,format=raw,cache=none,aio=native,discard=on,detect-zeroes=on: Could not open '/storage/DSM_VirtualDSM_64570.boot.img': filesystem does not support O_DIRECT
dsm exited with code 0
PS D:\Docker\Synology>

 

My docker-compose.yml:

version: "3"
services:
    dsm:
        container_name: dsm
        image: kroese/virtual-dsm:latest
        environment:
            DISK_SIZE: "4096G"
            ALLOCATE: "N"
            CPU_CORES: "4"
            RAM_SIZE: "4096M"
        devices:
            - /dev/kvm
            - /dev/vhost-net
        cap_add:
            - NET_ADMIN                       
        ports:
            - 5000:5000
        volumes:
            - /opt/dsm:/storage
        restart: on-failure
        stop_grace_period: 1m

Im fresh in docker and maybe im doing something wrong.
Platform: Windows 11 Pro.
 

[marc_all]

Le 08/08/2023 à 17:56, JohnyByk a dit :

My docker-compose.yml:

version: "3"
services:
    dsm:
        container_name: dsm
        image: kroese/virtual-dsm:latest
        environment:
            DISK_SIZE: "4096G"
            ALLOCATE: "N"
            CPU_CORES: "4"
            RAM_SIZE: "4096M"
        devices:
            - /dev/kvm
            - /dev/vhost-net
        cap_add:
            - NET_ADMIN                       
        ports:
            - 5000:5000
        volumes:
            - /opt/dsm:/storage
        restart: on-failure
        stop_grace_period: 1m

Im fresh in docker and maybe im doing something wrong.
Platform: Windows 11 Pro.
 

Hello,

Are you sure about the disksize "4096G"? (it is 4 To)

You create the image file on /opt/dsm (see in the volumes  part).

The /opt dir is in the virtual image from WSL2 (it should probably have a size of about a few GB and for backup reasons it is probably better on another drive or in /mnt/C but not in the virtual disk (virtual disk in a virtual disk seems odd).

 

Personally I have put (see my port is page 3) to have it on  a real disk and I have no problem.

 

volumes:
            - /mnt/d/DSM:/storage

 

 

[sonic9]

can be "converted" to Lxc(proxmox container) ?

[hendry]

@kroese how to mount host volume to be detected and used in docker DSM ? i want to install surveillance station on docker DSM but write the recorded video to host volume. could it be done ?

[ipman]

@kroese Can I also define multiple volumes in docker compose? So that not all network shares are on one storage in vDSM but I can use several storages in vDSM.

Edited August 29, 2023 by ipman

[JuanjOD]

Great work. Working like a charm on Open Media Vault (virtualized on Virtualbox running on Linux Mint). 

 

Next is installing OMV on baremetal and configure DSM inside.

 

Just had a weird behaviour when I changed the 5000 host port to 5002 to avoid problems with other synology devices. OMV restarted the DSM instalation and all that was previously configurated in DSM was lost... Maybe its something about OMV...

 

 

 

Edited August 31, 2023 by Polanskiman
Picture changed at the request of user.

[JuanjOD]

The DSM is perfectly working as a container inside Openmediavault. I finally used a Dell Optiplex 3050 i5-6500T with one nvme disk and a sata ssd disk mirrored with RAID 1 and 16GB of DDR4 RAM. The container was deployed with its own ip address. 

 

Installed Synoloy Photos and it works really fast. I'm loving the way Photos perform on this setup. Configured a Hyperbackup task against my old DS220J and also works perfectly.

 

The only problem I'm having is about the notifications... For some reason personalized smtp doesn't work and Telegram notifications gives an error message saying text notifications are not enabled.

 

Any clues?

 

 

Edited September 8, 2023 by JuanjOD

[CassTG]

Ok Sir/Madam all i can say is you are a legend.

 

Just built a new proxmox (shock horror lol) node and decided i would try this rather than the redpill way as i had done on my last machine.

 

Literally had this up and running on first attempt kept it simple then added to the config to get all the basics running.

 

I do get the email notification error others have mentioned but not fussed about that.

 

As i went with bridge network, did have to open up more ports so SMB and NFS would work, but literally flawless install

 

Here is my current config for those who are new to this (i use docker run rather than Yaml files)

 

docker run -d  \
-p 5000:5000 \
-p 5001:5001 \
-p 5353:5353 \
-p 139:139 \
-p 445:445 \
-p 111:111 \
-p 892:892 \
-p 2049:2049 \
-e DISK_SIZE=240G \
-e ALLOCATE=N \
-e CPU_CORES=4 \
-e RAM_SIZE=2048M \
-v synologydata:/storage \
--device=/dev/kvm \
--cap-add NET_ADMIN \
--stop-timeout 60 \
--restart unless-stopped \
kroese/virtual-dsm:latest

 

Switched to -d and --restart unless-stopped so that if the node or vm reboots it will auto start

Edited October 8, 2023 by CassTG

[nokia banana]

I have some noobs docker network and kvm questions on how to use this. 

 

1. How does the guest DSM access a storage device? Does it take control of a SATA controller? Can i expose a block device from the host OS into DSM and have DSM sees it as if an internal hard drive? (In particular, i'm thinking about exposing either one of LUKS device, cached LVM device, and/or ZFS volume to DSM; offloading the encryption and low level disk managements to the host OS)

 

2. To make networking easier and not fighting with ports with the host, is there a way to expose an entire NIC to DSM? 

 

Thanks!

[JuanjOD]

On 9/8/2023 at 5:48 PM, JuanjOD said:

The only problem I'm having is about the notifications... For some reason personalized smtp doesn't work and Telegram notifications gives an error message saying text notifications are not enabled.

 

Any clues?

 

 

 

 

 

I'll reply myself.... After seeing DNS resolving issues on DSM log, I added Google's DNS to DSM network config. Now notifications works like a charm. 

 

I also just got a notification on DSM upgrade (update 2). Anyone applied already? It's safe to upgrade? I'm not really in a hurry as changes does not affect my installation, but it would be good to know.

[JuanjOD]

On 10/14/2023 at 1:35 AM, nokia banana said:

2. To make networking easier and not fighting with ports with the host, is there a way to expose an entire NIC to DSM? 

 

I created a create a macvlan network and assigned a unique IP address to the container. Did not specify any ports. Working perfectly

[JuanjOD]

On 10/14/2023 at 1:35 AM, nokia banana said:

1. How does the guest DSM access a storage device? Does it take control of a SATA controller? 

 

On my system, using OMV as mentioned earlier, During the installation, 2 volumes were created. I think one is for bootloader while the other one is for data (you specify the size on the contaner creation). They looks like this:

 

Edited November 16, 2023 by JuanjOD

[kroese]

On 8/28/2023 at 5:43 AM, hendry said:

@kroese how to mount host volume to be detected and used in docker DSM ?

 

This is not possible because the stock  VirtualDSM kernel has no virtiofs module (like Xpenology has).

 

On 8/29/2023 at 4:13 PM, ipman said:

@kroese Can I also define multiple volumes in docker compose? So that not all network shares are on one storage in vDSM but I can use several storages in vDSM.

 

Yes.. See https://github.com/vdsm/virtual-dsm#how-do-i-add-multiple-disks 

 

On 10/14/2023 at 1:35 AM, nokia banana said:

How does the guest DSM access a storage device? Does it take control of a SATA controller? Can i expose a block device from the host OS into DSM and have DSM sees it as if an internal hard drive?

 

You cannot expose SATA controllers. You can either just bind a folder, where a virtual disk image (data.img) will be created, or passthrough a whole block device (not recommended). See https://github.com/vdsm/virtual-dsm#how-do-i-change-the-location-of-the-virtual-disk

 

On 10/14/2023 at 1:35 AM, nokia banana said:

To make networking easier and not fighting with ports with the host, is there a way to expose an entire NIC to DSM? 

 

You cannot expose a NIC. But like @JuanjOD already explained, you can create a macvlan network. See https://github.com/vdsm/virtual-dsm#how-do-i-assign-an-individual-ip-address-to-the-container

 

2 hours ago, JuanjOD said:

I also just got a notification on DSM upgrade (update 2). Anyone applied already? It's safe to upgrade?

 

Yes it is always safe to upgrade. Because this container uses a completely stock (unmodified) version of DSM, which means that upgrades can never break the system. With XPenology that is different, because it modifies a lot of things (bootloader, kernel modules, etc).

 

Also this update should fix the problem with notifications that you had (see https://github.com/vdsm/virtual-dsm/issues/313#issuecomment-1788824213 ).

 

For all other people: I wont be replying here anymore. Because almost every question was already answered in the FAQ.

 

If you have any issue, it should be reported to https://github.com/vdsm/virtual-dsm/issues and not here.

[JuanjOD]

11 minutes ago, kroese said:

Also this update should fix the problem with notifications that you had (see https://github.com/vdsm/virtual-dsm/issues/313#issuecomment-1788824213 ).

 

 

Actually, the notification problem solved as soon as I added DNS servers con DSM network config. It was a DNS resolution issue (at least in my case).

 

Thanks for your work

Edited November 16, 2023 by JuanjOD

[JuanjOD]

Perfectly updated to DSM 7.2.1-69057 Update 1. Great!

 

 

 

[Tdubs]

This is awesome!  

 

Question

What is the ram usage folks are seeing with Running DSM 7.3 and the Photo synce app on windows 11 boxes?

 

I am trying to decide if I need to go to 16GB or 24gb ram.  THis machine will only really be used to as a NAM on DSM 7.3 for Photos from the phones inside the home network.

 

Thanks!

[Tdubs]

3 hours ago, Tdubs said:

This is awesome!  

 

Question

What is the ram usage folks are seeing with Running DSM 7.3 and the Photo synce app on windows 11 boxes?

 

I am trying to decide if I need to go to 16GB or 24gb ram.  THis machine will only really be used to as a NAS on DSM 7.3 (in WSL2 on a windows 11 Machnie) for Photos from the phones inside the home network.

 

Thanks!

Fixed some spelling errors...

 

Edited November 28, 2023 by Tdubs

[RacerX]

I tested with bookworm on my old arm/v4  and it gives a manifest error pull fails , however when I tested on my Intel Nuc it worked this a charm.  Very easy to setup...thanks

[JuanjOD]

On 11/28/2023 at 10:20 PM, Tdubs said:

Fixed some spelling errors...

 

I'm using an Open Media Vault box (Dell optiplex Micro format,  i5-4 with 16GB of RAM, RAID 1 with SATA SSD+NVME). 8 GB assigned to DSM container. I'm using it only with Photos and a daily backup with hyperbackup to my DS220J Synology NAS. This is lightning fast in ANY condition even with face recognition, etc enabled. Reindex are incredibly fast too... Don't know how it will perform in the Redmon's thing though

Edited November 30, 2023 by JuanjOD

[bkbartk]

Hello,
I'm not sure I should post the question here or someone else.
I think using this as a docker is a really great way of using this.
I tried using the source and add it to a home assistant add-on, but ran into some troubles.
what I did was, grep the complete source from github, and add config.yml (as required by home assistant)

the config.yml looks like this
 

name: "virtual_dsm"
description: "virtual_dsm"
version: "0.1.5"
slug: "dsm"
init: false
arch:
  - aarch64
  - amd64
  - armhf
  - armv7
  - i386
ports:
  5000/tcp: 5000
options:
    dsm:
        container_name: dsm
        image: vdsm/virtual-dsm:latest
        # environment:
        #     DISK_SIZE: "16G"
        #     RAM_SIZE: "1G"
        #     CPU_CORES: "1"
        devices:
            - /dev/kvm
        device_cgroup_rules:
            - 'c *:* rwm'
        cap_add:
            - NET_ADMIN
        volumes:
            - /var/dsm:/storage
        restart: on-failure
        stop_grace_period: 2m
schema:
    dsm:
        container_name: str
        image: str
        # environment:
        #     DISK_SIZE: str
        #     RAM_SIZE: str
        #     CPU_CORES: str
        devices:
            - str
        device_cgroup_rules:
            - str
        cap_add:
            - str
        volumes:
            - str
        restart: str
        stop_grace_period: str

while starting port 5000 is accessible (blue screen with white installation text)
but then I run info some error:
 

❯ Starting Virtual DSM for Docker v...
❯ For support visit https://github.com/vdsm/virtual-dsm

❯ Install: Downloading installer...
❯ Install: Downloading DSM_VirtualDSM_69057.pat...

     0K ........ ........ ........ ........  9% 14.7M 21s
 32768K ........ ........ ........ ........ 18% 14.0M 20s
 65536K ........ ........ ........ ........ 27% 13.9M 18s
 98304K ........ ........ ........ ........ 36% 15.2M 15s
131072K ........ ........ ........ ........ 46% 18.2M 12s
163840K ........ ........ ........ ........ 55% 18.3M 10s
196608K ........ ........ ........ ........ 64% 18.1M 8s
229376K ........ ........ ........ ........ 73% 18.3M 6s
262144K ........ ........ ........ ........ 83% 16.8M 4s
294912K ........ ........ ........ ........ 92% 18.1M 2s
327680K ........ ........ ........ ..      100% 18.3M=21s

❯ Install: Extracting downloaded image...
❯ Install: Preparing system partition...
❯ Install: Extracting system partition...
❯ Install: Installing system partition...
❯ Creating a preallocated disk image in raw format with a size of 16G...
RTNETLINK answers: Operation not permitted
❯ ERROR: Failed to create bridge. Please add the following setting to your container: --cap-add NET_ADMIN

I know the structure in 'docker-compose' is somewhat different, but else It won't be accepted.
the original 'docker-compos.yml' is also in the add on

can anyone help me creating a working config? or should I ask the home assistant community?

or shou

[bkbartk]

is there a limit to the text in the post? it's cut off,
anyway
adding
 

privileged:
  - NET_ADMIN

fixed this error,
so now I have no error, but it's not working, web interface inaccessible.
I will take a look at this later,

any suggestions are welcome

[TheRealMaN_]

On 10/8/2023 at 10:37 PM, CassTG said:

Ok Sir/Madam all i can say is you are a legend.

 

Just built a new proxmox (shock horror lol) node and decided i would try this rather than the redpill way as i had done on my last machine.

 

Literally had this up and running on first attempt kept it simple then added to the config to get all the basics running.

 

I do get the email notification error others have mentioned but not fussed about that.

 

As i went with bridge network, did have to open up more ports so SMB and NFS would work, but literally flawless install

 

Here is my current config for those who are new to this (i use docker run rather than Yaml files)

 

docker run -d  \
-p 5000:5000 \
-p 5001:5001 \
-p 5353:5353 \
-p 139:139 \
-p 445:445 \
-p 111:111 \
-p 892:892 \
-p 2049:2049 \
-e DISK_SIZE=240G \
-e ALLOCATE=N \
-e CPU_CORES=4 \
-e RAM_SIZE=2048M \
-v synologydata:/storage \
--device=/dev/kvm \
--cap-add NET_ADMIN \
--stop-timeout 60 \
--restart unless-stopped \
kroese/virtual-dsm:latest

 

Switched to -d and --restart unless-stopped so that if the node or vm reboots it will auto start

 

Please provide more info, do you run docker in LXC container? can you share your LXC config?
I use Docker in several LXCs, but don't have a /dev/kvm device, does it need to be passed through from the proxmox host?

Edited February 14 by TheRealMaN_

[smilenkovski]

@kroese

Just to say Hi and Thank You.

[smilenkovski]

@kroese

Hi,

at post #43 quote:
The last two you can find by creating an instance of VirtualDSM in VMM temporarily.

I don't understand that sentence. Can You, please, elaborate?