NEW: Virtual DSM in Docker

[Rayqusa]

On 5/4/2023 at 8:25 PM, kroese said:

 

According to their website they use KVM for all their VPS'es. And they do support nested virtualization. So you should be able to use KVM on that VPS.

vultr only support KVM only for baremetal, for cloud doesnt support KVM. thus fucki expensive

[Orphée]

OVHcloud Eco has some low cost baremetal available in their Kimsufi or SoYouStart offers.

You may have a look.

[frozendark]

Great job for doing this

i have tested on ubuntu server 22.04.2 and it works fine.

i just have one question:

I don't have much experience with docker. I understood and created a macvlan network, but how exactly docker-compose file should be modified. Can you give me an example of how it should be passed in docker-compose because in the github page .. I read but I didn't understand

This is ok ?

 

services:
    dsm:
        container_name: dsm
        image: kroese/virtual-dsm:latest
        environment:
            CPU_CORES: "2"
            DISK_SIZE: "256G"
            ALLOCATE: "N"
            RAM_SIZE: "2048M"
        networks:
            vdsm:             
                ipv4_address: 192.168.X.XX
        devices:
            - /dev/kvm
            - /dev/vhost-net
        cap_add:
            - NET_ADMIN        
        ports:
            - 5000:5000
        volumes:
            - /home/user/dsm/storage:/storage
        restart: on-failure
        stop_grace_period: 1m
        networks:
            vdsm:
                external: true

Thaks

Edited May 15, 2023 by frozendark

[kroese]

No, the second "networks" needs to be seperated from "services", so like this:

 

services:
    dsm:
        container_name: dsm
        image: kroese/virtual-dsm:latest
        environment:
            CPU_CORES: "2"
            DISK_SIZE: "256G"
            ALLOCATE: "N"
            RAM_SIZE: "2048M"
        networks:
            vdsm:             
                ipv4_address: 192.168.X.XX
        devices:
            - /dev/kvm
            - /dev/vhost-net
        cap_add:
            - NET_ADMIN        
        ports:
            - 5000:5000
        volumes:
            - /home/user/dsm/storage:/storage
        restart: on-failure
        stop_grace_period: 1m

 

networks:
  vdsm:
    external: true

[claud-xp]

Has anyone tested the FTP service? I can't get it to work, any suggestions?

[Orphée]

11 hours ago, claud-xp said:

Has anyone tested the FTP service? I can't get it to work, any suggestions?

Did you publish FTP port in your docker-compose.yml / Dockerfile ?

 

This is not magic, as you did publish 5000 port, you have to do the same for FTP (21 ?) port.

 

Any service you run under docker has to be explicitly published/exposed if you want it to be accessible from outside docker container.

[claud-xp]

2 hours ago, Orphée said:

Did you publish FTP port in your docker-compose.yml / Dockerfile ?

 

This is not magic, as you did publish 5000 port, you have to do the same for FTP (21 ?) port.

 

Any service you run under docker has to be explicitly published/exposed if you want it to be accessible from outside docker container.

 

This Is my docker-compose file:

 

version: "3"

services:

    dsm:

        container_name: dsm

        image: kroese/virtual-dsm:latest

        environment:

            DISK_SIZE: "16G"

        devices:

            - /dev/kvm

            - /dev/vhost-net

        cap_add:

            - NET_ADMIN                       

        ports:

            - 5000:5000

            - 21:21

            - 20:20

        volumes:

            - /opt/dsm:/storage

        restart: on-failure

        stop_grace_period: 1m

[roxlol]

Finally I had install DSM on a vps,thank you very much for this solution.

 

[DeltaWhiplash]

hello does anyone know how to activate active backup for business on this insane project?
I tried

https://localhost:5001/webapi/auth.cgi?api=SYNO.API.Auth&version=3&method=login&account=USER&passwd=PASS&format= cookie

gaved me

{"data":{"did":"hXZEffEtPTNF7qvZFBLhTL_jFgpYyDPjv0eVjuhLVgdOs-zGbFWxZKKZnyjJDzSlfEjvtkToXCCqhWHreTqdPA","sid":"vbrT9-hOphWVwSAuvmSlGRsMNF5onBNPurPNdaXPorwJehcNKQxpEKrLa1OBa7c68r9XZiaA2xdFmzcShiu-t8"},"success":true}

then 

https://localhost:5001/webapi/entry.cgi?api=SYNO.ActiveBackup.Activation&method=set&version=1&activated=true&serial_number=0000000000000

{"error":{"code":120,"errors":{"name":"serial_number","reason":"type"}},"success":false}

 

[kroese]

Its explained here: https://github.com/kroese/virtual-dsm/issues/266 how to do it.

[hendry]

hi @kroese
is it possible to bind mount host volume to DSM storage, so every updated file from DSM storage will affect automatically to the host drive and vice versa ?

have u ever tried to using virtio 9p driver ? 

and do you know about posix acl (access permission where synology used) ? because i tried uxora docker xpenology, and the problem is i cannot run synology office app since the acl that synology used is different when running inside docker using virtio 9p mount.

does your virtual dsm can run synology office ?

i'm using qnap as the host of docker. your virtual dsm can work inside qnap ?

 

 

 

What are the differences compared to the standard DSM?

There are only two minor differences: the Virtual Machine Manager package is not provided, and Surveillance Station doesn't include any free licenses.

do you mean, this virtual dsm cannot run surveillance station for my cctv camera ?

 

Edited July 7, 2023 by hendry

[kroese]

@hendry It's not possible to mount a physical drive because the container runs a completely stock (unmodified) image of Virtual DSM. So it does not contain any of the required kernel drivers like virtiofs or 9p, because Synology does not include them.

 

And I prefer to keep it that way, otherwise it will experience the same problems as Xpenology (broken system after installing updates, etc). 

 

 

[hendry]

9 minutes ago, kroese said:

@hendry It's not possible to mount a physical drive because the container runs a completely stock (unmodified) image of Virtual DSM. So it does not contain any of the required kernel drivers like virtiofs or 9p, because Synology does not include them.

 

And I prefer to keep it that way, otherwise it will experience the same problems as Xpenology (broken system after installing updates, etc). 

 

 

that's why it's no need to use bootloader such as redpill kernel ?

hmm.. i see.. 

can you make so it can bind mount host drive simultaneously, i mean can you make it using the bootloader and implement 9p drivers or any else ?

Edited July 7, 2023 by hendry

[marc_all]

Hello,
I have tried to install virtualDSM in WSL2 (Win 11) and it is working with your method but the other computers in the network can't see the virtual dsm.

 

So I have tried to have an external IP

sudo docker network create -d macvlan --subnet=192.168.1.0/24  --gateway=192.168.1.1 --ip-range=192.168.1.70/28  -o parent=eth0 vdsm

 

And after that (for the docker-compose)

----------------------------------------------------------------

version: "3"
services:
    dsm:
        container_name: dsm
        image: kroese/virtual-dsm:latest
        environment:
            CPU_CORES: "2"
            DISK_SIZE: "128G"
            RAM_SIZE: "4096M"
        networks:
            vdsm:
                ipv4_address: 192.168.1.70
        devices:
            - /dev/kvm
        cap_add:
            - NET_ADMIN
        ports:
            - 5000:5000
        volumes:
            - /mnt/d/DSM:/storage
        restart: always
        stop_grace_period: 1m
networks:
    vdsm:
        external: true

 

-------------------------------------------------------------------

And I have this error message

dsm    | ❯ ERROR: Failed to download https://global.synologydownload.com/download/DSM/release/7.0.1/42218/DSM_VirtualDSM_42218.pat, reason: 6
dsm    | ❯ Starting Virtual DSM for Docker v4.10...
dsm    | ❯ Install: Downloading installer...

 

 

If I add the environment DHCP and             - /dev/vhost-net and

device_cgroup_rules:
            - 'c 236:* rwm'

I have an error Unsupported config option for services.dsm: 'device_cgroup_rules'

Can you please tell me what I do wrong?

Thank you very much

[kroese]

I have only experience with macvlan under Linux, im not sure if its completely supported under Windows. So it might be easier to just keep it in bridge mode.

 

When you said that other computers could not see the Virtual DSM, did you mean they cannot connect to port 5000 of your Windows machine? Maybe you need to open those ports in the Windows Firewall for them to be able to reach that port. Because if you can connect to that port from your local machine but not from other places, the only logical explanation would be that a firewall blocks access to it. 

[marc_all]

Hello,

Thank you for the time to help.

It works locally so it seems to be a problem with the bridging (using IP:5000 didn't work)

 

I have made a lot of tests, killed my network but finally I saw this web site (It didn't use the macvlan)

 

https://jwstanly.com/blog/article/Port+Forwarding+WSL+2+to+Your+LAN/

With

netsh interface portproxy add v4tov4 listenport=5000 listenaddress=0.0.0.0 connectport=5000 connectaddress=172.29.253.62

 

And I saw it working perfectly.

I am not sure what made it work exactly

But here are my rules (in cvase someone needs it in the future).

I will do other tests to try to understand all but it seems possible.

Thanks again

Adresse         Port        Adresse         Port
--------------- ----------  --------------- ----------
0.0.0.0         5001        172.29.253.62   5001
192.168.1.187   5000        172.29.253.62   5000
192.168.1.187   5001        172.29.253.62   5001
172.17.0.1      5000        172.29.253.62   5000
172.17.0.1      5001        172.29.253.62   5001
172.29.253.62   5000        172.29.253.62   5000
172.29.253.62   5001        172.29.253.62   5001
*               5000        localhost       5000
0.0.0.0         80          172.29.253.62   80
0.0.0.0         443         172.29.253.62   443
0.0.0.0         10000       172.29.253.62   10000
0.0.0.0         3000        172.29.253.62   3000
0.0.0.0         5000        172.29.253.62   5000

[comtech]

On 6/26/2023 at 2:49 PM, roxlol said:

Finally I had install DSM on a vps,thank you very much for this solution.

 

 

Can i ask wich vps provider you used? 
I tried several but they al don't support kvm.

[Poechi]

On 6/26/2023 at 2:49 PM, roxlol said:

Finally I had install DSM on a vps,thank you very much for this solution.

 

Nice! Which hosting provider? Any other steps involved?

[klm-bps]

Hello.
 

Thanks to you I managed to install Xpenology on Windows 11 home with docker desktop.

I used the following command:

docker run --rm -it --name dsm -p 5000:5000 -p 5001:5001 --cap-add NET_ADMIN --device="/dev/kvm:/dev/kvm" kroese/virtual-dsm:latest
 

Now, how can I modify environment variables such as

 environment:

Citation

    ALLOCATE: "N"

 

or
 

Citation

environment:
    CPU_CORES: "3"
    RAM_SIZE: "3048M

as stated in the FAQ

 

I don't know where I should go and how to make these changes.

Maybe I need to do it by installing via "docker-compose.yml", but I don't know how to do it from the desktop docker software under windows 11.

Thanks for your help!

 

[kroese]

59 minutes ago, klm-bps said:

how can I modify environment variables 

 

You can use the -e flag so add "-e CPU_CORES=3 -e ALLOCATE=N" to the docker run command.

[klm-bps]

il y a 52 minutes, kroese a dit :

 

You can use the -e flag so add "-e CPU_CORES=3 -e ALLOCATE=N" to the docker run command.

 

Thank you for your reply !

 

Sorry, this is the 1st time I'm using Docker and I'm still struggling.

 

How do you open a docker command with Desktop Docker under windows 11?

 

I guess I have to open a CMD command prompt under windows, and run the following command:

 

docker -e CPU_CORES=3 -e ALLOCATE=N

Can I do this after I've installed Xpenology and started using it?

Do I need to run the command when docker is run?

How will it know which docker to apply these settings to?

Thanks for your help.

[marc_all]

Hello,

 

Personally I have installed Docker on WSL2 and I simply start bash  (as in a linux distro)

(I used this tutorial to make the installation)

https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop

 

If you use docker desktop (without WSL2)

From https://docs.docker.com/desktop/install/windows-install/

you should have an application Docker Desktop (but I have never used it so I have no idea)

 

For your question with  

Citation

  ALLOCATE: "N"

 

(and so on) you must create a file docker-compose.yml (in a directory) with the example you had

After that you do the command docker-compose up (if all is fine after the test you add -d to make a daemon)

Please pay attention there is only one file docker-compose per directory.

Personally I make a directory Compose and inside it one diurectory per container

(dsm, caddy, portainer...)

 

I hope it helped

 

Edited August 3, 2023 by marc_all

[klm-bps]

Il y a 6 heures, kroese a dit :

 

Vous pouvez utiliser l'indicateur -e afin d'ajouter "-e CPU_CORES=3 -e ALLOCATE=N" à la commande docker run.

I didn't succeed, but I'm going to persevere: see picture

I have another problem: when I switch off DSM properly, the containers disappear completely...

Thanks

[kroese]

41 minutes ago, klm-bps said:

I have another problem: when I switch off DSM properly, the containers disappear completely...
 

 

That is because you use the "-rm" flag which means: remove. But your questions are all about general Docker use, they have nothing to do with my container.

[klm-bps]

Il y a 22 heures, kroese a dit :

 

That is because you use the "-rm" flag which means: remove. But your questions are all about general Docker use, they have nothing to do with my container.

 

Le 03/08/2023 à 16:31, marc_all a dit :

Hello,

 

Personally I have installed Docker on WSL2 and I simply start bash  (as in a linux distro)

(I used this tutorial to make the installation)

https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop

 

If you use docker desktop (without WSL2)

From https://docs.docker.com/desktop/install/windows-install/

you should have an application Docker Desktop (but I have never used it so I have no idea)

 

For your question with  

(and so on) you must create a file docker-compose.yml (in a directory) with the example you had

After that you do the command docker-compose up (if all is fine after the test you add -d to make a daemon)

Please pay attention there is only one file docker-compose per directory.

Personally I make a directory Compose and inside it one diurectory per container

(dsm, caddy, portainer...)

 

I hope it helped

 

f


 

Thank you for your answers !

I can now use DSM correctly with a yml file. I followed some tutorials on the internet to understand the basics of docker.
 

All I have to do now is find out how to change the ALLOCATE: "N" regalge, because if you use it as soon as you install the container, you get an error message that the paratition size is too small, but I'll look for it myself, it doesn't concern the container directly.
 

Thanks a lot!