RESOLVED
/var/lib/docker/volumes/ <-- thats where the data is saved

I fucked up colossaly
I run a Nextcloud all in one instance on my ubuntu VM with Hetzner.
I run a backup of my phone to said instance.
I forgot to set a sensible limit on my Nextcloud.
I somehow changed to hourly backups....
My Vm filled up in a very short time.
My VM has not one byte of free space left.

What I tried to do:
open Console inside the Nextcloud Aio Container with docker exec -t -i , to find the space and delete stuff, but no chance, not enough space.

There is nothing else on the VM i know of that i could delete to make space (nothing else i put)

has somebody an idea how to fix this gracefully?

I have some ideas but they involve a lot of work or money...

• just pay for a bigger VM but i dont need more cores only more space, even that i dont really need (incures a monthly cost, cannot downsize afterwards as more space means you cannot downsize)
• clone my vm onto a bigger one, delete stuff, clone it back to the original one... (sound like a lot of work, incrues (little) cost, and feels like a lot of risk involved)

I've got a problem with my nextcloud and as tildes is my favourite nice place to ask for tech-support, maybe somebody here can help me with that.

I can start and run Nextcloud AIO without any problems. I can update the subcontainers without any problems. But the update of the Mastercontainer always fails and I don't kno why, only that it has to be something with docker.sock and permissions, but I could not resolve the issues, and google does not seem to be helpful (or I'm looking for the wrong stuff).

my update logs:

time="2023-06-14T12:47:59Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2023-06-14T12:48:00Z" level=debug msg="Making sure everything is sane before starting"
time="2023-06-14T12:48:00Z" level=info msg="Watchtower 1.5.3"
time="2023-06-14T12:48:00Z" level=info msg="Using no notifications"
time="2023-06-14T12:48:00Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2023-06-14T12:48:00Z" level=info msg="Running a one time update."
time="2023-06-14T12:48:00Z" level=debug msg="Checking containers for updated images"
time="2023-06-14T12:48:00Z" level=debug msg="Retrieving running containers"
time="2023-06-14T12:48:00Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!" error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" module=api status_code=-1
time="2023-06-14T12:48:00Z" level=error msg="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x9f4a22]

goroutine 1 [running]:
github.com/containrrr/watchtower/pkg/metrics.NewMetric({0x0, 0x0})
	/home/runner/work/watchtower/watchtower/pkg/metrics/metrics.go:31 +0x22
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc0002fd830)
	/home/runner/work/watchtower/watchtower/cmd/root.go:375 +0x15e
github.com/containrrr/watchtower/cmd.Run(0xc00033c300?, {0xc000281300?, 0x4?, 0x4?})
	/home/runner/work/watchtower/watchtower/cmd/root.go:168 +0x570
github.com/spf13/cobra.(*Command).execute(0xc00033c300, {0xc0000300b0, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00033c300)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/containrrr/watchtower/cmd.Execute()
	/home/runner/work/watchtower/watchtower/cmd/root.go:71 +0x52
main.main()
	/home/runner/work/watchtower/watchtower/main.go:13 +0x17

my startup command

sudo docker run \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart unless-stopped \
--publish 8080:8080 \
-e APACHE_PORT=11000 \
-e APACHE_IP_BINDING=127.0.0.1 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
nextcloud/all-in-one:latest

output after start:

Trying to fix docker.sock permissions internally...
Creating docker group internally with id 998
WARNING: No swap limit support
Initial startup of Nextcloud All-in-One complete!
You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
E.g. https://internal.ip.of.this.server:8080

If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
https://your-domain-that-points-to-this-server.tld:8443
++ head -1 /mnt/docker-aio-config/data/daily_backup_time
+ BACKUP_TIME=04:00
+ export BACKUP_TIME
+ export DAILY_BACKUP=1
+ DAILY_BACKUP=1
++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
+ '[' '' '!=' automaticUpdatesAreNotEnabled ']'
+ export AUTOMATIC_UPDATES=1
+ AUTOMATIC_UPDATES=1
+ set +x
{"level":"info","ts":1686746753.2700157,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
{"level":"info","ts":1686746753.2748601,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
[14-Jun-2023 12:45:53] NOTICE: fpm is running, pid 106
[14-Jun-2023 12:45:53] NOTICE: ready to handle connections

I tried to change permissions on /var/run/docker.sock
I tried to change permissions on /lib/systemd/system/docker.sock

same but with restart of docker.sock
same but with restart of docker.sock and docker.service

nothing helped

does somebody know where I go wrong or can me point in the right direction to resolve this problem? It's not a game stopper as I can update the container manually, but it is annoying.

edit: it runs on a ubuntu server 20.04.6 LTS