6 votes

Docker Nextcloud AIO Mastercontainer update failing

I've got a problem with my nextcloud and as tildes is my favourite nice place to ask for tech-support, maybe somebody here can help me with that.

I can start and run Nextcloud AIO without any problems. I can update the subcontainers without any problems. But the update of the Mastercontainer always fails and I don't kno why, only that it has to be something with docker.sock and permissions, but I could not resolve the issues, and google does not seem to be helpful (or I'm looking for the wrong stuff).

my update logs:

time="2023-06-14T12:47:59Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2023-06-14T12:48:00Z" level=debug msg="Making sure everything is sane before starting"
time="2023-06-14T12:48:00Z" level=info msg="Watchtower 1.5.3"
time="2023-06-14T12:48:00Z" level=info msg="Using no notifications"
time="2023-06-14T12:48:00Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2023-06-14T12:48:00Z" level=info msg="Running a one time update."
time="2023-06-14T12:48:00Z" level=debug msg="Checking containers for updated images"
time="2023-06-14T12:48:00Z" level=debug msg="Retrieving running containers"
time="2023-06-14T12:48:00Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!" error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" module=api status_code=-1
time="2023-06-14T12:48:00Z" level=error msg="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x9f4a22]

goroutine 1 [running]:
github.com/containrrr/watchtower/pkg/metrics.NewMetric({0x0, 0x0})
	/home/runner/work/watchtower/watchtower/pkg/metrics/metrics.go:31 +0x22
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc0002fd830)
	/home/runner/work/watchtower/watchtower/cmd/root.go:375 +0x15e
github.com/containrrr/watchtower/cmd.Run(0xc00033c300?, {0xc000281300?, 0x4?, 0x4?})
	/home/runner/work/watchtower/watchtower/cmd/root.go:168 +0x570
github.com/spf13/cobra.(*Command).execute(0xc00033c300, {0xc0000300b0, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00033c300)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/containrrr/watchtower/cmd.Execute()
	/home/runner/work/watchtower/watchtower/cmd/root.go:71 +0x52
main.main()
	/home/runner/work/watchtower/watchtower/main.go:13 +0x17

my startup command

sudo docker run \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart unless-stopped \
--publish 8080:8080 \
-e APACHE_PORT=11000 \
-e APACHE_IP_BINDING=127.0.0.1 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
nextcloud/all-in-one:latest

output after start:

Trying to fix docker.sock permissions internally...
Creating docker group internally with id 998
WARNING: No swap limit support
Initial startup of Nextcloud All-in-One complete!
You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
E.g. https://internal.ip.of.this.server:8080

If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
https://your-domain-that-points-to-this-server.tld:8443
++ head -1 /mnt/docker-aio-config/data/daily_backup_time
+ BACKUP_TIME=04:00
+ export BACKUP_TIME
+ export DAILY_BACKUP=1
+ DAILY_BACKUP=1
++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
+ '[' '' '!=' automaticUpdatesAreNotEnabled ']'
+ export AUTOMATIC_UPDATES=1
+ AUTOMATIC_UPDATES=1
+ set +x
{"level":"info","ts":1686746753.2700157,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
{"level":"info","ts":1686746753.2748601,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
[14-Jun-2023 12:45:53] NOTICE: fpm is running, pid 106
[14-Jun-2023 12:45:53] NOTICE: ready to handle connections

I tried to change permissions on /var/run/docker.sock
I tried to change permissions on /lib/systemd/system/docker.sock

same but with restart of docker.sock
same but with restart of docker.sock and docker.service

nothing helped

does somebody know where I go wrong or can me point in the right direction to resolve this problem? It's not a game stopper as I can update the container manually, but it is annoying.

edit: it runs on a ubuntu server 20.04.6 LTS

11 comments

  1. [2]
    0d_billie
    Link
    I haven't tried using watchtower as I prefer manual upgrades, so the help I can give is probably limited. But judging by the error accessing /var/run/docker.sock, have you checked that docker and...

    I haven't tried using watchtower as I prefer manual upgrades, so the help I can give is probably limited. But judging by the error accessing /var/run/docker.sock, have you checked that docker and containerd both active and running and not showing any errors?

    systemctl status docker
    systemctl status containerd
    
    2 votes
    1. Don_Camillo
      Link Parent
      sudo systemctl status docker docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running)...

      sudo systemctl status docker

      docker.service - Docker Application Container Engine
           Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
           Active: active (running) since Wed 2023-06-14 12:44:56 UTC; 4h 3min ago
      TriggeredBy: ● docker.socket
             Docs: https://docs.docker.com
         Main PID: 20426 (dockerd)
            Tasks: 37
           Memory: 35.9M
           CGroup: /system.slice/docker.service
                   ├─20426 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
                   ├─20608 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 8096 -container-ip 172.17.0.3 -container-port 8096
                   ├─21804 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.18.0.2 -container-port 8080
                   └─21810 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 8080 -container-ip 172.18.0.2 -container-port 8080
      
      Jun 14 12:44:56 Kloud systemd[1]: Started Docker Application Container Engine.
      Jun 14 12:44:56 Kloud dockerd[20426]: time="2023-06-14T12:44:56.921315248Z" level=info msg="API listen on /run/docker.sock"
      Jun 14 12:45:14 Kloud dockerd[20426]: time="2023-06-14T12:45:14.869572098Z" level=info msg="ignoring event" container=d840f523190b46472539282d8df470af5bc08961f4b25d79c81c06fe63590285 module=libcontainerd namespace=moby topic=/tasks>
      Jun 14 12:45:14 Kloud dockerd[20426]: time="2023-06-14T12:45:14.891215004Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=d840f523190b46472539282d8df470af5bc08961f4b25d79c81c06fe63590285 daemon>
      Jun 14 12:46:07 Kloud dockerd[20426]: time="2023-06-14T12:46:07.542799342Z" level=error msg="Handler for POST /v1.41/networks/bridge/disconnect returned error: container 31d5e7506025e2b2f9477d1197d1c794f84fadb2e9bb77815035e3e9f825e>
      Jun 14 12:47:58 Kloud dockerd[20426]: time="2023-06-14T12:47:58.573827441Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
      Jun 14 12:47:58 Kloud dockerd[20426]: time="2023-06-14T12:47:58.573898090Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
      Jun 14 12:47:59 Kloud dockerd[20426]: time="2023-06-14T12:47:59.303740811Z" level=error msg="Handler for POST /v1.41/networks/bridge/disconnect returned error: container 31d5e7506025e2b2f9477d1197d1c794f84fadb2e9bb77815035e3e9f825e>
      Jun 14 12:48:00 Kloud dockerd[20426]: time="2023-06-14T12:48:00.167197875Z" level=info msg="ignoring event" container=3b3ec73bd415593f5e1f1617169292dcb178fec3edec4968e183f0f05994342c module=libcontainerd namespace=moby topic=/tasks>
      Jun 14 12:48:04 Kloud dockerd[20426]: time="2023-06-14T12:48:04.820666686Z" level=error msg="Handler for POST /v1.41/networks/bridge/disconnect returned error: container 31d5e7506025e2b2f9477d1197d1c794f84fadb2e9bb77815035e3e9f825e>
      

      sudo systemctl status containerd

      ● containerd.service - containerd container runtime
           Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
           Active: active (running) since Wed 2023-06-14 12:10:38 UTC; 4h 43min ago
             Docs: https://containerd.io
         Main PID: 610 (containerd)
            Tasks: 45
           Memory: 92.3M
           CGroup: /system.slice/containerd.service
                   ├─  610 /usr/bin/containerd
                   ├─20622 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 8235ae8b06dad03e569f04a5ed6debaedaf161c672f53e2a2d3bcfa228f7b9bb -address /run/containerd/containerd.sock
                   ├─20649 /usr/bin/containerd-shim-runc-v2 -namespace moby -id e1a5c99b139a09ae70d5b59ed1692f7084e126c43d27ba41c6e83d3ef898a191 -address /run/containerd/containerd.sock
                   └─69769 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b7425eb0cf9e7ee6c26cc4dac111a40bfd995317e4e1d3dea2401d23f11e6cf1 -address /run/containerd/containerd.sock
      
      Jun 14 16:51:16 Kloud containerd[610]: time="2023-06-14T16:51:16.379288620Z" level=info msg="cleaning up dead shim"
      Jun 14 16:51:16 Kloud containerd[610]: time="2023-06-14T16:51:16.391699175Z" level=warning msg="cleanup warnings time=\"2023-06-14T16:51:16Z\" level=info msg=\"starting signal loop\" namespace=moby pid=70325 runtime=io.containerd.r>
      Jun 14 16:51:23 Kloud containerd[610]: time="2023-06-14T16:51:23.821308898Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
      Jun 14 16:51:23 Kloud containerd[610]: time="2023-06-14T16:51:23.822073796Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
      Jun 14 16:51:23 Kloud containerd[610]: time="2023-06-14T16:51:23.822246669Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
      Jun 14 16:51:23 Kloud containerd[610]: time="2023-06-14T16:51:23.822594499Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/cd7cbc9a9286d25dbae4bef2f5828b361eb97fd619162>
      Jun 14 16:51:25 Kloud containerd[610]: time="2023-06-14T16:51:25.308388881Z" level=info msg="shim disconnected" id=cd7cbc9a9286d25dbae4bef2f5828b361eb97fd619162f541d9a47fe553b91a8
      Jun 14 16:51:25 Kloud containerd[610]: time="2023-06-14T16:51:25.308475816Z" level=warning msg="cleaning up after shim disconnected" id=cd7cbc9a9286d25dbae4bef2f5828b361eb97fd619162f541d9a47fe553b91a8 namespace=moby
      Jun 14 16:51:25 Kloud containerd[610]: time="2023-06-14T16:51:25.308491557Z" level=info msg="cleaning up dead shim"
      Jun 14 16:51:25 Kloud containerd[610]: time="2023-06-14T16:51:25.321960316Z" level=warning msg="cleanup warnings time=\"2023-06-14T16:51:25Z\" level=info msg=\"starting signal loop\" namespace=moby pid=70494 runtime=io.containerd.r>
      ~
      
  2. [7]
    th0mcat
    Link
    Looking at these logs, it looks like Watchtower cannot see /var/run/docker.sock. Do you have that bind mounted to Watchtower like you have it bind mounted to Nextcloud (--volume...

    Looking at these logs, it looks like Watchtower cannot see /var/run/docker.sock. Do you have that bind mounted to Watchtower like you have it bind mounted to Nextcloud (--volume /var/run/docker.sock:/var/run/docker.sock:ro)?

    2 votes
    1. [6]
      Don_Camillo
      Link Parent
      watchtower is running inside of the Nextcloud Mastercontainer. So I can only give binds to the mastercontainer. I'm pretty shure this here is my problem (output when running the Mastercontainer),...

      watchtower is running inside of the Nextcloud Mastercontainer. So I can only give binds to the mastercontainer.

      I'm pretty shure this here is my problem (output when running the Mastercontainer), but I don't know how to resolve this

      Trying to fix docker.sock permissions internally...
      Creating docker group internally with id 998
      
      1. [2]
        th0mcat
        Link Parent
        Huh, interesting that the AIO image has watchtower built in. Sorry I can't be of more help! If I think of something else I'll let you know

        Huh, interesting that the AIO image has watchtower built in. Sorry I can't be of more help! If I think of something else I'll let you know

        1. Don_Camillo
          Link Parent
          thank you very much for your help anyway!

          thank you very much for your help anyway!

      2. Don_Camillo
        Link Parent
        hmm, I was able to get rid of that message by sudo chmod 666 /var/run/docker.sock but It still does not work

        hmm, I was able to get rid of that message by

        sudo chmod 666 /var/run/docker.sock

        but It still does not work

      3. [2]
        0d_billie
        Link Parent
        This sounds... unusual. Is this the normal configuration for Nextcloud AIO?

        watchtower is running inside of the Nextcloud Mastercontainer.

        This sounds... unusual. Is this the normal configuration for Nextcloud AIO?

  3. [2]
    burntcookie90
    Link
    Searching their issues leads me to https://github.com/nextcloud/all-in-one/issues/1800 Could you try that? If it doesn't help, maybe it would be useful to file on the repo directly so they can...

    Searching their issues leads me to https://github.com/nextcloud/all-in-one/issues/1800

    Could you try that? If it doesn't help, maybe it would be useful to file on the repo directly so they can hopefully help a bit more.

    1 vote
    1. Don_Camillo
      Link Parent
      thank you for the link, this seems to be a MacOS related issue, and the solution does not really help me -/

      thank you for the link, this seems to be a MacOS related issue, and the solution does not really help me -/

      2 votes