29 votes

Tildes dev environment on Apple Silicon // Apple M1

Posted by Maxi
Tags: ask.experts, tech support, apple.silicon, development.web

I'm having some trouble getting the local developer environment set up on Apple hardware, specifically it seems because Virtualbox, which is used as the provider, is not properly functioning on Apple hardware.

Is there anyone here who has managed to get it up and running?

19 comments

  1. [2]
    Maxi
    Link
    With the default VM box "debian/contrib-buster64" and the apple m1 developer preview of virtualbox vagrant up fails on startup with Stderr: VBoxManage: error: The object is not ready VBoxManage:...

    With the default VM box "debian/contrib-buster64" and the apple m1 developer preview of virtualbox vagrant up fails on startup with

    Stderr: VBoxManage: error: The object is not ready VBoxManage: error: Details: code E_ACCESSDENIED (0x80070005), component SessionMachine, interface IMachine, callee nsISupports VBoxManage: error: Context: "GetCPUExecutionCap(&u32)" at line 1200 of file VBoxManageInfo.cpp
    I did some digging and it seemes like the virtualbox developer preview does not function well, and has very limited compatibility with various linux distros https://forums.virtualbox.org/viewtopic.php?f=8&t=107344#p528167

    It seems like at the minimum I would have to find another provider that would be compatible, but google does not provide much help with this.

    4 votes
    1. the-boy-sebastian
      Link Parent
      you can use VMWare or Docker as a backend, but I'm not too sure on how you'd have to modify the Vagrantfile to achieve that

      you can use VMWare or Docker as a backend, but I'm not too sure on how you'd have to modify the Vagrantfile to achieve that

      2 votes
  2. [3]
    sixthgear
    Link
    Had similar issues. For me the VirtualBox VM just kept locking up on “vagrant up” without much explanation.

    Had similar issues. For me the VirtualBox VM just kept locking up on “vagrant up” without much explanation.

    4 votes
    1. [2]
      Maxi
      Link Parent
      Did you manage to get it working? Supposedly changing the provider to VMware could perhaps solve it? Though I don't know how compatible it is with Apple Silicone, or if that would break something...

      Did you manage to get it working? Supposedly changing the provider to VMware could perhaps solve it? Though I don't know how compatible it is with Apple Silicone, or if that would break something else specific to the tildes environment.

      1 vote
      1. sixthgear
        Link Parent
        No, I haven't solved it yet. I have had some success (not with Tildes) with VMs on HyperKit via Multipass on M1, but I don't think thats an option here. I'll be watching this post to see if anyone...

        No, I haven't solved it yet. I have had some success (not with Tildes) with VMs on HyperKit via Multipass on M1, but I don't think thats an option here.

        I'll be watching this post to see if anyone else has any luck, and I'll update here too if I can figure anything out.

        2 votes
  3. [6]
    sixthgear
    Link
    @Deimos, @Maxi, moving here from the other topic I wrote this before I saw the update from @Maxi which confirms the same thing. Seems like there's no quick win here. After some digging, I found...

    @Deimos, @Maxi, moving here from the other topic

    I wrote this before I saw the update from @Maxi which confirms the same thing. Seems like there's no quick win here.

    After some digging, I found this thread regarding M1/M2 support with VirtualBox.

    In a way the ARM64 package "slipped out", and it's not expected to work reliably. The implementation isn't complete yet (which is what you saw), and in top of that the performance is known to be extremely low. It isn't anywhere near production ready, we know. This will not change for VirtualBox 7, and the "Technology Preview" marker will stay for the foreseeable future, indicating that it won't be supported at all.

    At best you'll get some really old 32-bit Linux to run to some degree, such as DSL 4.4.10. No chance even with Ubuntu 16.04 i386.

    ...

    And just for clarity (based on additional questions being raised in the forums): VirtualBox is not and never has been an ARM hypervisor. The M1/M2 developer preview relies on an Intel code simulator module to run x86 (32bit) single core guests. Incredibly slowly. It's currently more of a toy than a serious solution to any problem.

    So it seems like a non-starter on AMD64 for the current development stack without some other type of workaround. Here's some other ideas we might explore:

    • Find out if vmware backend works better than virtualbox (might be a similar situation!)
    • Replace vagrant with a Dockerfile setup. But perhaps that's too big of a change for current developers?
    • Here's a weird hybrid approach. But this feels silly, lol.
    3 votes
    1. [4]
      Deimos
      (edited )
      Link Parent
      I definitely prefer Vagrant overall, since I really like having the dev environment be a VM that's practically identical to production. But if it's difficult or impossible to get it to work on...

      I definitely prefer Vagrant overall, since I really like having the dev environment be a VM that's practically identical to production. But if it's difficult or impossible to get it to work on M1/M2 that's a problem, that will prevent a lot of people from being able to set up a dev version.

      I think if there's some kind of alternative Vagrant provider that can actually work that's probably the simplest option. For example, when I did a bit of searching about this I saw some mentions of a QEMU provider, but I don't know if it would work for this setup.

      Docker is an option if necessary, but it will probably end up being more annoying to set up initially and support.

      Either way, I appreciate you and @Maxi (and anyone else) trying to look into it, it's difficult for me to figure anything out since I have no way to actually test it myself.

      3 votes
      1. [2]
        sixthgear
        Link Parent
        The Bad News I tried two alternate provider plugins that seemed promising on M1. vagrant-libvirt was the most promising as it supports almost everthing the virtualbox provider does. Sadly, it...

        The Bad News

        I tried two alternate provider plugins that seemed promising on M1.

        • vagrant-libvirt was the most promising as it supports almost everthing the virtualbox provider does. Sadly, it seems to have a bug causing it to exit with no errors on vagrant up. Hopefully this will be fixed soon.
        • vagrant-qemu worked! Unfortunately it has no nfs share support and also limited networking.

        The Good News
        In my travails I stumbled upon lima which provides a pretty good experience for mac users. I was able to spin up a debian 10 image and now I'm just working through the ansible playbook to weed out any errors.

        So best case, vagrant-libvirt gets sorted out and Apple silicon users can use qemu for development. Worst case we add one more yml file (which looks very similar to the Vagrantfile) to use Lima.

        vmType: qemu
arch: default # x86_64, aarch64

cpus: 4
memory: 4GiB
disk: 50GiB

images:
- location: "https://cloud.debian.org/images/cloud/buster/20230601-1398/debian-10-genericcloud-amd64-20230601-1398.qcow2"
  arch: "x86_64"
  digest: "sha512:d8956fecfff4a176190b77484657e581bb1f7ceaacce6e905e5ab62a899d120e9a7a120c9c4acad89fb29b3e3b6c6666f9579e0e0409bb50ac642bf800b5b918"
- location: "https://cloud.debian.org/images/cloud/buster/20230601-1398/debian-10-generic-arm64-20230601-1398.qcow2"
  arch: "aarch64"
  digest: "sha512:c1cdfa11b2aaec7cc4694ae3f9ce39a73d399e1c546b462778026c3d3c6340b16f78026833f7f03238f5a5dc4a43a42146fa542bc345ee7e75f9dfee48e86954"

mounts:
- location: "~/Projects/tildes/tildes"
  mountPoint: "/opt/tildes"
  writable: true
- location: "~/Projects/tildes/ansible"
  mountPoint: "/srv/ansible"
  writable: true

portForwards:
- guestPort: 443
  hostPort: 4443
- guestPort: 9090
  hostPort: 9090

provision:
- mode: system
  script: |
    #!/bin/bash
    apt update
    apt install -y git
    apt install -y python3-distutils && curl -s https://bootstrap.pypa.io/get-pip.py | sudo python3
    python3 -m pip install ansible
- mode: user
  script: |
    #!/bin/bash
    cat << EOF > /tmp/ansible_local_inventory.ini
    [dev]
    localhost ansible_connection=local

    [app_server]
    localhost ansible_connection=local
    EOF
    ansible-galaxy install --role-file=/srv/ansible/requirements.yml --roles-path=/srv/ansible/roles
    ansible-playbook /srv/ansible/playbook.yml -i /tmp/ansible_local_inventory.ini

        Still a work in progress. I need to find out how to deal with those local paths, and sort out some permissions errors during the provisioning. But if you're ok with another build file I think this could work.

        (@Maxi)

        4 votes
        1. Maxi
          Link Parent
          Oooh that looks very promising. A single extra build file shouldn't be a problem, even if @Deimos doesn't want it into the main branch/repo, it can always be kept in e.g. a separate fork that's...

          Oooh that looks very promising. A single extra build file shouldn't be a problem, even if @Deimos doesn't want it into the main branch/repo, it can always be kept in e.g. a separate fork that's just kept up to date with the main one.

          1 vote
      2. Maxi
        Link Parent
        Thanks! Me and another guy are now looking in to a few different options that'd enable local development on M1/M2 hardware. Going full Docker would be the nuclear option, which would require the...

        Thanks!

        Me and another guy are now looking in to a few different options that'd enable local development on M1/M2 hardware. Going full Docker would be the nuclear option, which would require the most work. We'll keep everyone update if we manage to get something working :)

        3 votes
    2. Maxi
      Link Parent
      Yeah I bumped into those same pages. I also tried a separate approach where I used Docker as a provider for Vagrant - that seemed like a dead end. I think @Deimos would need to weigh in on what...

      Yeah I bumped into those same pages.

      I also tried a separate approach where I used Docker as a provider for Vagrant - that seemed like a dead end.

      I think @Deimos would need to weigh in on what direction he would want to take things. Switching from Vagrant to Docker is a pretty big core change to the project.

      We could always make a Dockerfile (as long as we can get it working without any major Ansible changes) and only use that for local M1/M2 development.

      1 vote
  4. [3]
    the-boy-sebastian
    Link
    Do you get any errors from Vagrant?

    Do you get any errors from Vagrant?

    1 vote
    1. [2]
      Maxi
      Link Parent
      Yes, see my other comment here: https://tildes.net/~tildes/165w/tildes_dev_environment_on_apple_silicon_apple_m1#comment-89uu I think I am forced to alter the Vagrantfile, I was hoping someone...

      Yes, see my other comment here: https://tildes.net/~tildes/165w/tildes_dev_environment_on_apple_silicon_apple_m1#comment-89uu

      I think I am forced to alter the Vagrantfile, I was hoping someone else on Apple Silicone had already figured out a solution :P

      Regardless, I guess once I figure out a way to get it working, I can put in a suggestion to update the documentation.

      4 votes
      1. the-boy-sebastian
        Link Parent
        Yeah, it's annoying, but kinda expected with new architectures.

        Yeah, it's annoying, but kinda expected with new architectures.

        2 votes
  5. [3]
    cfabbro
    (edited )
    Link
    This is really far outside my wheelhouse, but if you all think this is worthy of making a Tildes Gitlab issue for, let me know and I can add one. It's so far outside my wheelhouse that I wouldn't...

    This is really far outside my wheelhouse, but if you all think this is worthy of making a Tildes Gitlab issue for, let me know and I can add one. It's so far outside my wheelhouse that I wouldn't even know what to title the issue or how to describe it though, so you would have to let me know that too. :P

    1 vote
    1. [2]
      Maxi
      Link Parent
      I’ll spend a bit of time later today or tomorrow and look into this, I can make either an issue or a pull request depending on what my findings are :p

      I’ll spend a bit of time later today or tomorrow and look into this, I can make either an issue or a pull request depending on what my findings are :p

      3 votes
      1. cfabbro
        Link Parent
        That would be much appreciated... since I have no idea what I'm doing when it comes to the dev environment. :P

        That would be much appreciated... since I have no idea what I'm doing when it comes to the dev environment. :P

        2 votes
  6. [2]
    Cock
    Link
    This was discussed in a previous comment I made here. In this comment thread, you will find old gitlab issues for this and attempts to get tildes dockerized. There was a previous attempt here that...

    This was discussed in a previous comment I made here.

    In this comment thread, you will find old gitlab issues for this and attempts to get tildes dockerized. There was a previous attempt here that got pretty far, but the guy never kept it updated.

    AFAIK, it will be a very in depth process to transfer over to docker from vagrant at this point in time, and will require ongoing maintenance.

    1 vote
    1. fifthecho
      Link Parent
      As someone with decades in IT Ops and deeper knowledge of containers than I like, let me dig a little into this tomorrow. A good suite of Dockerfiles and a Compose YAML shouldn’t be more work to...

      As someone with decades in IT Ops and deeper knowledge of containers than I like, let me dig a little into this tomorrow.

      A good suite of Dockerfiles and a Compose YAML shouldn’t be more work to maintain than a Vagrant box.

      4 votes