36
votes
Resources and help for setting up a Tildes dev environment
I've been trying to set up a dev enviornment for Tildes, mainly so that I can actually test my MR (!136), and I've been running into a few issues.
However, since we also have a new influx of people who might be interested in contributing to Tildes, it seems like a good time to collect resources on setting up the dev environment, as well as helping anyone running into issues.
So, if you have issues or advice, post them here! I'll be adding my questions in a comment shortly.
Relevant wiki pages:
Edit: A more recent post on setting up the dev environment on Apple Silicon / M1 Macs
I personally ran into issues with getting vagrant set up; it seems like it only works with VirtualBox, but vagrant appears to not support VirtualBox 7.0, only VirtualBox 6.1, which appears to be a broken package in Fedora... (I don't remember the exact details for this, I'll find the logs when I get a chance)
Does anyone know how to get the vagrant setup running on something other than VirtualBox, or general advice?
Edit: I think I've managed to get the virtualbox 6.1 install unbroken, so I'll see how far I get this time.
Edit 2: Nope, it's still broken...
With the default provider
libvirt
I getand VirtualBox complains about the kernel module being missing, so I guess I'll work on fixing that.
Update: After upgrading my entire distro (probably not necessary), going back to VirtualBox 7.0, and then manually installing a recent version of vagrant, I've managed to get vagrant running. (Support for VirtualBox 7.0 appears to have been added very recently.)
edit- just saw your update. But I'll leave the comment here in case anyone else wants to learn about the exciting world of default providers for Vagrant :P
It's been a while since I worked with Vagrant, so idk how much help I can be beyond this comment, but check this out -- https://developer.hashicorp.com/vagrant/docs/providers/basic_usage
tl:dr -
There are a few ways to do it. The quickest/easiest way is to just add a flag to the up command like this:
vagrant up --provider=hyperv
. Swapping "hyperv" for whichever supported provider you want. I believe you'd have to remember to include that flag each time your run the command though (could be mistaken).The best approach, imo, would be to set the provider in the Vagrantfile with
config.vm.provider "hyperv"
, again being sure to swap "hyperv" out for your preference.The link above has more detailed instructions and examples, so check it out.
Thanks for the help; I think I've got it working on VirtualBox... but I'm still getting weird issues that might be from the VirtualBox version being too new, or something like that. (If only this worked in a virtual machine that didn't need a specific versioned kernel module and distro support...)
I think the main issue with the providers is that debian/contrib-buster64 only supports the
virtualbox
provider, so we're stuck with that anyway.Just wanted to offer a quick thank you for posting this. I made an account here a while ago, but after a few months stopped visiting very often.
Now that reddit is destroying itself, I came to remember that I enjoyed my time here, and stopped back to see what is happening and to see if I might be able to contribute as I am a professional software dev.
I appreciate that I was quickly able to find a post about it; thanks again.
After getting vagrant working, the basic setup works; now I'm just running into issues with the code-formatting tools.
Following the instructions from the wiki; I'm running
invoke code-style-check --full
, but it looks like it can't findstylelint
.invoke code-style-check --full
outputI guessed that it might be a missing
npm install stylelint
somewhere in the install script, so I tried that, but it seems to run into errors when installing, so I'm not sure what to do next. (I'm currently rebuilding the vagrant VM to test with a clean environment, but that takes a bit.)npm install stylelint
output@Deimos do you have a general idea of what would be causing this / how I should approach fixing it?
Edit:
It looks like a buggy interaction between vagrant, VirtualBox, and
npm install
; the quick fix is to runnpm ci
to install the packages correctly. This merge request should make it work automatically, hopefully.Worked for me on a clean Vagrant VM running (from the host):
Got and adapted the command from
git_hooks/pre-commit
.Also can I ask you to stop pinging Deimos please, he's already swamped as is with the huge influx of users. Other people will be happy to answer your troubleshooting questions, as demonstrated in this comment section.
That said, it is great you and others have started to pick out issues from the issue tracker and start coding bugfixes. I am working on coordinating with Deimos on a solution for maintaining the repo, mostly through DMs so far, not ready to speak on it yet.
I wasn't sure whether to ping the second time; I only did because of basically the opposite response in the previous thread.
I'll avoid pinging again; good luck with the maintenance / app development!
That command gives the exact same error for me on a clean vagrant VM.
Commands and abbreviated output
From the wiki:
For me,
vagrant provision
gives a few changed states, so could that be an issue?Logs
The only things I can see are potentially the VirtualBox version mismatching?
(since at least in the
npm install
case, it looks like a filesystem issue with a host-shared folder?)Tool versions: (on Fedora 38)
I can post the full
vagrant up
/vagrant provision
logs if needed.It looks like there are some semi-known issues with
vagrant
andnpm install
?I'm not sure if that's causing issues with the install script itself, but it matches the errors I get when trying to manually install
stylelint
https://github.com/hashicorp/vagrant/issues/11842
A working workaround:
and then everything appears to work. (Basically putting
node_modules
in a VM's filesystem instead of in a shared folder to work around the issue.)Ah, sorry, that bug slipped my mind. Indeed it breaks on a clean git checkout. I filed an MR for that before (I think it is the same issue, at least): https://gitlab.com/tildes/tildes/-/merge_requests/133
It clicked when you mentioned the
node_modules
directory is not being created.Yep, it looks like
npm ci
does the right thing; thanks!I'll check from a clean VM and mention the error messages in the MR so it's easier for others to find in the meantime (though hopefully it'll end up merged at some point).
I had trouble setting up my IDE to detect packages, as they were installed on Vagrant and not on my host machine.
Python was installed on my host, but in a more recent version than the VM so when I tried to install the packages from requirements it didn’t work.
As Im not a Python dev and I’m using PyCharm community, the Vagrant integration and ssh interpreter are not available.
What I figured out with the help of ChatGPT :
Then I got some errors and had to install some packages like python3-dev, libluajit-5.1-dev, etc.
After that, set the local virtual environment as the interpreter in PyCharm:
Go to File > Settings > Project: <Your Project Name> > Python Interpreter.
Click on the gear icon and select Add.
In the left panel of the dialog that opens, select Virtualenv Environment.
In Location, select the venv directory you just created. Then click OK.
Has anyone been successful in getting this running on an M series apple processor? Virtualbox has limited support for it and any instance I create gets aborted. Not sure I know a way around this.
In theory you might be able to use a different provider? (See @hamstergeddon's comment)
The current
Vagrantfile
uses debian/contrib-buster64 which only supports VirtualBox, but it looks like generic/debian10 is similar, with support for more providers. The current setup relies on synced folders though, and I'm not sure how well those work with non-VirtualBox providers. (You'll also have to change the cpu/memory config in the Vagrantfile to apply to the provider you choose.)I haven't managed to get vagrant to run with anything other than VirtualBox due to Fedora/SELinux issues, but hopefully you'll find something that works.
Hmm.. I wish they were using docker instead.
I'm not particularly well-versed in docker, but you might be able to use a docker image instead of vagrant?
The vagrant setup appears to just be setting up a debian 10 image with some bind mounts and forwarded ports, and then using ansible to deploy a dev environment; as long as there aren't nested containers or other kernel-level things being used, the same should be possible with a docker container.
I saw 2 attempts of this talked about in gitlab. One got it running but I don't think he finished it, I believe there were cron jobs that they didn't get added. If someone can get docker builds maintained that would make development easier and people deploying their own in k8s would be nice.
Here is as far as I got.. it is erroring on the "Create group for app user" on ansible startup. I am stuck at this point.
I'm interested in getting this working but i currently have 0 knowledge on tildes backend, if i get a chance this weekend I'll take a crack at it.
Edit: took a look at the latest attempt. There appears to be a lot more they have set up then yours. Source
Yeah this was more of just translating the Vagrantfile to a Dockerfile and seeing if it'd be that easy, and it wasn't of course! So I stopped there. I have never worked with ansible and the setup that is involved with it. Let me know if you get that guys fork working, it'd be great to hear.
Edit:
Ran the docker-compose with this guys fork in an up to date tildes fork. It seems that it has since broken with recent changes.
By any chance is there a way to quickly generate dummy data so I've got a more prod-like environment to make changes on? Or would I need to manually create any test data via
invoke shell
or the UI? Or alternatively, is there a SQL file floating out there somewhere with dummy data?Thanks for posting this! I wish I knew Python so I could contribute on the backend (I'm a PHP guy), but maybe I'll see if there are any frontend things I can work on to help out (I'm also a JS guy!).
Tangentially related, but has anyone forked/used the Tildes source to spin up their own website yet?
I believe some communities have tried it in the past, but I don't think there are currently other active instances. (the main example).
I don't think there are currently good docs for how to deploy it (issue #564) but it should be doable, with some work.
Interesting. Thank you.
I am trying to get everything set up so that I can start making contributions. I'm following the Tildes guide, although I didn't sync my fork to the official repository because I didn't understand what I was supposed to look at in the first link.
When I run
vagrant up
(or, more accurately,vagrant provision
) everything proceeds swimmingly untilTASK [nodejs : Install npm packages defined in package.json] *******************
At which point I get the following failure message (I have spaced it out to make it easier to read):
"msg": "npm WARN prepare removing existing node_modules/ before installation\nnpm ERR! code EPROTO\nnpm ERR! syscall symlink\nnpm ERR! path ../semver/bin/semver\nnpm ERR! dest /opt/tildes/node_modules/.bin/semver\nnpm ERR! errno -71\nnpm ERR! EPROTO: protocol error, symlink '../semver/bin/semver' -> '/opt/tildes/node_modules/.bin/semver'\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR! /home/vagrant/.npm/_logs/2023-09-10T00_56_41_305Z-debug.log",
"rc": 185,
"stderr": "npm WARN prepare removing existing node_modules/ before installation\nnpm ERR! code EPROTO\nnpm ERR! syscall symlink\nnpm ERR! path ../semver/bin/semver\nnpm ERR! dest /opt/tildes/node_modules/.bin/semver\nnpm ERR! errno -71\nnpm ERR! EPROTO: protocol error, symlink '../semver/bin/semver' -> '/opt/tildes/node_modules/.bin/semver'\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR! /home/vagrant/.npm/_logs/2023-09-10T00_56_41_305Z-debug.log\n",
I get that this has to do with NPM not working inside vagrant, but I can't figure out what to do. My assumption is that it is trying to run
/usr/bin/npm ci --no-bin-links
from inside Vagrant but I can't tell how I can either fix or work around this. I tried runningnpm install --no-bin-links
; that gave me the following:I'm totally lost with this. I vaguely understand what it's trying to do, but I have no clue what I need to do. Perhaps I should have changed some settings when I was installing VirtualBox or VagrantAny help would be greatly appreciated.
Does the issue happen on a clean Git checkout? I.e. clone into a new empty directory.
The
npm ci
change was to fix an issue with node_modules directory not being created. However I can imagine cases where the node_modules was already created a different way, and then the switch tonpm ci
might have some permissions or symlink conflicts, or something like that.Are you saying to start vagrant in an empty directory? There was nothing else in the folder when I cloned tildes with git.
Ah okay. That might not be the issue then.
An even more fundamental thing is the fact you appear to be running on Windows. Tildes and its dependencies probably expect a Unix environment, Linux in particular. Maybe you will have better luck running in WSL?
Edit: But I have no idea how Vagrant will interact with WSL. Maybe someone else knows more about running this stuff on Windows.
I thought the whole point of running Vagrant/VirtualBox was that it didn't matter what OS, etc you were running.
Thanks for the help, though. And thanks for all the years of reddit is fun and for recommending Tildes!
To some degree. In this case I think there's a complication because it's mounting your local host directory into the guest VM, and there is likely a filesystem mismatch, as the error message mentions a problem with symlinks.
I just did a quick search and there might be a workaround: https://stackoverflow.com/questions/33186123/npm-install-errors-on-vagrant-homestead-windows-eproto-protocol-error-symlink
See the answers about modifying the Vagrantfile and adding a setting to allow symlinks.
It's supposed to run on Windows, and work has been completed in the past to help it run there. However the build process may have bit rotted so that it's no longer functional.
Update for posterity (pasted from my other comment):
Edit 2: Found a proper fix. Don't run as administrator; that's bad and shouldn't be required.
https://gitlab.com/tildes/tildes/-/merge_requests/153
After @Deimos merges this, Vagrant provisioning should work on Windows again.
Old bad solution
https://gitlab.com/tildes/tildes/-/merge_requests/152Make sure your VM is stopped (
vagrant halt
)and then apply that Vagrantfile patchand then while running Windows command prompt as Administrator run:vagrant up --provision
Edit: Apparently a patch is unnecessary; symlinks may be enabled by default, as long as running as Administrator.
(cc @Merry, @Deimos from the other thread)