dodger's recent activity

  1. Let's talk best-practice Jenkins on AWS ECS

    ~comp Text 704 words
    [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly] I've had some...

    [seen on reddit but no discussion - if it's not okay to seek out better discussion here after seeing something fall flat on reddit, I am very sorry and I'll delete promptly]

    I've had some experience in this realm for a while now, but I'm having a little trouble with one issue in particular. Before I divulge, I'll present my thoughts on best practice and and what I've been able to implement:

    • Terraform everything (in accordance to terragrunt's "style guide" i.e. organization)
      THIS IS A BIG ONE: for the jenkins master task, make sure to use the following args to make sure jenkins jobs aren't super slow as hell to start:
    -Djava.awt.headless=true -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

    THIS IS A GAME CHANGER (more-so on k8s clusters when the ecs plugin isn't used... hint, it's shit).

    • Create an EFS (in a separate terraform module) and mount it to the jenkins ECS cluster at /var/jenkins_home. Makes jenkins much more reliable through outages and easier to upgrade.
    • Run a logging agent (via docker container) like logspout or newrelic or whatever IN USER_DATA and not as a task - that way you get logs if there are issues during user_data/cloud_init... this I'm actually not sure about. Running a container outside the context of an ECS task means the ECS agent can't really track it and allocate mem/cpu properly... but it does help with user_data triage.
    • Use pipelines and git plugins to drive jobs. All jenkins jobs should be in source control!
    • Make sure you setup docker cleanup jobs on DAY 1! If you hace limited access to your cluster and you run out of disk due to docker cache, networks, volumes, etc... you're screwed till the admin ssh's in and runs a prune. Get a docker system prune going or the equivalent for each docker resource with appropriate filters... i.e. filter for anything older than a few days and is dangling.
    • Use Jenkins Global Libraries to make Jenkinsfiles cleaner (I always just use vars instead of groovy/java style packages because it's easier and less ugly)
      Jenkinsfiles should mostly call other bash files, make files, python scripts to generate and load prop files, etc. The less logic you put in a Jenkinsfile (which is just modified groovy) the better. String interpolation, among other things, is a fuckery that we don't have time to triage.
    • (out-of-scope) Move to using k8s/EKS instead of ECS asap because the ECS plugin for jenkins is absolute shit and it doesn't use priority correctly (sorry whoever developed it and... oh wait abandoned it and hasn't merged anything for years... for for real it's cool, just give admin to someone else).
    • (cultural) Stop calling them slaves. "Hey @eng, we're rotating slaves due to some cache issues. If you have been affected by race conditions in that past, our new update and slave rotation should fix that. Our update may have killed your job that was running on an old slave, just wait a few and the new slaves will be ready" <--This just doesn't look good.
      Hope that was some good stuff for you guys. Maybe I'm preaching to the choir, but I've seen some pretty shit jenkins setups.

    NOW FOR MY QUESTION!

    Has ANYONE actually been able to setup a proper jenkins user on ECS that actually works for both a master and ephemeral jenkins-agents so that they can mount and use the docker.sock for builds without hitting permission issues? I'm talking using the ecs plugin and mounting docker.sock via that.

    I have always resorted to running jenkins master and agents as root, which means you have to chmod files (super expensive time and cpu for services with tons of files). Running microservices as root is obviously bad practice, and chmod-ing a zilliion files is shit for docker cache and time... so I want to get jenkins users able to utilize the docker.sock. THIS IS SPECIFICALLY FOR THE AWS ECS AMI! I don't care about debian or old versions of docker where you could use DOCKER_OPTS. That doesn't work on the AWS Linux image.

    Thanks! And happy Friday!

    5 votes

  2. Comment on Slack outage: All workspaces affected in ~comp

    dodger
    Link Parent
    Yeesh, same. You'd think they'd have better redundancy and disaster recovery automation being a medium that can be heavily utilized for ops automation.

    Yeesh, same. You'd think they'd have better redundancy and disaster recovery automation being a medium that can be heavily utilized for ops automation.

    1 vote

  3. Comment on What do you Tilderinos think about chillhop and other down-tempo genres? in ~music

    dodger
    Link Parent
    Still drinking out of my somafm mug from 5 years ago! Used to LOVE defjam radio on somafm, but like all stations on there it sometimes depends on the dj. Some really fun and interesting stuff on...

    Still drinking out of my somafm mug from 5 years ago! Used to LOVE defjam radio on somafm, but like all stations on there it sometimes depends on the dj. Some really fun and interesting stuff on there thats worth a browse.

    1 vote

  4. Comment on The Ante programming language in ~comp

    dodger
    Link
    You had me at "integrated build system". Can't wait to try this. Planning to create an official docker image or vscode/other ide plugins?

    You had me at "integrated build system". Can't wait to try this. Planning to create an official docker image or vscode/other ide plugins?

    4 votes

  5. Comment on After six days, Portland’s ICE blockade is a city of more than eighty tents in ~news

    dodger
    Link
    I get protesting, but is portland really a great hub for this? Why not pick a right wing state/city and ruin their days/commutes? Everyone in portland already probably agrees.

    I get protesting, but is portland really a great hub for this? Why not pick a right wing state/city and ruin their days/commutes? Everyone in portland already probably agrees.

  6. Comment on Summer Games Done Quick 2018 is live in ~games

    dodger
    (edited )
    Link Parent
    Watching the early LoZ runs with... forget his name but the guy with usually painted fingernails... he and his companion's commentary on the mechanics behind the glitches and history behind...

    Watching the early LoZ runs with... forget his name but the guy with usually painted fingernails... he and his companion's commentary on the mechanics behind the glitches and history behind evolving runs - that shit was awesome. I get so bored when people dont explain what they are doing during a glitch or run. The runners personality makes or breaks the experience and AGDQ has had some killer and entertaining runners. Looking forward to this.

    Edit: The LoZ runner from a few years ago was "cosmo". Did some real fun runs on ocarina and WW.

    1 vote

  7. Comment on What are the positive aspects of Microsoft's acquisition of GitHub, if any? in ~comp

    dodger
    Link Parent
    Agreed. It's easy to opt-out but the defaults suck.

    Agreed. It's easy to opt-out but the defaults suck.

    1 vote

  8. Comment on What are the positive aspects of Microsoft's acquisition of GitHub, if any? in ~comp

    dodger
    Link Parent
    Git itself is opensource, anyone can contribute. Github may employ people to work specifically on git, but they seem to mostley provide features outside of git to enhance the co-coding experience....

    Git itself is opensource, anyone can contribute. Github may employ people to work specifically on git, but they seem to mostley provide features outside of git to enhance the co-coding experience. If microsoft forks git and creates a "proprietary" knock off, they have to release the source code since git is "copy left" - there is no reason for them to do this and instead would contribute to git itself. Can't provide sources this second but hopefully someone can back me up on this.

    Guido (creator of Python) works at dropbox - they employ him to spend 50% of his time on dropbox work and 50% working on Python (or something like that). A few companies do this kind of thing for a few reasons, so github might do the same for git or other OS programs.

    2 votes

  9. Comment on Podcast recommendations in ~talk

    dodger
    Link Parent
    I somehow hated the show but absolutely love the podcast.

    I somehow hated the show but absolutely love the podcast.

    1 vote

  10. Comment on Podcast recommendations in ~talk

    dodger
    Link
    The Way I Heard it - mike rowe. Small stories usually with a twist or surprise due to mike's way of telling the story. Learn a ton and be entertained and all in 5-10min episodes. Plus it's hard...

    The Way I Heard it - mike rowe. Small stories usually with a twist or surprise due to mike's way of telling the story. Learn a ton and be entertained and all in 5-10min episodes. Plus it's hard not to enjoy mike rowe's voice and story telling cadence.

    I also second anything with scotty ox! "Solo bolo" series with john ralfio and all the best of's with PFT arw amazing.

    2 votes

  11. Comment on Broken Social Scene - Cause = Time (2002) in ~music

    dodger
    Link
    Love this album so much! Never seen the music video, looks like they had fun. Personally could listen to Looks Just Like the Sun on repeat all day: https://www.youtube.com/watch?v=9adQts8ZH9Y

    Love this album so much! Never seen the music video, looks like they had fun.

    Personally could listen to Looks Just Like the Sun on repeat all day: https://www.youtube.com/watch?v=9adQts8ZH9Y

  12. Comment on Reddit is adding native video ads starting next week in ~tech

    dodger
    Link Parent
    Not to disagree or request ~ start doing anything different, but imgur usersub gets real weird real quick. Plus I like how, on reddit, devastating news is padded in cute animal gifs. TildesLite...

    Not to disagree or request ~ start doing anything different, but imgur usersub gets real weird real quick. Plus I like how, on reddit, devastating news is padded in cute animal gifs. TildesLite isn't a terrible idea.

  13. Comment on Reddit is adding native video ads starting next week in ~tech

    dodger
    Link Parent
    I am in utter shock every time I see someone surfing FB and try loading a video... it's awful. I wonder what their tech stack is that leads to such a shit experience. React and GraphQL are great...

    I am in utter shock every time I see someone surfing FB and try loading a video... it's awful. I wonder what their tech stack is that leads to such a shit experience. React and GraphQL are great and turning frontend and api design on their head (tho I prefer vue for complex frontends myself), yet they can't stream a video reliably at all. Maybe they'll fix it eventually but I don't think I have friends that use it much anymore anyways.

    4 votes

  14. Comment on Reddit is adding native video ads starting next week in ~tech

    dodger
    Link Parent
    Once ~ goes open-source, I'm sure there will be spin-offs for different content. Maybe they can even link to each other like sister-sites but keep their domains focused. Keeping ~ true to it's...

    Once ~ goes open-source, I'm sure there will be spin-offs for different content. Maybe they can even link to each other like sister-sites but keep their domains focused. Keeping ~ true to it's mission statement is IMO a great idea and I'm loving it.

    That being said, I love HQG (and sequel and prequel memes). I would be happy to spin up a tildes instance for HQG when the time comes. Until then, patiently waiting for Tildes Open-Source Day.

    2 votes

  15. Comment on Chasing the American dream has got me jaded in ~talk

    dodger
    Link Parent
    Exactly. 40hrs/week (or really more like 60 in my work) plus a couple hours of commute a day... there's no time for life. I don't understand why 5 days of slaving away for 2 days of rest /...

    sneeze facing the wrong way, I look back at the clock and its already 9

    Exactly. 40hrs/week (or really more like 60 in my work) plus a couple hours of commute a day... there's no time for life. I don't understand why 5 days of slaving away for 2 days of rest / self-improvement is tolerated in modern day society. I get why it's shoved down our throats (rich want to get richer), but I'm getting real sick of it.

  16. Comment on Chasing the American dream has got me jaded in ~talk

    dodger
    Link Parent
    This is great. Granted the fisherman would probably have a nicer living situation and money for healthcare etc after the businessman's route, but it's still a great little story for framing hustle...

    This is great. Granted the fisherman would probably have a nicer living situation and money for healthcare etc after the businessman's route, but it's still a great little story for framing hustle vs happiness.

    2 votes

  17. Chasing the American dream has got me jaded

    ~talk Text 280 words
    I live in the US. I used to play music every day. I used to skate almost every day. I used to surf TWO times a day. I used to write songs and poetry. I used to contemplate the cosmos and reality...

    I live in the US.

    I used to play music every day. I used to skate almost every day. I used to surf TWO times a day. I used to write songs and poetry. I used to contemplate the cosmos and reality and come up with fun projects with friends. I used to garden. I used to spend more time with my pets.

    Now I work.

    I work and stress so hard that I wear myself out, drink too much to "help me sleep", still stay up stressing about my job - how I can improve my standing, and stress about needing to do side projects to supplement my income AND stress that I don't do any of the fun things I used to do.

    This is all to be able to afford a living space and ensure that if a disaster (personal or natural) happens I won't be completely screwed.

    Any friends that did have loans are still paying them off. I have been fortunate enough (and diligent enough) to pay off all my loans, but I had to ditch my hopes and dreams.

    Will I ever be able to afford a decent house in a metropolitan area? No. Will I ever do the things that used to make be happy? I don't see how. Will I ever be truly happy? I have no idea.

    Is anyone else in this situation? What are you doing to mitigate? Moving to a more affordable area (leaving friends and family)? Are you learning a new trade to up your financial standing? Are you as bummed out as I am that we have to work so hard just for a mediocre standard of living?

    39 votes

  18. Comment on What have you been playing, and what do you think of it? in ~games

    dodger
    Link Parent
    I can see where you can get tunnel vision here - but the game is designed for you to be able to explore and mess around and investigate new things if you get tired of the grind. Yes, if you plant...

    I can see where you can get tunnel vision here - but the game is designed for you to be able to explore and mess around and investigate new things if you get tired of the grind. Yes, if you plant too much to try and min/max you are gonna have to also water that shit, but you don't HAVE to min max. Each character has an arc and they game does have some depth - you just got to minimize mundane daily routines (unless you kinda like them like some people do) and get out there and talk to people and discover the story.

    Now I will say that multiplayer is game changing. I will never go back to a one player game ever - it's so much more fun to have a crew and all have specialties but then all charge into some objective together. If you need some people to play with (I think the max is more than 4 now) DM me. Can have you join one of my personal online farms to see if you'll click well with the rest of my group. (this is an open invitation to any stardew friendly folk).

    1 vote

  19. Comment on What have you been playing, and what do you think of it? in ~games

    dodger
    Link
    Been mostly playing low key games lately. Not enough time to get into big world games. Multiplayer Stardew Valley is fantastic. Friends and I play almost daily. Into the Brink (same team that did...

    Been mostly playing low key games lately. Not enough time to get into big world games.

    Multiplayer Stardew Valley is fantastic. Friends and I play almost daily.

    Into the Brink (same team that did FTL) is also super addicting and satisfying.

    3 votes

  20. Comment on Ensuring users read documentation in ~tildes

    dodger
    Link Parent
    That and a link to them on the NewThread page - kinda like how github has a link to contributing guidelines when you try and make a PR (if the repo has them).

    That and a link to them on the NewThread page - kinda like how github has a link to contributing guidelines when you try and make a PR (if the repo has them).

    4 votes