I'm not buying it. For something like Make or a CI/CD YAML file, I want my team to be copy pasting files and changing the config defines at the top. I don't want 37 different attempts to figure...
I'm not buying it.
For something like Make or a CI/CD YAML file, I want my team to be copy pasting files and changing the config defines at the top. I don't want 37 different attempts to figure out how to make a functional deploy.
Tools that enable this pattern often discourage broad learning
There isn't a ton of "broad learning" available for this kind of thing. Look up what the build steps are, provide the required parameters, hope it works.
Build pipelines are IKEA furniture. Nobody is missing out on a learning experience by grabbing the instruction booklet and throwing it together using the steps defined by someone else. It is the final step in moving the thing that you care about into a useful position by using tools that you probably don't care about.
I don't think there is anything wrong with copying over bits and pieces that work as long as you understand what they do and how they work. Which is I think what you are more or less arguing for....
I don't think there is anything wrong with copying over bits and pieces that work as long as you understand what they do and how they work. Which is I think what you are more or less arguing for. But the author, to me, anyway seems to talk about blindly copying over things like this verbatim and praying that it works for your new situation.
I wholeheartedly agree. As a software engineer who focuses more on Java dev, I want my CI/CD pipelines to just work and let me focus on writing business logic code. Will I ever be an expert at...
I wholeheartedly agree. As a software engineer who focuses more on Java dev, I want my CI/CD pipelines to just work and let me focus on writing business logic code. Will I ever be an expert at writing Gitlab CI yamls? No, but it's not what I'm primarily paid for.
Maybe if my job involved managing the pipelines across multiple teams, I'd put in the effort to learn this stuff from scratch and truly understand the deep internals. But reinventing the wheel for the sake of it just doesn't seem like a beneficial use of my time. I'll pick up the understanding over time as I need to make changes to the configuration, but for creating a new repo for a new microservice, I'm going to use what's in place already.
I mean as a java dev you likely work with gradle or maven. Unless you are in a very complex environment most CI/CD pipelines aren't that more complex. I mean they are effectively syntaxsugar on...
I mean as a java dev you likely work with gradle or maven. Unless you are in a very complex environment most CI/CD pipelines aren't that more complex. I mean they are effectively syntaxsugar on top of VMs/Containers to do certain steps in a certain order.
If you have any amount of experience with command lines and scripts then pipelines aren't really all that daunting imho.
Aye, using Maven as the build system, which is also abstracted away with the use of a parent pom in all our microservices. There is a certain complexity in our setup with various starters...
Aye, using Maven as the build system, which is also abstracted away with the use of a parent pom in all our microservices. There is a certain complexity in our setup with various starters available (like integration with databases and/or Kafka) but a well structured codebase will make things optional.
If your scripts always spin up a postgres Docker to run migration tests, then yeah, it's wasteful to do it on a DB-less microservice pipeline. Most places I've worked at has reached this level to have options available. The only one that didn't was a startup where the idea of templates or parent poms were rejected by the CTO for some reason and everything was copy-pasted to each microservice. Made upgrades so painful and slowed down a lot of development over time.
I wish. Code that defines a pipeline or any other infrastructure is still code and should be subject to the same engineering practices we use application development, even if the pipeline...
Build pipelines are IKEA furniture
I wish.
Code that defines a pipeline or any other infrastructure is still code and should be subject to the same engineering practices we use application development, even if the pipeline applications don't let you. I have 50+ repositories at work. Every time we need to change something in the pipelines configuration it is tedious and very error prone because there is a lot manual work.
Ideally you want one (versioned) script that is retrieved and executed by all the pipelines / dev team, but that is not straightforward to do, at least in the setups I have worked with.
The way we do it at my workplace is that the SRE team (the ones in charge of the Gitlab/kubernetes infrastructure) provides a series of templates for others to use that have a whole lot of...
The way we do it at my workplace is that the SRE team (the ones in charge of the Gitlab/kubernetes infrastructure) provides a series of templates for others to use that have a whole lot of functionality baked in. Here's a simplified psuedoexample of what a docker build looks like for everyone else:
And with that, the image is built, uploaded to our registry, a security scan is performed, and the build fails if there is a threshold of intolerable security (IE undocumented patchable fixes).
Yes, templates make a lot of sense here, except that once they're used everywhere, they can be difficult to change. If you need to make a change and also want your builds to remain consistent, you...
Yes, templates make a lot of sense here, except that once they're used everywhere, they can be difficult to change. If you need to make a change and also want your builds to remain consistent, you need to test all the builds using the updated template. Or alternately, don't change the old, working builds, and live with the inconsistent versions.
Which one of these makes sense depends on how centralized the organization is and how urgent the change is.
There's something sort of wrong with systems where the entire configuration has to happen in a single file. These tools are complex, so their configuration is necessarily complex as well. They...
There's something sort of wrong with systems where the entire configuration has to happen in a single file. These tools are complex, so their configuration is necessarily complex as well. They should offer plugins, templating, and/or other methods of abstraction.
If your entire org/team needs nearly identical configurations of something, then it'd be ideal if the parts that are the same are abstracted/shared (templates, etc).
I can think of some CI systems that work this way, and I'd love to see more of these highly configurable tools do it too.
I haven't experimented with it much, but Justfiles allow importing, as one example.
One does not follow the other, also most pipelines are already an abstraction over more complex other parts. That is the entire idea of the yaml syntax used by github actions, azure pipelines,...
There's something sort of wrong with systems where the entire configuration has to happen in a single file. These tools are complex, so their configuration is necessarily complex as well.
One does not follow the other, also most pipelines are already an abstraction over more complex other parts. That is the entire idea of the yaml syntax used by github actions, azure pipelines, gitlab ci/cd, it is effectively nothing more than "run this in this order".
They should offer plugins, templating, and/or other methods of abstraction.
Most of them do? In fact I am hard pressed to think of a pipeline solution where there isn't at least some form of templating. Github and Azure pipelines also offer specific plugins.
Most companies I have worked at also use this to offer standardized templates for certain deployment actions and such.
I think this a decent writeup for the core message: Spend time thinking about how/why you are reusing code. Reusing code is great, unless you don't know exactly why you're doing so.
I think this a decent writeup for the core message: Spend time thinking about how/why you are reusing code.
Reusing code is great, unless you don't know exactly why you're doing so.
In my experience, there are broadly three types of technical companies: A small company A big company A dysfunctional big company In a small company, there are only a few teams, or maybe even a...
In my experience, there are broadly three types of technical companies:
A small company
A big company
A dysfunctional big company
In a small company, there are only a few teams, or maybe even a few employees. So every technical person has to wear many hats. This means that the developers need to figure out the CI and the security and everything. This is neither good or bad, it is just something that happens. And maybe there is a lot of copy/pasting of configuration and build. Maybe that's also ok.
In a big company, there are separate teams for IT/CI, security, and development. The development team is supposed to concentrate on implementing requirements, and the other teams are supposed to figure out standards for the overall architecture of aws, kubernetes, or whatever, plus they do security scans and realtime monitoring and all that.
In a fully functioning big company, the CI and security teams would either fix the problems or provide very good documentation on how to fix them.
But many work in dysfunctional big companies where the CI and security teams exist not to really fix things, but just identify issues and then issue emails and warnings to the developers that they need to fix this stuff RIGHT NOW but you can't just look up on stackoverflow because we have a bunch of undocumented internal stuff and by the way fix it RIGHT NOW or we are blocking your app. So yeah we are going to copy/paste the shit out of the config just to get them out of our hair.
I'm not buying it.
For something like Make or a CI/CD YAML file, I want my team to be copy pasting files and changing the config defines at the top. I don't want 37 different attempts to figure out how to make a functional deploy.
There isn't a ton of "broad learning" available for this kind of thing. Look up what the build steps are, provide the required parameters, hope it works.
Build pipelines are IKEA furniture. Nobody is missing out on a learning experience by grabbing the instruction booklet and throwing it together using the steps defined by someone else. It is the final step in moving the thing that you care about into a useful position by using tools that you probably don't care about.
I don't think there is anything wrong with copying over bits and pieces that work as long as you understand what they do and how they work. Which is I think what you are more or less arguing for. But the author, to me, anyway seems to talk about blindly copying over things like this verbatim and praying that it works for your new situation.
Which I think is a valid concern.
I wholeheartedly agree. As a software engineer who focuses more on Java dev, I want my CI/CD pipelines to just work and let me focus on writing business logic code. Will I ever be an expert at writing Gitlab CI yamls? No, but it's not what I'm primarily paid for.
Maybe if my job involved managing the pipelines across multiple teams, I'd put in the effort to learn this stuff from scratch and truly understand the deep internals. But reinventing the wheel for the sake of it just doesn't seem like a beneficial use of my time. I'll pick up the understanding over time as I need to make changes to the configuration, but for creating a new repo for a new microservice, I'm going to use what's in place already.
I mean as a java dev you likely work with gradle or maven. Unless you are in a very complex environment most CI/CD pipelines aren't that more complex. I mean they are effectively syntaxsugar on top of VMs/Containers to do certain steps in a certain order.
If you have any amount of experience with command lines and scripts then pipelines aren't really all that daunting imho.
Aye, using Maven as the build system, which is also abstracted away with the use of a parent pom in all our microservices. There is a certain complexity in our setup with various starters available (like integration with databases and/or Kafka) but a well structured codebase will make things optional.
If your scripts always spin up a postgres Docker to run migration tests, then yeah, it's wasteful to do it on a DB-less microservice pipeline. Most places I've worked at has reached this level to have options available. The only one that didn't was a startup where the idea of templates or parent poms were rejected by the CTO for some reason and everything was copy-pasted to each microservice. Made upgrades so painful and slowed down a lot of development over time.
I wish.
Code that defines a pipeline or any other infrastructure is still code and should be subject to the same engineering practices we use application development, even if the pipeline applications don't let you. I have 50+ repositories at work. Every time we need to change something in the pipelines configuration it is tedious and very error prone because there is a lot manual work.
Ideally you want one (versioned) script that is retrieved and executed by all the pipelines / dev team, but that is not straightforward to do, at least in the setups I have worked with.
The way we do it at my workplace is that the SRE team (the ones in charge of the Gitlab/kubernetes infrastructure) provides a series of templates for others to use that have a whole lot of functionality baked in. Here's a simplified psuedoexample of what a docker build looks like for everyone else:
And with that, the image is built, uploaded to our registry, a security scan is performed, and the build fails if there is a threshold of intolerable security (IE undocumented patchable fixes).
And this then gets used by dozens of teams.
Yes, templates make a lot of sense here, except that once they're used everywhere, they can be difficult to change. If you need to make a change and also want your builds to remain consistent, you need to test all the builds using the updated template. Or alternately, don't change the old, working builds, and live with the inconsistent versions.
Which one of these makes sense depends on how centralized the organization is and how urgent the change is.
There's something sort of wrong with systems where the entire configuration has to happen in a single file. These tools are complex, so their configuration is necessarily complex as well. They should offer plugins, templating, and/or other methods of abstraction.
If your entire org/team needs nearly identical configurations of something, then it'd be ideal if the parts that are the same are abstracted/shared (templates, etc).
I can think of some CI systems that work this way, and I'd love to see more of these highly configurable tools do it too.
I haven't experimented with it much, but Justfiles allow importing, as one example.
One does not follow the other, also most pipelines are already an abstraction over more complex other parts. That is the entire idea of the yaml syntax used by github actions, azure pipelines, gitlab ci/cd, it is effectively nothing more than "run this in this order".
Most of them do? In fact I am hard pressed to think of a pipeline solution where there isn't at least some form of templating. Github and Azure pipelines also offer specific plugins.
Most companies I have worked at also use this to offer standardized templates for certain deployment actions and such.
I think this a decent writeup for the core message: Spend time thinking about how/why you are reusing code.
Reusing code is great, unless you don't know exactly why you're doing so.
In my experience, there are broadly three types of technical companies:
In a small company, there are only a few teams, or maybe even a few employees. So every technical person has to wear many hats. This means that the developers need to figure out the CI and the security and everything. This is neither good or bad, it is just something that happens. And maybe there is a lot of copy/pasting of configuration and build. Maybe that's also ok.
In a big company, there are separate teams for IT/CI, security, and development. The development team is supposed to concentrate on implementing requirements, and the other teams are supposed to figure out standards for the overall architecture of aws, kubernetes, or whatever, plus they do security scans and realtime monitoring and all that.
In a fully functioning big company, the CI and security teams would either fix the problems or provide very good documentation on how to fix them.
But many work in dysfunctional big companies where the CI and security teams exist not to really fix things, but just identify issues and then issue emails and warnings to the developers that they need to fix this stuff RIGHT NOW but you can't just look up on stackoverflow because we have a bunch of undocumented internal stuff and by the way fix it RIGHT NOW or we are blocking your app. So yeah we are going to copy/paste the shit out of the config just to get them out of our hair.