I've been feeling all of this my org. DevOps is down to one employee onshore and one off. Shipping scores of applications on dozens of platforms. We have to know it all. 40-50 FE and 10 BE. It...
I've been feeling all of this my org. DevOps is down to one employee onshore and one off.
Shipping scores of applications on dozens of platforms. We have to know it all. 40-50 FE and 10 BE.
It isn't sustainable. The disdain and apathy for runtime environments by developers is palpable. The relative safety of deployments has actually led to an expansion of 'not my problem'.
As a developer who is reasonably familiar with server configuration and database design and administration, I can tell you firsthand that the reason more developers lack such knowledge is because...
As a developer who is reasonably familiar with server configuration and database design and administration, I can tell you firsthand that the reason more developers lack such knowledge is because it is in no way encouraged or rewarded. It appears that you are learning this for yourself at present.
I absolutely agree with this. I speak from experience with two of the big tech companies. Management wants developers to do all their own ops, but then turns around and asks why they're spending...
I absolutely agree with this. I speak from experience with two of the big tech companies. Management wants developers to do all their own ops, but then turns around and asks why they're spending time on things that aren't pushing code. There's a total lack of respect for reliability and systems engineering...even when it saves huge amounts of money due to host optimizations and reduced downtime!
It's really frustrating because devs know management has a disdain for operations, so they push off version upgrades and patches until a vulnerability is found (we're still using Java 8). I feel similar whenever front end and back end specialists are pushed to become generalists. We need really good specialists that can handle tough problems!
It sounds like we have a similar background. I’m older as well, and learned most of what I know about programming and related fields purely out of curiosity.
It sounds like we have a similar background. I’m older as well, and learned most of what I know about programming and related fields purely out of curiosity.
Honest question: Can the "build guys" be classified as devops? My earlier company didn't have any official devops position but a lot of these so called "build guys" were just former BE engineers...
Honest question: Can the "build guys" be classified as devops? My earlier company didn't have any official devops position but a lot of these so called "build guys" were just former BE engineers who wanted some change.
To me it's more a philosophy and not an explicit job role. I'm the opposite in that I'm wary of companies who promise I get to work with an explicitly named "DevOps team" while still keeping the...
To me it's more a philosophy and not an explicit job role. I'm the opposite in that I'm wary of companies who promise I get to work with an explicitly named "DevOps team" while still keeping the old "over the wall" way of working.
My anecdotal data says what mostly happens is you have this team that's not really integrated into things and the company isn't getting the core purpose of the movement by having a specific named roles and teams after it.
To me the ideal team that can execute these principles would have wide authority on platform decisions and a blend of skills, not necessarily a "DevOps Engineer" job in a "DevOps team" with a wishlist of half of IT skills under the sun that realistically should be 2-3 people.
I've been feeling some of these issues and in general I'm currently feeling like the ideal team is a small set of people with broad responsibilities would work best. I.e. 3-4 developers with...
I've been feeling some of these issues and in general I'm currently feeling like the ideal team is a small set of people with broad responsibilities would work best. I.e.
3-4 developers with specialties in CI/CD, Instrumentation, IaC, etc.
2 QAs with specialties in IaC, Automation
1 Dedicated Technical writer with real domain knowledge
1 DevOps/SRE with a Specialty in Security, deployment techniques, etc.
Then of course the over arching organization should provide support for documentation style expectations, compliance needs, release documentation/processes, and service requirements.
I left off a product manager and engineering manager because I'm not really sure how important they are. In my experience product managers absolutely suck, lack creativity, and don't lead to innovative products. But the need to interface and translate the needs of 'stakeholders' is important. Similarly, an engineering manager can help organize work, and make controversial decisions. But how that all works idk.
I think a good business analyst is massively underrated. Product managers can be awful when all they do is manage timelines, but the two roles are invaluable when they get the right features...
I think a good business analyst is massively underrated. Product managers can be awful when all they do is manage timelines, but the two roles are invaluable when they get the right features prioritized and connect developers to clients. Developers should develop, and having good project managers and business analysts should let them focus on that. This also allows managers to work with multiple or larger teams since they no longer have to handle all the project manager responsibilities.
It’s a good article but it’s written in the abstract (without referencing specific companies) and it doesn’t talk that much about how procedures differ between companies. I wonder much is about...
It’s a good article but it’s written in the abstract (without referencing specific companies) and it doesn’t talk that much about how procedures differ between companies. I wonder much is about problems specific to where they worked?
I think they are right, although in a naive way. DevOps in theory really can't work in the same way that the mention of agile makes some people go violent. Expecting everyone in teams to be...
Expecting everyone in teams to be T-shaped to the degree that the the roof of the T is comically wide and thick simply isn't all that realistic. So naturally there are very few, if any, companies where DevOps is actually practiced as you still have people that specialize in the different disciplines. Most companies still end up with infrastructure teams managing away the things that are too specialized for regular DevOps teams. A lot of companies still sneak back in a QA type of role into the whole process. Simply because their software is critical enough they can't afford to test in production. For a similar reason a lot of companies end up adding back a bunch of checks and balances before something can go to production as well.
But it is still framed in that decentralized hellscape that is called DevOps so while secretly a lot of the underlying actions have remained the same now nobody knows who is responsible anymore.
There are a few companies where this isn't the case and where things run rather well. But those are the companies, in my experience, where they never fully tried to shift to "DevOps" but instead kept in a lot of the old checks and balances with just a slight shift of responsibilities towards the teams.
Yes, it seems like it's a question of how flexible you want everyone to be in sticking to their roles. You probably still want people to have roles, because it's useful to give different bundles...
Yes, it seems like it's a question of how flexible you want everyone to be in sticking to their roles. You probably still want people to have roles, because it's useful to give different bundles of tasks a name, and it takes time to learn to do them well.
Sometimes roles get too rigid, though, which can be frustrating too, because you could do it, but you're not allowed to. (Based on anecdotes, union jobs are supposedly like this sometimes.)
Hmm, my team still does devops and it's been fine. It's not perfectly balanced, we do have some who prefer the ops work and some who prefer dev but at the end of the day, everyone does at least a...
Hmm, my team still does devops and it's been fine. It's not perfectly balanced, we do have some who prefer the ops work and some who prefer dev but at the end of the day, everyone does at least a little of both.
I actually enjoy doing both. I like to be able to fiddle in the whole stack to get something working, tested or fixed. But I also came from an ops background before doing more dev.
I'd hate if I could never lean into the silly frontend framework of the week or the 15 year old Factory Impl. And, honestly, I wouldn't be able to generate good IaC let alone Helm charts if I...
I'd hate if I could never lean into the silly frontend framework of the week or the 15 year old Factory Impl. And, honestly, I wouldn't be able to generate good IaC let alone Helm charts if I didn't spend a decent amount of time fiddling across the dev stack.
Guess I'm in a lucky spot in that my team kind of evolved into it naturally in some way without external consultants preaching the Gartner bible and beating buzzwords into our head. In our case...
Guess I'm in a lucky spot in that my team kind of evolved into it naturally in some way without external consultants preaching the Gartner bible and beating buzzwords into our head. In our case we're a small team with a wide variety of skills and do a little bit of everything and support some variety of platforms.
I'd hazard a guess that one reason is we're not forcibly conformed into "DevOps Engineers" put into a "DevOps Team" that is essentially still a separate operating entity from the rest of the company. We have a SaaS product and wanted a way to rapidly prototype new services/offerings by independent teams (internal or external), so we had to adapt to that requirement. Dev decided microfrontends, Ops decided Kubernetes, we needed to talk to each other frequently to get each other to do the side of the work and we learned a bit of how the other side works. Eventually that changed to adjusting access control so dev can be trusted to self-service privileged tasks on the cluster and ops can submit patches to existing apps if we find a better way to do things.
But even right now I can't even tell you the DORA metrics off the top of my head, and we're likely ignoring half the metrics in our dashboards knowing it will bite us someday. But the overarching concept just made sense from existing requirements and we adapted practices piecemeal instead of being shoved in on advice of an external consultancy because that's how you "should" do things today. Things like observability, secrets injection or a service mesh became real scaling problems that had to be tackled naturally because we wanted to stick to things like a "no prod access" rule, or having the web inter-service traffic talk to each other over TLS.
We're not perfect. We're not 10x Google SRE types, we generally average 1-2 automated production deployments per week from these apps we run. We have crusty apps that need manual renewal of TLS certs until cert-manager can be setup to take over. But everything is being built at a nice pace (not big tech pace) and I don't feel for one moment that I've been forced to rushing something out to the detriment of a glaring security issue.
I've been feeling all of this my org. DevOps is down to one employee onshore and one off.
Shipping scores of applications on dozens of platforms. We have to know it all. 40-50 FE and 10 BE.
It isn't sustainable. The disdain and apathy for runtime environments by developers is palpable. The relative safety of deployments has actually led to an expansion of 'not my problem'.
As a developer who is reasonably familiar with server configuration and database design and administration, I can tell you firsthand that the reason more developers lack such knowledge is because it is in no way encouraged or rewarded. It appears that you are learning this for yourself at present.
I absolutely agree with this. I speak from experience with two of the big tech companies. Management wants developers to do all their own ops, but then turns around and asks why they're spending time on things that aren't pushing code. There's a total lack of respect for reliability and systems engineering...even when it saves huge amounts of money due to host optimizations and reduced downtime!
It's really frustrating because devs know management has a disdain for operations, so they push off version upgrades and patches until a vulnerability is found (we're still using Java 8). I feel similar whenever front end and back end specialists are pushed to become generalists. We need really good specialists that can handle tough problems!
It sounds like we have a similar background. I’m older as well, and learned most of what I know about programming and related fields purely out of curiosity.
Honest question: Can the "build guys" be classified as devops? My earlier company didn't have any official devops position but a lot of these so called "build guys" were just former BE engineers who wanted some change.
To me it's more a philosophy and not an explicit job role. I'm the opposite in that I'm wary of companies who promise I get to work with an explicitly named "DevOps team" while still keeping the old "over the wall" way of working.
My anecdotal data says what mostly happens is you have this team that's not really integrated into things and the company isn't getting the core purpose of the movement by having a specific named roles and teams after it.
To me the ideal team that can execute these principles would have wide authority on platform decisions and a blend of skills, not necessarily a "DevOps Engineer" job in a "DevOps team" with a wishlist of half of IT skills under the sun that realistically should be 2-3 people.
I've been feeling some of these issues and in general I'm currently feeling like the ideal team is a small set of people with broad responsibilities would work best. I.e.
Then of course the over arching organization should provide support for documentation style expectations, compliance needs, release documentation/processes, and service requirements.
I left off a product manager and engineering manager because I'm not really sure how important they are. In my experience product managers absolutely suck, lack creativity, and don't lead to innovative products. But the need to interface and translate the needs of 'stakeholders' is important. Similarly, an engineering manager can help organize work, and make controversial decisions. But how that all works idk.
I think a good business analyst is massively underrated. Product managers can be awful when all they do is manage timelines, but the two roles are invaluable when they get the right features prioritized and connect developers to clients. Developers should develop, and having good project managers and business analysts should let them focus on that. This also allows managers to work with multiple or larger teams since they no longer have to handle all the project manager responsibilities.
It’s a good article but it’s written in the abstract (without referencing specific companies) and it doesn’t talk that much about how procedures differ between companies. I wonder much is about problems specific to where they worked?
I think they are right, although in a naive way. DevOps in theory really can't work in the same way that the mention of agile makes some people go violent.
Expecting everyone in teams to be T-shaped to the degree that the the roof of the T is comically wide and thick simply isn't all that realistic. So naturally there are very few, if any, companies where DevOps is actually practiced as you still have people that specialize in the different disciplines. Most companies still end up with infrastructure teams managing away the things that are too specialized for regular DevOps teams. A lot of companies still sneak back in a QA type of role into the whole process. Simply because their software is critical enough they can't afford to test in production. For a similar reason a lot of companies end up adding back a bunch of checks and balances before something can go to production as well.
But it is still framed in that decentralized hellscape that is called DevOps so while secretly a lot of the underlying actions have remained the same now nobody knows who is responsible anymore.
There are a few companies where this isn't the case and where things run rather well. But those are the companies, in my experience, where they never fully tried to shift to "DevOps" but instead kept in a lot of the old checks and balances with just a slight shift of responsibilities towards the teams.
Yes, it seems like it's a question of how flexible you want everyone to be in sticking to their roles. You probably still want people to have roles, because it's useful to give different bundles of tasks a name, and it takes time to learn to do them well.
Sometimes roles get too rigid, though, which can be frustrating too, because you could do it, but you're not allowed to. (Based on anecdotes, union jobs are supposedly like this sometimes.)
Hmm, my team still does devops and it's been fine. It's not perfectly balanced, we do have some who prefer the ops work and some who prefer dev but at the end of the day, everyone does at least a little of both.
I actually enjoy doing both. I like to be able to fiddle in the whole stack to get something working, tested or fixed. But I also came from an ops background before doing more dev.
I'd hate if I could never lean into the silly frontend framework of the week or the 15 year old Factory Impl. And, honestly, I wouldn't be able to generate good IaC let alone Helm charts if I didn't spend a decent amount of time fiddling across the dev stack.
Agree.
Guess I'm in a lucky spot in that my team kind of evolved into it naturally in some way without external consultants preaching the Gartner bible and beating buzzwords into our head. In our case we're a small team with a wide variety of skills and do a little bit of everything and support some variety of platforms.
I'd hazard a guess that one reason is we're not forcibly conformed into "DevOps Engineers" put into a "DevOps Team" that is essentially still a separate operating entity from the rest of the company. We have a SaaS product and wanted a way to rapidly prototype new services/offerings by independent teams (internal or external), so we had to adapt to that requirement. Dev decided microfrontends, Ops decided Kubernetes, we needed to talk to each other frequently to get each other to do the side of the work and we learned a bit of how the other side works. Eventually that changed to adjusting access control so dev can be trusted to self-service privileged tasks on the cluster and ops can submit patches to existing apps if we find a better way to do things.
But even right now I can't even tell you the DORA metrics off the top of my head, and we're likely ignoring half the metrics in our dashboards knowing it will bite us someday. But the overarching concept just made sense from existing requirements and we adapted practices piecemeal instead of being shoved in on advice of an external consultancy because that's how you "should" do things today. Things like observability, secrets injection or a service mesh became real scaling problems that had to be tackled naturally because we wanted to stick to things like a "no prod access" rule, or having the web inter-service traffic talk to each other over TLS.
We're not perfect. We're not 10x Google SRE types, we generally average 1-2 automated production deployments per week from these apps we run. We have crusty apps that need manual renewal of TLS certs until
cert-managercan be setup to take over. But everything is being built at a nice pace (not big tech pace) and I don't feel for one moment that I've been forced to rushing something out to the detriment of a glaring security issue.