Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics. Vast majority of our work is Python and SQL (at...
But when I look into jobs in that world, a lot of them require higher degrees in mathematics or science that I don't have.
Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics.
I'd really love to work in Python. I'd like to learn Rust.
Vast majority of our work is Python and SQL (at least at my level), understanding things like Terraform / CloudFormation are really essential to picking up the Cloud Engineering aspects on what it is we do.
I've read on here comments from users saying there are a ton of opportunities in this world, but I'm not quite sure how to frame myself on a resume to make me look like a good candidate. My education isn't in tech, but much of my work experience is. Maybe I'm just overthinking it all. Admittedly, I've been so exhausted from work and life recently that I haven't had the time to really start applying for anything.
Worth taking some down time mate? The problem with tech that goes unspoken is that it can be bloody exhausting. It can be CONSTANT knowledge work that makes your brain burn and we never truly get the downtime that we need.
Honestly, having a look at a few courses a la Data Camp / Data Quest can help you out. Looking at Udemy for AWS/Azure/GCP (fuck GCP) can be helpful for knowledge. But it's very often about experience that we look for when recruiting.
You said you've been parsing sales data, talk to that. What have you been doing? How have you been modelling the data for additional use cases beyond what you're doing? How do you ensure reliability of the data? What monitoring have you put in place to ensure that data gets to the place its supposed to, on time and of high quality? How do you ensure lower cloud utilisation and costs (more DevOps) to ensure you aren't breaking the bank? Do you prefer Inmon or Kimball data modelling?
They're all questions to give you a rough idea of what they hell we talk about. Data Engineering is a wide remit of skills and use cases. Everything from "Streaming TBs of data in IoT" to "SFTP, Model in Kimball and provide to a BI team" and the later skills are often more missing, but more important.
You're probably doing Infra as Code in Web Dev to be honest? It's not supremely different! The Experience does feel that way, it's more about how you approach problems. Data Engineering is about...
Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be much easier to learn when you have an actual use case. I suppose I could do a Udemy crash course just to put it on my resume. But if I’ve never worked professionally in it, the experience just seems a bit superficial.
You're probably doing Infra as Code in Web Dev to be honest? It's not supremely different!
The Experience does feel that way, it's more about how you approach problems. Data Engineering is about problem solving the hard stuff so others can get away with not having to!
What kind of positions should I be looking for with my skill set and level? As I said initially, often times when I search Data Engineering I get jobs with monster requirements. Not just academic, but 15 years in the field like you. People often say job postings are a wishlist, but this isn’t the difference between asking 5 years and having 3. Surely there’s a spot for someone like me who’s not entry level but not senior either.
You get that. People really do not know how to translate requirements from DE teams into what they are recruiting. Honestly, give yourself a crack at some of the Data Engineer roles! Make sure your resume/CV is setup to show the types of behaviours and technology that we want to see (Usually in the JD! Hell JD's can be HUGE fonts of knowledge around what people do and do not want for a DE)
Taking time off to search is not in the cards. But it’s also hard to find the time to self-study and do courses right now. I just have to work with what my life is. And that’s keeping this job while working on getting the next one.
Was more time to chill mate. Burnout is rife in technology at the moment. Everything ok?
With the current project I head up at work, there’s no real opportunity to take it further with reporting (beyond my current email logs and logging in the database) and modeling. We have other software that does that. This project is just ingesting client sales data, parsing and scrubbing it, and then sending it off to a third party for more action. I also grab response data from that third party, process and log that, to make sure the data we sent them processed correctly on their end. It’s a lot of fun and I wish I could do more with it. This is on my resume and it’s a project I would be leaning on heavily for getting a job in Data.
Sounds like you're clarifying a lot of data and user requirements for said data. You basically need to extrapolate on that with the tools, what you're doing and how you're designing it to be sustainable. If you're comfortable in AWS then get yourself a Cloud Engineer course on Udemy and you'll see the kinds of things we're looking at.
Again, Data Engineering is a weird thing where you might need DBMS skills (Data Modelling) one day and then Cloud focused skills the next. It's so painfully broad!
Anytime buddy. I hear this. I'm on the exit path at my place and SERIOUSLY aiming to get gardening leave. Need the downtime to just... be.
Anytime buddy.
Everything's great. I just have a demanding work life and a full social life, so... man, it really adds up! Right now I just need to find the time and headspace to actually start applying for jobs. It's tough!
I hear this. I'm on the exit path at my place and SERIOUSLY aiming to get gardening leave. Need the downtime to just... be.
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them. Currently I run a "People Analytics" team for a major corp (3 + Me). I have...
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.
Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.
Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.
Linkedin mostly. The Data Community on there is actually pretty good for sticking out for each other. There was a lot of poaching and support during the big tech layoffs this year Where are you...
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.
Linkedin mostly. The Data Community on there is actually pretty good for sticking out for each other.
There was a lot of poaching and support during the big tech layoffs this year
Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.
Where are you based? Some places like the Hybrid option and we have to adapt to that. But sometimes it's the education factor. Are you going for analytics director / head of type roles?
Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.
Then honestly you need to stipulate that or train into Data Science in a big way matey! There's a real problem with folks being pushed into Leadership who just want to do the graft, it's about finding a role you enjoy and really hammering that!
Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go. Most importantly, harden your SQL skills: make...
Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go.
Most importantly, harden your SQL skills: make sure you know how to write sprocs and views well. Be able to talk about indexing strategies. Consider doing a portfolio project to gain experience with Postgres (if you want startup roles/have good taste) or MS SQL (if you want to go corporate).
Webdev is a massive asset: in interviews, pitch the fact that you know how to make internal QA dashboards and web interfaces that can monitor data jobs, and build basic visualizations of your data for end-users.
Keep leaning in to Python where you can. Use it as an enhanced version of shell scripting to automate routine tasks and workflows.
Rust is not a good use of your time YET. Most experienced DEs will lean into Rust/Go/Java (as way to make performant, cross-platform binaries when needed) but it's nearly always just a nice-to-have for non-senior roles.
Many roles will ask you about cloud experience, but the questions will typically come from a screening call with a recruiter. Based on what you've said, your best option here is to make sure you can fully provision and configure a VPS from root using bash. Pitch your shell and sysadmin skills as a form of generalist cloud knowledge; IME, the first-line recruiters eat that particular spin up like catnip, and hiring managers are fine with it as long as you're transparent about what you don't know.
You have a great base for DE and you're selling yourself well. I don't think you'll have much difficulty landing a role if you polish up your weak points. Good luck!
Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics.
Vast majority of our work is Python and SQL (at least at my level), understanding things like Terraform / CloudFormation are really essential to picking up the Cloud Engineering aspects on what it is we do.
Worth taking some down time mate? The problem with tech that goes unspoken is that it can be bloody exhausting. It can be CONSTANT knowledge work that makes your brain burn and we never truly get the downtime that we need.
Honestly, having a look at a few courses a la Data Camp / Data Quest can help you out. Looking at Udemy for AWS/Azure/GCP (fuck GCP) can be helpful for knowledge. But it's very often about experience that we look for when recruiting.
You said you've been parsing sales data, talk to that. What have you been doing? How have you been modelling the data for additional use cases beyond what you're doing? How do you ensure reliability of the data? What monitoring have you put in place to ensure that data gets to the place its supposed to, on time and of high quality? How do you ensure lower cloud utilisation and costs (more DevOps) to ensure you aren't breaking the bank? Do you prefer Inmon or Kimball data modelling?
They're all questions to give you a rough idea of what they hell we talk about. Data Engineering is a wide remit of skills and use cases. Everything from "Streaming TBs of data in IoT" to "SFTP, Model in Kimball and provide to a BI team" and the later skills are often more missing, but more important.
Hope that helps? Ask away mate. Happy to help.
Source: Me, Data dude for 15 years.
You're probably doing Infra as Code in Web Dev to be honest? It's not supremely different!
The Experience does feel that way, it's more about how you approach problems. Data Engineering is about problem solving the hard stuff so others can get away with not having to!
You get that. People really do not know how to translate requirements from DE teams into what they are recruiting. Honestly, give yourself a crack at some of the Data Engineer roles! Make sure your resume/CV is setup to show the types of behaviours and technology that we want to see (Usually in the JD! Hell JD's can be HUGE fonts of knowledge around what people do and do not want for a DE)
Was more time to chill mate. Burnout is rife in technology at the moment. Everything ok?
Sounds like you're clarifying a lot of data and user requirements for said data. You basically need to extrapolate on that with the tools, what you're doing and how you're designing it to be sustainable. If you're comfortable in AWS then get yourself a Cloud Engineer course on Udemy and you'll see the kinds of things we're looking at.
Again, Data Engineering is a weird thing where you might need DBMS skills (Data Modelling) one day and then Cloud focused skills the next. It's so painfully broad!
Anytime buddy.
I hear this. I'm on the exit path at my place and SERIOUSLY aiming to get gardening leave. Need the downtime to just... be.
I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.
Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.
Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.
Linkedin mostly. The Data Community on there is actually pretty good for sticking out for each other.
There was a lot of poaching and support during the big tech layoffs this year
Where are you based? Some places like the Hybrid option and we have to adapt to that. But sometimes it's the education factor. Are you going for analytics director / head of type roles?
Then honestly you need to stipulate that or train into Data Science in a big way matey! There's a real problem with folks being pushed into Leadership who just want to do the graft, it's about finding a role you enjoy and really hammering that!
Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go.
You have a great base for DE and you're selling yourself well. I don't think you'll have much difficulty landing a role if you polish up your weak points. Good luck!