14 votes

Career advice: Getting into data engineering or similar

Hey all. I'm looking for a little career advice. For the past 1.5 years I've been working as a full-stack web developer for a proprietary eCom framework. It's LAMP and pretty antiquated and not very interesting. However, I have been able to work on a fun (for me) project parsing sales data with Python and Pandas. Before this job, I worked for myself for a good amount of time doing something creative outside of tech, and before that I was a Senior IT Consultant in a big market.

I'd really like to get out of web development. I don't really enjoy frontend work, and I don't really care about the web or eCom that much. I've been considering moving toward DevOps as I've got a lot of skills in that department, but I've also recently started thinking about Data Engineering. But when I look into jobs in that world, a lot of them require higher degrees in mathematics or science that I don't have. I've got a bevy of tech skills, I'm very into Linux, and I really enjoy working with data. I have some personal experience with Prometheus and Grafana, though nothing professional.

Ultimately, I'm not sure how I should be breaking into Data, or breaking out of web development. Maybe there's a career path I'm neglecting to look at, but would be perfect for. I would be looking for remote only jobs, as I live in a very small market that has very few tech jobs. I'd really love to work in Python. I'd like to learn Rust. I've read on here comments from users saying there are a ton of opportunities in this world, but I'm not quite sure how to frame myself on a resume to make me look like a good candidate. My education isn't in tech, but much of my work experience is. Maybe I'm just overthinking it all. Admittedly, I've been so exhausted from work and life recently that I haven't had the time to really start applying for anything.

Anyhow, any advice would be greatly appreciated.

9 comments

  1. [7]
    Pioneer
    Link
    Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics. Vast majority of our work is Python and SQL (at...

    But when I look into jobs in that world, a lot of them require higher degrees in mathematics or science that I don't have.

    Absolutely not the case mate. I'm Senior Leadership in Data Engineering and I've got 3 GCSE's and the rest spell FUDGE... Don't fret the academics.

    I'd really love to work in Python. I'd like to learn Rust.

    Vast majority of our work is Python and SQL (at least at my level), understanding things like Terraform / CloudFormation are really essential to picking up the Cloud Engineering aspects on what it is we do.

    I've read on here comments from users saying there are a ton of opportunities in this world, but I'm not quite sure how to frame myself on a resume to make me look like a good candidate. My education isn't in tech, but much of my work experience is. Maybe I'm just overthinking it all. Admittedly, I've been so exhausted from work and life recently that I haven't had the time to really start applying for anything.

    Worth taking some down time mate? The problem with tech that goes unspoken is that it can be bloody exhausting. It can be CONSTANT knowledge work that makes your brain burn and we never truly get the downtime that we need.

    Honestly, having a look at a few courses a la Data Camp / Data Quest can help you out. Looking at Udemy for AWS/Azure/GCP (fuck GCP) can be helpful for knowledge. But it's very often about experience that we look for when recruiting.

    You said you've been parsing sales data, talk to that. What have you been doing? How have you been modelling the data for additional use cases beyond what you're doing? How do you ensure reliability of the data? What monitoring have you put in place to ensure that data gets to the place its supposed to, on time and of high quality? How do you ensure lower cloud utilisation and costs (more DevOps) to ensure you aren't breaking the bank? Do you prefer Inmon or Kimball data modelling?

    They're all questions to give you a rough idea of what they hell we talk about. Data Engineering is a wide remit of skills and use cases. Everything from "Streaming TBs of data in IoT" to "SFTP, Model in Kimball and provide to a BI team" and the later skills are often more missing, but more important.

    Hope that helps? Ask away mate. Happy to help.

    Source: Me, Data dude for 15 years.

    10 votes
    1. [4]
      simplify
      Link Parent
      Thanks for all of this! Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be...

      Thanks for all of this!

      Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be much easier to learn when you have an actual use case. I suppose I could do a Udemy crash course just to put it on my resume. But if I’ve never worked professionally in it, the experience just seems a bit superficial.

      What kind of positions should I be looking for with my skill set and level? As I said initially, often times when I search Data Engineering I get jobs with monster requirements. Not just academic, but 15 years in the field like you. People often say job postings are a wishlist, but this isn’t the difference between asking 5 years and having 3. Surely there’s a spot for someone like me who’s not entry level but not senior either.

      Taking time off to search is not in the cards. But it’s also hard to find the time to self-study and do courses right now. I just have to work with what my life is. And that’s keeping this job while working on getting the next one.

      With the current project I head up at work, there’s no real opportunity to take it further with reporting (beyond my current email logs and logging in the database) and modeling. We have other software that does that. This project is just ingesting client sales data, parsing and scrubbing it, and then sending it off to a third party for more action. I also grab response data from that third party, process and log that, to make sure the data we sent them processed correctly on their end. It’s a lot of fun and I wish I could do more with it. This is on my resume and it’s a project I would be leaning on heavily for getting a job in Data.

      4 votes
      1. [3]
        Pioneer
        Link Parent
        You're probably doing Infra as Code in Web Dev to be honest? It's not supremely different! The Experience does feel that way, it's more about how you approach problems. Data Engineering is about...

        Is it required to have some level of knowledge in Terraform or CloudFormation before landing the job? I pick up on things quickly and those seem like systems that would be much easier to learn when you have an actual use case. I suppose I could do a Udemy crash course just to put it on my resume. But if I’ve never worked professionally in it, the experience just seems a bit superficial.

        You're probably doing Infra as Code in Web Dev to be honest? It's not supremely different!

        The Experience does feel that way, it's more about how you approach problems. Data Engineering is about problem solving the hard stuff so others can get away with not having to!

        What kind of positions should I be looking for with my skill set and level? As I said initially, often times when I search Data Engineering I get jobs with monster requirements. Not just academic, but 15 years in the field like you. People often say job postings are a wishlist, but this isn’t the difference between asking 5 years and having 3. Surely there’s a spot for someone like me who’s not entry level but not senior either.

        You get that. People really do not know how to translate requirements from DE teams into what they are recruiting. Honestly, give yourself a crack at some of the Data Engineer roles! Make sure your resume/CV is setup to show the types of behaviours and technology that we want to see (Usually in the JD! Hell JD's can be HUGE fonts of knowledge around what people do and do not want for a DE)

        Taking time off to search is not in the cards. But it’s also hard to find the time to self-study and do courses right now. I just have to work with what my life is. And that’s keeping this job while working on getting the next one.

        Was more time to chill mate. Burnout is rife in technology at the moment. Everything ok?

        With the current project I head up at work, there’s no real opportunity to take it further with reporting (beyond my current email logs and logging in the database) and modeling. We have other software that does that. This project is just ingesting client sales data, parsing and scrubbing it, and then sending it off to a third party for more action. I also grab response data from that third party, process and log that, to make sure the data we sent them processed correctly on their end. It’s a lot of fun and I wish I could do more with it. This is on my resume and it’s a project I would be leaning on heavily for getting a job in Data.

        Sounds like you're clarifying a lot of data and user requirements for said data. You basically need to extrapolate on that with the tools, what you're doing and how you're designing it to be sustainable. If you're comfortable in AWS then get yourself a Cloud Engineer course on Udemy and you'll see the kinds of things we're looking at.

        Again, Data Engineering is a weird thing where you might need DBMS skills (Data Modelling) one day and then Cloud focused skills the next. It's so painfully broad!

        3 votes
        1. [2]
          simplify
          Link Parent
          Just wanted to say thanks again for the info and pep talk. I know I've got a lot of skills that would translate well to Data Engineering. I think it's a place I could really thrive. Everything's...

          Just wanted to say thanks again for the info and pep talk. I know I've got a lot of skills that would translate well to Data Engineering. I think it's a place I could really thrive.

          Was more time to chill mate. Burnout is rife in technology at the moment. Everything ok?

          Everything's great. I just have a demanding work life and a full social life, so... man, it really adds up! Right now I just need to find the time and headspace to actually start applying for jobs. It's tough!

          1 vote
          1. Pioneer
            Link Parent
            Anytime buddy. I hear this. I'm on the exit path at my place and SERIOUSLY aiming to get gardening leave. Need the downtime to just... be.

            Anytime buddy.

            Everything's great. I just have a demanding work life and a full social life, so... man, it really adds up! Right now I just need to find the time and headspace to actually start applying for jobs. It's tough!

            I hear this. I'm on the exit path at my place and SERIOUSLY aiming to get gardening leave. Need the downtime to just... be.

            1 vote
    2. [2]
      Notcoffeetable
      Link Parent
      I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them. Currently I run a "People Analytics" team for a major corp (3 + Me). I have...

      I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.

      Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.

      Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.

      1 vote
      1. Pioneer
        Link Parent
        Linkedin mostly. The Data Community on there is actually pretty good for sticking out for each other. There was a lot of poaching and support during the big tech layoffs this year Where are you...

        I have a follow up question. Where do you find opportunities? I hear about all these "opportunities" but don't see them.

        Linkedin mostly. The Data Community on there is actually pretty good for sticking out for each other.

        There was a lot of poaching and support during the big tech layoffs this year

        Currently I run a "People Analytics" team for a major corp (3 + Me). I have a PhD in Mathematics and 5 years as lead dev previous to that. I would like to take a higher (or similar) paying job with remote/hybrid but I seem to get little traction.

        Where are you based? Some places like the Hybrid option and we have to adapt to that. But sometimes it's the education factor. Are you going for analytics director / head of type roles?

        Edit: I'd be more interested in doing actual work on interesting projects. My days are filled primarily with management and glad handing execs. I want to do math again.

        Then honestly you need to stipulate that or train into Data Science in a big way matey! There's a real problem with folks being pushed into Leadership who just want to do the graft, it's about finding a role you enjoy and really hammering that!

        4 votes
  2. [2]
    FluffyKittens
    Link
    Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go. Most importantly, harden your SQL skills: make...

    Pioneer's answers are excellent, so I'm just going to build on that and try to give you advice to bridge where you are now vs. where you want to go.

    • Most importantly, harden your SQL skills: make sure you know how to write sprocs and views well. Be able to talk about indexing strategies. Consider doing a portfolio project to gain experience with Postgres (if you want startup roles/have good taste) or MS SQL (if you want to go corporate).
    • Webdev is a massive asset: in interviews, pitch the fact that you know how to make internal QA dashboards and web interfaces that can monitor data jobs, and build basic visualizations of your data for end-users.
    • Keep leaning in to Python where you can. Use it as an enhanced version of shell scripting to automate routine tasks and workflows.
    • Rust is not a good use of your time YET. Most experienced DEs will lean into Rust/Go/Java (as way to make performant, cross-platform binaries when needed) but it's nearly always just a nice-to-have for non-senior roles.
    • Many roles will ask you about cloud experience, but the questions will typically come from a screening call with a recruiter. Based on what you've said, your best option here is to make sure you can fully provision and configure a VPS from root using bash. Pitch your shell and sysadmin skills as a form of generalist cloud knowledge; IME, the first-line recruiters eat that particular spin up like catnip, and hiring managers are fine with it as long as you're transparent about what you don't know.

    You have a great base for DE and you're selling yourself well. I don't think you'll have much difficulty landing a role if you polish up your weak points. Good luck!

    4 votes
    1. simplify
      Link Parent
      Thanks so much for the advice! All of what you mentioned is totally actionable and a nice roadmap to help me fill in gaps in my knowledge. Lately I've felt the pangs of imposter syndrome when...

      Thanks so much for the advice! All of what you mentioned is totally actionable and a nice roadmap to help me fill in gaps in my knowledge. Lately I've felt the pangs of imposter syndrome when thinking about applying for jobs, and even sometimes at my current job (though I can easily admit "I don't know, let's figure it out"). But the reality is, all my coworkers are constantly asking me questions and I'm consistently helpful, which is nice. When I think of my strengths, they all point to DevOps and Data Engineering. I like helping, I like orchestrating, I like keeping things in order. Hopefully I can find the right place to let my skills shine.

      2 votes