10 votes

Hey ~comp, how does on-call work?

I'm starting a new job soon and I believe I'll eventually be in the on-call rotation. I've never done on-call in a formal sense. Past jobs I was always pretty junior and I tended to work on mobile apps that don't have all that many emergency problems. The closest I've ever come to a real on-call was helping some charities run their website.

I don't really perform well under pressure and I'm pretty terrified. I've heard that a lot of places have automatic rollbacks and other methods of automatically dealing with things, so pages only really happen when something is really broken.

How do people deal with issues like that? What happens if I'm not able to figure something out within a good time limit?

5 comments

  1. fkr637
    Link
    I think the specifics will probably depend on how your company chooses to handle things and what kind of disaster recovery setup they have, etc. As far as your concerns about performance under...

    I think the specifics will probably depend on how your company chooses to handle things and what kind of disaster recovery setup they have, etc. As far as your concerns about performance under pressure, I imagine your co-workers would offer help if you explain your fears to them. I know it can be difficult to express vulnerability in that way but see if you can find the most helpful people you work with. In my experience, the most helpful people are some of the smartest and most reliable people too.

    4 votes
  2. treed
    Link
    On-call varies a lot. At my last company where I did on-call, we had a few policies that helped bring new people into the rotation: A ramp up period where we tried to make sure the new employee...

    On-call varies a lot. At my last company where I did on-call, we had a few policies that helped bring new people into the rotation:

    • A ramp up period where we tried to make sure the new employee was familiarized with all of our systems before they were eligible for the rotation.
    • In adding someone to a rotation, they had two (sometimes more) trial oncalls. The first was shadowing someone, where you'd both receive the same pages, and the person you were shadowing would actually be doing most of the work, but explaining to the shadower as they went along. Then the new employee is shadowed by someone else, just as a second pair of eyes. Once the new person had been through the shadowing on both sides and felt reasonably confident, they could be added to solo oncall shifts. Of course, they could always escalate if they felt unsure about taking actions that could be dangerous, and the amount of these escalations went down over time.
    • Every alert was required to have associated documentation. Official policy was that an alert without documentation was fair game to be deleted. Of course in many cases this documentation was fairly minimal/general as you can't really document everything that can go wrong up front, but it would usually at least point you in the right direction.

    Any/all of those can help bring people in, and I'd be a little surprised if an org with any degree of maturity weren't doing some of them.

    2 votes
  3. zellyn
    Link
    Make sure your team (a) has an oncall runbook that they keep up to date, (b) works hard to constantly reduce the burden of being oncall by not tolerating un-actionable pages for long without...

    Make sure your team (a) has an oncall runbook that they keep up to date, (b) works hard to constantly reduce the burden of being oncall by not tolerating un-actionable pages for long without fixing the cause, and (c) holds some kind of blameless post-mortem when production outages do occur.

    There are quite healthy oncall rotations, and completely unhealthy oncall rotations that will burn you out. Run away from the second kind before you burn out :-)

    2 votes
  4. Th3irdEye
    Link
    In the case of my company I work for, about 3 months after I started they just threw me into rotation. It's not so bad in my experience. It probably varies wildly company to company but for me it...

    In the case of my company I work for, about 3 months after I started they just threw me into rotation. It's not so bad in my experience. It probably varies wildly company to company but for me it was just a few calls a week. Usually late at night. Usually something that is relatively simple or that could safely wait until the next day. On occasion there would be a major incident and in that case there is not much you can usually do besides escalate it to someone. Make sure you write down as much information as you can about the event. Start making phone calls and give all that info to the necessary people. I wouldn't say that being on-call is very high pressure. A lot of times you are there to act as a facilitator. Someone to get the right people the right information. I wouldn't stress over it too much. I'm sure you will do fine.

  5. Overman
    Link
    I used to work at a company that required us to be on-call a couple times a year. It's one of the reasons I left. I signed up for 40-hour weeks and though extra unpaid hours were not in my...

    I used to work at a company that required us to be on-call a couple times a year. It's one of the reasons I left. I signed up for 40-hour weeks and though extra unpaid hours were not in my contract, there was enough pressure to do it that it felt impossible to say no, and no one had ever refused. (There was also pressure to work more than 40-hour weeks, but it wasn't so overt and I didn't give in to that.) Some higher-ranking developers were, basically, constantly "on-call" because only they knew how certain things worked.

    I understand that companies want to provide 24/7 support. But I simply do not want to work outside of work. It's not something I'd ever sign up for willingly.

    I did one week of on-call at that company and it was a disaster. Already sleep-deprived after a week of issues, I was woken up on the sixth day after two hours of sleep and hit with a real doozie: complex and time-sensitive. I honestly don't know how someone would be expected to deal with that, though at least I had the option to punt it off to one of the permanently on-call guys.

    If I did it over again, I'd tell them from the start that I simply refuse to do on-call, and just quit if they have a problem with it. Life's too short to be doing bullshit for other people or bending over backwards for a company that doesn't care about me (and they didn't pay that well compared to many SV companies).