1.4.all

On Pager Duty – organizational challenges

In my last post I wrote about the personal aspects and challenges of On Call Duty (OCD). A few people replied in the comments or their own blogs and added...

In my last post I wrote about the personal aspects and challenges of On Call Duty (OCD). A few people replied in the comments or their own blogs and added valuable comments. This episode will focus on the challenges the organization might face when organizing OCD coverage. My experience here is based on my work for germany based companies. The german labor laws are of course special and your own experience and circumstances in your company/country might vary significantly.

Labour law compliance

There are several aspects of the german labour law that apply to OCD. The important terms here are “working time” and “rest time”. German labour law requires a certain amount of uninterrupted rest time for certain work time. Roughly speaking: when you're working your normal 8 – 10 hours a day you have to have at least 11 hours of uninterrupted rest time. This is not a recommendation but requirement by german labour law. The other aspect is the compensation. German labour law also requires the company to compensate your working time.

When an engineer is on call duty she has to be reachable and prepared to start working outside of her normal working time. This time off normal working hours however is considered as rest time until the engineer is called to an actual support mission. The time spent on the support mission is surely considered as working time as well as possible commute times to the work place.

The first organizational challenge is the requirement of 11h of uninterrupted rest time after 8h of work (ArbZG §3 & §5 and most important §7). Imagine you are OCD and still do your normal work during the day. You work your usual 8-9 hours and go home. After that amount of work on a day, the law requires 11h of uninterrupted rest time. In reality this means, you don't have any working time left for a possible support mission. Theoretically this situation will immediately violate german labour law if you're called out for a support mission.

So formally you can't be OCD and still do your normal day job without risking to violate german labor law. There is a little bit of flexibility in the law that might allow you to do OCD if the you're get very little support calls ( < 1 call/week, with short support mission time). You have to compensate the over time quickly and don't work more than 8h/day on average.

If for whatever reason you tend to get calls quite often (> 3 calls/week) and your support missions are getting long ( >1h) you have to consider taking the OCD engineer off the normal work rotation. Otherwise the OCD engineer is very likely to work more than allowed by law. Additionally you might anyway exhaust your staff quickly when they work the normal 8h and additionally be on a 4h support mission during the night. For these scenarios you need to have multiple persons available to cover OCD.

This especially applies for the 11h of uninterrupted rest time. Think of a scenario where the engineer worked part of the normal work day (6h) and left the office at about 1pm. This is when the rest time starts. She has to rest at least until 1am the next day. Now she get's a call at 10pm that day and has to work for about 2 hours. This means the rest time is reset and starts at 12pm. As she has worked 8h now the 11h rest time are mandatory. She can't neither work nor be OCD until 11am on the new day. Somebody else would have to take over OCD at 12pm for her.

To be completely in line with the german labor law you should rather consider shift work or take the OCD engineer off the normal work and have a second on standby if support missions are lengthy.

I strongly recommend to get help from a lawyer when setting up the contracts for OCD with your employees. There are several exceptions that might apply for your company. The workers council also must be involved if there is one.

The above mentioned regulations might not strictly apply if there is a special frame contract for your industry sector. The ArbZG §7 allows special rules for on call duty and similar businesses. But the details have to be regulated and the health of the employee must be a priority.

Special attention has to be payed to the terms of the OCD regulation. The law destincts between on call duty and working on demand. For the latter the on demand engineer is bound to a certain place to start working immediately when called to a support mission. This applies for instance to doctors in hospitals and fire fighters. Their time waiting for a support mission is also considered working time, not rest time.

Compensation

The second aspect is the compensation. You usually will have a contract for 8 working hours per day. This is the maximum allowed regular work time in germany. Additional work (be it over time or support mission) must be compensated. Not necessarily with money but at least with time off in the same calendar month. Usually your normal hourly rate (based on your normal work contract) will apply for additional work spent during support missions. There might be a different rate for night and weekend work

This is the compensation for the mere work. Good companies will also pay you a lump sum for each day you're on call. This should compensate your for your limited freedom during your off work hours where you have to be reachable and ready to start working.

For those whose company refuses to compensate the OCD time apart from the mere support mission time I recommend having a look at the above mentioned labour law. If the OCD limits your freedom to move and chains you to your home desk or company campus etc. the law has special regulations. A compensation might be mandatory by law.

The amount of the lump sum and the rate for mission time may vary based on your goals. A higher lump sum but low mission compensation will usually encourage your staff to avoid being called outside offic hours as the compensation doesn't earn them much money. The lump sum should make sure there are enough people volunteering for OCD. This model only works long term if the engineers have the power and authority to influence the stability of the applications and thus the chance of being called at night.

If that's not the case you rather want to set the mission compensation higher to compensate for work outside office hours, you could also more easily make those costs transparent to the product owner who can influence the stability of the application.

You usually want some tracking of the OCD assignment and the support mission that occurred. Ideally you create and incident ticket to track for each call. This could be used to generate the monthly billing statements for the OCD engineers. I recommend to set the minimum mission time when being called to 30 min or even 1h for the billing. A record by the minute seems exaggerated as one should also incorporate the time it takes to get back to sleep or other activities after finishing the support mission. This might also reduce the amount of “let the OCD engineer quickly fix this” mentality.

In all my previous and current companies the OCD period was always one full week. Means an engineer was on duty for 7 days and then handed over to the next. This unit seems to be a good compromise between annoyance and organizational efforts ( hand over, billing, planning).

The specifics of your OCD should be fixed in a contract between the company and the individual. Otherwise the normal German labour law applies without the special exemptions for OCD regarding rest time etc.

Your company might have to fulfill certain service level agreements with customers or partners. This might include uptime of the service and mean time to repair. So you might also require your OCD engineer to start working as soon as possible after being called for a support mission. I've had OCD contracts that required me start working within 30min after receiving the call. That doesn't sound too complicated, but in reality it's not that easy to always meet this requirements. If the contractual required reaction time is very low the it might no longer be considered on call duty but working on demand. As mentioned above this has significant implications on the consideration of the working vs. rest time.

• February 2nd, 2013
• Tagged with: ocd
  
• Comments(2) »

On Pager Duty – a reply

Recently Mathias Meyer of paperplanes.de wrote a blog post about on call duty (OCD) for his startup Travis CI. Apparently that field seemed new to him and he summarized his...

Recently Mathias Meyer of paperplanes.de wrote a blog post about on call duty (OCD) for his startup Travis CI. Apparently that field seemed new to him and he summarized his experience and findings and asked for other people’s experience in this special field.

As I’m working in the field of web operations since about 1999 I was on call most of the time on the last 13 years. And of course I’ve gather my share of experience and opinion on this topic and thought a reply might be worthwhile. Quickly I started drafting a reply. But quickly I realized that this can be quite an extensive topic and slowed down a bit. I started to draw a mind map of what I see related to this topic.

That looks like material for a series of blog posts to me. And in order to give at least a bit of feedback rather sooner than a full reply in half a year I’m going start with my experiences around the mere on call duty itself first.

What does it mean to be “on call”?

Being on call to me mean being reachable by phone (or pager) all the time – 24h a day, 7 days a week. No matter what. And not only being reachable, but also being able to work on an issue.

This has several implications. One must not only carry a working phone/pager, but also make sure those devices have network coverage to accept the call. This limits my personal freedom of moving and traveling during those on call shifts. Most of the country sides in Germany (and most other countries actually) have pretty bad mobile network coverage. Some facilities explicitly block mobile reception like Cinemas, Hospitals and their likes. There are also some other areas where carrying a phone is not easy or acceptable like swimming pools, sauna etc.

The next aspect is the ability to quickly start working on issues. Depending on the Service Level Agreements you have with your customers and partners you have to start fixing a problem within minutes of detection. Lucky enough mobile computer and 3G/4G carriers enable us to work from almost everywhere (given the above mentioned restrictions). So I don’t have to stay in the office during my on call duty but can travel around to a certain extend. But it also means I have to carry around my work equipment.

In my company we’re using our normal work Laptops in combination with mobile hotspots. Some people use their normal smartphones to establish an internet connection but mostly we use additional 3G/4G equipment. One reason for this is, that we usually have to attend an incident conference call while working on the issue. Using the phone as a hotspot while being in a conference call will not work very well if at all. Separate WiFi equipment usually also comes with it’s own battery and thus extends the working time off a power socket a bit.

To be able to work from remote location you of course need a secure way to enter your compute environment. Means you have to have some sort of VPN and remote access in place to let you in from untrusted networks (the evil Interweb). To add more flexibility on the work equipment using open standards is the best way to ensure you get in when you need it. Using super proprietary VPN equipment might limit your ability to login with mobile equipment like iPads or Smartphones.

Although it’s technically possible to do SSH connections from mobile Phones or Tables I still prefer a Laptop for On Call work. It gives me more flexibility and is usually a bit quicker. And time usually is a sensitive factor when being on call on a support mission. Your customers want you to keep a possible service degradation or downtime as short as possible and you want to get back to bed or other activities asap as well.

[Update 2013-01-10 10:20pm] Another aspect of “being able to work” just came to my mind. Not only do you need your technical equipment ready to start working but also yourself. Taking a support mission working in a root shell after 5 beers might not be the wisest decision. So staying sober during OCD is mandatory…

Being on call definitely has an impact to family life. If you’re just on your own you might not mind being called in the middle of the night and spent 3 hours recovering a database. But when you have a family, maybe with young kids, it’s becoming an issue.

You potentially wake up the family when being called. Now you have the choice of fixing the issue or calming down a crying baby. Parents know the choice is easy. When you work half the night, you might be grumpy in the morning when kids expect you to be the lion king on their birthday party… Wife usually don’t appreciate taking call during dinner in a restaurant and opening a laptop on a dining table.

Summarized on call duty will put a virtual chain on you. It limits your ability to do what you would normally do. It requires you to plan your activities around potential support missions and affects also the people who are with you.

This is why on call duty usually deservers a special compensation (either as money or time off) in companies.

In my next article in this OCD series I’ll write about the organizational challenges of organizing on call duty. You may have a quick peek at the subtopics in the mind map linked at the above preview image.

• January 3rd, 2013
• Tagged with: ocd
  
• Comments(5) »