Site Reliability Workbook

In 2016, Google dropped Site Reliability Engineering on the operations world, and the operations world was never the same.

Narrow, Rigid Incentives Narrow Your Success

Breaking Down the SLO Wall Between Customer and Cloud Provider

Treat Your Configuration as Code

Promote Toil Reduction as a Feature

Simplicity Is End-to-End, and SREs Are Good for That

Model and Enforce Blameless Behavior

Case Study 2: When Load Shedding Attacks

Event Processing/Data Transformation to Order or Structure Data

Pitfall 1: Failing to Recognize Configuration as a Programming Language Problem

Balancing Release Velocity and Reliability

Case Study 2: Perceived Overload After Organizational and Workload Changes

Your Users, Not Your Monitoring, Decide Your Reliability

Forming, Storming, Norming, Performing.

It turns out that automation doesn’t eliminate humans, after all; rather, it challenges us to reassert human needs across all scales.

I love that the language and the process create a dispassionate contract between operational considerations and delivering new functionality.

I cringe when I hear someone say 'SRE versus DevOps.' For me, they are inseparable in time and space, as labels describing the sociotechnical systems that deliver modern infrastructure with software.

Modern architecture and security practices evolve from slides, checklists, and hope to enabling the right behaviors with running code.

Like most success disasters, the SRE book created an opportunity to respond with human effort (“Hire more people! Do more speaking engagements!”) or with something more scalable.

SRE is a journey as much as it is a discipline.

We would like to thank the following reviewers for providing valuable (and sometimes pointed) feedback.

His leadership, thoughtfulness, tenacity, and wit are nothing short of inspirational!

The enterprise world, for example, often treats operations as a cost center, which makes meaningful improvements in outcomes difficult if not impossible.

Some more traditionally minded businesses possess the cultural instinct to root out the mistake maker and punish them.

So, in a way, class SRE implements interface DevOps.

For SRE, any manual, structurally mandated operational task is abhorrent.

It turns out that you can get a lot more done if you “blur the lines” and have SREs instrument JavaScript, or product developers qualify kernels.

SRE believes in the same things as DevOps but for slightly different reasons.

As Tolstoy almost but never quite said, effective operations approaches are all alike, whereas broken approaches are all broken in their own way.

These people will quit if they’re tasked with too much operational work and aren’t given the opportunity to use their engineering skill set.

The practice of rename-and-shame is a hollow one, unlikely to yield benefit.

One could even claim that without SLOs, there is no need for SREs.

If you do manage to create an experience that is 100% reliable for your customers, and want to maintain that level of reliability, you can never update or improve your service.

If a single outage is responsible for 1,500 errors, that error costs 50% of the error budget.

Especially if you’re just starting your SLI journey, pick an aspect of your system that’s relevant but easy to measure—you can always iterate and refine later.

For example, if you have 1 billion records for the previous 10 years, but the user wants only the records from today (which are unavailable), then they will be unhappy even though almost all of their data is readable.

This methodology assumes that creating such a system is both possible and practical.

If you have a large outage on the final day of a month, your user doesn’t suddenly forget about it on the first day of the following month.

The team responsible for the production environment who are tasked with defending this SLO have agreed that it is defensible without Herculean effort, excessive toil, and burnout.

Even if the SLOs are totally ad hoc, this fact should be documented so that future engineers reading the document don’t make bad decisions based upon ad hoc data.

It’s pointless to improve the recall of your system if you lower the precision such that the team must constantly respond to unimportant events.

This way you can track your progress toward meeting the aspirational SLO, but you won’t be in a perpetual state of emergency.

The numbers prove that addressing the release problem provides much more benefit than investing resources in investigating the server failure.

The probability that both services will experience an outage at the same time is so low that two zones should provide 99.9999% availability. However, this reasoning assumes that both services are wholly independent, which is almost never the case.

To make a choice to deliberately lower the perceived customer experience is a Rubicon to be crossed extremely thoughtfully, if at all.

As we swung wildly between these two goals, the ops and dev teams developed a frustrated and strained relationship.

In our experience, an error budget/SLO approach has led both teams to make similar decisions when presented with the same facts, as it removes a good deal of subjectivity from the conversation.

By applying an SLO calculation to the problem and removing human subjectivity from the scenario, we were able to better quantify customer impact and reduce our release windows from five to two to minimize customer pain.

Even when GCP SLO graphs are green (i.e., above 99.95%), Evernote’s view of the same SLO might be very different.

This approach gives developers freedom to push code when they want, but also makes them jointly responsible for the operations of their service.

The primary (and often only) way we measured the reliability of the applications deployed to our stores was by tracking the number of support calls our internal support desk receives.

Armed with an easy-to-remember acronym, we set out to evangelize SLOs to the enterprise.

We called this framework TPS Reports, a play on the term we used for volume and performance testing (transactions per second), and, of course, to poke fun at the idea that multiple managers might want to review this data.

After tracking SLOs for about 50 services at the beginning of the year, by the end of the year we were tracking SLOs for 800 services.

We’d like to emphasize that The Home Depot is a traditional enterprise; if we can introduce such a large change successfully, you can too.

Data more than four to five minutes stale might significantly impact how quickly you can respond to an incident.

A monitoring system that supports computing percentiles will let you see if 50%, 5%, or 1% of your requests are too slow, whereas the arithmetic mean can only tell you—without specifics—that the request time is slower.

We tend to use logs to find the root cause of an issue, as the information we need is often not available as a metric.

These scripts were hard to maintain and also used data that wasn’t available to the metrics-based monitoring system.

Treating system configuration as code and storing it in the revision control system are common practices that provide some obvious benefits: change history, links from specific changes to your task tracking system, easier rollbacks and linting checks, and enforced code review procedures.

When you’re trying to correlate an outage with a rollout, it’s much easier to look at a graph/dashboard linked from your alert than to trawl through your CI/CD system logs after the fact.

Resist the temptation of exporting a handful of metrics just because they are easy to generate.

It’s very likely that your alerting rules will not fire for months or years after you configure them, and you need to have confidence that when the metric passes a certain threshold, the correct engineers will be alerted with notifications that make sense.

Precision is 100% if every alert corresponds to a significant event.

Taking this example to an extreme, you could receive up to 144 alerts per day every day, not act upon any alerts, and still meet the SLO.

A series of 100% error spikes lasting 5 minutes every 10 minutes never triggers an alert, despite consuming 35% of the error budget.

We can enhance the multi-burn-rate alerts in iteration 5 to notify us only when we’re still actively burning through the budget—thereby reducing the number of false positives.

If a system receives 10 requests per hour, then a single failed request results in an hourly error rate of 10%.

Additionally, if an issue affects real users but doesn’t affect artificial traffic, the successful artificial requests hide the real user signal, so you aren’t notified that users see errors.

A 100% outage for a service with a target monthly availability of 99.999% would exhaust its budget in 26 seconds—which is smaller than the metric collection interval of many monitoring services.

A problem space that once seemed intractable (or too risky) will become feasible once you get comfortable with “letting the robots do the work.”

Toil can slowly deflate team morale. Time spent working on toil is generally time not spent thinking critically or expressing creativity.

Instead of the short-term endorphin hit of fixing those machines every morning, I now had the more cerebral pleasure of knowing that I’d fixed the problem properly.

Toil, like a crumbling bridge or a leaky dam, hides in the banal day to day.

A four 9s service that adds nine critical four 9s dependencies is now a three 9s service.

In our experience, while it may seem counterproductive, rejecting a toil-intensive task should be the first option you consider.

Teams are free to choose their own approaches, but they have to own the toil generated by unsupported tools or legacy systems.

They tend to operate like a magical black box in that they “mostly work,” but few people understand how they work.

This tactic is still avoidance, but is a bit like refinancing high-interest technical debt into low-interest technical debt.

This growth changed the nature of datacenter manual repairs from occasional and interesting to frequent and rote—two signals of toil.

Every manual risk assessment was an opportunity for human error that could result in an outage.

The new system freed the engineering team from a large volume of toilsome work, giving them more time to pursue productive projects elsewhere.

In one particularly high-impact episode, a technician decided to expedite the “press button and wait for results” experience by initiating concurrent drains on every line card waiting for repairs at the datacenter, resulting in network congestion and user-visible packet loss.

We spent nearly three years (2012–2015) collecting data on over 650 discrete memory error problems before realizing this exercise was probably overkill, or at least shouldn’t block our repair automation project.

Make sure your automation doesn’t create new toil—for example, by opening unnecessary tickets that need human attention.

It’s much more painful to take something away from users than never offer it in the first place.

False positives (erroneously reporting action required) or false negatives (failing to notify a user that you were taking something away) were both unacceptable, and errors here would mean extra work in the form of lost credibility and customer service.

At Google, this practice of changing reality to fit the code rather than the other way around is called 'buying the gnome.'

A complex system that works is invariably found to have evolved from a simple system that worked.

Frequently, the cost of complexity does not directly affect the individual, team, or role that introduces it—in economic terms, complexity is an externality.

In practice, the grass isn’t always greener.

As Dijkstra said, 'If we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent.”'

At one point, we added tests to make sure we had removed all infinite loops in the query flow.

We were like cave dwellers who could only light fires by running with a torch lit from the last campfire.

Systems inevitably creep toward complexity as they evolve, so the fight for simplicity requires continuous attention and commitment—but it is very much worth pursuing.

In this situation, engineers might start to forget crucial aspects of the service they are responsible for.

Our on-call engineer gets paged about a hundred times in a typical 24-hour shift. A lot of pages get ignored, while the real problems are buried under the pile.

At Google, the overall goal of being on-call is to provide coverage for critical services, while making sure that we never achieve reliability at the expense of an on-call engineer’s health.

The team also held lab sessions to perform common debugging and mitigation tasks to help everyone build muscle memory and gain confidence in their abilities.

If your playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, we recommend implementing automation.

In a world of live migrations and network latency, we needed to take a much more holistic approach to monitoring.

One of the benefits of keeping our process lightweight is that we can explicitly free the on-call from any expectations of project work.

Members of the team heroically responded to the daily onslaught of pages but couldn’t keep up; there simply was not enough time in the day to find the root cause and properly fix the incoming issues.

The team’s services were subject to an unprecedented rate of change, and the changes themselves caused a significant portion of the on-call load.

Follow a 'detect, roll back, fix, and roll forward' strategy rather than a 'detect, continue to roll forward despite identifying the bug, fix, and roll forward again' strategy.

Generally, it is better to “roll back, fix, and roll forward” rather than “roll forward, fix, and roll forward again.”

Receiving a page creates a negative psychological impact. To minimize that impact, only introduce new paging alerts when you really need them.

Explaining away a page as “transient,” or taking no action because the system “fixed itself” or the bug inexplicably “went away,” invites the bug to happen again and cause another page, which causes trouble for the next on-call engineer.

The answer is simple: collect data!

To avoid boiling the frog, it is important to pay attention to the health of on-call engineers over time, and ensure that production health is consistently and continuously prioritized.

“Fairness” doesn’t mean a completely uniform distribution of each type of shift across team members.

In our experience, you need a bare minimum of five people per site to sustain on-call in a multisite, 24/7 configuration, and eight people in a single-site, 24/7 configuration.

As one on-call engineer puts it: “I take a quick look at the page subject and know they are duplicates. So I just ignore them.”

If you’ve spent a little time with that colleague, you’d feel much worse about what happened, and strive to be considerate by being more careful in the future.

ICS was established in 1968 by firefighters as a way to manage wildfires.

This example shows how failing to declare an incident early on can leave a team without the tools to respond to an incident quickly and efficiently.

The team had not followed the best practice of performing rollouts only during business days to ensure developers are around in case something goes wrong.

The following incident illustrates what happens when a team of experts tries to debug a system with so many interactions that no single person can grasp all the details.

The first Seattle SRE, Il-Seong, arrived at the office, lightly caffeinated and ready for the day.

The outage was over. All that remained was cleanup, and writing a truly epic postmortem.

Ultimately, customers do not care whether or not you fully understand what caused an outage. What they want is to stop receiving errors.

The UPS batteries in the disk trays did not swap power usage to the backup batteries on the third and fourth lightning strikes because the strikes were too closely spaced.

The stressful and communication-intensive nature of the game forces players to cooperate.

We also record all of the phone calls involved in a major incident so we can learn from the real-time communication feed.

No one wants to write these announcements under extreme stress with no guidelines.

By practicing during less critical situations, your team develops good habits and patterns of behavior for when lightning strikes—figuratively and literally.

Staying calm and following the response structure during an emergency takes practice, and practice builds “muscle memory.”

Within minutes, the disks of all satellite machines, globally, were erased.

The team (especially maxone@, logantwo@) never wrote any documentation to tell SREs not to run the automation multiple times, which is ridiculous.

Let’s plan for a future where we’re all as stupid as we are today.

The value of a postmortem is proportional to the learning it creates.

The bug removed the machine constraint on the decom action, sending all satellite machines to decommission.

The decom workflow doesn’t cross-check decom requests with other data sources; as a result, there were no objections to the request to trash (many) geographically diverse machines.

Add a big-red-buttona disable approach to decom workflows.

A postmortem with no action items is ineffective.

The longer you wait, the more they will fill the gap with the products of their imagination. That seldom works in your favor!

Use the Wheel of Misfortune when training new engineers: a cast of engineers reenacts a previous postmortem, assuming roles laid out in the postmortem.

Postmortems are letters you write to future team members: it’s very important to keep a consistent quality bar, lest you accidentally teach future teammates a bad lesson.

With this level of tracking, we can ensure that action items don’t slip through the cracks, leading to increasingly unstable services.

No service is 100% available 100% of the time: clients can be inconsiderate, demand can grow fifty-fold, a service might crash in response to a traffic spike, or an anchor might pull up a transatlantic cable.

Each time the BGP route “flaps,” all in-progress TCP streams are reset as the unfortunate user’s packets are directed to a new frontend with no TCP session state.

By putting GFE backends into a mode in which they fail health checks while continuing to respond to in-flight requests, we can gracefully remove GFE backends from service without disrupting any user requests.

The actual launch requests per second (RPS) rate was nearly 50x that estimate—enough to present a scaling challenge for nearly any software stack.

These error responses served to effectively synchronize client retries, producing a “thundering herd” problem, in which many client requests were issued at essentially the same time.

Autoscaling runs into problems when machines are not serving (known as unhealthy instances) but are still counted toward the utilization average.

Autoscaler is a powerful tool; if misconfigured, it can scale out of control.

Dressy’s development teams investigate and notice a problem: their load balancing is inexplicably drawing all user traffic into region A, even though that region is full-to-overflowing and both B and C are empty.

As far as the load balancer system was concerned, each successive dropped request was a reduction in the per-request CPU cost.

Unless carefully configured, autoscaling can result in disastrous consequences—for example, potentially catastrophic feedback cycles between load balancing, load shedding, and autoscaling when these tools are configured in isolation.

Google has learned (the hard way) that the people designing distributed systems need to develop and continuously exercise the muscle of turning a whiteboard design into concrete estimates of resources.

The final design is the end of a story of twists and turns.

If every log entry can be stored and indexed in an average of one disk write per log entry, we see that IOPS is a limiting factor for our query logs: (5 × 10^5 queries/sec) / (200 IOPS/disk) = 2.5 × 10^3 disks or 2,500 disks.

If a logged query is in batch 1, and its click is in batch 2, the click and query will never be joined.

While we could calculate the impact of using higher IOPS drives (e.g., SSD), our exercise is focused on demonstrating that the system can scale to an arbitrarily large size.

If our log sharder process sends duplicate log entries to two shards, the system can continue to perform at full speed and process accurate results even when a LogJoiner fails.

This requirement places a limit on the sequential throughput for the system.

Google’s experience has shown that the ability to reason from an abstract requirement to a concrete approximation of resources is critical to building healthy and long-lived systems.

Ideally, SREs should be involved in this work from its early stages: Google’s SRE teams regularly consult with teams developing a data processing pipeline to ensure that the pipeline can be easily released, modified, and run without causing issues for customers.

New products that are uploaded to their system don’t have structured data or consistent labels (e.g., some vendors may include extra information about color, size, and features of the dress using different categories and formats).

Dressy has noticed that occasionally a new model doesn’t get published for over 24 hours, and the recommendations trigger intermittent errors.

If you don’t have access to such data, you can generate it. For example, use test accounts to calculate the expected output. Once you have this “golden data,” you can compare expected and actual output.

Both jobs think they are correct, but the user doesn’t see the data.

Each of the components and transformations shown in your diagram can get stuck, causing data to stop flowing through the system.

Often, you’ll have to wait for the complete cycle of processing to finish before you can discover any customer-impacting issues.

To be safe, it’s still best to build an emergency shutdown into your client logic to allow you to stop processing and isolate fine-grained chunks of processing work characterized by large resource usage or errors.

Constantly running the number of workers required for peak capacity is an expensive and inefficient use of resources.

Alerting on the symptom (versus the cause) reduces monitoring gaps.

With two-phase mutation, the mutations themselves are stored in a temporary location. A separate verification step (or pipeline) can run against these potential mutations to validate them for correctness.

To score your system, read the descriptions for each characteristic below, and select the best matching milestone.

There is a solution to automatically quarantine and replay a failed work unit.

Stale data is almost always better than incorrect data.

Recovering from this kind of data corruption is labor-intensive and difficult to automate.

Most importantly, we pay royalties to artists based on delivered events. It’s imperative that we have a reliable means of event storage and delivery.

Once decoupled, data collection and delivery act as independent failure domains, which limits the impact of any production issues and results in a more resilient system.

A skew can negatively impact jobs, since they might first underreport and then overreport values for some time periods.

In this regard, we use the analogy that event delivery should behave like a postal service: your mail should be delivered on time, intact, and unopened.

When Autoscaler is presented with constantly increasing CPU usage that has no correlation with the amount of work performed, it will scale indefinitely until it uses all of the resources it can find.

Nevertheless, we believe that the ability to delegate responsibility is worth the occasional feeling of powerlessness.

Because previous iterations were far less reliable, our engineers were paged every few nights.

In contrast, changing a single configuration option can have dramatic changes on functionality—for example, one bad firewall configuration rule may lock you out of your own system.

In the following philosophy, our ideal configuration is no configuration at all.

Perhaps counterintuitively, limited configuration options can lead to better adoption than extremely versatile software—onboarding effort is substantially lower because the software mostly works 'out of the box.'

While one might argue that adding one or two small steps incurs little cost, the life of an engineer is often an endless chain of individually small steps.

Consider not only the act of configuration itself, but also the decision paralysis users might experience when presented with many options, the time it takes to correct the configuration after taking a wrong turn, the slower rate of change due to lower confidence, and more.

To answer the age-old question of whether configuration is code or data, our experience has shown that having both code and data, but separating the two, is optimal.

Validating that the configuration is semantically meaningful, to the maximum extent possible, can help prevent outages and decrease operational costs.

Configuration that requires external resources that can change outside of its hermetic environment can be very hard to roll back.

Trivial configuration changes can impact a production system in dramatic ways, so we need to deliberately design configuration to mitigate these risks.

If you’re not intentionally designing a language, then it’s highly unlikely the “lan‐ guage” you’ll end up with is a good one.

If our configuration strategy starts with the objective of using a data-only format, programming language features tend to creep through the back door.

In an extreme case, it is impossible to debug your config without spending money by reserving cloud resources.

Once you’re able to generate configuration in the necessary formats, you can easily unify, synchronize, and eliminate repetition across your entire config corpus.

The variants are hard to read and maintain because the important differences are obscured.

Template overrides provide a useful escape hatch to change specific details that might normally be considered too low-level.

This strategy makes the collection (and individual animals) easier to extend, and you can reference entities by name (e.g., animals.cat) instead of referencing brittle indexes (e.g., animals[0]).

In our experience, configuration changes tend to dominate outage root causes over time in a system.

Generally, hermeticism means that Jsonnet code is always interchangeable with the expanded JSON it represents.

Unfortunately, due to bugs or deliberate attacks, configuration may take an arbitrary amount of CPU time or memory.

The logic underpinning this approach is that usually the canary deployment is performed on a much smaller subset of production, or affects a much smaller subset of the user base than the control portion.

The business wants to ship new features and product improvements as quickly as possible with 100% reliability!

Introducing the change to actual production traffic also enables us to identify problems that might not be visible in testing frameworks like unit testing or load testing, which are often more artificial.

Figure 16-3 shows that the impact of the change is greatly reduced when we use a canary; in fact, the errors are barely visible!

As British statistician George Box said, “Essentially, all models are wrong, but some are useful.”

Canarying is a balancing act, informed both by cold analysis of worst-case scenarios and the past realistic track record of a system.

Too many metrics can bring diminishing returns, and at some point, the returns are outweighed by the cost of maintaining them, or the negative impact on trust in the release process if they are not maintained.

The response by the canary may change the content of the second request, which may land on the control, altering the control’s behavior.

Because monitoring likely looks at the system as a whole, it will detect an overall error rate of only 1%.

It can be especially hard to artificially simulate load in mutable systems (systems with caches, cookies, request affinity, etc.).

The whole team starts to feel stressed and frustrated as they work harder but don’t feel like they are making progress.

A team is in a state of operational overload when it can’t make progress toward key priorities because urgent issues continually preempt project work.

Even if you can finish all the tickets quickly and the actual workload is low, you feel overloaded when you first look at the ticket queue.

We didn’t realize we were in the grip of a sunk cost fallacy.

It turned out that almost none of these tasks were impactful or important enough to resume.

We then started assuming we couldn’t depend on other people to get their work done, which eroded feelings of trust and dependability within the team.

The team deliberately didn’t try to find the source for every single alert, but focused on relieving the stress from being paged and ticketed continuously for issues we already knew about.

Hope is not a strategy, but it certainly helps team morale.

The path toward a healthy, friendly, and happy work atmosphere can be hard to visualize when you’re mired in overload.

A team can function only as well as its individual members.

Perceived overload is a special form of overload that can’t be measured by the amount of toil or operational work.

Fixing architectural mistakes becomes more difficult later in the development cycle.

SREs can have an explicit goal to support the developer team’s release velocity and ensure the success of all approved launches.

SREs are not traditional operations engineers. They do not support manual work such as running a job for deployment.

If an engagement isn’t working, we expect teams to not shy away from planning for disengagement.

In a stable environment, the absence of a roadmap may be a signal that the SRE team can merge with another team, move service management work back to the development teams, expand scope, or dissolve.

Readers will remember how the words made them feel, not necessarily what was written.

Interactions conducted solely via code reviews or documentation can quickly become drawn out and frustrating.

We’ve found that the costs of this configuration ultimately outweigh the benefits. Although that location will tend to become really good at executing those responsibilities, this fosters an “us versus them” mentality, hinders distribution of knowledge, and presents a risk for business continuity.

Abuse SREs taught the Ares team that the easiest way to launch a new service in production (when you’re already running large distributed services) is to minimize the additional cognitive load that the service imposes.

Although decommissioning two SRE teams was not a pleasant experience, doing so resolved the continual tension over where to invest engineering effort.

SRE and development teams need to be willing to address issues head on and identify points of tension that need resetting.

If your user is worried that your platform is responsible for instability they’re experiencing, then telling them “our monitoring looks fine; the problem must be on your end” won’t make them any less grumpy.

Remember, in the absence of a stated SLO, your customer will inevitably invent one and not tell you until you don’t meet it!

The application they thought was operating at “five 9s” (99.999%; everybody thinks they’re getting five 9s) is probably achieving only 99.5%–99.9% when measured against their shiny new SLOs.

Develop a healthy muscle memory between the teams for effective ways to communicate during a crisis.

Even if an organization doesn’t have SRE staff, we believe that it is worthwhile to set SLOs for critical customer applications and to implement an error budget policy, if only because an implicit 100% SLO means a team can only ever be reactive.

SREs must have time to make tomorrow better than today.

Be careful to avoid renaming a team from “Operations” to “SRE” without first applying SRE practices and principles!

A team that is constantly firefighting doesn’t have time to address risk in a more permanent way.

In a lot of environments, automation eliminates portions of work, but not jobs as a whole; while this might be a step on the path to job losses, it does at least have the virtue of freeing up time to do something better (and more sellable to a future employer) than nonautomated toil.

This model was not sustainable, and the team would need to grow linearly with products to keep up with this support burden.

Otherwise, your team risks attrition as SREs move on to more interesting opportunities. The slow bleed from attrition can put production at risk.

In our experience, finding qualified SRE candidates is difficult, so growing a team quickly with new hires often isn’t realistic.

If this is the case, be vigilant to ensure that the team that is not colocated ('Office 2') doesn’t become a night shift that has little contact with the product development team, takes more than its fair share of toil, or is assigned only the less interesting or impactful projects.

Being the hero that fixes easy problems is fun during office hours. But if it has some amount of personal cost, the motivation to make sure it never happens again is sharp and immediate.

The software engineer is trained in production systems and practices and eventually goes on-call for that service.

In short, you should have fewer SREs than the organization would like, and only enough SREs to accomplish their specialized work.

Site Reliability Engineers are frequently in the middle of this complicated and rapidly shifting landscape, responsible for balancing the risks inherent in change with product reliability and availability.

Because SRE is often on the front line when problems occur, it is uniquely motivated to lead the change needed to ensure products are available 24/7/365.

The case studies we refer to in this chapter deal with larger, organizational changes where iteration is counterproductive: frequent, wrenching org-chart changes can sap employee confidence and negatively impact company culture.

At its worst, the entire Waze SRE team spent most of a two-week period firefighting 24/7, eventually resorting to restarting some components of the message queue hourly to keep messages flowing and tens of millions of users happy.

Eliminating one bottleneck in a system often highlights another one.

Now, engineers began to put pressure on dependencies that had not moved, as they were the impediment to faster development velocity—not the SRE team!

Those waiting for the big use case—a nonspecific, singing-and-dancing solution of the future—waited a long time, got frustrated, and used their own software engineering skills to create their own niche solution.

There was a lot of state to be swapped between running a service and building a serious piece of software.

But in real life, the hardest engineering work ends up being the evolution of many small/constrained systems into fewer, more general systems—without disturbing already running services that many customers depend on.

Despite the allure of easier management and less specific deep knowledge, the costs of migrating away from the familiar (with all its warts and toil) were generally a barrier.

Perceptions of how the sausage is made turned out to be more important than we anticipated from the get-go.

He who lives by the crystal ball soon learns to eat ground glass.

May your queries flow and your pagers stay silent.

Even if the numbers in the SLO are not strongly evidence-based, it is necessary to document this so that future readers can understand this fact, and make their decisions appropriately.

Halting change is undesirable; this policy gives teams permission to focus exclusively on reliability when data indicates that reliability is more important than other product features.

Binary and configuration pushes account for over two-thirds of outage triggers, while hardware failures represent a mere 2%.

modeling and enforcing blameless behavior

Failing to recognize configuration as programming language problem.

SREs sharing ownership with developers, 6 sustaining an effective ongoing relationship between SRE and development teams, 380-382

tradeoffs between feature velocity and reliability, 19

case study, Google Home software bug and failure to communicate, 177-180

failing to recognize configuration as programming language problem, 317

Prioritizing mitigation above all else, 185

scenario, a survive the week culture, 171-173

Reliability as most important feature, 391; decided by users, not monitoring, 392.

planning for long-term breaks, 170 planning for part-time work schedules, 170 planning for short-term swaps, 169

principles, 4-7: managing by SLOs, 5; moving fast by reducing cost of failure, 6; sharing ownership with developers, 6; using same tooling regardless of job title, 7; working to minimize toil, 5

on-call engineers in survive the week culture, 171-173

SRE teams having complete self-determination of workloads.

The green thornytail lizard is rather small, at about 3.5 inches long, and is difficult to spot in the wild because it tends to live high in the forest canopy.

Narrow, Rigid Incentives Narrow Your Success

Promote Toil Reduction as a Feature

Balancing Release Velocity and Reliability

Case Study 2: Perceived Overload After Organizational and Workload Changes

I love that the language and the process create a dispassionate contract between operational considerations and delivering new functionality.

One could even claim that without SLOs, there is no need for SREs.

If you do manage to create an experience that is 100% reliable for your customers, and want to maintain that level of reliability, you can never update or improve your service.

The team responsible for the production environment who are tasked with defending this SLO have agreed that it is defensible without Herculean effort, excessive toil, and burnout.

It’s pointless to improve the recall of your system if you lower the precision such that the team must constantly respond to unimportant events.

Precision is 100% if every alert corresponds to a significant event.

We can enhance the multi-burn-rate alerts in iteration 5 to notify us only when we’re still actively burning through the budget—thereby reducing the number of false positives.

Toil can slowly deflate team morale. Time spent working on toil is generally time not spent thinking critically or expressing creativity.

They tend to operate like a magical black box in that they “mostly work,” but few people understand how they work.

We spent nearly three years (2012–2015) collecting data on over 650 discrete memory error problems before realizing this exercise was probably overkill, or at least shouldn’t block our repair automation project.

Frequently, the cost of complexity does not directly affect the individual, team, or role that introduces it—in economic terms, complexity is an externality.

At Google, the overall goal of being on-call is to provide coverage for critical services, while making sure that we never achieve reliability at the expense of an on-call engineer’s health.

Receiving a page creates a negative psychological impact. To minimize that impact, only introduce new paging alerts when you really need them.

ICS was established in 1968 by firefighters as a way to manage wildfires.

The outage was over. All that remained was cleanup, and writing a truly epic postmortem.

Within minutes, the disks of all satellite machines, globally, were erased.

Let’s plan for a future where we’re all as stupid as we are today.

As far as the load balancer system was concerned, each successive dropped request was a reduction in the per-request CPU cost.

Google has learned (the hard way) that the people designing distributed systems need to develop and continuously exercise the muscle of turning a whiteboard design into concrete estimates of resources.

Both jobs think they are correct, but the user doesn’t see the data.

Once decoupled, data collection and delivery act as independent failure domains, which limits the impact of any production issues and results in a more resilient system.

In contrast, changing a single configuration option can have dramatic changes on functionality—for example, one bad firewall configuration rule may lock you out of your own system.

If you’re not intentionally designing a language, then it’s highly unlikely the “lan‐ guage” you’ll end up with is a good one.

The logic underpinning this approach is that usually the canary deployment is performed on a much smaller subset of production, or affects a much smaller subset of the user base than the control portion.

Too many metrics can bring diminishing returns, and at some point, the returns are outweighed by the cost of maintaining them, or the negative impact on trust in the release process if they are not maintained.

A team is in a state of operational overload when it can’t make progress toward key priorities because urgent issues continually preempt project work.

Fixing architectural mistakes becomes more difficult later in the development cycle.

If your user is worried that your platform is responsible for instability they’re experiencing, then telling them “our monitoring looks fine; the problem must be on your end” won’t make them any less grumpy.