Site Reliability Workbook

• The Site Reliability Workbook serves as a practical companion to Google's original SRE book, focusing on implementation rather than theory.
• Industry experts from companies like LinkedIn, Stripe, and Gremlin praise the book for translating Google's internal practices into actionable steps for organizations of all sizes.
• The text emphasizes moving beyond abstract concepts to address specific operational challenges like toil, data pipelines, and service level management.
• A key goal of this volume is to help engineers who could not see themselves in Google's original story apply SRE principles to their own unique contexts.
• The workbook highlights that reliability is a human-centric endeavor created through the interaction of users, engineers, and technology.

In 2016, Google dropped Site Reliability Engineering on the operations world, and the operations world was never the same.

• The text outlines the publication history and legal disclaimers for 'The Site Reliability Workbook', a collaboration between O'Reilly and Google.
• It establishes a clear distinction between the views of the authors and the publisher, emphasizing that readers use the technical instructions at their own risk.
• The table of contents introduces the relationship between Site Reliability Engineering (SRE) and DevOps, highlighting shared values like breaking down silos.
• Key SRE principles are introduced, such as treating operations as a software problem and managing through Service Level Objectives (SLOs).
• The text emphasizes the importance of minimizing 'toil' and automating manual tasks to allow for more efficient scaling.
• It advocates for a culture of shared ownership and reducing the cost of failure to enable faster innovation.

Narrow, Rigid Incentives Narrow Your Success

• The text outlines the structural necessity of Service Level Objectives (SLOs) for Site Reliability Engineering (SRE) teams.
• It details a step-by-step methodology for defining Service Level Indicators (SLIs) and calculating initial reliability targets.
• The framework emphasizes the importance of error budgets and stakeholder agreement in operational decision-making.
• Advanced topics include modeling user journeys, grading interaction importance, and managing dependency-based reliability.
• Real-world case studies from Evernote and The Home Depot illustrate the cultural and technical journey of adopting SLOs.

Breaking Down the SLO Wall Between Customer and Cloud Provider

• The text outlines the lifecycle of Service Level Objectives (SLOs), from initial creation and evangelization to automation using the VALET framework.
• It explores the expansion of SLO applications into batch processing and software testing environments.
• A comprehensive monitoring strategy is defined by key features such as speed, calculation capabilities, user interfaces, and alerting mechanisms.
• The management of monitoring systems is treated as a software engineering discipline, emphasizing configuration as code and loose coupling.
• Metrics are categorized by purpose, focusing on system saturation, traffic status, and the impact of dependencies.
• The section concludes with methodologies for implementing purposeful metrics and testing the logic behind automated alerts.

Treat Your Configuration as Code

• The text outlines various alerting methodologies based on Service Level Objectives (SLOs), including burn rate and multiwindow alerts.
• It addresses the specific challenges of maintaining error budget alerting for low-traffic services through artificial traffic or infrastructure changes.
• A significant portion is dedicated to defining 'toil' as repetitive, manual, and automatable work that lacks long-term value.
• The 'Toil Taxonomy' categorizes operational burdens such as release shepherding, migrations, and troubleshooting opaque architectures.
• Strategies for eliminating toil emphasize engineering solutions, self-service methods, and gaining management support to treat reduction as a feature.

Promote Toil Reduction as a Feature

• The text outlines strategies for increasing uniformity and assessing risk within automated systems to reduce manual toil.
• It emphasizes the importance of using open source tools and feedback loops to continuously improve operational efficiency.
• Detailed case studies illustrate real-world applications of automation, such as hardware repair and decommissioning legacy storage systems.
• A significant portion of the content focuses on measuring and regaining simplicity within complex technical environments.
• The section concludes by transitioning from theoretical frameworks to the practical application of SRE principles.

Simplicity Is End-to-End, and SREs Are Good for That

• The text outlines the structural evolution of on-call setups, comparing Google's team formation with Evernote's cloud transition.
• It details the practical mechanics of pager load and the importance of maintaining healthy team dynamics during on-call shifts.
• Incident response strategies are explored through the Incident Command System and specific roles required for effective crisis management.
• Real-world case studies illustrate diverse failure modes, including software bugs, service faults, and physical power outages.
• The section emphasizes the necessity of rigorous training through drills and proactive preparation before incidents occur.
• A critical distinction is made between 'good' and 'bad' postmortems, advocating for a blameless culture to incentivize organizational learning.

Model and Enforce Blameless Behavior

• Establishing a healthy postmortem culture involves rewarding outcomes, sharing findings openly, and addressing cultural failures.
• Standardized tools and templates are essential for consistent and effective postmortem documentation.
• Google Cloud Load Balancing (GCLB) utilizes technologies like Anycast and Maglev to achieve low latency and high availability.
• Managing system load requires a combination of autoscaling, handling unhealthy machines, and implementing kill switches.
• Strategies to avoid overloading backends and traffic imbalances are critical for maintaining stateful and stateless system stability.
• Non-Abstract Large System Design (NALSD) focuses on concrete design processes and initial requirements for massive scale.

Case Study 2: When Load Shedding Attacks

• The text outlines the architectural transition from single-machine systems to complex distributed systems for data processing.
• It categorizes pipeline applications into three primary domains: event processing, data analytics, and machine learning.
• A comprehensive set of best practices is established, emphasizing service level objectives, dependency failure planning, and resource autoscaling.
• Technical design requirements focus on ensuring data integrity through idempotent mutations, two-phase commits, and checkpointing.
• The section includes a detailed case study of Spotify's event delivery system to illustrate real-world operational challenges and architecture.
• Production readiness is highlighted as a critical phase involving failure prevention, response planning, and security policy adherence.

Event Processing/Data Transformation to Order or Structure Data

• Configuration should be treated as a distinct discipline that separates high-level philosophy from low-level mechanics.
• Effective configuration systems ask users questions that align closely with their ultimate goals rather than exposing internal implementation details.
• A major pitfall in system design is failing to recognize that configuration management is essentially a programming language problem.
• Using general-purpose scripting languages like Python or Ruby for configuration is often a mistake that leads to unmanageable complexity.
• Reliability is improved by separating the configuration source from the resulting data and implementing strict change tracking.
• Configuration-induced toil can be mitigated by avoiding ad hoc language features and preventing the interleaving of evaluation with side effects.

Pitfall 1: Failing to Recognize Configuration as a Programming Language Problem

• The text outlines the lifecycle of configuration management, including versioning, source control, and testing methodologies.
• It evaluates the trade-offs between checking in configuration early versus evaluating it at build time or runtime.
• The concept of canarying is introduced as a method to balance release velocity with system reliability.
• A robust canary process requires careful selection of metrics that are representative, attributable, and capable of indicating problems.
• The section explores deployment alternatives such as blue/green deployments, traffic teeing, and artificial load generation.
• Strategies are provided for minimizing risks to Service Level Objectives (SLOs) and error budgets during the release process.

Balancing Release Velocity and Reliability

• The text outlines strategies for identifying and recovering from team overload through specific case studies.
• It distinguishes between actual work overload caused by staffing shortages and perceived overload following organizational shifts.
• A framework is provided for recognizing symptoms of burnout and restoring team health through mitigation strategies.
• The SRE Engagement Model defines a seven-phase service lifecycle from architecture and design to abandonment.
• Effective SRE relationships require aligning business priorities, identifying risks, and establishing clear ground rules.

Case Study 2: Perceived Overload After Organizational and Workload Changes

• Effective SRE engagement requires sustaining ongoing relationships through regular service reviews and open communication.
• Operational priorities must be dynamically adjusted based on Service Level Objectives (SLOs) and error budgets to maintain system health.
• Scaling SRE involves managing multiple services with single teams and adapting structures to distributed environments.
• Reliability is framed as a partnership where users, rather than internal monitoring, are the ultimate judges of success.
• Practicing SRE with customers involves shared dashboards, collaborative risk analysis, and disciplined measurement.
• The lifecycle of SRE teams begins with establishing core practices even before dedicated roles are officially filled.

Your Users, Not Your Monitoring, Decide Your Reliability

• The text outlines the progression from bootstrapping a single SRE to managing distributed teams and complex service architectures.
• It details the psychological stages of team development—Forming, Storming, Norming, and Performing—within an SRE context.
• Strategies for scaling include geographical splits, service complexity management, and the creation of specialized Launch Coordination Engineering teams.
• Operational continuity is maintained through 'Mission Control' practices, SRE exchanges, and horizontal projects that span multiple teams.
• The section introduces formal organizational change management frameworks like Lewin’s Three-Stage Model and Kotter’s Eight-Step Process.
• It emphasizes that SRE is not just a technical role but a cultural shift requiring structured models to manage emotional and organizational transitions.

Forming, Storming, Norming, Performing.

• The text outlines case studies on scaling Waze and adopting common tooling within SRE teams.
• Mark Burgess emphasizes that SRE has evolved from technical descriptions to user-facing services and business-aligned objectives.
• Reliability is framed as a series of promises made to stakeholders that must be continuously assessed and repaired.
• Automation is revealed not as a replacement for humans, but as a challenge to reassert human needs at a global scale.
• The transition from ad hoc to planned change is highlighted as a critical evolutionary step for growing infrastructure.

It turns out that automation doesn’t eliminate humans, after all; rather, it challenges us to reassert human needs across all scales.

• Andrew Clay Shafer highlights how SRE principles like embracing risk and eliminating toil align with the core philosophy of DevOps.
• Service Level Objectives (SLOs) are praised for creating a dispassionate contract that balances operational stability with feature delivery.
• The shift from physical media to software-as-a-service has elevated the role of operations within the value chain.
• DevOps is described as an emergent optimization required for the rapid delivery of highly available internet software.
• The author reflects on his role in articulating successful patterns borrowed from the world's most advanced infrastructures.
• Early collaborations with Google, such as their use of Puppet at scale, provided a window into the future of automated IT management.

I love that the language and the process create a dispassionate contract between operational considerations and delivering new functionality.

• Google's historical culture of secrecy regarding internal tools like Borg eventually gave way to the public sharing of Site Reliability Engineering principles.
• The DevOps movement emerged in 2008 as a way to bridge the gap between developers and operations, emphasizing that system administration is becoming software development.
• The CAMS/CALMS framework (Culture, Automation, Lean, Metrics, Sharing) serves as a foundational bridge between DevOps philosophy and SRE practices.
• DevOps is defined as a broad set of principles for optimizing human performance in software operations, while SRE is viewed as a specific, advanced implementation of those principles.
• The relationship between DevOps and SRE is analogous to the relationship between Agile and Extreme Programming (XP).
• While software is often credited with 'eating the world,' the author emphasizes that human systems and hardware infrastructure are equally critical components.

I cringe when I hear someone say 'SRE versus DevOps.' For me, they are inseparable in time and space, as labels describing the sociotechnical systems that deliver modern infrastructure with software.

• Software and humans form a single interconnected technosocial system where reliability is an existential requirement.
• Google's SRE practices set a global standard for scale, proving that reliability processes can function at the highest levels of complexity.
• Adopting SRE is not just about operations; it transforms architecture, security, and compliance by replacing checklists with running code.
• True learning in SRE is defined by changing organizational behavior and outcomes rather than simply collecting information or changing job titles.
• The transition from principles to practice requires converting potential internal resistance into allies by integrating SRE across all departments.

Modern architecture and security practices evolve from slides, checklists, and hope to enabling the right behaviors with running code.

• The unexpected success of the first SRE book revealed a global demand for scalable reliability practices beyond Google's internal teams.
• Readers frequently questioned how to translate abstract SRE principles into concrete practices for diverse organizational contexts.
• A common misconception persists that SRE is only viable at 'Google scale' or within Google's specific corporate culture.
• This second volume serves as a companion guide focused on implementation details and practical application rather than just theory.
• The authors address criticisms regarding their previous insular focus by exploring the relationship between SRE and the broader DevOps movement.
• The book maintains an 'opinionated' stance, presenting SRE not as a universal truth but as a proven, successful methodology for reliability.

Like most success disasters, the SRE book created an opportunity to respond with human effort (“Hire more people! Do more speaking engagements!”) or with something more scalable.

• The authors emphasize that SRE is an evolving journey rather than a fixed gospel, requiring constant updates to previously held beliefs.
• This volume serves as a practical companion to the first SRE book, shifting focus from high-level philosophy to concrete implementation.
• The scope is expanded beyond Google's internal practices to include perspectives from diverse firms like Spotify, Evernote, and The Home Depot.
• The text explicitly addresses the relationship between SRE and the broader DevOps community for the first time.
• Readers are encouraged to use the book as a conversation starter and a starting point for their own organizational discussions.
• The book includes supplemental code examples and materials intended for practical use in professional documentation and programming.

SRE is a journey as much as it is a discipline.

• The text outlines the legal requirements for citing the book, including specific formatting for titles, authors, and ISBNs.
• O’Reilly Safari is introduced as a comprehensive membership-based training platform featuring content from over 250 publishers.
• Contact information is provided for the publisher, including physical addresses, phone numbers, and social media links for technical inquiries.
• The book is a massive collaborative effort involving over 100 contributors, including authors, technical writers, and reviewers.
• Executive sponsorship from Google leadership was essential in positioning this volume as a worthy companion to the original SRE book.
• Specific chapter contributors and reviewers are meticulously listed to credit their technical input and feedback.

We would like to thank the following reviewers for providing valuable (and sometimes pointed) feedback.

• The text provides an exhaustive list of contributors and authors for chapters 9 through 21 of a technical publication.
• It highlights collaborative efforts between Google employees and external organizations like Spotify, Niantic, and The New York Times.
• The preface acknowledges the critical role of industry reviewers and technical writing teams in refining the book's content.
• Special recognition is given to Niall Richard Murphy for his leadership and dedication to the project even after leaving Google.
• The editors express personal gratitude to family members and support staff who facilitated the book's completion under an ambitious timeline.

His leadership, thoughtfulness, tenacity, and wit are nothing short of inspirational!

• The authors acknowledge the personal and professional support systems that enabled the creation of the book.
• Operations is described as a difficult discipline with best practices that are highly context-dependent and rarely adopted.
• The historical roots of operational research date back to World War II military improvements and ancient process management.
• A common industry failure is treating operations as a cost center, which prevents meaningful improvements in system reliability.
• Site Reliability Engineering (SRE) is introduced as a specific implementation of the DevOps philosophy.
• The text highlights a growing industry revolution born from dissatisfaction with traditional IT operations models.

The enterprise world, for example, often treats operations as a cost center, which makes meaningful improvements in outcomes difficult if not impossible.

• DevOps and Site Reliability Engineering (SRE) are presented as similar reactions to the inefficiencies of traditional enterprise IT management.
• The CALMS acronym (Culture, Automation, Lean, Measurement, and Sharing) serves as the core philosophical framework for DevOps practices.
• Breaking down silos is essential to prevent local optimization and knowledge hoarding that ultimately harm business outcomes.
• The philosophy views accidents as normal systemic occurrences rather than individual failures, advocating for better safeguards over punishment.
• Focusing on speeding up recovery is considered more profitable and effective than attempting to prevent all accidents through rigid control.
• Small, frequent changes are preferred over large, infrequent updates to reduce risk and improve organizational agility.

Some more traditionally minded businesses possess the cultural instinct to root out the mistake maker and punish them.

• Effective change management involves breaking updates into small, low-risk subcomponents rather than batching them for manual review.
• Organizational culture is prioritized over tooling, as a healthy culture can overcome technical failures while the reverse is rarely true.
• DevOps is a broad philosophy of collaboration, whereas Site Reliability Engineering (SRE) acts as a concrete implementation of that philosophy.
• Objective measurement is essential for breaking down silos and establishing a shared reality for incident resolution and business goals.
• The core tenet of SRE is treating operations as a software engineering problem to be solved with software engineering approaches.

So, in a way, class SRE implements interface DevOps.

• SRE rejects the goal of 100% availability, instead using Service Level Objectives (SLOs) as collaborative, blameless targets for performance.
• Toil is defined as manual, repetitive operational work that SREs strive to minimize in favor of engineering projects that provide lasting value.
• Google enforces a 50% cap on toil to ensure engineers have the capacity to automate systems and improve scalability.
• The 'wisdom of production' suggests that hands-on operational experience provides essential, messy reality that informs better system design.
• Automation is treated as a mechanism to 'automate this year's job away,' allowing teams to take on new services or different challenges over time.
• Reducing the cost of failure through SRE practices allows organizations to move faster while maintaining necessary reliability.

For SRE, any manual, structurally mandated operational task is abhorrent.

• Reducing mean time to repair (MTTR) directly increases developer velocity by preventing expensive late-stage problem discovery.
• Rigid boundaries between developers and operators are counterproductive, especially when they create power imbalances or pay discrepancies.
• SREs and product developers should share a holistic view of the entire stack, from frontend code to physical hardware.
• Blurring the lines of authority allows SREs to modify application code and developers to qualify kernels, removing incentives to guard specific functions.
• Unified tooling is essential; using different tools for different roles leads to catastrophic divergence and wasted effort.

It turns out that you can get a lot more done if you “blur the lines” and have SREs instrument JavaScript, or product developers qualify kernels.

• Both SRE and DevOps are built on the foundational belief that continuous change is necessary for improvement and that silos must be dismantled through collaboration.
• Change management is ideally executed through small, automated actions where testing and application are integrated into the workflow.
• Measurement is the lifeblood of both philosophies, with SRE focusing on SLOs to drive service improvements and DevOps focusing on feedback loop duration.
• Blameless postmortems are a shared cultural requirement to manage the 'brute reality' of production failures without unhelpful emotional reactions.
• While DevOps acts as a broad, context-sensitive philosophy for organizational change, SRE provides a highly opinionated and prescriptive framework for running services.
• SRE often supports the same technical practices as DevOps, such as CI/CD, but justifies them through operational stability rather than just the business case.

SRE believes in the same things as DevOps but for slightly different reasons.

• DevOps and SRE share a significant conceptual overlap and require specific organizational conditions to be effective.
• Narrow or rigid incentives tied strictly to launch or reliability metrics often lead to gaming the system rather than achieving true performance.
• Successful adoption requires a feedback loop between design and production, ideally involving SRE engagement during the design phase.
• The traditional separation of software and operations teams creates divergent incentives that encourage passing blame for production incidents.
• Organizations should foster blameless postmortems and allow teams the authority to be radical within their mission to optimize performance.
• The threat of withdrawing support from operationally difficult products serves as a necessary motivator for developers to fix underlying issues.

As Tolstoy almost but never quite said, effective operations approaches are all alike, whereas broken approaches are all broken in their own way.

• Google institutionalized the practice of withdrawing SRE support from products that fail to meet engineering standards or create excessive operational load.
• SRE and product development operate as separate organizations with mutual stakes, where product success funds SRE growth.
• Specialization in reliability requires dedicated job ladders and peer communities to reward unique skill sets and prevent burnout.
• While small startups lack the scale to withdraw support, they must still balance growth needs against the accumulation of technical debt.
• A data-driven partnership between SRE and development teams replaces subjective conflict with objective decisions about service prioritization.
• The 'SRE way' relies on high-level management support to overcome resistance from engineering teams accustomed to traditional operations.

These people will quit if they’re tasked with too much operational work and aren’t given the opportunity to use their engineering skill set.

• SRE teams should work with developers to improve products before maintenance shifts to ensure quality and expertise retention.
• Organizations must maintain parity of esteem between SRE and product development through equal career incentives and financial rewards.
• While SRE offers concrete suggestions for operational change, DevOps provides a wider focus that often meets with less initial resistance.
• The core foundations of a successful SRE journey include SLOs, monitoring, alerting, toil reduction, and simplicity.
• Service Level Objectives (SLOs) are the most critical component of SRE, enabling data-driven decisions regarding service reliability.

The practice of rename-and-shame is a hollow one, unlikely to yield benefit.

• Engineering time is a scarce resource that must be balanced between new feature development and maintaining service reliability.
• Service Level Objectives (SLOs) serve as the primary tool for making data-informed decisions regarding the opportunity cost of reliability work.
• The core identity of Site Reliability Engineering is so tied to SLOs that the role arguably cannot exist without them.
• Effective SLO adoption requires organizational commitment to an error budget policy that formalizes how to prioritize work when reliability targets are missed.
• SLOs define the threshold of user happiness; exceeding the target is unnecessary, while falling below it leads to customer dissatisfaction and churn.

One could even claim that without SLOs, there is no need for SREs.

• Reliability is a tool for customer happiness, but 100% availability is an unrealistic and counterproductive target for any service.
• The marginal utility of adding 'nines' of reliability decreases as costs increase, especially since users rarely experience perfect uptime due to external system failures.
• A 100% reliability goal prevents necessary updates and improvements, as change is the primary source of system outages.
• Service Level Indicators (SLIs) should be expressed as a ratio of good events to total events to provide an intuitive 0-100% scale.
• The difference between 100% and the chosen SLO creates an 'error budget' that allows teams to balance feature velocity with stability.

If you do manage to create an experience that is 100% reliable for your customers, and want to maintain that level of reliability, you can never update or improve your service.

• Standardizing SLIs into a consistent format (numerator, denominator, and threshold) enables the use of automated tooling for alerting and reporting.
• SLIs should be divided into a 'specification' (the desired user outcome) and an 'implementation' (the actual measurement method).
• Different measurement implementations, such as server logs versus client-side telemetry, offer varying trade-offs in accuracy, coverage, and infrastructure cost.
• The primary goal for beginners is to establish a measurement feedback loop rather than achieving perfect accuracy on the first attempt.
• While current performance can serve as a baseline, setting SLOs based solely on past performance can inadvertently commit a team to unnecessarily strict standards.
• To begin, teams should identify their specific users and focus on the most common ways those users interact with the system.

If a single outage is responsible for 1,500 errors, that error costs 50% of the error budget.

• Begin SLI implementation by creating a high-level architecture diagram that identifies key components, data flows, and critical dependencies.
• Categorize system components into three primary types: request-driven services, data pipelines, and storage systems to simplify SLI selection.
• Avoid overcomplicating the initial setup by choosing relevant but easily measurable metrics, refining them through iteration over time.
• Select five or fewer SLI types that represent the most critical functionality to ensure focus on what truly impacts the customer experience.
• Use multiple SLO thresholds for a single SLI, such as latency, to capture both the typical user experience and the 'long tail' of dissatisfied users.

Especially if you’re just starting your SLI journey, pick an aspect of your system that’s relevant but easy to measure—you can always iterate and refine later.

• Service quality metrics should account for graceful degradation, such as using generic imagery when specific data stores are unavailable.
• Pipeline metrics focus on freshness, correctness, and coverage to ensure data is updated, accurate, and processed within time windows.
• Storage durability SLIs must prioritize the data users actually want to access rather than just the total volume of data stored.
• When implementing SLIs, teams should start with the path of least resistance, such as existing web server logs or cloud monitoring dashboards.
• The choice of data source—ranging from application logs to client-side instrumentation—impacts how closely the metric reflects the actual user experience.
• For API and HTTP services, availability is typically measured by the ratio of non-5XX status codes to total requests.

For example, if you have 1 billion records for the previous 10 years, but the user wants only the records from today (which are unavailable), then they will be unhappy even though almost all of their data is readable.

• The example uses client-side implementation for SLIs because it correlates more closely with user experience and is easier to integrate.
• Coverage SLIs are calculated by comparing the number of records successfully processed against the total number of records the pipeline expected to process.
• Correctness can be measured by injecting known data into the system or by using an independent, more expensive secondary calculation method for sampling.
• White-box monitoring systems collect metrics like request totals and response codes to calculate availability and latency over specific time windows.
• While histograms are often used to approximate latency, counting explicit slow requests is generally more accurate but requires more complex configuration.

This methodology assumes that creating such a system is both possible and practical.

• Service Level Objectives (SLOs) are derived by rounding down Service Level Indicators (SLIs) to manageable, significant figures.
• Error budgets are calculated based on these objectives, defining the specific number of allowed failures over a set period.
• Rolling windows are recommended for tracking user experience because they prevent the 'reset' effect of calendar months and should be measured in full weeks to account for weekend traffic variance.
• Calendar-aligned windows are better suited for high-level business planning and quarterly project headcount allocation.
• Shorter time windows facilitate rapid tactical course corrections, while longer periods are necessary for making strategic infrastructure decisions.
• A four-week rolling window is suggested as an ideal general-purpose interval for balancing operational and strategic needs.

If you have a large outage on the final day of a month, your user doesn’t suddenly forget about it on the first day of the following month.

• Historical performance data should be analyzed to correlate SLO violations with actual incidents and response actions.
• If no historical data exists, low-fidelity monitoring like periodic health checks can serve as an initial data source.
• Effective SLOs require a three-way consensus between product managers, developers, and SREs to ensure the targets are both user-satisfying and defensible.
• An error budget policy must be established to define specific actions and responsibilities when reliability thresholds are breached.
• The negotiation of an error budget policy serves as a litmus test for whether the chosen SLOs are realistic or require further iteration.

The team responsible for the production environment who are tasked with defending this SLO have agreed that it is defensible without Herculean effort, excessive toil, and burnout.

• Error budget policies mandate specific actions when reliability targets are missed, such as halting feature development or freezing production changes.
• If stakeholders refuse to honor an error budget policy during an outage, the policy must return to the approval stage for renegotiation.
• Comprehensive SLO documentation must include the rationale behind the numbers, even if they were originally chosen on an ad hoc basis.
• The frequency of SLO reviews should scale with the maturity of the organization's reliability culture, starting monthly and moving to quarterly.
• Error budget policies require a clear escalation path to resolve disagreements regarding the calculation or the appropriateness of mandated actions.
• Visual dashboards and reports are essential for tracking budget consumption rates and identifying trends before the budget is fully exhausted.

Even if the SLOs are totally ad hoc, this fact should be documented so that future engineers reading the document don’t make bad decisions based upon ad hoc data.

• Error budgets provide a quantifiable metric for assessing the severity of outages and prioritizing incidents based on their impact.
• Improving SLO targets requires gathering data on user satisfaction through diverse channels like support tickets, social media sentiment, and manual surveys.
• The quality of an SLO is measured by its correlation with real-world user pain, which can be statistically analyzed using tools like Spearman’s rank correlation.
• Discrepancies between error budget loss and support ticket spikes indicate a lack of SLO coverage, necessitating adjustments to either the SLO or the SLI.
• Refining SLOs involves a delicate balance between recall (capturing all significant events) and precision (avoiding false alarms that fatigue the team).

It’s pointless to improve the recall of your system if you lower the precision such that the team must constantly respond to unimportant events.

• SLI implementations can be improved by moving measurements closer to the user or increasing the coverage of system functionality.
• Aspirational SLOs allow teams to track progress toward higher reliability goals without triggering immediate error budget penalties.
• Iterative improvements should prioritize high return on investment, favoring cheaper and quicker adjustments in early stages.
• Error budget policies dictate team actions when thresholds are breached, such as halting feature launches to focus on reliability.
• In extreme cases, teams may declare an emergency to deprioritize external demands until the service meets specific exit criteria.
• The severity of an incident can be quantified by calculating the exact percentage of the monthly error budget it consumed.

This way you can track your progress toward meeting the aspirational SLO, but you won’t be in a perpetual state of emergency.

• Data-driven analysis of error budgets helps prioritize fixing frequent release failures over rare hardware issues.
• Services with high performance and low toil may be moved to less intensive support tiers to free up engineering resources.
• An SLO decision matrix provides specific actions based on the intersection of performance, operational toil, and customer satisfaction.
• Mature SLO cultures should transition from technical metrics to modeling critical user journeys that reflect the actual customer experience.
• Measuring complex user journeys often requires advanced techniques like joining log events or client-side instrumentation.
• Not all system interactions are equal, necessitating a grading system for the importance of different user requests.

The numbers prove that addressing the release problem provides much more benefit than investing resources in investigating the server failure.

• Bucketing allows for different SLOs based on customer tiers or request types, such as prioritizing premium users over free ones.
• Responsiveness SLOs should vary based on user expectations, distinguishing between interactive page loads and background tasks like downloads.
• Tracking SLO compliance per individual customer can be noisy due to low request volumes, but aggregate data reveals significant systemic issues.
• Critical dependencies must have reliability guarantees that match or exceed the reliability requirements of the services that rely on them.
• Engineering around unreliable components through caching or graceful degradation is often more effective than relying on theoretical availability math.
• Redundancy across zones rarely provides the calculated mathematical reliability because of shared dependencies and common failure domains.

The probability that both services will experience an outage at the same time is so low that two zones should provide 99.9999% availability. However, this reasoning assumes that both services are wholly independent, which is almost never the case.

• Teams must decide whether to halt releases or enact change freezes when external dependencies cause outages.
• Experimenting with lower reliability, such as intentional latency, can help identify the mathematical relationship between technical metrics and business outcomes.
• Deliberately lowering customer experience is a 'Rubicon' that should only be crossed with a sufficient error budget and careful thought.
• Data from reliability experiments can be misleading if users lack alternatives, as they may stay despite being unhappy until a competitor emerges.
• SLOs and error budgets provide a necessary framework for balancing reliability against the pace of engineering innovation.

To make a choice to deliberately lower the perceived customer experience is a Rubicon to be crossed extremely thoughtfully, if at all.

• Google's Customer Reliability Engineering (CRE) team partners with GCP customers to implement SLO-based reliability frameworks.
• Evernote transitioned from physical datacenters to the public cloud to shift focus from infrastructure maintenance to product engineering.
• The company struggled with a traditional 'ops/dev split' where conflicting goals led to strained relationships and production instability.
• After experimenting with various ownership models, Evernote adopted an SRE approach to balance service delivery with feature evolution.
• The move to SLOs was part of a broader technology revamp aimed at increasing engineering velocity without sacrificing quality for 220 million users.

As we swung wildly between these two goals, the ops and dev teams developed a frustrated and strained relationship.

• Service Level Objectives (SLOs) provide a common frame of reference for operations and development teams, reducing subjectivity in decision-making.
• Evernote transitioned to Google Cloud Platform (GCP) and introduced SLOs to align internal teams and manage their new infrastructure partnership.
• The initial SLO focus was kept simple, prioritizing service availability and content syncing from the customer's perspective.
• The first SLO iteration defined a 99.95% uptime target measured over a calendar month to maintain organizational focus during reviews.
• Measurement is conducted via independent third-party probers (Pingdom) that check service endpoints from multiple geographic locations to avoid false positives.
• A node is only officially marked as 'down' for SLO calculations if two geographically separate probers fail consecutively.

In our experience, an error budget/SLO approach has led both teams to make similar decisions when presented with the same facts, as it removes a good deal of subjectivity from the conversation.

• Evernote treats maintenance windows as downtime in SLO calculations because uninformed users perceive any service interruption as a failure.
• The company uses a monthly review process between Evernote and Google teams to prioritize fixes and allocate resources based on error budget performance.
• Adopting the principle 'perfect is the enemy of good' allowed the team to implement initial SLOs quickly rather than spending months on exhaustive edge cases.
• SLOs are reviewed every six months to ensure they remain relevant and balance desired metrics with technical feasibility.
• The implementation of objective SLOs improved collaboration between operations and development by replacing subjective quality assessments with shared data.
• Evernote actively works to break down the 'virtual wall' between customer and cloud provider by sharing performance characteristics with Google.

By applying an SLO calculation to the problem and removing human subjectivity from the scenario, we were able to better quantify customer impact and reduce our release windows from five to two to minimize customer pain.

• Evernote sought to align its reliability objectives with Google Cloud Platform to ensure shared responsibility for service performance.
• Standard global SLAs often mask localized issues, as Evernote's specific footprint can be lost in a provider's global uptime averages.
• By sharing real-time SLO dashboards, Google provides Evernote with context-specific notifications regarding the exact impact of infrastructure issues.
• Shared SLOs provide a framework for major incidents, triggering rapid mobilization on shared conference bridges when critical thresholds are breached.
• Evernote is evolving its SLO practice from simple uptime to probing individual API calls and client-side metrics for a better view of quality of service.
• Data-driven conversations based on SLOs have improved internal focus and external support, leading to more effective service improvements.

Even when GCP SLO graphs are green (i.e., above 99.95%), Evernote’s view of the same SLO might be very different.

• The Home Depot transitioned from monolithic software packages to a microservices architecture to support 1.5 billion annual transactions.
• The shift introduced a 'freedom and responsibility' culture where developers have full-stack ownership and joint responsibility for operations.
• Service Level Objectives (SLOs) were adopted as a common language to manage dependencies and accountability across small, independent teams.
• Prior to SLOs, the organization struggled with fragmented monitoring tools and 'working backward' from outages, which wasted significant time.
• The company launched the 'FiRE Academy' and used evangelism tactics like road shows and t-shirts to embed reliability concepts into the corporate culture.
• A strategic focus on automation and a common vernacular (the VALET acronym) was implemented to reduce friction in adopting these new metrics.

This approach gives developers freedom to push code when they want, but also makes them jointly responsible for the operations of their service.

• The organization automated the collection of Service Level Indicators (SLIs) to simplify the eventual creation of Service Level Objectives (SLOs).
• Management established annual goals for development managers to ensure SLO adoption and measurement across all services.
• Initial analysis revealed that existing metrics were inconsistent, poorly named, and lacked sufficient data across different teams.
• The team prioritized availability and latency for API calls as the primary SLOs to facilitate reliability contracts between microservices.
• Infrastructure utilization was excluded from the SLO framework because users are unaffected by it as long as performance and capacity remain stable.
• Traffic volume SLOs were implemented to ensure systems were sized correctly for retail peaks like Black Friday, despite being driven by external user behavior.

The primary (and often only) way we measured the reliability of the applications deployed to our stores was by tracking the number of support calls our internal support desk receives.

• The organization transitioned from arithmetic averages to percentile-based targets (90th, 95th, and 99th) to better measure service performance.
• Error tracking was standardized using HTTP response codes, distinguishing between client-side 4xx errors and service-side 5xx errors for SLO purposes.
• The 'VALET' acronym was created to represent five key metrics: Volume, Availability, Latency, Errors, and Tickets.
• The framework incorporates 'Tickets' as a metric for manual intervention, effectively measuring the 'software operation level' of a service.
• A comprehensive evangelism strategy was launched, including weekly executive reports, internal workshops, and a dedicated reliability blog.
• The shift to SLOs supported a 'freedom and responsibility' culture where developers became accountable for the operational health of their own software.

Armed with an easy-to-remember acronym, we set out to evangelize SLOs to the enterprise.

• The VALET framework gained internal popularity through marketing campaigns and inclusion in annual performance reviews.
• To scale beyond manual spreadsheets, the team developed 'TPS Reports' to automate data collection via Google Cloud's BigQuery.
• Automation allowed for real-time querying of service metrics and integration with chatbots for instant status reporting.
• A dedicated VALET service was created to track SLO trends at daily, weekly, and monthly granularities across disparate monitoring platforms.
• The system was designed with an integration layer to collect data from both cloud-native and on-premises applications.
• While the setup offers flexibility in monitoring tools, it creates a disconnect between real-time alerting thresholds and long-term SLO trending.

We called this framework TPS Reports, a play on the term we used for volume and performance testing (transactions per second), and, of course, to poke fun at the idea that multiple managers might want to review this data.

• The VALET Dashboard enables service registration, SLO objective setting, and customized metric tracking across five key categories.
• Automated reporting fostered a new culture of regular SLO reviews, allowing teams to adjust unrealistic targets or prioritize reliability fixes.
• Standardized automation led to rapid organizational adoption, scaling from 50 to 800 tracked services within a single year.
• The VALET framework proved flexible enough to be adapted for batch processing applications and destructive chaos engineering tests.
• Future goals include implementing a formal error budget culture where feature velocity is paused to address reliability breaches.

After tracking SLOs for about 50 services at the beginning of the year, by the end of the year we were tracking SLOs for 800 services.

• The organization aims to refine VALET metrics by differentiating between specific service operations and individual consumers rather than tracking at the service level.
• Future monitoring goals include measuring end-user latency to account for external factors like third-party tags, internet speed, and CDN caching.
• Automation is being integrated to use VALET data as a safety gate for verifying service health before proceeding with regional deployments.
• Product managers are being empowered to set SLOs by translating technical metrics into business-relevant tiers based on service criticality.
• The Home Depot successfully scaled from zero to 800 SLO-supported services in under a year through executive buy-in and a clear incentive structure.
• Implementing an SLO culture is an ongoing process that bridges the gap between product development and operations to improve customer experience.

We’d like to emphasize that The Home Depot is a traditional enterprise; if we can introduce such a large change successfully, you can too.

• SLO implementation is highly adaptable and can be tailored to unique organizational environments beyond Google's specific model.
• Metrics and structured logging are identified as the two most essential data sources for Site Reliability Engineering needs.
• Monitoring serves five core purposes: alerting, incident diagnosis, visual information display, long-term trend analysis, and experimental comparison.
• Data freshness is critical for incident response, as delays of more than five minutes can lead to false correlations or delayed paging.
• A robust monitoring system must support long-term data retention to enable the analysis of system growth and historical trends.
• Precomputing answers to common queries through new time series can mitigate latency issues when retrieving vast amounts of data.

Data more than four to five minutes stale might significantly impact how quickly you can respond to an incident.

• Aggregated data is sufficient for growth planning, but detailed metrics help identify historical patterns at the cost of storage.
• Monotonically incrementing counters are the ideal building blocks for calculating request rates and SLO burn-based alerting.
• Statistical functions like percentiles are superior to arithmetic means because they reveal latency outliers that averages mask.
• Dashboards should be tailored to specific audiences, using consistent formats like heatmaps or histograms for clear communication.
• Effective alerting requires classification by severity and suppression logic to prevent notification fatigue during widespread outages.
• Ad hoc drill-downs across machine types or server versions allow teams to find correlations during real-time troubleshooting.

A monitoring system that supports computing percentiles will let you see if 50%, 5%, or 1% of your requests are too slow, whereas the arithmetic mean can only tell you—without specifics—that the request time is slower.

• Monitoring systems primarily rely on two data sources: numerical metrics for real-time visibility and structured logs for granular event records.
• Metrics are ideal for rapid alerting and high-level dashboards because they provide near real-time data with lower overhead.
• Logs offer high-fidelity information necessary for root-cause analysis and accurate reporting but often involve processing delays.
• Google recommends converting specific log events into counter metrics to centralize alerting and improve management efficiency.
• A case study on App Engine shows that moving HTTP status codes from logs to metrics significantly reduced debugging complexity by providing scale and context.

We tend to use logs to find the root cause of an issue, as the information we need is often not available as a metric.

• Adding HTTP status code labels to metrics allowed for granular graphing and more accurate alerting thresholds.
• A team managing 50 services struggled with complex, service-specific log processing scripts that were difficult to maintain.
• The lack of consistency between metrics-based alerts and log-based SLO data led to slow response times and manual triage.
• A unified library was developed to determine user impact at request time, exporting the decision simultaneously to both logs and metrics.
• Standardization reduced false positives and allowed the team to reuse tooling across diverse services, increasing scalability.
• The team faced a trade-off between the speed of metrics and the rich, entity-specific data found only in logs during incident investigation.

These scripts were hard to maintain and also used data that wasn’t available to the metrics-based monitoring system.

• The team transitioned from manual log queries to automated scripts to reduce cognitive load and improve response times.
• Monitoring configurations should be treated as code to enable version control, peer reviews, and easier rollbacks.
• Centralized monitoring frameworks foster consistency, allowing engineers to switch teams and collaborate on debugging more effectively.
• Automating basic metric collection ensures that every new component has immediate visibility without manual setup.
• Loose coupling between collection, storage, and visualization components allows the monitoring stack to evolve alongside the business.
• Stable interfaces between monitoring components make it easier to swap out individual tools for better alternatives.

Treating system configuration as code and storing it in the revision control system are common practices that provide some obvious benefits: change history, links from specific changes to your task tracking system, easier rollbacks and linting checks, and enforced code review procedures.

• Modern monitoring has shifted from monolithic systems like Zabbix to modular stacks involving Prometheus, InfluxDB, and Grafana.
• Decoupling dashboarding from alerting allows for smoother migrations between monitoring backends and provides a unified view of disparate data sources.
• SLI metrics are essential for identifying SLO violations but often lack the granularity required to diagnose the underlying cause of an issue.
• Effective debugging requires monitoring 'intended changes' such as binary versions, command-line flags, and dynamic configuration timestamps.
• Standardization efforts like OpenMetrics and OpenCensus are unifying how software exposes metrics across different programming languages.
• Visualizing deployment changes directly on dashboards is more efficient for correlation than searching through CI/CD logs during an outage.

When you’re trying to correlate an outage with a rollout, it’s much easier to look at a graph/dashboard linked from your alert than to trawl through your CI/CD system logs after the fact.

• Monitoring direct dependencies is essential because their failures or changes directly impact your service's performance.
• Standardizing metrics through lower-level RPC client libraries ensures consistency and automatic monitoring for new dependencies.
• Opaque APIs with generic methods like 'Query' require specialized metrics or architectural rewrites to provide meaningful signals.
• Saturation monitoring must track both hard-limited resources like RAM and soft-limited resources like thread pools or log volume.
• Language-specific resources, such as Java's heap or Go's goroutines, require specialized tracking to prevent performance degradation.
• Metrics should be designed with purpose, ideally showing dramatic changes only when the system enters a problematic state.

Resist the temptation of exporting a handful of metrics just because they are easy to generate.

• Debugging metrics should provide specific insights into system behavior when alerts are triggered rather than just signaling a problem.
• Monitoring and alerting code should ideally be subject to the same rigorous testing standards as production software development.
• Google utilizes a three-tiered testing approach covering binary reporting, monitoring configurations, and alerting routing.
• If synthetic testing is unavailable, engineers should use a running system that exports known metrics to validate that alerts will fire correctly.
• SREs must be intimately familiar with monitoring features to effectively identify abnormal behavior during emergencies.
• Effective monitoring strategies are context-dependent and require iterative refinement to ensure they remain visible and useful.

It’s very likely that your alerting rules will not fire for months or years after you configure them, and you need to have confidence that when the metric passes a certain threshold, the correct engineers will be alerted with notifications that make sense.

• Service Level Objectives (SLOs) provide the highest-quality signal for determining when on-call engineers should intervene to protect the error budget.
• Effective alerting strategies are evaluated based on four key attributes: precision, recall, detection time, and reset time.
• Precision and recall ensure that alerts are meaningful and that no significant events are missed, while detection and reset times impact budget consumption and engineer fatigue.
• The text outlines a progression of six alerting methods, moving from simple, inadequate thresholds to complex, high-fidelity solutions.
• A basic but flawed approach involves alerting when the error rate over a short window, such as 10 minutes, exceeds the total SLO threshold.

Precision is 100% if every alert corresponds to a significant event.

• Short alert windows provide fast detection for total outages but suffer from low precision, potentially triggering hundreds of unnecessary alerts per day.
• Increasing the alert window size improves precision by ensuring an event consumes a significant portion of the error budget before notifying staff.
• Longer alert windows suffer from poor reset times, where an alert may continue to fire for hours or days after the underlying issue is resolved.
• Using a fixed duration parameter is discouraged because it does not scale with incident severity, leading to poor recall and delayed detection of critical outages.
• Fluctuating metrics can reset duration timers, meaning a service that is frequently failing may never actually trigger an alert.

Taking this example to an extreme, you could receive up to 144 alerts per day every day, not act upon any alerts, and still meet the SLO.

• Standard window-based alerts can fail to trigger during intermittent 100% error spikes that significantly deplete the error budget.
• Burn rate is defined as the speed at which a service consumes its error budget relative to the SLO window.
• Alerting based on burn rate improves precision and detection time by focusing on the percentage of budget spent rather than raw error rates.
• A burn rate of 1 exhausts the entire 30-day error budget exactly at the end of the window, while higher rates like 1,000 exhaust it in minutes.
• Implementing multiple burn rate thresholds allows for tiered responses, such as paging for rapid exhaustion and ticketing for slower budget drains.
• Recommended configurations include paging for a 2% budget spend in one hour and ticketing for a 10% spend over three days.

A series of 100% error spikes lasting 5 minutes every 10 minutes never triggers an alert, despite consuming 35% of the error budget.

• Multiple burn rates allow SREs to prioritize alerts based on how quickly an error budget is being exhausted.
• High burn rates that threaten the budget within hours trigger immediate pages, while slower burns generate tickets for the next business day.
• Using multiple windows improves precision and recall but increases the complexity of managing various thresholds and reset times.
• Alert suppression is necessary in multi-burn-rate systems to prevent a single incident from triggering multiple redundant notifications.
• The 'Multiwindow, Multi-Burn-Rate' approach adds a short window (typically 1/12th of the long window) to verify that an error spike is still active.
• Short windows significantly improve alert reset times, stopping the notification shortly after the issue is resolved rather than waiting for a long window to clear.

We can enhance the multi-burn-rate alerts in iteration 5 to notify us only when we’re still actively burning through the budget—thereby reducing the number of false positives.

• Multi-window and multi-burn-rate alerting provides a flexible framework for balancing precision and recall in SLO monitoring.
• Alerting parameters are categorized by severity, with 'Page' alerts triggered by high burn rates and 'Ticket' alerts by sustained lower burn rates.
• Low-traffic services present a unique challenge where a single failed request can disproportionately consume the error budget and trigger false alarms.
• To manage low-traffic systems, organizations can generate artificial traffic to provide a consistent signal for monitoring tools.
• Alternative strategies for low-traffic services include combining smaller services or modifying the product to reduce the impact of individual failures.

If a system receives 10 requests per hour, then a single failed request results in an hourly error rate of 10%.

• Synthetic traffic can provide monitoring signals but often fails to cover all states or hide real user errors.
• Combining related microservices into a single monitoring group can reduce false positives and increase signal precision.
• Modifying client behavior with retries and fallbacks can mitigate the impact of individual failures on the error budget.
• Lowering the SLO or extending the time window ensures that engineers are only paged for truly significant events.
• Choosing between these strategies requires balancing product expectations against the operational cost of low-signal alerts.

Additionally, if an issue affects real users but doesn’t affect artificial traffic, the successful artificial requests hide the real user signal, so you aren’t notified that users see errors.

• Low availability targets (e.g., 90%) may prevent standard alerts from firing because even a total outage consumes the error budget too slowly for typical thresholds.
• Extremely high availability targets (99.999%) offer a budget of only 26 seconds per month, which is often shorter than monitoring collection intervals.
• Defending high-reliability SLOs requires architectural solutions like canary deployments rather than just reactive alerting to prevent total budget exhaustion.
• Customizing alerting parameters for every individual microservice creates unsustainable cognitive load and operational toil.
• Scaling alerting is best achieved by grouping request types into standardized buckets based on criticality and latency requirements.

A 100% outage for a service with a target monthly availability of 99.999% would exhaust its budget in 26 seconds—which is smaller than the metric collection interval of many monitoring services.

• Categorizing request classes into specific availability and latency buckets balances user happiness with operational simplicity.
• Effective alerting strategies should focus on actionable threats to the error budget rather than every minor threshold breach.
• Toil is defined as repetitive, manual, and predictable work that can consume a team if not strictly bounded.
• Google maintains a 50% limit on operational work to ensure SREs have time for project-based system optimization.
• Identifying the characteristics of toil, such as manual execution and repetitiveness, is the first step toward automation.

A problem space that once seemed intractable (or too risky) will become feasible once you get comfortable with “letting the robots do the work.”

• Toil is characterized by reactive, manual work that lacks enduring value and fails to prevent future occurrences of the same issue.
• Operational tasks often scale linearly with infrastructure size, creating a burden that grows alongside the fleet unless automated.
• Persistent toil leads to career stagnation, burnout-induced turnover, and the erosion of team morale by stifling creativity.
• Engineers should prioritize root-cause fixes and software patches over scripts that merely mask symptoms or automate remediation.
• The true cost of toil is the displacement of high-value engineering work that requires human judgment and critical thinking.

Toil can slowly deflate team morale. Time spent working on toil is generally time not spent thinking critically or expressing creativity.

• Fixing the root cause of a technical issue is more valuable than repeatedly treating symptoms, as it removes the need for manual intervention.
• Objective measurement of toil is necessary because intuition and experience are not repeatable, transferable, or consistent across a team.
• Toil reduction projects should be prioritized based on a cost-benefit analysis to ensure maximum return on engineering investment.
• Automation offers intangible benefits beyond time savings, including improved morale, reduced burnout, and fewer human-error outages.
• The first steps in measuring toil involve identifying the specific work and selecting a universal unit of measure, such as minutes or hours.

Instead of the short-term endorphin hit of fixing those machines every morning, I now had the more cerebral pleasure of knowing that I’d fixed the problem properly.

• Toil can be measured through objective units like patches, tickets, or hardware operations to track the success of reduction efforts.
• Business processes are the most common source of toil, often disguised as necessary human-to-machine interfaces via ticketing systems.
• Ticket-driven toil is insidious because it successfully delivers results while quietly accumulating and dispersing effort across a team.
• Production interrupts act as time-sensitive janitorial tasks, such as manual restarts or disk clearing, that divert attention from high-value work.
• Release shepherding and technology migrations frequently generate toil through manual configuration changes and one-off scripting.

Toil, like a crumbling bridge or a leaky dam, hides in the banal day to day.

• Migration work often masquerades as project work but frequently meets the criteria for toil due to its repetitive nature and lack of new business value.
• Cost engineering and capacity planning involve significant toil, ranging from contract negotiations to optimizing workloads against specific cloud billing models.
• The complexity of distributed microservice architectures introduces opaque failure modes that require manual, ad hoc troubleshooting when sophisticated tracing tools are absent.
• System availability decreases mathematically with every new dependency, where a service with nine dependencies can drop from four 9s to three 9s of availability.
• Effective toil management requires a formal strategy to identify, quantify, and eliminate repetitive tasks at their source.
• While the nuances of toil vary between organizations, common reduction tactics include refactoring tooling for cheaper resources and addressing brittle architectures.

A four 9s service that adds nine critical four 9s dependencies is now a three 9s service.

• Adopt a data-driven approach to quantify the return on investment for toil reduction projects.
• Eliminate toil at the source by collaborating with product developers to create operationally friendly software.
• Consider rejecting or delaying toil-intensive tasks to analyze the cost of inaction or to process them in efficient batches.
• Leverage Service Level Objectives (SLOs) to prioritize service health over individual operational tasks.
• Implement human-backed interfaces as an interim step toward full automation to map complex domains without overengineering.
• Transition to self-service methods to reduce manual workload by up to 90% for common requests.

In our experience, while it may seem counterproductive, rejecting a toil-intensive task should be the first option you consider.

• Transitioning to self-service interfaces like APIs or web forms reduces the manual burden of provisioning and configuration.
• Securing management support is essential because toil reduction requires a short-term trade-off in feature development for long-term engineering health.
• Toil reduction should be promoted as a feature by coupling it with business goals like security, scalability, or reliability.
• Adopting a 'cattle' philosophy over 'pets' increases uniformity, making diverse production environments easier to manage through interchangeable units.
• Automation should be implemented incrementally, starting with high-priority items and using gained time to refine the system.
• Defensive software is critical when automation possesses administrative powers to prevent large-scale outages.

Teams are free to choose their own approaches, but they have to own the toil generated by unsupported tools or legacy systems.

• Automation requires defensive input validation and safeguards equivalent to human-perceived alerts to ensure safety.
• Effective automation should not literally transcribe human workflows but rather simplify and decompose them into reusable software libraries.
• Engineers should leverage open source and third-party tools to reduce development costs for one-off migrations or partial automation.
• Continuous improvement of automation relies on both user feedback and quantitative metrics like latency, error rates, and time saved.
• Legacy systems often persist as 'black boxes' that require significant toilsome ritual to maintain due to their fragility and lack of documentation.
• Automation should be designed to fail safely by defaulting back to human operators when encountering unsafe or ambiguous conditions.

They tend to operate like a magical black box in that they “mostly work,” but few people understand how they work.

• Encapsulation acts as a stopgap measure by wrapping brittle legacy systems in modern APIs and automation to lower technical debt interest.
• Incremental replacement requires defining common interfaces and using release engineering techniques like canarying to safely migrate users.
• Legacy system specifications are often defined by historical usage rather than documentation, necessitating production-sized datasets for testing.
• Retirement involves shifting custodial ownership of remaining legacy components to the final straggling users to align business incentives.
• Google's case studies demonstrate that infrastructure reaching a point of sublinear scaling requires aggressive toil mitigation to remain viable.
• Effective toil reduction in large organizations requires overcoming institutional inertia through persistence and reevaluating entrenched processes.

This tactic is still avoidance, but is a bit like refinancing high-interest technical debt into low-interest technical debt.

• The transition to software-defined networking and Clos topologies dramatically increased the number of switches, making manual repairs unsustainable.
• While modern network architectures reduced the impact of individual component failures, the sheer volume of issues created an overwhelming backlog for engineers.
• The complexity of new layouts led to technician confusion regarding switch stages and the potential user impact of shutting down specific hardware.
• Manual repair workflows—consisting of draining traffic, repairing hardware, and undraining—became textbook examples of repetitive, low-value toil.
• Human error during manual repairs, such as reintroducing unconfigured switches due to multitasking, highlighted the need for automated systems.
• Google targeted line-card repairs as the first stage for automation to manage the scale of their growing datacenter infrastructure.

This growth changed the nature of datacenter manual repairs from occasional and interesting to frequent and rote—two signals of toil.

• The team faced a scaling crisis where manual repairs could not keep pace with hardware failure volumes, leading to human error and fabric instability.
• Engineers lacked a programmatic way to prioritize serious failures or perform risk assessments before isolating network capacity.
• A new automation framework was developed to handle triage, traffic draining, and validation without requiring constant engineer intervention.
• The system implemented a 'strike policy' where software first attempts a reboot and only escalates to physical replacement on the second failure.
• Technicians now interact with a UI that uses safety interlocks, such as a red 'stop' indicator, to prevent hardware work until traffic is safely diverted.
• Automated risk assessment eliminated the primary source of outages caused by manual errors during device isolation.

Every manual risk assessment was an opportunity for human error that could result in an outage.

• The transition from Saturn to Jupiter fabrics required a six-fold increase in scale, necessitating more ambitious automation to handle the volume of failures.
• Engineers moved from a manual 'push-button' repair model to a fully automated workflow that drains switches without human intervention.
• The new system incorporates automated configuration, verification, and link testing (BERT) before returning hardware to service.
• Technicians are now only involved in the repair process when hardware replacement is strictly necessary or when automated power-cycles fail.
• By automating toil, the engineering team was able to shift focus from maintenance to developing next-generation datacenter topologies.

The new system freed the engineering team from a large volume of toilsome work, giving them more time to pursue productive projects elsewhere.

• Automation simplified Jupiter switch repairs by treating memory errors as hardware failures rather than troubleshooting them individually.
• The transition from manual engineering decisions to automated config-based symptoms significantly reduced operational toil.
• Poorly designed user interfaces, such as the 'Prep component' button, introduced risks by providing insufficient feedback and failing to sync with switch states.
• Relying on human expertise proved dangerous when technicians bypassed slow automated processes, leading to network congestion and packet loss.
• Building automation from reusable, modular components allowed for easier adaptation across successive generations of network fabric.

In one particularly high-impact episode, a technician decided to expedite the “press button and wait for results” experience by initiating concurrent drains on every line card waiting for repairs at the datacenter, resulting in network congestion and user-visible packet loss.

• Overanalyzing technical problems can lead to 'analysis paralysis,' as seen in a three-year delay to automate memory error repairs while seeking perfect diagnostics.
• Imperfect automation is often superior to manual processes, especially when applied to non-critical systems like management links that don't carry customer traffic.
• Automation requires long-term maintenance and continuity planning, as legacy systems often outlive their expected lifespans due to hardware shortages.
• Implementing defense-in-depth through secondary safety checks and hard limits prevents automation from causing large-scale outages during atypical events.
• Securing management support and establishing error budgets for automation projects is essential to survive the inevitable failures during early deployment.

We spent nearly three years (2012–2015) collecting data on over 650 discrete memory error problems before realizing this exercise was probably overkill, or at least shouldn’t block our repair automation project.

• Simplifying or decommissioning complex systems is often more effective than attempting to automate inherently flawed workflows.
• Automation must be designed to avoid creating 'new toil,' such as generating unnecessary tickets that burden other teams.
• Google's Corp Data Storage team faced significant toil managing a fleet of Netapp appliances for home directories and team shares.
• Legacy storage protocols like NFS/CIFS proved inefficient for a global workforce due to latency issues and high operational costs.
• The transition to modern alternatives like Google Drive and Cloud Storage was driven by the need for better security and reduced manual intervention.
• Successful toil reduction requires seeking feedback from those directly involved in the work to identify UI flaws and safety gaps.

Make sure your automation doesn’t create new toil—for example, by opening unnecessary tickets that need human attention.

• Google's Corporate Data Services (CDS) used a tool called Moonwalk to analyze employee usage patterns and validate the need for alternative storage solutions.
• The decision to migrate away from specialized hardware was driven by high operational toil, latency issues, and the need to improve Google's own cloud products.
• No single replacement existed; instead, the team fragmented use cases across specialized tools like G Suite, Colossus, and internal systems like 12x20.
• The decommissioning process was a multiyear, iterative effort divided into four distinct projects: Moira, Tekmor, Migra, and Azog.
• A critical first step in the strategy was discouraging new adoption and targeting the least active user groups to build confidence in the migration tools.
• The project emphasized user experience and clear communication to ensure employees understood how to archive or migrate their data effectively.

It’s much more painful to take something away from users than never offer it in the first place.

• The Moonwalk system was developed to analyze access patterns across 2.5 billion files and 300 terabytes of data to inform business decisions.
• A self-service portal called Moira was built using Python and Flask to handle complex user scenarios like litigation holds and temporary leaves without manual ticketing.
• The team prioritized high-accuracy communication to avoid 'false positives' or 'false negatives' that would damage credibility and increase support toil.
• A 'low and slow' two-year engineering approach allowed a lean team of three to build tools as needed while minimizing disruption to 60,000 users.
• The project required substantial software development to bridge gaps between existing storage systems and the new archiving requirements.

False positives (erroneously reporting action required) or false negatives (failing to notify a user that you were taking something away) were both unacceptable, and errors here would mean extra work in the form of lost credibility and customer service.

• The Moira team successfully reduced user count from 65,000 to 50 by moving from a generalized filesystem to application-specific solutions.
• The project traded general flexibility for improved scalability, latency tolerance, and a BeyondCorp security model.
• Toil reduction is most effective when paired with broader business justifications, such as security upgrades, rather than just operational ease.
• Self-service interfaces and 'engineer behind the curtain' automation allow for human fallback while maintaining high-velocity migration.
• The team practiced 'melting snowflakes' by forcing nonconforming data to match automation expectations to achieve zero-touch processing.

At Google, this practice of changing reality to fit the code rather than the other way around is called 'buying the gnome.'

• Organizational nudges like codelabs and cookbooks can reduce manual onboarding by teaching users self-service best practices.
• Toil grows linearly with service complexity and scale, making automation the gold standard for maintaining engineering efficiency.
• Deciding to eliminate toil requires a calculated ROI analysis to ensure the cost of automation doesn't exceed the manual labor saved.
• SRE teams must be relentless in identifying and quantifying toil to prevent it from rapidly consuming all available engineering time.
• Simplicity in software and architecture is a primary goal for SREs because simple systems are inherently more reliable and easier to repair.

A complex system that works is invariably found to have evolved from a simple system that worked.

• Cyclomatic code complexity (CCN) is a well-established metric for code, but it fails to capture the nuances of necessary versus accidental complexity.
• Measuring system-level complexity by counting entities and communication paths is impractical due to exponential growth in large systems.
• Practical proxies for system complexity include training time for new members, explanation time for architecture, and administrative diversity in configurations.
• Hyrum's Law suggests that as systems age, users depend on every implementation detail, making the system increasingly fragile.
• Complexity acts as an economic externality, where the cost of a change is often borne by those maintaining the system rather than those who introduced it.
• SREs are uniquely positioned to champion end-to-end simplicity because their role requires a holistic view of the entire production environment.

Frequently, the cost of complexity does not directly affect the individual, team, or role that introduces it—in economic terms, complexity is an externality.

• SREs provide a holistic view of production systems that product developers often lack due to their focus on narrow subsystems.
• Maintaining canonical system diagrams is essential for onboarding new engineers and establishing a common vocabulary for troubleshooting.
• Abstract data structures like key/value bags can create hidden complexity compared to structured types like Protocol Buffers which force upfront design.
• The 'Omega' project case study illustrates the danger of attempting to replace complex legacy systems with clean-slate designs.
• Wholesale system migrations often fail to account for the cost of maintaining two systems simultaneously while the original continues to evolve.
• SREs should review design documents to identify where added complexity can be simplified before implementation.

In practice, the grass isn’t always greener.

• Omega's design concepts influenced both the evolution of Borg and the creation of the open-source Kubernetes system.
• Rewriting a system requires comparing the potential outcome against the result of investing equal effort into improving the existing system.
• Simplification is a form of efficiency that saves engineering time and cognitive load rather than just hardware resources.
• Leadership must prioritize and celebrate code removal and simplification projects as much as new feature launches to ensure they are career-beneficial.
• Specific architectural risks like amplification from retries and cyclic dependencies can be identified through system diagramming.
• Maintaining a small group of SREs with a holistic view of the entire stack helps prevent fragmentation and drives cross-system simplification.

As Dijkstra said, 'If we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent.”'

• Google's Display Ads business faced complexity issues after integrating multiple acquisitions like DoubleClick and AdMob.
• Interconnected backends created 'system smells' such as redundant lookups and the risk of infinite query loops.
• SRE teams drove standardization by drafting uniformity standards for data copying, lookups, and monitoring templates.
• Consolidating server backends allowed for unified auction logic, reducing the need for request rewriting and redundant processing.
• The initiative demonstrated that well-defined standards provide a blueprint for removing complexity that gains managerial support.
• A separate case study highlights the move toward a shared microservices platform to reduce the overhead of bespoke production stacks.

At one point, we added tests to make sure we had removed all infinite loops in the query flow.

• A common platform with automated configurations improves reliability and simplifies debugging across diverse service teams.
• The shift to managed microservices empowers developers to handle their own releases and monitoring through unified workflow tools.
• High-quality platform standards enable tiered SRE engagement, allowing developers to run hundreds of services with minimal direct SRE oversight.
• Standardization is a long-term investment that succeeds by providing incremental productivity wins rather than requiring massive, delayed-payoff refactors.
• The pDNS case study highlights the danger of circular dependencies, where a service unintentionally relies on itself for startup.
• Implementing explicit whitelists for service communication prevents accidental dependency bloat and ensures a more reliable system path.

We were like cave dwellers who could only light fires by running with a torch lit from the last campfire.

• SREs are uniquely positioned to identify and mitigate system complexity due to their end-to-end understanding of architecture.
• Simplicity is a core goal because it directly correlates with system reliability and ease of operation.
• SRE leadership must explicitly empower and reward teams for fighting the natural creep toward complexity as systems evolve.
• Operational work is categorized into four main areas: on-call, customer requests, incident response, and postmortems.
• SRE practices focus on applying software engineering solutions to automate and scale tasks that are traditionally manual operations.
• The transition from SRE foundations to practices introduces Non-Abstract Large System Design (NALSD) as a critical skill.

Systems inevitably creep toward complexity as they evolve, so the fight for simplicity requires continuous attention and commitment—but it is very much worth pursuing.

• SRE work is categorized into three main buckets: operational tasks, project work, and administrative overhead.
• Google mandates that SREs spend at least 50% of their time on project work to prevent burnout and ensure sustainable engineering.
• Documentation is not a standalone activity but an integrated facet of both operational and project tasks.
• A feedback loop should exist where operational issues directly inform strategic projects aimed at reducing toil and increasing reliability.
• Teams must avoid 'operational underload,' where too little hands-on work leads to a loss of service knowledge and expertise.
• Regular reviews using defined rubrics are essential for tracking team health and maintaining the ideal balance between work types.

In this situation, engineers might start to forget crucial aspects of the service they are responsible for.

• On-call responsibilities involve being available to diagnose, mitigate, and fix production incidents with appropriate urgency.
• While Google is a large organization, its SRE principles for on-call rotations are designed to be applicable to companies of any size or maturity level.
• Common challenges in on-call rotations include high pager volume, knowledge gaps due to turnover, and confusion over roles between SRE and DevOps.
• Effective on-call management requires balancing mathematical logistics with human factors like incentives and team dynamics.
• Not all SRE teams require on-call rotations, but they remain a primary method for handling escalations that have not yet been automated.

Our on-call engineer gets paged about a hundred times in a typical 24-hour shift. A lot of pages get ignored, while the real problems are buried under the pile.

• Google SRE teams prioritize a balance where at least 50% of time is dedicated to project work to prevent burnout and address root causes.
• The target pager load is limited to two incidents per 12-hour shift to ensure engineers have sufficient time for thorough follow-up and recovery.
• Psychological safety is considered vital, supported by clear escalation paths and procedures to mitigate the high stress of being on-call.
• Compensation for out-of-hours work is provided through time-off or cash, structured to prevent engineers from overworking for financial gain.
• A case study of a new Mountain View SRE team shows that even junior teams can successfully onboard services by focusing on automation and efficiency.
• Effective on-call rotations require adequate staffing levels to maintain service continuity across different time zones without sacrificing project velocity.

At Google, the overall goal of being on-call is to provide coverage for critical services, while making sure that we never achieve reliability at the expense of an on-call engineer’s health.

• The team utilized a comprehensive checklist of focus areas, including traffic draining and rollback procedures, to ensure readiness before going on-call.
• Training combined self-directed research, hands-on 'starter projects', and impromptu peer-led knowledge sharing sessions.
• Engineers were coached to explore services from first principles rather than relying solely on years of specific expertise.
• The 'Wheel of Misfortune' role-playing exercises allowed the team to practice debugging real-world incidents in a safe, collaborative environment.
• Clear operational guidelines were established for shift handoffs, prioritizing user impact mitigation over immediate root cause analysis.

The team also held lab sessions to perform common debugging and mitigation tasks to help everyone build muscle memory and gain confidence in their abilities.

• Playbooks provide essential instructions for responding to automated alerts, reducing stress and human error while improving recovery times.
• There is a strategic tension between maintaining general playbooks for flexibility and detailed, step-by-step guides for speed and consistency.
• Teams should prioritize automating deterministic tasks rather than letting playbooks become static lists of manual commands.
• Structured documentation and shadowing programs create psychological safety, allowing new engineers to act confidently during high-pressure incidents.
• Investing in onboarding materials and architecture diagrams facilitates smoother knowledge transfer during global team transitions.

If your playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, we recommend implementing automation.

• The transition from on-prem datacenters to Google Cloud Platform required a massive migration of 3.5 PB of data and thousands of servers in just 70 days.
• Legacy monitoring focused on low-level hardware failures, which became obsolete in a cloud environment characterized by live migrations and transient network latency.
• The team shifted focus toward Service Level Objectives (SLOs) and high-level indicators like API responsiveness rather than granular infrastructure metrics.
• Adopting a holistic monitoring approach reduced the time spent chasing transient issues, leading to improved engineer sleep and job satisfaction.
• Alerts were restructured into a three-tier priority system (P1, P2, P3) to maintain a high signal-to-noise ratio for a small, scrappy engineering team.
• The new system ensures that only immediately actionable, SLO-impacting events trigger a page to the on-call engineer.

In a world of live migrations and network latency, we needed to take a much more holistic approach to monitoring.

• Incident response is tiered by severity, with P1 events triggering a full incident management team including a manager, scribe, and communications lead.
• On-call engineers are explicitly freed from project work to focus on immediate remediation and identifying tooling gaps, fostering a cycle of continuous improvement.
• Monthly service review meetings use SLOs and 'pager budgets' as barometers for team health and technical accountability.
• Collaboration with Google's Customer Reliability Engineering (CRE) team reduces MTTR by providing expert insight into cloud abstraction layers during outages.
• A lightweight on-call process allows the team to shift focus from constant firefighting to strategic projects like platform improvement and production readiness.

One of the benefits of keeping our process lightweight is that we can explicitly free the on-call from any expectations of project work.

• Pager load is defined as the number of paging incidents an engineer receives over a typical shift, which can lead to burnout if not strictly managed.
• Engineers should only be required to respond within minutes for revenue-impacting outages, while less severe issues should be handled via tickets or automated repair.
• A team in 'operational overload' often lacks the time to find root causes, leading to a cycle of temporary mitigation and high staff turnover.
• Even teams with mature monitoring and symptom-based alerting can fail if they take on too many dependencies from external developer groups.
• Management intervention is necessary when pager budgets are consistently exceeded to prevent the loss of experienced engineers.

Members of the team heroically responded to the daily onslaught of pages but couldn’t keep up; there simply was not enough time in the day to find the root cause and properly fix the incoming issues.

• The team faced high operational burden from alerts outside their control, such as network congestion requiring escalation.
• Simultaneous feature delivery and migration from legacy infrastructure to a new cluster management system increased the rate of change and pager load.
• Management approved a dedicated project to reduce pager load by analyzing three main factors: production bugs, alerting, and human processes.
• Production bugs are categorized as preexisting or new, with the latter often reaching production despite automated testing.
• Strategies to mitigate preexisting bugs include reducing system complexity, regular software updates, and destructive testing like Chaos Monkey.
• The speed of bug identification and mitigation is a critical metric for managing the overall health of production systems.

The team’s services were subject to an unprecedented rate of change, and the changes themselves caused a significant portion of the on-call load.

• Implement a rigorous post-incident process by asking how every production bug could have been detected during preproduction testing.
• Prioritize load testing and synthetic traffic in staging environments to catch bugs that only manifest under specific request volumes or mixes.
• Adopt a 'detect, roll back, fix' strategy to maintain a low tolerance for new bugs, ensuring releases are frequent enough to make rollbacks inexpensive.
• Prioritize fixing existing production bugs over delivering new features to reduce pager load and simplify the identification of new issues.
• Utilize error budgets to halt feature development when SLO violations occur, shifting focus toward system stabilization.
• Minimize human error by replacing manual production changes with automation driven by human-developed intent configuration.

Follow a 'detect, roll back, fix, and roll forward' strategy rather than a 'detect, continue to roll forward despite identifying the bug, fix, and roll forward again' strategy.

• Manual production changes are prone to human error and should be replaced by automated systems that can validate safety before deployment.
• Identification delay can be minimized by linking alerts directly to monitoring consoles that correlate black-box and white-box data.
• Regular 'Wheel of Misfortune' exercises and small, frequent releases help engineers identify root causes more rapidly during high-pressure events.
• A searchable timeline of all system changes is essential for quickly correlating new bugs with recent code or configuration updates.
• Mitigation speed is improved by prioritizing rollbacks over 'roll forward' fixes, especially when operating under tight error budgets.
• System design should favor feature isolation and avoid irreversible changes like API-incompatible releases to ensure quick recovery.

Generally, it is better to “roll back, fix, and roll forward” rather than “roll forward, fix, and roll forward again.”

• Draining requests away from buggy infrastructure elements can mitigate customer impact in seconds, offering a faster alternative to full rollouts or rollbacks.
• Effective on-call rotations require that all alerts be immediately actionable by a human to prevent alert fatigue and maintain a high signal-to-noise ratio.
• Adopting SLO-based alerting requires a shared commitment across development and maintenance teams to prioritize service level objectives over other work.
• New alerts should undergo a rigorous review process similar to code, including a week of production testing to vet for false positives before becoming paging alerts.
• A team-wide 'pager budget' should be used to govern the introduction of new alerts, ensuring the on-call load remains sustainable and psychologically healthy.

Receiving a page creates a negative psychological impact. To minimize that impact, only introduce new paging alerts when you really need them.

• Root causes often extend beyond technical bugs into team processes like code review standards.
• If a root cause is unknown, engineers should implement additional logging or monitoring to capture data for the next occurrence.
• Dismissing alerts as transient or self-fixing invites future outages and burdens subsequent on-call engineers.
• Point fixes address immediate issues, while systemic fixes use automation to eliminate entire classes of future bugs.
• Prioritizing long-term engineering work can be justified by calculating the break-even point between implementation time and the cost of repeated pages.
• Effective follow-up includes preemptive ticket-based alerts that trigger before a system becomes service-impacting.

Explaining away a page as “transient,” or taking no action because the system “fixed itself” or the bug inexplicably “went away,” invites the bug to happen again and cause another page, which causes trouble for the next on-call engineer.

• Engineers should use every paging event as an opportunity to ask systemic questions about prevention, testing, and alert surfacing.
• SRE and developer teams must prioritize production bugs over project work, reserving specific time for system health improvements.
• Effective bug resolution requires a structured methodology for cataloging and tracking follow-up actions, especially after serious postmortems.
• Manual tracking of on-call load is unsustainable; teams should instead link every paging alert to a specific bug in a tracking system.
• Collecting structured data on incident causes allows for objective reporting on which components or bugs are the most disruptive.
• Data-driven analysis enables teams to prioritize engineering efforts based on actual impact rather than anecdotal evidence.

The answer is simple: collect data!

• Linking structured data to bugs allows for automated prioritization and better support team visibility.
• High-quality data collection requires documented policies and peer-to-peer reinforcement of expectations.
• Teams must monitor pager load trends, such as a 21-day trailing average, to prevent operational overload.
• Regular meetings between SRE and developer teams are essential to address outstanding production bugs.
• On-call shifts should be limited to 12 hours to prevent exhaustion and maintain mental health.
• Sustainability is achieved by splitting shifts or shortening rotations rather than requiring 24-hour availability.

To avoid boiling the frog, it is important to pay attention to the health of on-call engineers over time, and ensure that production health is consistently and continuously prioritized.

• Personal life changes such as parenthood or illness often necessitate a shift in how on-call duties are managed.
• A healthy and equitable on-call rotation requires prioritizing flexibility as a core principle rather than a rigid set of rules.
• Manual scheduling becomes unsustainable as teams grow and must account for diverse constraints like religious requirements and vacations.
• Fairness in scheduling is not defined by uniform distribution but by intelligently meeting the varied preferences of team members.
• Automated tools should be used to rebalance loads and accommodate constraints while ensuring that generated schedules remain stable for planning.
• Teams must establish processes for both urgent short-term swaps, like illness, and non-urgent swaps for personal activities.

“Fairness” doesn’t mean a completely uniform distribution of each type of shift across team members.

• Effective on-call rotations require decentralization and peer review to balance scheduling flexibility with operational safety.
• Strict response SLOs necessitate commute coverage planning to ensure emergencies are handled while engineers are in transit.
• Sustainable rotations require a minimum of six engineers per site for multisite setups and nine for single-site configurations to handle staff reductions.
• Part-time schedules are compatible with on-call duties through modified shift lengths or split daily hours.
• On-call compensation and workload for part-time employees should be proportionally adjusted to maintain a healthy project-to-ops balance.
• Teams must proactively plan for long-term breaks to prevent burnout and accommodate changes in personal circumstances.

In our experience, you need a bare minimum of five people per site to sustain on-call in a multisite, 24/7 configuration, and eight people in a single-site, 24/7 configuration.

• High pager load and time pressure force engineers to rely on intuition and heuristics rather than data-driven reasoning.
• A 'survive the week' culture often emerges when feature developers prioritize new code over operational health and reliability.
• Low morale and friction occur when on-call rotations are bloated with noisy, unactionable alerts that some engineers simply ignore.
• Simply renaming an operations team to SRE is ineffective without a fundamental shift in ownership and responsibility.
• Empowering SREs to own site operations and monitoring rules can resolve conflicts between feature development and reliability.
• Successful teams must transition from a passive 'ops-centric' model to a proactive, shared roadmap for system health.

As one on-call engineer puts it: “I take a quick look at the page subject and know they are duplicates. So I just ignore them.”

• SREs are encouraged to modify code directly to build ownership and technical authority within the production environment.
• Feature developers and SREs act as explicit collaborators on reliability rather than working in silos.
• Google utilizes a 'fun budget' for offsite activities to foster personal bonds that lead to increased professional accountability.
• Stronger team relationships create a protective mentality where engineers are more careful to avoid paging colleagues unnecessarily.
• The SRE role transcends traditional operations by focusing on strategic engineering and automation rather than just day-to-day tasks.
• Effective incident response requires a dual focus on both technical resolution and the coordination of communication flows.

If you’ve spent a little time with that colleague, you’d feel much worse about what happened, and strive to be considerate by being more careful in the future.

• Incident management provides a structured framework to reduce chaos and allow teams to focus on resolution rather than coordination during a crisis.
• Effective response systems are built on four pillars: clear command lines, defined roles, continuous record-keeping, and early incident declaration.
• Google's incident response system is rooted in the Incident Command System (ICS), originally developed by firefighters in 1968 to manage wildfires.
• The 'Three Cs' of incident management—Coordinate, Communicate, and Control—are the essential goals for any successful response framework.
• Primary response roles include the Incident Commander (IC), who oversees the 3Cs, the Operations Lead (OL) for mitigation, and the Communications Lead (CL) for stakeholder updates.

ICS was established in 1968 by firefighters as a way to manage wildfires.

• Incident response teams, led by Communications and Operations Leads, are designed to scale dynamically based on the severity of the crisis.
• A software bug in Google Assistant version 1.88 caused devices to fetch speaker recognition data 50 times more frequently than intended.
• Initial detection by an on-call developer led to a bug report rather than a formal incident declaration, allowing the problematic rollout to continue.
• Misdiagnosis of the root cause led the team to request a quota increase for a different bug, which failed to address the underlying traffic surge.
• The failure to escalate early resulted in a widespread service degradation where users were met with error messages instead of assistance.
• The case study highlights how informal management of technical anomalies can lead to significant customer-facing outages.

This example shows how failing to declare an incident early on can leave a team without the tools to respond to an incident quickly and efficiently.

• A miscommunication between client and server developers led to a prolonged troubleshooting process and delayed the identification of the root cause.
• The team violated best practices by continuing a major software rollout over a weekend when staff availability was limited.
• Despite a surge in user reports across social media and support forums, the bug was not escalated to the highest priority for several days.
• The incident was temporarily mitigated multiple times by simply increasing quota limits rather than addressing the underlying technical trigger.
• The resolution ultimately relied on the 'heroic efforts' of off-duty developers, highlighting a failure in formal incident management and work-life balance.

The team had not followed the best practice of performing rollouts only during business days to ensure developers are around in case something goes wrong.

• Effective incident management prioritizes immediate impact mitigation followed by a rigorous root-cause analysis to prevent recurrence.
• Declaring an incident early is critical for preventing miscommunication and ensuring relevant teams are looped in for faster resolution.
• Centralized communication, whether through physical 'war rooms' or virtual channels like IRC, is a core principle of the IMAG protocol.
• A case study of Google Kubernetes Engine (GKE) illustrates the complexity of debugging systems where no single person understands every interaction.
• The GKE incident began with a 60% failure rate in cluster creation across multiple zones, triggered by external dependencies in the open-source codebase.

The following incident illustrates what happens when a team of experts tries to debug a system with so many interactions that no single person can grasp all the details.

• SREs and developers collaborate to investigate a persistent failure in GKE certificate signing that specifically impacts European clusters.
• The incident response scales rapidly, involving multiple teams including Cloud Networking and Compute Engine to eliminate infrastructure root causes.
• Il-Seong takes over as Incident Commander to establish a formal structure, assigning specific roles for operations and communications to manage the growing team.
• The investigation identifies a corrupted image on DockerHub as the likely culprit, as the image pull fails in Europe but succeeds in the US.
• The team faces a difficult mitigation choice between waiting for an external dependency (DockerHub) to fix the image or reconfiguring clusters to use Google Container Registry.

The first Seattle SRE, Il-Seong, arrived at the office, lightly caffeinated and ready for the day.

• Engineers pursued multiple mitigation paths simultaneously, including fixing broken clusters and rebuilding binaries with updated configurations.
• A breakthrough occurred when an SRE discovered that Docker pull requests could be intercepted using an existing GCR mirror, bypassing the corrupted external source.
• The incident lasted nearly seven hours, involving 41 unique users and generating a massive 26,000-word IRC log and 28 postmortem action items.
• While escalation paths and customer communication were handled well, the response suffered from initial coordination gaps and a lack of formal structure.
• The postmortem identified a critical need for 'generic mitigations'—blunt instruments like rollbacks or traffic redirection that can alleviate user pain before a root cause is found.

The outage was over. All that remained was cleanup, and writing a truly epic postmortem.

• Generic mitigations like rollbacks can stop service 'bleeding' quickly before a root cause is fully identified.
• A case study shows that a generic rollback could have resolved an outage two hours earlier than the bespoke fix.
• Engineering teams should develop general-purpose mitigation tools during normal operations rather than during emergencies.
• Customers prioritize service restoration over the technical understanding of why a failure occurred.
• The ideal incident response workflow places mitigation as the second step, immediately following impact assessment.
• Google uses hardware redundancies like backup generators and UPS batteries to mitigate power grid fluctuations automatically.

Ultimately, customers do not care whether or not you fully understand what caused an outage. What they want is to stop receiving errors.

• A rare sequence of four lightning strikes in two minutes overwhelmed the UPS systems at a Google datacenter in Belgium.
• While servers remained powered, disk trays lost power, causing widespread read/write errors for Google Compute Engine virtual machines.
• The incident required complex coordination between Persistent Disk and GCE SRE teams to migrate unaffected VMs before rebooting hardware.
• The Incident Commander had to pivot from documented procedures to managing the rapid development of new tools for VM migration.
• A dedicated Communications Lead managed simultaneous updates for company leadership, internal teams, and external customers.
• Post-incident follow-up focused on diagnosing the specific UPS failure mode and replacing hardware to prevent recurrence.

The UPS batteries in the disk trays did not swap power usage to the backup batteries on the third and fourth lightning strikes because the strikes were too closely spaced.

• A major power failure resulted in only 0.000001% data loss due to multi-datacenter redundancy and effective incident management.
• Clear leadership roles, such as the Incident Commander and Operations Lead, allowed for delegated tasks and coordinated decision-making.
• PagerDuty evolves its incident response processes every few months to match business growth and documents these practices publicly.
• The 'Failure Friday' program uses manual failure injection to train new Incident Commanders and familiarize engineers with response vernacular.
• Time-bound simulation games like 'Keep Talking and Nobody Explodes' are used to replicate the high-stress urgency of real-world incidents.
• Teamwork is prioritized to ensure engineers do not feel isolated during high-impact, high-stress scenarios.

The stressful and communication-intensive nature of the game forces players to cooperate.

• PagerDuty utilizes open meetings and thorough documentation to reduce resolution time and prevent future incident recurrence.
• The company records all real-time phone communication during major incidents to ensure accurate postmortem analysis and timeline recreation.
• A case study of a 10-hour NTP clock drift incident demonstrates the transition from individual team alerts to a formal Incident Commander-led response.
• Rotating engineers and Incident Commanders every four hours during long incidents prevents fatigue and introduces fresh perspectives to problem-solving.
• The incident response toolkit relies on PagerDuty for metadata, Slack as a chronological ledger, and conference calls for rapid decision-making.

We also record all of the phone calls involved in a major incident so we can learn from the real-time communication feed.

• Effective incident management requires establishing a formal process and common language before a crisis occurs to prevent miscommunication.
• Training should focus on concrete actions and past scenarios, defining specific roles like Incident Commander and Communications Lead.
• A mitigation-first response is encouraged, allowing on-call engineers to delegate and escalate tasks as needed during an emergency.
• Pre-selecting and practicing with specific communication channels prevents decision fatigue and confusion during high-stress events.
• Maintaining stakeholder trust requires regular status updates and pre-approved templates for public announcements to avoid drafting under pressure.
• Preparation strategies include simulation exercises, reviewing previous failures, and maintaining a ready-to-use list of essential contacts.

No one wants to write these announcements under extreme stress with no guidelines.

• Pre-prepared communication lists and established incident criteria are essential for saving time during critical outages.
• Effective incident management requires clear roles, defined audiences, and agreed-upon communication channels.
• Regular practice through drills like Google's 'Disaster Recovery Testing' (DiRT) builds muscle memory and identifies procedural gaps.
• Simulating emergencies using 'Wheel of Misfortune' or treating minor issues as major ones allows for low-stakes training.
• Post-drill reports are vital for analyzing outcomes and closing the gaps in a team's response capabilities.
• Involving diverse roles—including SREs, developers, and marketing—ensures the entire organization is prepared for disaster.

By practicing during less critical situations, your team develops good habits and patterns of behavior for when lightning strikes—figuratively and literally.

• Establishing incident procedures ahead of time is critical because collaboration needs expand rapidly as incidents grow in size.
• The Incident Command System provides a scalable structure that requires regular practice to build the muscle memory needed to combat panic.
• A blameless postmortem culture is essential for building more reliable systems and driving positive organizational change.
• Postmortems must be written well and acted upon to prevent repeat outages and should be shared widely across the organization.
• Transitioning to a postmortem culture is a significant cultural shift that can be started with small, basic procedures and tuned over time.

Staying calm and following the response structure during an emergency takes practice, and practice builds “muscle memory.”

• A failed automation process for decommissioning a single satellite rack led to a catastrophic global disk erasure across all Google satellite machines.
• The root cause was an API bug where an empty list filter was interpreted as 'no filter,' causing the command to apply to every machine in the fleet.
• Despite the massive scale of the data wipe, users only experienced slight latency increases because traffic was successfully rerouted to core datacenters.
• The incident highlights the necessity of idempotency and sanity checks in automated workflows to prevent unintended global consequences.
• A 'bad postmortem' example illustrates the danger of blaming individuals rather than identifying systemic technical failures and automation bugs.

Within minutes, the disks of all satellite machines, globally, were erased.

• A routine rack decommissioning failed when automation was manually rerun, leading to the accidental erasure of satellite machines.
• The incident caused increased latency and lost ad revenue as traffic was rerouted from edge satellites to the core network.
• The postmortem document is used as a negative example due to its lack of technical context, missing impact data, and unprofessional tone.
• A significant cultural failure is highlighted through the 'blame game' where specific individuals are criticized for documentation gaps and ignorance.
• The text emphasizes that without concrete numerical data and detailed root cause analysis, a postmortem cannot effectively prevent future failures.
• The 'Where we got lucky' section reveals a lack of systemic reliability, relying on the core's unexpected capacity to survive the surge.

The team (especially maxone@, logantwo@) never wrote any documentation to tell SREs not to run the automation multiple times, which is ridiculous.

• A postmortem serves as a factual record that must detail the incident, mitigation efforts, and user impact to facilitate organizational learning.
• Effective action items must be preventative rather than just mitigative, focusing on system changes rather than trying to make humans less error-prone.
• Action items require clear prioritization, specific measurable language, and formal tracking bugs to ensure they are actually implemented.
• Blameful narratives that call out specific individuals create a culture of risk-aversion and lead to the suppression of critical facts.
• Subjective or animated language, such as personal judgments or dramatic exclamations, erodes psychological safety and distracts from factual analysis.
• Official ownership and accountability are essential for transforming postmortem findings into concrete improvements.

Let’s plan for a future where we’re all as stupid as we are today.

• Effective postmortems require a single owner to ensure accountability and the completion of follow-up action items.
• Transparency is vital; documents should be shared widely across the organization and even with customers to restore trust and maximize learning.
• Timeliness is critical as delays lead to the loss of key details and missed opportunities to prevent recurring incidents.
• Mature cultures utilize machine-readable tags and metadata to allow for automated downstream analytics of failure patterns.
• A real-world example illustrates how a bug in turndown automation accidentally triggered a global satellite frontend outage by wiping all machine disks.
• Rapid mitigation often involves shifting traffic to core clusters, though this may result in increased latency for users.

The value of a postmortem is proportional to the learning it creates.

• A manual re-execution of a 'run once' workflow triggered a longstanding input validation bug in the Traffic Admin server.
• The bug interpreted an empty machine list as a lack of constraints, leading to the accidental decommissioning and disk-wiping of the majority of global satellite machines.
• The outage caused significant latency spikes and revenue loss in display, video, and search advertising due to lost queries.
• The Traffic team spent 48 hours in an 'all hands on deck' effort to rebuild the global satellite infrastructure.
• The incident highlighted a lack of idempotency in the MDB API calls and a failure of safety checks when handling empty machine lists.
• Despite the failure, the system's ability to evacuate the edge to core clusters allowed for mitigation without total service collapse.

The bug removed the machine constraint on the decom action, sending all satellite machines to decommission.

• A lack of sanity checks and rate-limiting in the Traffic Admin server allowed a flawed command to rapidly decommission geographically diverse satellite machines.
• The recovery process was severely hampered by high-latency TFTP transfers and infrastructure that could not handle the scale of simultaneous reinstalls.
• The outage revealed critical technical debt, including concurrency regressions in automation tools and overly strict SSH timeouts on remote links.
• Monitoring safety checks failed to trigger during the initial removal of targets but delayed the recovery by blocking the re-addition of those same targets.
• The incident was contained only because core clusters and YouTube's CDN were managed through separate infrastructure, preventing a total global collapse.
• Post-incident action items focus on auditing all systems capable of 'turning live servers into paperweights' and implementing strict input rejection.

The decom workflow doesn’t cross-check decom requests with other data sources; as a result, there were no objections to the request to trash (many) geographically diverse machines.

• The text outlines a structured list of emergency response action items categorized by type, priority, and ownership.
• Key technical mitigations include implementing safety checks for administrative servers to prevent destructive operations on large numbers of nodes.
• A 'big-red-button' mechanism is proposed to provide a manual emergency stop for decommissioning workflows during catastrophic failures.
• Monitoring improvements focus on detecting rapid machine loss and ensuring that configuration changes remain reversible.
• The section includes a glossary of internal infrastructure terms like Borg, MDB, and Diskerase to provide context for Google's operational environment.
• Action items are tracked via specific bug IDs to ensure accountability and follow-through in the postmortem culture.

Add a big-red-buttona disable approach to decom workflows.

• The YouTube CDN remained unaffected during the incident, highlighting the resilience of specific edge network components.
• Recovery to normal traffic balance took approximately 48 hours, with 50% of edge traffic restored within the first 36 hours.
• A high-quality postmortem must include quantifiable metrics, such as cache hit ratios and traffic levels, to provide objective context.
• Effective action items require clear ownership, prioritization, and a verifiable end state to ensure accountability.
• The core of a successful postmortem culture is blamelessness, focusing on system design flaws rather than individual human error.
• Depth in investigation involves exploring impact across multiple teams and identifying both the proximate trigger and the underlying root cause.

A postmortem with no action items is ineffective.

• Postmortems must be written promptly to ensure accuracy and to prevent stakeholders from filling information gaps with negative assumptions.
• Effective documentation balances conciseness with depth by linking to raw data and logs rather than including them in the main text.
• Leadership must actively model and enforce blameless behavior to ensure psychological safety and honest reporting.
• Language should focus on systemic improvements and process failures rather than individual mistakes or lack of training.
• Incentives must be balanced between the act of writing a postmortem and the actual completion of the resulting action items.
• Including all incident participants in the authoring process prevents the oversight of key contributing factors.

The longer you wait, the more they will fill the gap with the products of their imagination. That seldom works in your favor!

• Organizations should reward postmortem contributions through formal recognition like peer bonuses, performance reviews, and promotions.
• Highlighting improved reliability and increased feature velocity serves as intrinsic motivation for teams to engage in the postmortem process.
• Public accolades and positioning authors as subject matter experts can satisfy the desire for peer acknowledgment and leadership status.
• Gamification techniques, such as leaderboards and 'FixIt' weeks, encourage engineers to close action items and resolve system weaknesses.
• Widespread sharing through reading clubs, cross-team reviews, and training exercises like the 'Wheel of Misfortune' ensures institutional learning.
• A culture is failing when employees view postmortems as a burden to be avoided rather than an opportunity for growth and impact.

Use the Wheel of Misfortune when training new engineers: a cast of engineers reenacts a previous postmortem, assuming roles laid out in the postmortem.

• High-visibility postmortems must be audited for blameful language to prevent team members from avoiding future reporting.
• Leadership must be redirected when they use blame-oriented rhetoric, shifting the focus from 'who' to 'why' warning signs were missed.
• Postmortems are 'letters to future team members' and require dedicated time to ensure quality and prevent the recurrence of incidents.
• Repeating incidents signal systemic issues, such as prioritizing feature velocity over reliability or applying temporary 'Band-Aids' to deep technical problems.
• Standardized templates and collaborative tools like Google Docs lower the barrier to entry and ensure consistency across different organizational domains.

Postmortems are letters you write to future team members: it’s very important to keep a consistent quality bar, lest you accidentally teach future teammates a bad lesson.

• Google utilizes automation and Google Apps Script to streamline postmortem authoring and data capture for easier analysis.
• Incident management tools automatically populate postmortems with critical data such as timelines, IRC logs, and root-cause services.
• A standardized postmortem checklist ensures authors perform thorough impact assessments and vet action items with technical leads.
• The 'Requiem' tool serves as a centralized repository for thousands of postmortems, enabling organization-wide search and metadata parsing.
• Action items are integrated into a centralized bug tracking system to prevent critical fixes from slipping through the cracks.
• Aggregated postmortem data allows teams to identify trends in incident duration, detection time, and systemic vulnerabilities.

With this level of tracking, we can ensure that action items don’t slip through the cracks, leading to increasingly unstable services.

• Cultivating a postmortem culture leads to fewer outages, better system design, and increased trust from users and engineers.
• No service can maintain 100% availability due to unpredictable factors like traffic spikes or physical infrastructure damage.
• Google utilizes a combination of tools and strategies rather than a single solution to stabilize network load and ensure reliability.
• Google Cloud Load Balancing (GCLB) externalizes Google's internal infrastructure to provide global load balancing for public use.
• Unlike traditional DNS-based load balancing, GCLB uses anycast to route traffic to the nearest node using a single virtual IP.
• Anycast improves latency and reliability by relying on the BGP routing mesh rather than client-side DNS cooperation.

No service is 100% available 100% of the time: clients can be inconsiderate, demand can grow fifty-fold, a service might crash in response to a traffic spike, or an anchor might pull up a transatlantic cable.

• Anycast deployments face challenges with site-level overloading and BGP route flapping that can reset active TCP sessions.
• Google uses 'stabilized anycast' to maintain connection coherence by mapping client IPs to the closest frontend site even when routing changes.
• Maglev is a custom, distributed packet-level load balancer that runs on commodity hardware and utilizes ECMP for high availability.
• The system achieves N+1 redundancy and massive scalability by spreading traffic across a pool of machines rather than using active/passive pairs.
• Maglev uses a combination of consistent hashing and connection tracking to route packets without needing to share state between individual load balancer instances.

Each time the BGP route “flaps,” all in-progress TCP streams are reset as the unfortunate user’s packets are directed to a new frontend with no TCP session state.

• GSLB manages global traffic by matching user demand to service capacity across distributed clusters.
• The Google Front End (GFE) terminates TCP and SSL sessions at the network edge to minimize latency.
• GFEs utilize a 'lame duck' mode to gracefully drain traffic from backends during updates without disrupting active requests.
• Persistent sessions between GFEs and backends reduce the overhead of re-establishing secure connections.
• Maglev and GFE components work together to terminate SSL as close to the user as possible.
• GSLB acts as the 'glue' that routes traffic to the nearest available GFE and backend instance groups.

By putting GFE backends into a mode in which they fail health checks while continuing to respond to in-flight requests, we can gracefully remove GFE backends from service without disrupting any user requests.

• Google Cloud Load Balancing (GCLB) provides a 99.99% availability SLA and acts as a global traffic manager to route around service failures.
• Standard release procedures like canarying and gradual rollouts are used to mitigate the impact of software regressions by isolating new code on small server subsets.
• The launch of Pokémon GO in 2016 saw traffic levels reach 50x the most optimistic estimates, presenting an unprecedented scaling challenge.
• Pokémon GO's interactive, globally shared world requires near-real-time updates to a state shared by all participants in a given area.
• Before migrating to GCLB, the game relied on regional Network Load Balancers (NLB) and Nginx proxies, which suffered from resource exhaustion and SSL latency.
• Within two days of migrating to GCLB, Pokémon GO became one of the largest services on Google's infrastructure, comparable to top-tier global services.

The actual launch requests per second (RPS) rate was nearly 50x that estimate—enough to present a scaling challenge for nearly any software stack.

• Niantic migrated to Google Cloud Load Balancing (GCLB) to mitigate SYN flood attacks and scale beyond the limitations of packet-level proxies.
• The migration revealed that actual client demand was 200% higher than previously monitored, triggering a massive cascading failure across the entire stack.
• A performance regression in the Google Front End (GFE) reduced global capacity by 50% as the system struggled with SSL reconnections to unresponsive backends.
• Synchronized client retries created a 'thundering herd' effect, causing traffic spikes that reached 20 times the previous global peak.
• Google SREs resolved the crisis by isolating Pokémon GO traffic into a dedicated pool and implementing administrative rate limits to allow the system to stabilize.
• Post-incident improvements included Niantic implementing jitter and exponential backoff to prevent future synchronized retry storms.

These error responses served to effectively synchronize client retries, producing a “thundering herd” problem, in which many client requests were issued at essentially the same time.

• Measuring load as close to the client as possible is essential for accurately predicting demand and preemptively scaling resources.
• Autoscaling can enhance service availability through horizontal or vertical scaling, but misconfiguration can lead to severe service degradation.
• A common failure mode occurs when unhealthy or 'zombie' instances are included in utilization averages, preventing the autoscaler from triggering.
• Best practices for handling unhealthy machines include using load balancer capacity metrics and implementing cool-down periods for new instances.
• Autohealing mechanisms can complement autoscaling by monitoring health metrics and automatically restarting non-responsive instances.
• Stateful systems require specialized strategies like consistent hashing or vertical scaling, as simple horizontal scaling often fails to resolve session-based bottlenecks.

Autoscaling runs into problems when machines are not serving (known as unhealthy instances) but are still counted toward the utilization average.

• Autoscalers are designed to scale up aggressively to prevent traffic loss but scale down cautiously to maintain stability.
• Services should be configured to stay far from system bottlenecks like CPU to provide a buffer for sudden load spikes.
• Unconstrained autoscaling can lead to 'runaway' resource consumption if triggered by software bugs or failing dependencies.
• Setting hard minimum and maximum bounds is essential for capacity planning and preventing total quota depletion.
• Engineers must have access to a fast, well-documented kill switch to manually override autoscaling during emergencies.
• Scaling one microservice can inadvertently starve downstream dependencies or peer services of shared resource quotas.

Autoscaler is a powerful tool; if misconfigured, it can scale out of control.

• Regional Managed Instance Groups (RMiGs) use background jobs to balance instance counts across zones, ensuring even quota usage and failure domain diversity.
• Complex systems often require the simultaneous use of load balancing, load-based autoscaling, and load shedding to maintain stability.
• Load shedding acts as a critical safety valve when traffic spikes exceed the speed of hardware provisioning or cause memory exhaustion.
• While load balancing, shedding, and autoscaling are often configured independently, they are deeply interconnected and can interact in unexpected ways.
• The 'Dressy' case study illustrates a failure mode where a utilization-aware load balancer paradoxically funnels all traffic into a single failing region.

Dressy’s development teams investigate and notice a problem: their load balancing is inexplicably drawing all user traffic into region A, even though that region is full-to-overflowing and both B and C are empty.

• A failure occurred when a load balancer misinterpreted error responses from load shedding as high-efficiency performance.
• The system created a dangerous feedback loop by routing more traffic to the region that was already failing and rejecting requests.
• The root cause was a lack of communication between independent load management systems designed by different engineers.
• Effective load management requires treating balancing, shedding, and autoscaling as a single unified system rather than isolated tools.
• Engineers should implement 'error weighting' in balancing logic to ensure failed requests are treated as high-cost rather than low-cost.
• Strategic precautions include setting minimum instance counts for failover and ensuring autoscaling triggers before load shedding begins.

As far as the load balancer system was concerned, each successive dropped request was a reduction in the per-request CPU cost.

• Unbounded request deadlines lead to resource exhaustion, increased latency, and potential system crashes.
• Effective traffic management requires servers to terminate long-running requests and clients to cancel obsolete ones.
• Autoscaling and load shedding can create catastrophic feedback cycles if they are not configured with a holistic view of the system.
• Reliability is the most critical feature of any production system and should be integrated into the design phase rather than deferred.
• Non-Abstract Large System Design (NALSD) is an iterative approach used by SREs to build robust, scalable systems with low operational costs.
• NALSD focuses on evolving a design from a simple problem statement into a sophisticated solution that defends against multiple failure modes.

Unless carefully configured, autoscaling can result in disastrous consequences—for example, potentially catastrophic feedback cycles between load balancing, load shedding, and autoscaling when these tools are configured in isolation.

• NALSD is a framework used by Google SREs to bridge the gap between theoretical whiteboard designs and concrete physical resource requirements.
• The 'Non-Abstract' component emphasizes that systems must eventually run on real hardware, necessitating rigorous estimates of CPU, RAM, and network constraints.
• The design process is iterative, focusing on sound reasoning and reasonable assumptions rather than achieving perfect final calculation values.
• The basic design phase evaluates if a concept is possible in principle and explores optimizations for speed and efficiency.
• The scaling phase tests feasibility against physical constraints like budget and hardware, while ensuring the system can fail gracefully.
• A practical application of this process is demonstrated through the design of a system to calculate click-through rates for Google AdWords.

Google has learned (the hard way) that the people designing distributed systems need to develop and continuously exercise the muscle of turning a whiteboard design into concrete estimates of resources.

• The design process is described as an iterative story of twists and turns where components are modified or replaced as they flounder.
• Initial requirements focus on calculating Click-Through Rate (CTR) by tracking impressions and clicks for specific search terms and ad IDs.
• System performance is defined by strict Service Level Objectives (SLOs), requiring 99.9% of queries to finish under one second with data no older than five minutes.
• The scale of the problem involves handling 500,000 search queries and 10,000 ad clicks per second for millions of advertisers.
• A single-machine starting point utilizes separate query and click logs joined by a unique identifier to maintain scalability and data integrity.
• To meet latency goals on one machine, the design suggests using an indexed SQL database to avoid inefficient full-log scans during dashboard generation.

The final design is the end of a story of twists and turns.

• Calculating resource requirements for log parsing requires aggressive rounding and scientific notation to account for metadata overhead and prevent premature scaling limits.
• A single day of query logs for a high-traffic system can generate approximately 100 TB of data, assuming a 2 KB entry size and a 2% click-through rate.
• Traditional hard disk drives (HDDs) are often limited by IOPS rather than capacity, potentially requiring thousands of disks to handle the write volume of query logs.
• Evaluating a RAM-only storage solution reveals a massive hardware footprint, requiring over 1,500 standard machines to hold a single day's data.
• Single-machine designs are fundamentally flawed for large-scale systems because they create numerous single points of failure that jeopardize Service Level Objectives (SLOs).
• The transition from a single-machine model to a distributed system like MapReduce is necessary to handle the join operations between query and click logs.

If every log entry can be stored and indexed in an average of one disk write per log entry, we see that IOPS is a limiting factor for our query logs: (5 × 10^5 queries/sec) / (200 IOPS/disk) = 2.5 × 10^3 disks or 2,500 disks.

• MapReduce serves as a reliable batch processor that scales horizontally by adding machines, but it fails to meet low-latency SLOs.
• The batch-oriented nature of MapReduce creates a 'boundary problem' where queries and clicks in different batches are never joined.
• A new architecture introduces QueryStore, a distributed system inspired by Bigtable, to store all query logs for on-demand lookup.
• The LogJoiner component processes a continuous stream of click logs, joining them with QueryStore data to create a ClickMap.
• To handle asynchronous data arrival, the system implements a retry mechanism for clicks that do not yet have a corresponding query log entry.
• The final CTR dashboard relies on two parallel components: the ClickMap for successful joins and the QueryMap for total impressions.

If a logged query is in batch 1, and its click is in batch 2, the click and query will never be joined.

• Calculations reveal that the QueryStore manages approximately 100 TB of daily log data, requiring efficient deletion of stale records.
• Network throughput for processing clicks and performing QueryStore lookups is estimated at a manageable aggregate of 400 Mbps.
• While the QueryMap storage requirement of 2 TB per day is small enough for a single machine's capacity, high write frequency exceeds single-drive IOPS limits.
• The design must transition to a sharded architecture to overcome the IO limitations of individual hardware units.
• Future iterations must address data management across shards and ensure system reliability to prevent data loss during machine failures.

While we could calculate the impact of using higher IOPS drives (e.g., SSD), our exercise is focused on demonstrating that the system can scale to an arbitrarily large size.

• The LogJoiner design scales horizontally by sharding query and click logs based on query_id, allowing parallel processing of massive data volumes.
• A log sharder component uses hashing and modulo operations to ensure consistent distribution of records across N available shards.
• Storage constraints for the 2 TB QueryMap are addressed by sharding on ad_id, enabling the data to fit within the RAM limits of individual 64 GB machines.
• System reliability is improved by replicating shards across different machines, ensuring that a single LogJoiner failure does not result in data loss or SLO violations.
• The design utilizes an 'error budget' to manage the residual risk of concurrent machine failures, with log reprocessing as a fallback for disaster recovery.
• Despite local redundancy, hosting all components in one datacenter remains a single point of failure, necessitating a move toward multidatacenter architecture.

If our log sharder process sends duplicate log entries to two shards, the system can continue to perform at full speed and process accurate results even when a LogJoiner fails.

• To ensure infrastructure resilience, data must be replicated across multiple datacenters to allow for seamless failover during outages.
• The consensus problem is addressed using algorithms like Paxos, which require multiple replicas to maintain state across geographically separated sites.
• Network latency between distant datacenters limits sequential throughput, necessitating the use of sharding and parallel processes to meet high transaction demands.
• Calculations for a multidatacenter AdWords design show that 25,500 tasks are required to handle over one million queries per second given a 25ms Paxos latency.
• The resulting hardware requirements are manageable, requiring approximately 64 machines per datacenter to host 4 TB of RAM with low network overhead.

This requirement places a limit on the sequential throughput for the system.

• The proposed system architecture meets high-throughput requirements including 10,000 ad clicks and 500,000 search queries per second.
• Horizontal scaling across all components ensures that end-to-end pipeline latency remains under the five-minute threshold.
• Non-Abstract Large System Design (NALSD) is defined as an iterative process that breaks software into logical components within a reliable ecosystem.
• The NALSD framework relies on four critical questions regarding possibility, optimality, feasibility, and resilience.
• Effective design involves separating components based on expected growth to allow independent scaling and remove single points of failure.
• Data processing pipelines are essential for transforming unbounded, global-scale data into structured insights for business-critical decisions.

Google’s experience has shown that the ability to reason from an abstract requirement to a concrete approximation of resources is critical to building healthy and long-lived systems.

• Data processing pipelines are complex systems that require significant labor and time to fix if designed poorly.
• Site Reliability Engineers (SREs) should be involved in the early design stages to ensure pipelines are maintainable and reliable.
• The Extract Transform Load (ETL) model remains a foundational paradigm for structuring, aggregating, and indexing data for downstream use.
• Modern pipeline applications often mirror cognitive processes, such as acquiring sensor data to train machine learning networks.
• Pipeline health is maintained through a balance of design patterns, lifecycle best practices, and specific optimization tradeoffs.
• Case studies like Spotify's demonstrate how hybrid cloud and in-house solutions manage business-critical event delivery.

Ideally, SREs should be involved in this work from its early stages: Google’s SRE teams regularly consult with teams developing a data processing pipeline to ensure that the pipeline can be easily released, modified, and run without causing issues for customers.

• ETL pipelines serve as the backbone for complex data manipulations, including preprocessing for machine learning, event counting, and indexing for search engines.
• Business intelligence leverages data aggregation from various sources to identify system failures and optimize decision-making processes.
• The 'Shave the Yak' case study illustrates how automated jobs can process player event data into aggregate tables for monthly feature analysis.
• Machine learning systems follow a structured five-stage lifecycle from feature extraction and model training to serving predictions to other services.
• The 'Dressy' example highlights the necessity of preprocessing pipelines to normalize unstructured vendor data into formats compatible with frameworks like TensorFlow.
• Recommendation strategies such as collaborative filtering, clustering, and content-based filtering are used to drive revenue through personalized user experiences.

New products that are uploaded to their system don’t have structured data or consistent labels (e.g., some vendors may include extra information about color, size, and features of the dress using different categories and formats).

• The development team utilizes Google Cloud Dataflow for streaming preprocessing to transform raw dress images and customer data into labeled formats.
• Multiple data sources, including BigQuery purchase history and product data, are integrated to train a TensorFlow model for personalized recommendations.
• The production workflow includes automated accuracy checks against test datasets before model binaries are stored in Google Cloud Storage and promoted.
• Operational challenges such as stale models or intermittent errors are addressed by monitoring for data bottlenecks, software bugs, and feature selection issues.
• Pipeline reliability is managed through Service Level Objectives (SLOs) and error budgets to balance system stability with the pace of new feature launches.
• Data freshness is identified as a critical metric for ensuring that recommendations remain relevant and accurate for the end user.

Dressy has noticed that occasionally a new model doesn’t get published for over 24 hours, and the recommendations trigger intermittent errors.

• Freshness and latency SLOs define the maximum acceptable age of data and the time required for pipeline completion.
• Correctness SLOs are essential for preventing financial or operational errors, such as billing discrepancies.
• When 'correct' output is unknown, 'golden data' from test accounts can be used to benchmark production accuracy.
• Data isolation and load balancing allow systems to prioritize high-value data processing during resource constraints.
• Effective monitoring should focus on end-to-end system health rather than isolated per-stage metrics to reflect actual user experience.

If you don’t have access to such data, you can generate it. For example, use test accounts to calculate the expected output. Once you have this “golden data,” you can compare expected and actual output.

• Measuring SLOs for individual pipeline stages can lead to excessive alerting and fails to capture the actual user experience.
• End-to-end monitoring is essential to detect data corruption bugs where individual stages report success despite data loss between components.
• Engineers must design systems to handle the failure of third-party dependencies by aligning their SLOs with the advertised SLAs of providers.
• Google uses planned outages and Disaster Recovery Testing (DiRT) to ensure pipelines can automatically fail over during regional outages.
• Manual failover risks prolonged outages and data staleness, potentially requiring expensive data restoration and reprocessing.
• Regularly practicing disaster recovery scenarios is necessary to ensure systems remain resilient to both common and uncommon failures.

Both jobs think they are correct, but the user doesn’t see the data.

• Effective system diagrams should map every component and transformation to help on-call engineers pinpoint failure points and data correctness bugs.
• Diagrams should integrate live monitoring links and historical runtime data to provide real-time status and foreshadow potential performance degradation.
• Process documentation must cover both routine releases and rare manual tasks like service turnup, with a goal of eventual automation.
• Every system alert requires a corresponding playbook entry that provides specific recovery steps for on-call engineers.
• The pipeline development lifecycle begins with prototyping to verify business logic and select the most appropriate programming model or language.
• A 1% dry run using production data on the full stack is a critical bridge between prototyping and full-scale deployment.

Each of the components and transformations shown in your diagram can get stuck, causing data to stop flowing through the system.

• Gradual scaling from dry runs to full production data is essential for identifying performance bottlenecks and preventing outages.
• Staging environments should use representative production data to catch integration issues that unit tests might miss.
• A/B comparisons between new and known good data help certify releases before they move to production.
• Canarying involves partially deploying a pipeline to monitor results and limit the impact of potential failures.
• Complex pipelines may require dry-run modes or two-phase mutations to safely compare canary results with live data.
• Automation of canary verification using existing SLO metrics ensures consistent health assessments during rollouts.

Often, you’ll have to wait for the complete cycle of processing to finish before you can discover any customer-impacting issues.

• Partial deployments allow for testing major features on a subset of real traffic to predict performance impacts before a full launch.
• Gradual data ramping (e.g., 1% to 100%) helps identify bugs or storage errors before they reach the majority of users.
• Hotspotting occurs when specific resources become overloaded due to excessive concurrent access or row-level lock contention.
• Infrastructure can combat hotspots by dynamically rebalancing work or breaking large units into smaller, manageable pieces.
• Emergency shutdown mechanisms should be built into client logic to isolate and skip problematic data patterns or users.
• Strategies to mitigate load include restructuring data access patterns and reducing lock granularity to avoid contention.

To be safe, it’s still best to build an emergency shutdown into your client logic to allow you to stop processing and isolate fine-grained chunks of processing work characterized by large resource usage or errors.

• Autoscaling mitigates service outages during workload spikes while reducing costs by decommissioning idle workers.
• Resource planning must account for hidden costs like network bandwidth, cross-region replication, and expensive data access patterns.
• Efficiency should be measured at each individual pipeline stage to pinpoint specific jobs causing resource spikes.
• Security best practices include avoiding unencrypted PII in temporary storage and enforcing the principle of least privilege.
• Resilient pipeline design should prioritize automated recovery to prevent manual intervention and SLO violations.

Constantly running the number of workers required for peak capacity is an expensive and inefficient use of resources.

• SREs prioritize a rigorous design phase to evaluate technology options based on user needs, product requirements, and system constraints.
• Pipeline selection involves a tradeoff between fully managed, automated solutions and flexible, hands-on management systems.
• Key performance features include optimizing for latency through streaming or batch APIs and ensuring data correctness via exactly-once semantics.
• Operational reliability is maintained through high availability features like multihoming, autoscaling, and black-box monitoring.
• Effective incident management requires minimizing MTTR and MTTD through fast rollbacks, regional draining, and symptom-based SLO alerting.
• Development lifecycles should incorporate canary environments and resource accounting dashboards to prevent production errors and manage costs.

Alerting on the symptom (versus the cause) reduces monitoring gaps.

• Leveraging existing automation and operational tools reduces maintenance costs and human error during complex tasks like regional migrations.
• The idempotent mutations pattern ensures that reprocessing data after a failure produces consistent results without duplicates.
• Two-phase mutations allow for a validation step by storing potential changes in a temporary location before final application.
• Checkpointing enables long-running pipelines to save partial state, preventing the need to restart from scratch after a failure or preemption.
• Implementing reusable code libraries standardizes monitoring and metrics across multiple pipelines, simplifying updates and alerting.
• A microservice approach to pipeline design can further streamline management and scalability.

With two-phase mutation, the mutations themselves are stored in a temporary location. A separate verification step (or pipeline) can run against these potential mutations to validate them for correctness.

• Applying microservices principles to data pipelines involves breaking monolithic structures into smaller, independently releasable and monitorable units.
• Google SRE teams utilize a pipeline maturity matrix to evaluate and onboard services, similar to the Production Readiness Review (PRR) process.
• The maturity matrix assesses five key characteristics: failure tolerance, scalability, monitoring, transparency, and testing.
• Systems are scored on a scale from 1 (Chaotic) to 5 (Continuous Improvement) to identify specific technical weaknesses.
• Google encourages the reuse of existing tools and open-source technologies to meet maturity standards rather than building custom solutions.
• High-maturity pipelines feature advanced capabilities like automatic dynamic resharding, load shedding, and multihomed automatic failover.

To score your system, read the descriptions for each characteristic below, and select the best matching milestone.

• The text outlines a maturity framework for data processing pipelines, ranging from 'Chaotic' to 'Continuous Improvement'.
• Key technical capabilities include built-in load shedding, work unit preemption, and automated quarantine/replay of failed tasks.
• Monitoring requirements evolve from no visibility to fine-grained execution maps, stack graphs, and historical run statistics.
• Ease of implementation is measured by automatic discoverability, global data registries, and zero-config machine-readable formats.
• Testing maturity progresses from no unit testing to small build dependency graphs, sanitizers, and built-in test data generation.
• Effective configuration management requires decoupling test environments from production while allowing for smart reuse of relevant components.

There is a solution to automatically quarantine and replay a failed work unit.

• Implementing integration testing for pipelines is often resource-intensive due to a lack of specialized third-party tools and documentation.
• Google tracks Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR) to measure the effectiveness of their response to SLO violations.
• Data delay is a primary failure mode where stale data is generally preferred over the propagation of incorrect or incomplete data.
• Batch pipelines require strict stage completion, whereas streaming systems like Dataflow allow for more flexible, event-time processing.
• Corrupt data caused by software bugs or configuration errors requires automated blocking policies and abuse detection to prevent user-facing issues.
• Effective debugging requires visual data flow diagrams, direct log access, and the ability to trace specific units of work through the system.

Stale data is almost always better than incorrect data.

• Immediate mitigation involves preventing corrupt data entry and draining affected regions to isolate the issue.
• Data recovery often requires labor-intensive reprocessing, which can be optimized through selective reprocessing or intermediate checkpoints.
• External dependencies like storage and network systems can cause failures through throttling, resource exhaustion, or storage bugs.
• Application and configuration errors are the most common causes of outages, necessitating robust nonproduction testing and rollback capabilities.
• Sudden resource growth requires a combination of automatic scaling and emergency resource planning for both the pipeline and its downstream dependencies.

Recovering from this kind of data corruption is labor-intensive and difficult to automate.

• Regional outages pose a severe threat to singly homed pipelines, potentially leading to stranded data and incorrect outputs.
• Multihomed pipelines with automatic failover provide a robust defense by allowing processing to drain from affected regions.
• Spotify's event delivery system manages hundreds of billions of daily events, ranging from user clicks to royalty-critical play counts.
• The system organizes data into delivered hourly buckets partitioned by type and time to simplify access control and ownership.
• Performance and Service Level Objectives (SLOs) for Spotify's infrastructure are defined by the successful delivery of these hourly buckets.

Most importantly, we pay royalties to artists based on delivered events. It’s imperative that we have a reliable means of event storage and delivery.

• Spotify decouples data collection from delivery using Google Cloud Pub/Sub to create independent failure domains and increase system resilience.
• The system ensures event type isolation by publishing different user actions to dedicated topics, preventing one event stream from impacting another.
• The ETL process follows a three-step microservice architecture: consuming streams, assigning hourly partitions, and deduplicating data via Dataproc.
• Resource management is handled dynamically through GCE Autoscaler, allowing engineering teams to enable or disable event types via simple configuration.
• Service Level Objectives (SLOs) for timeliness, completeness, and skewness are used to manage customer expectations and guide system design.
• The architecture relies on Google Cloud Storage (GCS) as the final destination for hourly buckets of processed event data.

Once decoupled, data collection and delivery act as independent failure domains, which limits the impact of any production issues and results in a more resilient system.

• Timeliness is measured by the delay between an hourly bucket's theoretical closing time and its actual delivery.
• The Datamon tool provides a visual interface for monitoring delivery status, using color-coded rectangles to track SLO compliance.
• Downstream data jobs are dependent on timely bucket delivery, creating a ripple effect of delays across the processing chain.
• Event delivery is categorized into high, normal, and low priority tiers to manage system performance during incidents.
• Skewness SLOs track the percentage of misplaced data, which is critical for finance-bearing events where accurate reporting is mandatory.
• Completeness metrics monitor for data loss caused by software bugs, service outages, or accidental deletion through internal auditing.

A skew can negatively impact jobs, since they might first underreport and then overreport values for some time periods.

• Spotify buckets events based on server arrival time rather than client production to avoid issues with offline buffering and device clock manipulation.
• The system functions like a postal service, focusing on delivery logistics while leaving data quality and business logic SLOs to the individual internal teams.
• To maintain scalability, the service is fully managed and hides system complexity behind a simplified API with limited, standardized functionalities.
• Event delivery must be explicitly enabled by owners who define financial-bearing status and timeliness requirements to ensure accountability.
• Documentation is treated as a software product, where every support request is viewed as a bug in either the code or the existing guides.
• The team aims for a self-service model where clear ownership and robust monitoring minimize the need for direct support as the customer base grows.

In this regard, we use the analogy that event delivery should behave like a postal service: your mail should be delivered on time, intact, and unopened.

• Proactive monitoring of system metrics and CPU usage is essential to resolve issues before Service Level Objectives (SLOs) are breached.
• Application logs provide critical health data but must be carefully filtered to prevent 'log drowning' and performance degradation.
• Capacity planning at Spotify involves provisioning components at 50% CPU usage during peak hours to provide a safety margin for traffic bursts.
• The transition from static resource allocation to GCE Autoscaler improved utilization but introduced risks of 'runaway' scaling.
• To mitigate autoscaling failures, the team implements instance limits, daemon CPU restrictions, and aggressive throttling when no useful work is detected.

When Autoscaler is presented with constantly increasing CPU usage that has no correlation with the amount of work performed, it will scale indefinitely until it uses all of the resources it can find.

• Spotify utilizes a testing pyramid philosophy consisting of unit, integration, and end-to-end tests to ensure system stability.
• The event delivery system is considered so critical that it requires a conservative, staged deployment process with manual approvals.
• Deployment stages include a staging environment with mirrored production traffic followed by a canary release to a subset of production instances.
• Incident response prioritizes damage mitigation and system stability, often involving immediate rollbacks if new code caused the failure.
• The team relies on external services like Google Cloud Pub/Sub to reduce operational load, accepting a trade-off between speed and direct control.
• Operational failures are frequently attributed to software bugs, performance regressions, or breaking changes in external service APIs.

Nevertheless, we believe that the ability to delegate responsibility is worth the occasional feeling of powerlessness.

• Complex system failures often arise from unpredictable edge cases combined with heavy production loads.
• Breaches in data completeness or skewness SLOs require manual intervention, including event redelivery and reshuffling.
• Spotify transitioned from a high-toil, unreliable system to a modularized product focused on meeting specific SLOs.
• A unified team structure for both development and operations allows maintenance experience to directly inform system improvements.
• Applying SRE best practices to data pipelines reduces manual work and increases reliability during periods of rapid growth or migration.
• Effective configuration design is a critical but often frustrating SRE task that requires deep system familiarity and clarity.

Because previous iterations were far less reliable, our engineers were paged every few nights.

• Configuration serves as a low-overhead human-computer interface for modifying system behavior without the need for expensive code redeployments.
• Infrastructure systems are composed of three overlapping components: the software, the data set, and the system configuration.
• While code changes are typically incremental and heavily reviewed, a single configuration change can have immediate, dramatic, and potentially catastrophic impacts.
• Configuration is often managed under high-pressure scenarios, such as incident response, making usability a critical factor for system reliability.
• Poorly designed configuration interfaces increase cognitive load and operator error, drawing parallels to early aviation accidents caused by confusing controls.

In contrast, changing a single configuration option can have dramatic changes on functionality—for example, one bad firewall configuration rule may lock you out of your own system.

• Configuration is divided into philosophy, which covers structure and abstraction, and mechanics, which covers language and deployment.
• Separating philosophy from mechanics allows designers to focus on usability and clarity rather than getting bogged down in implementation details like XML or Lua.
• The ultimate goal of configuration philosophy is to move toward 'no configuration at all,' where systems automatically adapt to their environment.
• Reducing the number of manual tunables decreases the surface area for human error and lowers the cognitive load on operators.
• Modern system complexity makes extensive human training, like that required for NASA spacecraft controls, no longer feasible for the industry.
• At its core, any configuration interface—whether a GUI or a text file—is essentially a series of questions being asked of the user.

In the following philosophy, our ideal configuration is no configuration at all.

• Software design faces a choice between infrastructure-centric views (maximizing knobs for perfection) and user-centric views (minimizing knobs to reduce chores).
• A user-centric approach requires deep research into specific use cases rather than providing a generic, highly configurable base.
• Limited configuration often leads to higher adoption rates because the software works 'out of the box' with lower onboarding effort.
• Configuration inputs exist on a spectrum between high-level goals (e.g., 'hot green tea') and low-level implementation details (e.g., water volume and steeping time).
• Focusing on high-level goals allows the underlying system to evolve and improve its implementation without requiring the user to change their inputs.
• Configuration questions are categorized into mandatory requirements for functionality and optional settings for optimization.

Perhaps counterintuitively, limited configuration options can lead to better adoption than extremely versatile software—onboarding effort is substantially lower because the software mostly works 'out of the box.'

• Systems should minimize mandatory configuration questions to reduce the cumulative cognitive load on engineers.
• Static defaults provide safe and effective baseline settings for the majority of users without requiring manual input.
• Dynamic defaults allow systems to automatically adjust parameters, such as thread counts or memory limits, based on the environment.
• When defaults fail for a significant user base, the underlying logic should be improved rather than adding more configuration knobs.
• Defaults act as a powerful nudge, as seen in organ donation rates, placing a heavy responsibility on designers to choose wisely.
• Excessive optional questions can lead to confusion and should be removed if they lack a clear use case.

While one might argue that adding one or two small steps incurs little cost, the life of an engineer is often an endless chain of individually small steps.

• Configuration knobs should only be added when motivated by a real, demonstrated need to avoid unnecessary complexity.
• A successful strategy for power users involves high-level defaults with optional overrides for fine-tuning low-level aspects.
• Designers should optimize for the total sum of hours spent configuring across an organization, accounting for decision paralysis and error correction.
• If a large portion of users require complex configurations, the initial product assumptions and common use cases likely need to be re-evaluated.
• Choosing between pure data formats like YAML and higher-level languages is a fundamental decision in configuration mechanics.

Consider not only the act of configuration itself, but also the decision paralysis users might experience when presented with many options, the time it takes to correct the configuration after taking a wrong turn, the slower rate of change due to lower confidence, and more.

• Configuration is best managed by separating high-level code or interfaces from the static data used by infrastructure.
• Infrastructure should consume plain static formats like Protocol Buffers, YAML, or JSON to ensure stability and interoperability.
• High-level interfaces can range from domain-specific languages like Jsonnet to web UIs, acting as a compilation layer for the raw data.
• Separating the interface from the data representation allows organizations to adapt to diverse cultural norms and technical requirements.
• Static configuration data enables powerful post-deployment analysis, such as using SQL to query configuration parameters across an entire fleet.
• Storing metadata about the source of the configuration helps track authorship and manage the lifecycle of specific features.

To answer the age-old question of whether configuration is code or data, our experience has shown that having both code and data, but separating the two, is optimal.

• Decoupling internal data structures from external interfaces prevents implementation details from leaking into configuration.
• Tooling is the critical factor that determines whether a configuration system is scalable or a chaotic nightmare.
• Semantic validation should be prioritized to catch errors like typos or unrealistic resource requests before they reach production.
• Standardized syntax tools like linters and autoformatters reduce cognitive load and facilitate automated system-wide updates.
• Robust change tracking and ownership are essential for incident response and postmortem analysis in critical systems.
• Automated rewriting tools help large organizations manage deprecations and avoid the accumulation of legacy configuration issues.

Validating that the configuration is semantically meaningful, to the maximum extent possible, can help prevent outages and decrease operational costs.

• Clear ownership of configuration snippets is essential for tracking changes and identifying responsibility during incidents.
• Versioning configuration through tools like Git allows for historical audits and tighter coupling between software versions and their settings.
• Safe deployment requires a gradual, rolling strategy rather than an all-at-once push to prevent global outages.
• Configuration must be hermetic, meaning it does not rely on external, volatile resources, to ensure rollbacks are predictable and effective.
• Systems should implement automatic rollbacks or safety prompts for changes that could result in a loss of operator control, such as network or display settings.

Configuration that requires external resources that can change outside of its hermetic environment can be very hard to roll back.

• Configuration design should be treated as a deliberate process similar to API and UI design rather than a byproduct of implementation.
• Separating configuration into philosophy and mechanics allows for better scoping and clarity in internal system design.
• Investing time in reducing mandatory questions and simplifying options leads to higher adoption rates and lower support requirements.
• As systems scale, simple data formats often evolve into 'replication toil' where redundant changes must be made across multiple locations.
• While automation and custom configuration languages can reduce duplication, they do not automatically eliminate configuration-related toil.

Trivial configuration changes can impact a production system in dramatic ways, so we need to deliberately design configuration to mitigate these risks.

• Complexity toil arises from emergent, undesirable behaviors in automated systems and typically compounds as engineering teams grow beyond ten people.
• One strategy for reducing toil is removing configuration entirely by allowing applications to handle defaults and dynamic scaling internally.
• When configuration is necessary, automation and specialized configuration languages can mitigate replication toil and reduce manual duplication.
• Effective configuration systems require hermetic evaluation to ensure reliable rollbacks and consistent replayability across environments.
• A common industry pitfall is failing to treat configuration design as a formal programming language problem, leading to poorly structured systems.
• Successful setups must separate configuration from data to enable better analysis and provide robust tooling like linters and debuggers.

If you’re not intentionally designing a language, then it’s highly unlikely the “lan‐ guage” you’ll end up with is a good one.

• Data-only formats often evolve into esoteric and complex programming languages through the back door.
• Features like count attributes or string interpolation create ad hoc languages that lack formal design and expressive power.
• Combining pure-data formats like YAML with templating engines like Jinja makes systems difficult for humans and tools to maintain.
• Ad hoc language features often lead to undocumented semantics and a lack of essential tooling like IDE support and linters.
• Niche domain-specific solutions suffer from a lack of community resources and learning materials like Stack Overflow.
• Designers should anticipate the need for programming constructs early rather than adding them incrementally as pitfalls.

If our configuration strategy starts with the objective of using a data-only format, programming language features tend to creep through the back door.

• Side effects during configuration runs violate hermeticity and prevent the separation of config from data.
• General-purpose scripting languages like Python or Ruby are often too heavyweight and require intrusive sandboxing for configuration tasks.
• Domain-specific languages (DSLs) like Jsonnet, Dhall, and HOCON provide a balance of power and safety for complex configurations.
• Jsonnet is an extension of JSON that adds computational constructs while remaining familiar to programmers through Python-like syntax.
• Integrating a configuration language often involves generating standard formats like JSON or YAML to ensure compatibility with existing applications.
• The most complex system failures often occur at the edges where configuration and production environments interact.

In an extreme case, it is impossible to debug your config without spending money by reserving cloud resources.

• Configuration languages use generic primitives like maps and lists to represent data and reduce manual toil.
• Serialization functions allow a single internal representation to output multiple formats like INI, XML, or YAML.
• Unifying configurations enables cross-application synchronization, such as defining a port once for both a web server and a firewall.
• Advanced tools like Jsonnet can manage nested configurations and output multiple files from a single execution.
• Kubernetes serves as a primary case study for configuration complexity because it lacks a built-in abstraction language.
• Users often adopt external configuration languages to manage large-scale Kubernetes infrastructure that outgrows basic YAML.

Once you’re able to generate configuration in the necessary formats, you can easily unify, synchronize, and eliminate repetition across your entire config corpus.

• Kubernetes manages cluster features like load balancing and autoscaling through JSON objects typically authored as YAML files.
• While YAML is more human-readable than raw JSON, it lacks the abstraction capabilities needed to manage multiple similar object variants efficiently.
• The practice of duplicating configuration files for minor variations leads to 'configuration-induced toil' and obscures important differences between environments.
• Jsonnet serves as a powerful alternative by allowing developers to create templates that emit valid JSON for the Kubernetes API.
• Using a configuration language like Jsonnet reduces maintenance by centralizing common logic and allowing specific overrides for different tiers or namespaces.

The variants are hard to read and maintain because the important differences are obscured.

• Jsonnet allows for the creation of abstract templates that can be instantiated multiple times to reduce configuration toil.
• The use of hidden fields and the 'self' reference enables complex object nesting and validation within templates.
• Templates function similarly to abstract classes in programming, requiring specific fields to be overridden to avoid runtime errors.
• Unlike simple function parameterization, templates provide an 'escape hatch' allowing users to override any low-level field in the parent object.
• Abstracting commonalities across configurations promotes modularity and separation of concerns between infrastructure and application teams.
• The workflow for adopting Jsonnet involves converting existing YAML to JSON and then manually applying abstraction constructs.

Template overrides provide a useful escape hatch to change specific details that might normally be considered too low-level.

• Custom applications should be designed to coexist with reusable configuration languages by separating language-level logic from application functionality.
• Applications should consume a single pure data file, allowing the configuration language to handle file imports and merging logic.
• Collections of named entities should be represented as objects rather than arrays to enable easier extension and direct referencing by name.
• Configuration structure should group logically related items into subtrees rather than grouping by type at the top level to facilitate functional abstraction.
• Data representations should remain simple and avoid embedding custom language features like string interpolation or complex abstractions that compete with the config language.
• While configuration languages can hide complexity through templates, the ultimate goal should be to remove configuration entirely whenever possible.

This strategy makes the collection (and individual animals) easier to extend, and you can reference entities by name (e.g., animals.cat) instead of referencing brittle indexes (e.g., animals[0]).

• Configuration changes are a leading cause of system outages, necessitating rigorous validation beyond simple syntax checks.
• Domain-specific validation should verify application-level constraints like value ranges, file existence, and required fields.
• Treating configuration as code requires software engineering disciplines such as versioning libraries to manage breaking changes.
• Source control is essential for configuration to enable historical records, peer reviews, and reliable rollbacks.
• Automated tooling, including linters, formatters, and precommit hooks, ensures consistent style and early error detection.
• Upstream template libraries and utility functions must be supported by unit tests to ensure they generate expected concrete outputs.

In our experience, configuration changes tend to dominate outage root causes over time in a system.

• Jsonnet supports unit testing through native assert statements and community-developed frameworks that allow for structured test suites.
• Hermeticity is a core requirement, ensuring that configuration code produces identical output regardless of the execution environment.
• Checking in generated JSON alongside Jsonnet allows reviewers to audit concrete changes but can lead to merge conflicts and unreadable files.
• Evaluating configuration at build time avoids desynchronization and reduces runtime complexity by embedding JSON directly into release artifacts.
• The timing of configuration evaluation—from pre-commit to runtime—involves trade-offs between auditability, build complexity, and risk exposure.

Generally, hermeticism means that Jsonnet code is always interchangeable with the expanded JSON it represents.

• Evaluating Jsonnet at runtime simplifies workflows by removing pre-evaluation steps but increases application footprint and risk exposure.
• The timing of evaluation depends on the environment, ranging from local client-side tools to server-side deployment daemons like Helm or Spinnaker.
• Configuration languages, even those that are not Turing complete, are vulnerable to resource exhaustion attacks through nonterminating loops or exponential memory growth.
• Simple formats like XML and YAML are not immune to programs that can practically function as nonterminating or resource-heavy 'bombs'.
• To mitigate Denial of Service (DOS) risks in multi-tenant environments, untrusted configuration code should be sandboxed using separate processes and resource limits like ulimit.

Unfortunately, due to bugs or deliberate attacks, configuration may take an arbitrary amount of CPU time or memory.

• Configuration languages should prioritize good tooling, hermeticity, and the separation of configuration from data.
• Release engineering encompasses all processes and artifacts required to move code from a repository into a production environment.
• Canarying is defined as a partial, time-limited deployment used as an A/B testing process to evaluate changes before a full rollout.
• Core principles of release engineering include reproducible builds, automated testing, and automated deployments.
• Automation reduces operational toil, enforces peer review, and allows for the measurement of release pipeline efficiency.
• Small, self-contained deployments are preferred to minimize risk and simplify the rollback process.

The logic underpinning this approach is that usually the canary deployment is performed on a much smaller subset of production, or affects a much smaller subset of the user base than the control portion.

• Automated releases create a virtuous cycle where smaller, frequent updates make rollbacks easier and bug fixes faster.
• While shipping speed and reliability are often seen as opposing goals, they can be balanced using SLOs and error budgets.
• Google's data indicates that the majority of system incidents are triggered by binary or configuration pushes.
• Separating components like binaries, libraries, and configurations allows different parts of a service to change at independent rates.
• Feature flags and experiment frameworks enable developers to decouple feature launches from binary releases, reducing risk.
• Canarying serves as a final safety check by exposing a small amount of real production traffic to new code before a full rollout.

The business wants to ship new features and product improvements as quickly as possible with 100% reliability!

• Canarying mitigates the inherent risk of deploying new features by exposing changes to a small fraction of real-world traffic.
• The process identifies defects that artificial testing frameworks, such as unit or load testing, often fail to capture.
• A successful canary process requires a deployment method for subsets, a robust evaluation process, and integration into the release pipeline.
• Confidence in a release candidate is built incrementally by gradually increasing the volume of traffic the new version handles.
• The ultimate goal is to detect bad release candidates with high confidence while avoiding false positives on good releases.

Introducing the change to actual production traffic also enables us to identify problems that might not be visible in testing frameworks like unit testing or load testing, which are often more artificial.

• Standard deployments without canary processes risk high error rates and prolonged outages if rollbacks are not readily available.
• Canary deployments reduce risk by isolating a release candidate to a small segment of production traffic for comparison against a control group.
• While canary errors may be invisible in aggregate traffic trends, per-version monitoring reveals the specific impact of a new release.
• Automated deployment systems can use canary error rates to trigger immediate rollbacks, preventing widespread user impact.
• Traffic splitting for canaries can be achieved through various methods including load balancer weights, proxy configurations, or DNS records.

Figure 16-3 shows that the impact of the change is greatly reduced when we use a canary; in fact, the errors are barely visible!

• Canary deployments minimize risk to Service Level Objectives (SLOs) by exposing only a small fraction of traffic to potential defects.
• The impact on an error budget is directly proportional to the size of the canary population and the duration of the deployment.
• A worst-case scenario model assumes a 100% failure rate in the canary, allowing for conservative planning and budget preservation.
• Over-investing in model complexity often leads to diminishing returns and incessant tuning without real business benefit.
• Canary duration must be balanced against development velocity to avoid bottlenecks in the release pipeline.
• Running multiple simultaneous canaries is discouraged due to increased mental overhead and the risk of signal contamination.

As British statistician George Box said, “Essentially, all models are wrong, but some are useful.”

• Canary deployments must be large and long enough to provide a representative sample of diverse queries and system behaviors.
• Traffic volume requirements vary based on request homogeneity; more diverse inputs require higher volumes to distinguish artifacts from real defects.
• Timing is critical as performance defects often only manifest under heavy load, making off-peak deployments less effective for testing.
• Selecting the right metrics is a balancing act between sensitivity to service health and the risk of false positives from overly strict thresholds.
• Service Level Indicators (SLIs) are recommended as a starting point for canary metrics because they correlate strongly with overall service health.
• The process of defining 'acceptable behavior' is time-consuming but necessary to avoid both false alarms and undetected system failures.

Canarying is a balancing act, informed both by cold analysis of worst-case scenarios and the past realistic track record of a system.

• Prioritize metrics that directly reflect user-perceivable problems, such as latency and HTTP return codes, over internal resource usage.
• Limit canary evaluations to a small set of high-signal metrics to avoid maintenance overhead and 'alert fatigue' that erodes trust.
• Exclude ambiguous signals like 404 errors from automated canary analysis to prevent external user behavior from triggering false positives.
• Ensure metrics are strictly attributable to the code change by isolating them from system-wide noise like log rotation or shared database load.
• Address infrastructure variance by increasing canary population size or selecting metrics with lower inherent volatility.
• Beware of shared failure domains where a malfunctioning canary might negatively impact the control group or vice versa.

Too many metrics can bring diminishing returns, and at some point, the returns are outweighed by the cost of maintaining them, or the negative impact on trust in the release process if they are not maintained.

• Canary metrics must be attributable to Service Level Indicators (SLIs) to ensure that observed changes actually impact service quality.
• Before/after evaluations are inherently risky because time-based variables and external noise can mask or mimic performance degradation.
• Temporal comparisons often fail to account for natural shifts in user behavior, such as the difference between business day and weekend traffic.
• A multi-stage gradual canary approach allows for safer testing by starting with high-confidence metrics like application crashes before scaling up.
• Imperfect isolation between canary and control groups can lead to cross-contamination where one environment's response alters the other's behavior.

The response by the canary may change the content of the second request, which may land on the control, altering the control’s behavior.

• Imperfect isolation in canary deployments can lead to 'bad behavior' bleeding into the control group, necessitating absolute SLOs for verification.
• Noninteractive systems like data pipelines require canary durations that exceed the processing time of a single work unit.
• To avoid mixed signals in multistage pipelines, work units should ideally be processed by workers from the same pool throughout their lifecycle.
• Standard service-wide monitoring often masks critical failures because a high error rate in a small canary population averages out to a negligible signal.
• Effective canary evaluation requires fine-grained monitoring data that can explicitly differentiate between canary and control populations.

Because monitoring likely looks at the system as a whole, it will detect an overall error rate of only 1%.

• Metrics used to evaluate canary deployments must have intervals equal to or shorter than the canary duration to avoid muddied signals.
• Blue/green deployment allows for zero-downtime cutovers and easy rollbacks but requires double the resource capacity.
• Blue/green setups can function as A/B canaries by splitting traffic between the standby and active environments.
• Artificial load generation provides high code coverage but fails to simulate the complex state coverage of real user traffic.
• Synthetic testing is risky in mutable systems, such as billing, where artificial requests might trigger unintended real-world consequences.

It can be especially hard to artificially simulate load in mutable systems (systems with caches, cookies, request affinity, etc.).

• Traffic teeing allows for representative testing by mirroring production traffic to a canary deployment while discarding responses.
• Teeing is complex to implement and can fail in stateful systems where shared resources like caches invalidate performance metrics.
• Canarying serves as a robust supplement to testing that increases development velocity and minimizes user impact from defects.
• Operational overload occurs when high volumes of pages and tickets prevent SRE teams from focusing on long-term engineering projects.
• Sustained stress from overload leads to human error, decreased reliability, and a loss of the team's ability to regulate daily work.

The whole team starts to feel stressed and frustrated as they work harder but don’t feel like they are making progress.

• Operational load consists of interrupts that prevent Site Reliability Engineers from focusing on long-term engineering projects.
• Operational overload occurs when urgent issues continually preempt priority work, leading to a loss of progress and increased error rates.
• Google SRE teams maintain a strict 50% cap on operational work to ensure engineers have time to automate and reduce future toil.
• Overload is categorized as an occupational stressor that can lead to decreased productivity and serious health issues if left unchecked.
• The text distinguishes between objective overload from shrinking teams and perceived overload, both of which have detrimental effects on team health.
• Effective recovery requires identifying the threshold of manageable work and implementing strategies to shift focus back to impactful projects.

A team is in a state of operational overload when it can’t make progress toward key priorities because urgent issues continually preempt project work.

• Work overload is categorized into actual overload, where tasks exceed available time, and perceived overload, which is a subjective feeling of being overwhelmed.
• Perceived overload often stems from rapid organizational changes coupled with a lack of communication between leadership and teams.
• The unpredictability of on-call work and ticket queues can trigger cognitive biases, causing workers to misjudge their workload before they even begin.
• External stressors such as job insecurity, health issues, and lack of sleep can transform even a manageable workload into a source of psychological distress.
• Proper prioritization and quantifying work through metrics like ticket volume are essential to distinguish between systemic capacity issues and psychological stress factors.

Even if you can finish all the tickets quickly and the actual workload is low, you feel overloaded when you first look at the ticket queue.

• A Google storage SRE team faced a crisis when two-thirds of its staff, including senior leadership, departed within a short window.
• The team suffered from siloed expertise, making it difficult for remaining members to cover operational duties across multiple services like Gmail and Drive.
• Psychological barriers, specifically the sunk cost fallacy, initially prevented the team from dropping in-progress work that was no longer sustainable.
• The team utilized a comprehensive whiteboard triage of all responsibilities to redefine priorities and identify tasks for elimination or hand-off.
• Recovery strategies included implementing low-effort automation and creating self-service documentation to reduce repetitive manual toil.
• A significant portion of the ticket backlog was found to be obsolete or non-actionable, allowing for a rapid reduction in perceived workload.

We didn’t realize we were in the grip of a sunk cost fallacy.

• A team successfully cleared a multi-month backlog in just two days by aggressively triaging non-critical tasks and documenting progress for later review.
• Post-triage analysis revealed that most 'tentative' tasks were actually low-impact and did not require resumption, highlighting the prevalence of busywork.
• To prevent future overload, the team established a hard limit of ten open tickets per person and implemented bi-weekly queue evaluations.
• A second case study illustrates how simultaneous triggers—onboarding noisy services and losing key personnel—can rapidly destabilize a healthy team.
• Knowledge drain and untuned monitoring created a feedback loop where engineers felt helpless and spent more time on pages than on operational improvements.

It turned out that almost none of these tasks were impactful or important enough to resume.

• New ticket SLOs and technical debt created an immediate workload surge that prevented SREs from recovering after on-call shifts.
• A lack of empathy from management led to a disconnect where team members felt unheard, transforming perceived stress into objective overload.
• The resulting exhaustion triggered a decline in productivity and a 'cascade of unhappiness' that spread across the entire team.
• Chronic overload eroded psychological safety, causing team members to stop trusting one another and cease collaboration.
• The team reached a breaking point where career development stalled and promotion rates hit an all-time low before management intervention occurred.
• Recovery began only after a new manager was assigned who prioritized participatory management and simple team-building exercises.

We then started assuming we couldn’t depend on other people to get their work done, which eroded feelings of trust and dependability within the team.

• Establishing psychological safety and stress relief is the foundational requirement for team recovery and clear thinking.
• Short-term tactical relief included silencing non-critical alerts and auditing spam to reduce immediate cognitive load.
• Structural changes like dedicated management and team rebalancing helped restore trust and provide fresh perspectives.
• Social bonding through non-work activities like lunches and games was used to ease tension and improve interpersonal safety.
• Mid-term sustainability focused on limiting operational work to on-call shifts and returning service ownership to developers.
• Knowledge sharing and training were implemented to build individual confidence and accelerate future troubleshooting.

The team deliberately didn’t try to find the source for every single alert, but focused on relieving the stress from being paged and ticketed continuously for issues we already knew about.

• The team addressed systemic overload by silencing noisy alerts, backfilling open roles, and integrating remote SREs to provide fresh perspectives.
• Management prioritized 'listening events' to understand pain points, shifting from top-down mandates to team-driven solutions.
• Long-term stability was pursued through service uniformity and aligning SLOs with backends to reduce cognitive load for engineers.
• Psychological safety was restored over a year, transforming the team's culture from a high-stress environment to one perceived as warm and safe by new members.
• The case study highlights that perceived overload is as damaging as objective workload, often triggered by staffing losses and increased ticket noise.
• A 'vicious cycle' occurs when stress levels prevent a team from trusting each other enough to collaborate on solving the underlying technical debt.

Hope is not a strategy, but it certainly helps team morale.

• Identifying overload from within a team is difficult, requiring a focus on behavioral symptoms like decreased morale and frequent complaints.
• Physical indicators of an overworked team include members working through illness or consistently logging uncompensated overtime.
• An unhealthy task queue, characterized by missed deadlines and a backlog that grows faster than it can be cleared, is a primary sign of systemic overload.
• Key metrics such as high toil ratios and long resolution times for on-call issues can quantify the extent of the team's burden.
• Managers should avoid micromanagement during crises, as giving team members more autonomy and control actually reduces perceived stress and improves performance.

The path toward a healthy, friendly, and happy work atmosphere can be hard to visualize when you’re mired in overload.

• Restoring psychological safety is the primary requirement for fixing a dysfunctional, overloaded team.
• Managers should identify psychosocial stressors and mitigate workload by reducing backlogs or silencing non-essential alerts.
• Empowering team members with specific technical leadership roles builds the self-confidence necessary to take interpersonal risks.
• Transparent and democratic decision-making, such as collaborative backlog grooming, helps teams regain a sense of control.
• Teams must prioritize project work that automates away repetitive toil to prevent future cycles of burnout.
• Establishing metrics and early warning signs allows teams to detect and address growing backlogs before they become unmanageable.

A team can function only as well as its individual members.

• Frequent operational interrupts create a self-enforcing cycle of overload that negatively impacts team health and productivity.
• Perceived overload is a psychological stressor that is difficult to measure or eliminate compared to standard toil.
• The primary goal of SRE is to maximize developer velocity while maintaining product reliability, though complexity can limit effectiveness.
• SRE teams must strategically choose where to focus their attention as microservice architectures often outpace team capacity.
• Effective SRE engagement involves understanding the goals of developer teams throughout the entire service lifecycle, from planning to scaling.
• Proactive communication and defining the SRE role are essential for successful collaboration regardless of organization size.

Perceived overload is a special form of overload that can’t be measured by the amount of toil or operational work.

• Early SRE involvement in architecture and design prevents costly redesigns by establishing best practices and validating infrastructure assumptions.
• During active development, SREs focus on productionization tasks such as capacity planning, redundancy, and sustainable monitoring practices.
• The Limited Availability phase requires defining Service Level Objectives (SLOs) to objectively measure reliability before a general launch.
• Shared operational responsibility between developers and SREs during early phases ensures both teams understand the service's failure modes.
• In General Availability, SREs manage the bulk of operations, but developers must remain in the on-call rotation to maintain operational perspective.

Fixing architectural mistakes becomes more difficult later in the development cycle.

• The transition phase requires SREs to support two systems simultaneously, necessitating adjusted headcount and staffing.
• Once a service is abandoned or unsupported, SREs hand back operational control to developers or assist in the final decommissioning of the service.
• Successful SRE engagement requires a deep understanding of business goals and regular communication between SRE and developer leadership.
• SREs and developers must align their goals by balancing long-term service viability with the need for feature and launch velocity.
• Developers should commit a percentage of engineering time to fixing reliability issues and technical debt in exchange for SRE support of release velocity.
• SRE teams are uniquely positioned to identify and gauge the impact of system risks to prevent disruption of feature flow.

SREs can have an explicit goal to support the developer team’s release velocity and ensure the success of all approved launches.

• The core mission of the SRE function at the New York Times is to empower product teams with tools and processes that maximize reliability for journalism distribution.
• A shared goals model is used to balance the SRE team's internal automation backlog with external requests from product development teams.
• Engagements are structured as either full-time embedded roles or part-time constrained projects based on SRE bandwidth and company strategy.
• Application owners remain directly responsible for making changes to their applications, as SREs are not traditional operations engineers for manual tasks.
• Successful engagements prioritize creating company-wide tooling and automation rather than developing one-off scripts for specific teams.
• Jointly defined milestones and readiness reviews ensure that both SREs and developers are aligned on priorities and expectations.

SREs are not traditional operations engineers. They do not support manual work such as running a job for deployment.

• Define explicit success stories to ensure both SRE and development teams understand the desired end state of their collaboration.
• Establish a structured roadmap including architecture reviews, joint planning sessions, and production readiness reviews.
• Maintain a constant feedback loop outside of standard Agile ceremonies to address friction and allow for early disengagement if necessary.
• Utilize a maturity matrix to assess service health across axes like observability and incident response before and after the engagement.
• Measure the impact of SRE work to ensure high-value contributions and to track the evolution of the partner team's operational capabilities.

If an engagement isn’t working, we expect teams to not shy away from planning for disengagement.

• SRE teams aim to automate service operations to a point where they can transition to new high-value engagements.
• Successful cooperation requires hard limits on operational work and measured Service Level Objectives (SLOs) to prioritize engineering tasks.
• Quarterly error budgets should be established to regulate release velocity and manage service capacity for unexpected growth.
• Developer involvement in daily operations is essential to ensure root causes of issues are visible and prioritized for fixing.
• Regular strategic planning and roadmaps help align SRE goals with product leadership and identify when a team should expand or dissolve.
• Investing time in interpersonal relationships between SREs and developers facilitates faster escalation and resolution during outages.

In a stable environment, the absence of a roadmap may be a signal that the SRE team can merge with another team, move service management work back to the development teams, expand scope, or dissolve.

• Quarterly 'state of production' and 'state of the product' talks align SRE efforts with developer accomplishments and future roadmaps.
• Annual face-to-face service reviews facilitate long-term planning and retrospective exercises to identify what to stop, keep, or start doing.
• When cooperation or service quality regresses, teams must pivot priorities through reliability hackathons, feature freezes, or 'all hands on deck' responses.
• Service Level Objectives (SLOs) and error budgets act as the primary mechanism for balancing feature velocity against tactical and strategic reliability fixes.
• A blameless culture is maintained by focusing on system behavior rather than human error, especially during high-stress situations.
• Effective communication requires emotional intelligence, such as delaying difficult conversations until participants are rested and calm.

Readers will remember how the words made them feel, not necessarily what was written.

• Prioritize high-bandwidth communication like video chats or in-person meetings to resolve complex issues and disambiguate tone.
• Foster a positive culture by explicitly thanking engineers for good code comments, design reviews, and failure scenario training.
• Google maintains a lean SRE-to-developer ratio of less than 10%, necessitating that one SRE team often supports multiple services.
• Scaling limited SRE resources is most effective when services share a single product goal, a similar tech stack, and a small number of developer teams.
• Large organizations should structure SRE teams to shadow the developer hierarchy or group them by technology stack to ensure alignment.
• Organizing SRE teams by technology rather than developer reporting structures helps prevent team churn during frequent corporate reorganizations.

Interactions conducted solely via code reviews or documentation can quickly become drawn out and frustrating.

• SRE teams should be restructured through sharding or merging based on service needs and operational load rather than building from scratch.
• Global distribution is recommended for 24/7 coverage, but teams should be colocated based on service adjacency to avoid the vulnerability of singleton teams.
• Rotating responsibilities between physical locations prevents knowledge silos and the development of an 'us versus them' mentality.
• SRE engagements are not indefinite and should be terminated if the work becomes primarily operational (toil) rather than impactful engineering.
• Handing back a service to developers is appropriate when the service is optimized, its business relevance has diminished, or it is reaching end-of-life.

We’ve found that the costs of this configuration ultimately outweigh the benefits. Although that location will tend to become really good at executing those responsibilities, this fosters an “us versus them” mentality, hinders distribution of knowledge, and presents a risk for business continuity.

• The inherent conflict between rapid abuse-fighting needs and SRE reliability goals led to friction between the CAT and Abuse SRE teams.
• To resolve this, a dedicated infrastructure team called 'Ares' was formed within CAT to unify and manage abuse-fighting systems.
• Abuse SREs mentored the Ares team on reducing cognitive load through system homogeneity and modular microservice architecture.
• The new infrastructure provided abstractions that allowed developers to focus on features without managing low-level production details.
• This transition enabled Abuse SRE to disengage from direct support while significantly increasing developer velocity and reducing time to production.
• Integration for new products became faster and more predictable as feature launches no longer required deep infrastructure overhauls.

Abuse SREs taught the Ares team that the easiest way to launch a new service in production (when you’re already running large distributed services) is to minimize the additional cognitive load that the service imposes.

• The Ares project succeeded by fostering a sense of shared destiny between SREs and product development through collaborative infrastructure decisions.
• When the cost of an SRE support relationship exceeds its perceived value, disbanding the SRE team can be the most logical outcome.
• A decade-long partnership at Google faced a crisis when a legacy data analysis pipeline hit scaling limits, necessitating a new system.
• Prioritizing a new system often leads to a 'frozen' state for legacy infrastructure, where even conservative changes become difficult despite growing usage.
• Organizational separation between teams maintaining the old system and those building the new one can lead to insurmountable communication breakdowns.
• Decommissioning the SRE teams and returning full control to the developers resolved tension by allowing a single team to balance operational needs against development velocity.

Although decommissioning two SRE teams was not a pleasant experience, doing so resolved the continual tension over where to invest engineering effort.

• Realigning ownership of legacy systems with developers can accelerate troubleshooting due to their deep internal knowledge.
• A healthy SRE-developer relationship involves temporary hand-offs to stabilize systems before SRE reassumes responsibility.
• Effective engagement management is as critical to service success as technical design decisions.
• Long-term success depends on aligning team goals and maintaining open communication rather than just defending SLOs.
• The rapid expansion of SRE practices outside of Google is making the future of the profession both exciting and difficult to predict.
• Reliability remains the foundational principle and most important feature of any service.

SRE and development teams need to be willing to address issues head on and identify points of tension that need resetting.

• Reliability is the most critical feature of any system because user trust is the foundation of a system's value and network effects.
• The only metric of reliability that truly matters is the user's experience, regardless of what internal monitoring dashboards indicate.
• When a service provides an API, it becomes a platform where reliability is a partnership between the provider and the consumer's implementation.
• Platform providers are often held accountable for instability even when the root cause lies within the user's own poorly architected integration.
• System evolution toward becoming a platform is inevitable; if you do not provide an official API, users will create unofficial ones to access your audience.
• High customer friction and support burdens act as a tax on innovation, diverting engineering energy away from system improvements.

If your user is worried that your platform is responsible for instability they’re experiencing, then telling them “our monitoring looks fine; the problem must be on your end” won’t make them any less grumpy.

• Internal platform teams are not exempt from SRE principles and often face the highest risk of being consumed by toil.
• To ensure customers build reliable systems on your platform, you must actively teach them SRE practices through direct collaboration.
• Static resources like books and diagrams are insufficient; you must 'do SRE' with a representative sample of users to keep lessons relevant.
• Service Level Objectives (SLOs) and Indicators (SLIs) serve as the essential shared language between platform providers and their users.
• Without defined SLOs, customers will invent their own arbitrary performance expectations and alert you whenever those invisible thresholds are crossed.
• Collaborating on reliability with customers helps accelerate your own SRE journey and prevents circular, unproductive arguments about performance.

Remember, in the absence of a stated SLO, your customer will inevitably invent one and not tell you until you don’t meet it!

• Adopting a shared language of SLOs and SLIs between providers and customers transforms vague complaints into productive, data-driven conversations.
• Auditing customer monitoring often reveals that up to half of existing alerts have zero impact on actual service objectives and should be disabled.
• Shared dashboards eliminate information asymmetry, ensuring both parties see the same performance data when an error budget is threatened.
• Initial data collection often provides a 'rude awakening' by revealing that perceived 'five 9s' availability is frequently much lower in reality.
• If users are satisfied despite lower-than-expected metrics, it indicates that the SLO targets should be renegotiated rather than over-engineered.
• Deep technical collaboration involves conducting design reviews to identify single points of failure and manual operational risks.

The application they thought was operating at “five 9s” (99.999%; everybody thinks they’re getting five 9s) is probably achieving only 99.5%–99.9% when measured against their shiny new SLOs.

• Reviewing customer error budgets helps identify reliability mistakes and the specific trade-offs users make to improve uptime.
• Operational rigor should be built through simulated disaster exercises like 'Wheel of Misfortune' to develop team muscle memory.
• Joint postmortems between providers and customers foster trust and reveal edge cases that can be integrated into platform enhancements.
• Because deep engagement is resource-intensive, teams must use principled selection criteria like revenue, feature, or workload coverage.
• The future of SRE involves blurring the lines between the service provider and the customer to achieve mutual operational success.

Develop a healthy muscle memory between the teams for effective ways to communicate during a crisis.

• SRE principles are applicable to any organization, regardless of size or whether they have dedicated SRE staff.
• The foundational requirement for SRE is the implementation of SLOs with consequences to guide business decisions.
• Adopting an error budget policy allows teams to move from reactive firefighting to data-informed reliability management.
• Initial SRE hires should possess a mix of skills in operations, software engineering, monitoring, and automation.
• The first SRE in an organization must navigate the tension between product velocity and system reliability.
• Strategic placement of the first SRE—whether embedded in product, operations, or a horizontal role—is critical for influence.

Even if an organization doesn’t have SRE staff, we believe that it is worthwhile to set SLOs for critical customer applications and to implement an error budget policy, if only because an implicit 100% SLO means a team can only ever be reactive.

• Embedding SREs into operations or product teams can break down silos and address immediate technical risks or production issues.
• Long-term organizational goals should dictate SRE placement, as embedding them in product teams early may make future centralization difficult.
• The first SRE's primary mission is to establish SLOs and error budgets while identifying the 'toil' required to maintain the system.
• A core SRE principle is ensuring engineers have time to make 'tomorrow better than today' by balancing operational work with project-based improvements.
• Warning signs of a failing SRE role include the SRE performing a disproportionate share of manual operations or their work becoming indistinguishable from standard development.
• In organizations without a centralized SRE department, creating a community for distributed SREs is vital to prevent isolation and maintain consistent standards.

SREs must have time to make tomorrow better than today.

• Organizations can establish SRE teams by creating new project-specific units, forming horizontal consulting teams, or converting existing operations teams.
• A core principle of SRE is the team's ability to regulate its own workload, though this is often difficult to implement from day one.
• The 'Forming' stage requires a mix of skills including software engineering for automation, system architecture knowledge, and methodical operational practices.
• Seeding new teams with internal transfers is recommended to leverage existing relationships and reduce the initial learning curve.
• Converting an operations team to SRE requires genuine cultural and procedural shifts; simply rebranding a team without changing practices can 'poison' the organization against SRE.
• The 'Storming' phase focuses on building cohesion through collaborative learning, such as SREcon video screenings or book clubs, to encourage critical thinking about practices.

Be careful to avoid renaming a team from “Operations” to “SRE” without first applying SRE practices and principles!

• New SRE teams often face the risk of spreading themselves too thin by managing too many services, leading to constant firefighting instead of permanent risk mitigation.
• Teams may become overly introspective regarding SRE principles, such as obsessing over perfect SLO definitions while neglecting actual service needs.
• A lack of diverse technical skills, particularly in software engineering, can prevent SREs from properly instrumenting products or automating away toil.
• Mitigation strategies include engaging early at the design stage and ensuring the product development team retains operational responsibility on day one.
• Horizontal SRE teams must avoid being perceived as 'gatekeepers' by focusing on delivering high-impact tools that benefit multiple existing teams.
• Converting an existing team into an SRE unit requires addressing fears of job loss due to automation and ensuring there is slack capacity for cultural change.

A team that is constantly firefighting doesn’t have time to address risk in a more permanent way.

• Transitioning to SRE requires senior leadership support and renegotiated responsibilities to create the necessary slack for cultural change.
• Addressing job security concerns is vital, emphasizing that automation replaces repetitive toil rather than eliminating roles, ultimately increasing engineer marketability.
• Performance evaluation metrics must shift to align with Service Level Objectives (SLOs) and the adoption of specific SRE technical skills.
• The 'Norming' phase is reached when teams agree on acceptable toil levels, establish sustainable on-call rotations, and implement error budget policies.
• A mature SRE team fosters a postmortem culture and utilizes training exercises like the 'Wheel of Misfortune' to prevent recurring incidents.

In a lot of environments, automation eliminates portions of work, but not jobs as a whole; while this might be a step on the path to job losses, it does at least have the virtue of freeing up time to do something better (and more sellable to a future employer) than nonautomated toil.

• Existing teams often fall into a reactive 'service team' trap, where work is driven by tickets and production issues rather than innovation.
• A high 'bus factor' and accumulated institutional knowledge can create a bottleneck, leading to constant interrupts from other departments.
• The core SRE strategy involves removing SREs from the critical path by empowering developers with self-service solutions.
• Process improvements included breaking out service configurations into team-based repositories and migrating to developer-friendly CI/CD pipelines.
• Implementing scheduled office hours helped batch interrupts and facilitated knowledge sharing without disrupting deep work.
• The successful transition allowed the team to reach a goal of over 50% project work, enabling focus on advanced resilience and observability features.

This model was not sustainable, and the team would need to grow linearly with products to keep up with this support burden.

• The 'performing' stage marks the transition of SRE from a reactive role to a strategic partner in architectural design and system reliability.
• SRE teams should define patterns for how software is built from the initial design phase to prevent high-cost reengineering later.
• Self-determination of workload is a critical principle that establishes SRE as an engineering peer to product development teams.
• Operational overload can be mitigated by reducing SLOs, transferring work back to developers, or even handing back service responsibility entirely.
• SRE engagement is not perpetual; teams must rotate to new challenges once a service is stabilized to prevent attrition and 'slow bleed' of talent.
• For SRE teams without partner developers, reliability-focused project work must take precedence over new features when SLOs are not met.

Otherwise, your team risks attrition as SREs move on to more interesting opportunities. The slow bleed from attrition can put production at risk.

• SRE teams may need to split when service complexity exceeds a single team's cognitive capacity or to facilitate 24/7 global coverage.
• New teams should be seeded with experienced SREs from existing teams rather than relying solely on new hires due to the difficulty of finding qualified candidates.
• Service splits can be organized by architectural layers, programming languages, or physical office locations to reduce operational overhead.
• When expanding SRE support, organizations should prioritize services based on financial impact and business relevance rather than just unreliability.
• Geographical splits provide a 'follow-the-sun' model that improves quality of life for engineers and ensures disaster recovery resilience.
• To avoid ownership gaps during a split, a senior SRE should be appointed as a technical lead across both teams or one team should be designated as the default owner.

In our experience, finding qualified SRE candidates is difficult, so growing a team quickly with new hires often isn’t realistic.

• Splitting on-call rotations into 12-hour shifts across multiple continents reduces engineer stress and eliminates the need for overnight shifts.
• Geographic distribution forces improvements in production maturity through better documentation, training, and standardization.
• A time zone separation of six to eight hours is ideal for balancing on-call coverage while maintaining enough overlap for team interaction.
• New remote teams should be seeded with experienced SREs from the original site to establish cultural norms and engineering practices.
• Management must be vigilant to prevent the remote office from becoming a 'night shift' that only handles toil and low-impact projects.
• Natural workload differences occur based on daily traffic peaks and release cadences, requiring intentional efforts to maintain parity between sites.

If this is the case, be vigilant to ensure that the team that is not colocated ('Office 2') doesn’t become a night shift that has little contact with the product development team, takes more than its fair share of toil, or is assigned only the less interesting or impactful projects.

• Effective multi-site SRE management requires balancing high-impact projects and operational toil fairly between offices to prevent resentment.
• Splitting projects across sites deliberately fosters interoffice interactions, knowledge sharing, and trust despite minor efficiency losses.
• Three-site rotations are discouraged because they eliminate the 'personal cost' of night-time pages, which reduces the incentive to automate away toil.
• Face-to-face interactions through regular travel budgets are essential for building the rapport needed to resolve technical disagreements.
• Leadership across sites must maintain strong personal relationships to advocate for each other and prevent an 'us versus them' mentality.
• The timing of team spin-ups should consider proximity to product development teams to ensure early involvement in the product lifecycle.

Being the hero that fixes easy problems is fun during office hours. But if it has some amount of personal cost, the motivation to make sure it never happens again is sharp and immediate.

• The Mission Control program embeds product developers into SRE teams for six months to foster cross-functional empathy and production expertise.
• SRE Exchange programs facilitate week-long peer observations between specialized teams to share best practices and identify operational gaps.
• Standardized training curriculums like SRE EDU ensure a baseline level of knowledge across the organization regarding tooling and incident management.
• Horizontal projects prevent the duplication of effort by tasking experienced SREs with building standardized, scalable platforms for the entire company.
• Internal mobility policies allow SREs to transfer between teams freely, preventing burnout and ensuring engineers remain engaged with their specific services.

The software engineer is trained in production systems and practices and eventually goes on-call for that service.

• Internal mobility allows SREs to transfer between teams to manage personal workload, avoid unhealthy services, and spread cultural consistency across offices.
• Investment in travel and industry conferences is essential for building internal communities and keeping engineers inspired by external innovations.
• Launch Coordination Engineering (LCE) teams apply SRE principles to smaller product teams through automation and standardized tooling.
• The 'Production Excellence' review process uses senior leadership to evaluate team health metrics like pager load and error budget usage.
• SRE staffing is intentionally kept below total organizational demand to ensure that engineers are only assigned to the most high-priority, high-value services.
• Standard SRE-to-developer ratios vary significantly based on service complexity, ranging from 1:5 for infrastructure to 1:50 for consumer applications.

In short, you should have fewer SREs than the organization would like, and only enough SREs to accomplish their specialized work.

• The core principles of SRE include SLOs with consequences, time for future-facing improvements, and workload regulation.
• SRE has evolved from a Google-specific practice into a global profession applicable across diverse organizational sizes.
• Organizational change management in SRE focuses on supporting individuals and teams through transitions rather than technical change control.
• SRE teams act as a bridge between rapid product innovation and the necessity of maintaining system reliability.
• The field utilizes established psychological frameworks, such as Lewin’s Three-Stage Model, to refine how technical organizations adapt to change.
• Error budgets serve as the primary mechanism for balancing the inherent risks of constant technological evolution.

Site Reliability Engineers are frequently in the middle of this complicated and rapidly shifting landscape, responsible for balancing the risks inherent in change with product reliability and availability.

• Lewin's three-stage model focuses on unfreezing current behaviors, executing change, and institutionalizing new patterns at a macro level.
• McKinsey’s 7-S Model balances business elements like strategy and systems with people-centric elements like shared values and style.
• Kotter’s Eight-Step Process is highly relevant to SRE, though the 'sense of urgency' is often pre-existing due to scaling and reliability pressures.
• The Prosci ADKAR model emphasizes individual goals—awareness, desire, knowledge, ability, and reinforcement—for successful transitions.
• While ADKAR is people-centric, its iterative nature can be difficult to implement within the high-pressure, time-constrained environment of global SRE teams.

Because SRE is often on the front line when problems occur, it is uniquely motivated to lead the change needed to ensure products are available 24/7/365.

• Emotion-based models like Bridges and Kübler-Ross help managers navigate the psychological impact of change on employee productivity.
• The Deming Cycle (PDCA) is effective for technical process improvements but fails to address the human motivations required for organizational shifts.
• Frequent iterations in organizational structure can be counterproductive, damaging company culture and employee confidence.
• Google SRE utilizes a mix of frameworks, including Kotter’s Eight-Step Process for core changes and ADKAR for global coordination.
• The Waze case study demonstrates how a grassroots technical response to scaling challenges can retroactively align with formal change models.
• Establishing a sense of urgency is the critical first step in Kotter’s model, as seen during Waze's reliability crisis with their messaging queue.

The case studies we refer to in this chapter deal with larger, organizational changes where iteration is counterproductive: frequent, wrenching org-chart changes can sap employee confidence and negatively impact company culture.

• The Waze message queue became a critical bottleneck, requiring 24/7 firefighting and hourly restarts to maintain service for millions of users.
• Extreme operational load on SREs halted feature velocity, prompting leadership to prioritize a custom-built reliability solution over new features.
• To create development time, a 'volunteer army' of developers reduced system load by trimming unnecessary messages and implementing compression.
• SREs engineered a dual-publishing client library that allowed for a gradual, risk-mitigated migration between the old and new messaging systems.
• Short-term wins, such as the backup path preventing outages during 'near misses,' validated the strategic shift and accelerated mass migration.

At its worst, the entire Waze SRE team spent most of a two-week period firefighting 24/7, eventually resorting to restarting some components of the message queue hourly to keep messages flowing and tens of millions of users happy.

• Successful migration to a new messaging system allowed Waze SRE to handle 1000 times the previous load with minimal manual effort.
• The resolution of one bottleneck revealed a new crisis: SRE's manual ownership of releases was severely hindering development velocity.
• A senior developer introduced a microservices framework, which SRE enhanced with reliability features to standardize and automate deployments.
• Initial 'quick-and-dirty' tools proved the value of automation, leading to a strategic shift toward continuous deployment using Spinnaker.
• The cyclical nature of change in SRE means that every improvement in efficiency creates the space to identify and tackle the next systemic pain point.

Eliminating one bottleneck in a system often highlights another one.

• Waze SRE overcame developer resistance to self-service releases by building centralized dashboards that provided visibility and confidence in deployment metrics.
• The adoption of Spinnaker was accelerated by partnering with volunteer development teams to prove the system's value before engineering leadership mandated the switch.
• Early adopters created a chain reaction of adoption as they pressured dependencies to migrate to avoid bottlenecks in development velocity.
• The transition resulted in over 95% of Waze services using continuous deployment with minimal human involvement.
• The case study highlights that grass-roots technical change requires a combination of executive support, cross-functional collaboration, and incremental wins.
• Applying formal change management theories, such as Kotter’s model, can help streamline and guide complex organizational transitions.

Now, engineers began to put pressure on dependencies that had not moved, as they were the impediment to faster development velocity—not the SRE team!

• Building new systems is necessary when current solutions hit a local maxima and cannot scale with long-term growth.
• Waze's in-house message queue development demonstrates how small engineering groups can drive significant reliability improvements.
• Google SRE emphasizes horizontal software adoption to create a virtuous cycle and prevent the constant reinvention of wheels.
• The Prosci ADKAR model is often more effective than Kotter's model for organizational evolutions that aren't triggered by immediate crises.
• Fragmented tooling at Google arose because developers were dissociated from users, leading to niche solutions that failed to meet universal needs.
• Standardizing technology across an organization creates efficiency gains by allowing teams to share automated practices and policies.

Those waiting for the big use case—a nonspecific, singing-and-dancing solution of the future—waited a long time, got frustrated, and used their own software engineering skills to create their own niche solution.

• Google SRE sought to replace fragmented, custom-built tools with a common monitoring engine and automated infrastructure.
• The strategy relied on 'critical mass' where engineers would eventually abandon high-maintenance custom tools for well-supported, universal solutions.
• Initial efforts utilized virtual teams of experts spending 80% of their time on horizontal projects while remaining on-call for their specific services.
• The virtual team model failed due to the cognitive load of context switching between on-call duties and complex software engineering.
• Geographic dispersion and time zone differences hindered consensus-building and code reviews within the decentralized structure.
• The organization ultimately pivoted to a centralized model with dedicated senior engineers to ensure focus and project velocity.

There was a lot of state to be swapped between running a service and building a serious piece of software.

• Google transitioned from disparate custom solutions to centralized, agile development teams focused on building automation for the majority of the company.
• Physical centralization of small teams (6–10 people) was prioritized to increase development efficiency and speed.
• The shift required SREs to adopt a product-centric mindset, focusing on customer requirements, roadmaps, and delivery commitments.
• The Viceroy project successfully unified monitoring under one framework, but still faced challenges with duplicated effort in dashboard customization.
• A 'zero-config' system was developed to provide opinionated, standard monitoring displays that met the needs of most services without manual setup.
• The ADKAR model illustrates how grassroots success provided the awareness and desire to fund a broader, company-wide tooling initiative.

But in real life, the hardest engineering work ends up being the evolution of many small/constrained systems into fewer, more general systems—without disturbing already running services that many customers depend on.

• Building horizontal software requires SREs to adopt product management roles, prioritizing user needs and experience over pure technical design.
• The perceived cost of migrating from familiar, fragmented systems to new common solutions remains the primary barrier to organizational adoption.
• Engineers often resist change due to the 'knowledge-to-ability gap,' viewing migrations as lateral moves rather than system improvements.
• To ensure success, migration costs must be reduced to nearly zero while providing immediate, tangible benefits like security or performance gains.
• Moving from 'best effort' internal tool development to reliable, staffed product teams is essential for building consumer trust.
• Google's approach favors a bottom-up adoption strategy, relying on delighting users rather than top-down mandates.

Despite the allure of easier management and less specific deep knowledge, the costs of migrating away from the familiar (with all its warts and toil) were generally a barrier.

• Internal adoption of new systems is driven more by the transparency and reliability of the development process than by technical superiority alone.
• Successful software projects require dedicated, production-experienced engineers and clear communication regarding timelines and feature goals.
• Targeting the 'grumpiest adopters' first ensures that a wide multitude of edge cases are addressed before a broader rollout.
• Adoption creates a network effect where increased scale justifies more development resources, which in turn further reduces migration costs.
• The primary metric for measuring the impact of organizational change and new tooling must be the rate of adoption by users.
• Effective change management requires a structured plan involving champions, beta testers, and executive sponsors to minimize barriers to entry.

Perceptions of how the sausage is made turned out to be more important than we anticipated from the get-go.

• SRE adoption requires constant reevaluation and iteration, moving from grassroots beginnings to structured coordination.
• Large enterprises are debunking the myth of being slow to change by aggressively and innovatively adopting SRE practices.
• Smaller firms are successfully implementing SRE principles incrementally rather than attempting to adopt the entire framework at once.
• There is a significant market opportunity for new products and services designed to help organizations transition to SRE models.
• The principles of SRE are beginning to merge with other disciplines, most notably security, leading to the rise of DevSecOps.
• The scope of SRE is expanding into non-technical fields, including legal teams interested in applying error budgets to their work.

He who lives by the crystal ball soon learns to eat ground glass.

• The authors envision SRE principles expanding beyond technical implementation into diverse fields like the legal profession.
• This volume serves as a launching point for a broader ecosystem of SRE content, including videos, podcasts, and future publications.
• The authors acknowledge and aim to rectify past alienation of the DevOps community by emphasizing that SRE and DevOps are complementary.
• SRE has transitioned from a Google-centric practice to a global community where non-Google practitioners are becoming the majority.
• The conclusion includes a practical example of an SLO document for a game service to demonstrate real-world application.
• The authors express gratitude to the community and maintain their SRE identity by planning a postmortem of the book-writing process.

May your queries flow and your pagers stay silent.

• Availability is defined by the ratio of successful HTTP requests to total requests, excluding 5XX status codes.
• Latency targets are tiered, aiming for 90% of requests under 200ms and 99% under 1,000ms.
• Data integrity is tracked through freshness, correctness, and completeness metrics within the score pipeline.
• Error budgets are calculated as the inverse of the SLO goal, providing a quantifiable limit for service failures.
• The document acknowledges that current SLO numbers may not yet strongly correlate with actual user experience.
• Measurement occurs at the load balancer, which may miss failures that prevent requests from reaching the infrastructure.

Even if the numbers in the SLO are not strongly evidence-based, it is necessary to document this so that future readers can understand this fact, and make their decisions appropriately.

• The policy establishes a formal mechanism to balance the pace of innovation with service reliability by using error budgets.
• Exceeding the error budget for a four-week window triggers a mandatory halt on all non-emergency releases and data changes.
• Teams are required to pivot resources toward reliability if the budget miss was caused by internal code bugs or procedural errors.
• Exceptions to the release freeze are granted for external factors like company-wide network failures or outages caused by other teams.
• Significant incidents consuming over 20% of the budget require a postmortem and a high-priority 'P0' action item to address root causes.
• The policy is framed as a permission to focus on stability rather than a punishment for missing performance targets.

Halting change is undesirable; this policy gives teams permission to focus exclusively on reliability when data indicates that reliability is more important than other product features.

• Statistical analysis of thousands of postmortems reveals that binary and configuration pushes account for 68% of all service outages.
• Software defects and development process failures are identified as the primary root causes for system instability, far outweighing hardware issues.
• The SRE engagement model spans the entire service lifecycle, from architecture and design to active development and eventual decommissioning.
• Effective postmortem culture relies on blamelessness, concrete action items, and the gamification of task completion to ensure systemic improvements.
• Advanced alerting strategies, such as multi-window burn rate alarms, are essential for managing error budgets and maintaining high availability goals.
• Automation is framed as a tool to reduce toil and increase uniformity, though it requires careful risk assessment and manual overrides like kill switches.

Binary and configuration pushes account for over two-thirds of outage triggers, while hardware failures represent a mere 2%.

• The text outlines strategies for maintaining service availability, including load shedding, health-checking handlers, and factoring in dependencies.
• It details the 'canarying' process, which balances release velocity with reliability by isolating changes and monitoring specific metrics before full deployment.
• A significant focus is placed on 'blameless culture,' emphasizing the use of blameless language and postmortems to mitigate damage from system failures.
• The index covers organizational change management theories, such as Kotter’s eight-step process and the Bridges Transition Model, as applied to SRE environments.
• Operational health is addressed through the management of 'toil,' the balance of on-call duties, and the use of error budget burn rates for alerting.

modeling and enforcing blameless behavior

• The text outlines the critical role of configuration management in system reliability, emphasizing that configuration should be treated as a programming language problem.
• It details the '3 Cs' of incident management—coordinate, communicate, and control—and the importance of maintaining open communication channels during failures.
• The index highlights common pitfalls in configuration systems, such as building ad hoc language features or interleaving evaluation with side effects.
• It explores the philosophy of configuration, suggesting that systems should ask users questions that align closely with their high-level goals.
• The section covers the integration of SRE practices with customers and the necessity of a blameless postmortem culture to learn from failures.

Failing to recognize configuration as programming language problem.

• The index highlights the critical role of Service Level Objectives (SLOs) and shared dashboards in maintaining data processing pipelines.
• Best practices for data pipelines include planning for dependency failures, implementing autoscaling, and reducing workload hotspotting.
• A detailed case study from Spotify explores event delivery system design, architecture, and customer integration strategies.
• The text outlines the structural relationship between SRE and development teams, emphasizing shared ownership and aligned business priorities.
• Operational readiness for pipelines requires specific design patterns like checkpointing, idempotency, and two-phase mutations.
• The interrelation between DevOps and SRE is defined through the elimination of silos, gradual change, and the crucial role of measurement.

SREs sharing ownership with developers, 6 sustaining an effective ongoing relationship between SRE and development teams, 380-382

• The text serves as a detailed index for Site Reliability Engineering (SRE) practices, focusing heavily on the mechanics of error budgets and service level objectives (SLOs).
• It outlines strategies for managing low-traffic services, such as generating artificial traffic or combining services to make error budget alerting more effective.
• The index highlights the cultural aspects of SRE, including the importance of avoiding finger-pointing in postmortems and managing the tension between feature velocity and system reliability.
• Specific case studies from companies like Spotify, Evernote, and Home Depot are referenced to illustrate real-world applications of data processing pipelines and SLO evangelism.
• Operational tactics for incident management are listed, such as draining requests from buggy systems, practicing emergency responses, and using 'Four Golden Signals' for monitoring.

tradeoffs between feature velocity and reliability, 19

• The text serves as an index for Site Reliability Engineering (SRE) topics, focusing heavily on Google Cloud infrastructure and load balancing technologies like GSLB and GCLB.
• It details incident response frameworks, specifically the Incident Command System (ICS) and the role of the Incident Commander (IC) in managing service failures.
• Case studies are highlighted, including the Pokémon GO migration to GCLB and specific software bug responses for Google Home and Google Kubernetes Engine.
• The document emphasizes the importance of Service Level Objectives (SLOs) and the VALET framework for measuring application health and performance.
• It addresses the human element of SRE, including avoiding blame, aligning incentives between development and operations, and managing global team distribution.

case study, Google Home software bug and failure to communicate, 177-180

• Explores the technical nuances of configuration management, specifically the use of Jsonnet and JSON for Kubernetes objects and the risks of treating configuration as a non-programming language.
• Details various load balancing strategies including Google Cloud Load Balancer (GCLB), Maglev, and anycast, alongside case studies like Pokémon GO.
• Addresses organizational change management through frameworks like Kotter’s eight-step process, Lewin’s model, and the Kübler-Ross Change Curve.
• Discusses operational health metrics such as latency SLIs, SLO quality improvement iterations, and the management of interrupts and work overload.
• Highlights the importance of postmortem quality, noting that omitting key details or limiting the audience can undermine the learning process.

failing to recognize configuration as programming language problem, 317

• The text outlines the critical role of measurements and Service Level Indicators (SLIs) in DevOps and SRE practices, specifically for tracking toil and system availability.
• It details the strategic selection of metrics for monitoring systems, emphasizing the importance of resource usage saturation, traffic status, and dependency tracking.
• The section covers incident response and postmortem best practices, highlighting the need for quantifiable action items and prioritizing mitigation above all else.
• It introduces Non-Abstract Large System Design (NALSD) through an AdWords case study, focusing on iterative design from single machines to distributed systems.
• Monitoring system requirements are defined, including the need for speed in data retrieval, treating configuration as code, and testing alerting logic.

Prioritizing mitigation above all else, 185

• The text outlines the complexities of on-call rotations, including strategies for managing pager load and balancing duties across geographically split teams.
• It details organizational change management theories, such as Kotter’s eight-step process and the Deming Cycle, as applied to SRE culture.
• A significant focus is placed on identifying and recovering from operational overload, distinguishing between actual load and perceived overload.
• The index highlights the importance of postmortem ownership and the role of release engineering in creating operational rigor.
• Case studies from Evernote, Waze, and PagerDuty provide real-world context for incident response and tooling adoption.

scenario, a survive the week culture, 171-173

• The index highlights the critical role of blameless postmortems in SRE culture, emphasizing organizational incentives and the sharing of results.
• Reliability is framed as a partnership between platform providers and users, prioritized as the most important feature of any system.
• Production readiness is managed through formal reviews (PRRs) and the careful balancing of release velocity against error budgets.
• Psychological safety is identified as a key factor in healthy on-call rotations and interpersonal risk-taking within engineering teams.
• Configuration management is treated as a programming language problem, with warnings against ad hoc features and manual replication toil.

Reliability as most important feature, 391; decided by users, not monitoring, 392.

• The index outlines critical Site Reliability Engineering (SRE) concepts including Service Level Indicators (SLIs) and Service Level Objectives (SLOs) as foundational metrics for system health.
• It details operational strategies for incident management, such as rollback procedures, postmortem root cause analysis, and the use of 'Requiem' for storage.
• Resource management is highlighted through data processing pipeline planning, scaling calculations for services like AdWords, and monitoring resource consumption.
• The text covers the human element of SRE, including on-call scheduling automation, managing personal circumstances, and planning for long-term breaks.
• Deployment and release strategies are categorized into rolling updates, canary deployments, and roll-forward methodologies to mitigate production risks.
• It emphasizes the evolution of SRE teams, from initial risk analysis and customer reviews to scaling SRE practices across larger environments.

planning for long-term breaks, 170 planning for part-time work schedules, 170 planning for short-term swaps, 169

• The index outlines the foundational role of Service Level Objectives (SLOs) in SRE, including their use in calculating error budgets and reliability targets.
• Case studies from Home Depot and Spotify illustrate the practical application of SLOs to batch applications, event delivery systems, and data collection automation.
• The SRE engagement model covers the entire service lifecycle, from architecture and design through to deprecation and the 'unsupported' phase.
• Core SRE principles focus on minimizing toil, sharing ownership with developers, and reducing the cost of failure to enable faster deployment.
• Organizational strategies for SRE include managing distributed teams, scaling to large environments, and implementing SRE practices even in the absence of dedicated SRE roles.

principles, 4-7: managing by SLOs, 5; moving fast by reducing cost of failure, 6; sharing ownership with developers, 6; using same tooling regardless of job title, 7; working to minimize toil, 5

• The index outlines the lifecycle of SRE teams, from bootstrapping the first hire to managing multi-team environments and mobility programs.
• A significant focus is placed on identifying and eliminating 'toil' through automation, with specific case studies on datacenter repairs and home directory decommissioning.
• Operational health is maintained through structured incident management, including the 'three Cs' and the avoidance of 'survive the week' cultures.
• The text details the transition of existing operations teams into SRE roles, highlighting the risks and mitigations involved in such organizational shifts.
• Service Level Objectives (SLOs) are presented as foundational tools for managing work overload and defining support tiers for various services.

on-call engineers in survive the week culture, 171-173

• The index details comprehensive strategies for toil management, including risk assessment, automation, and the importance of management support.
• Operational efficiency is highlighted through the use of Service Level Objectives (SLOs), uniform tooling, and self-service methods to reduce manual workloads.
• Training and professional development are emphasized as critical for incident response, new SRE team onboarding, and managing work overload.
• The text outlines technical concepts such as traffic volume metrics, configuration versioning, and the 'wisdom of production' in SRE culture.
• The section concludes with biographies of the editors, showcasing a diverse mix of backgrounds in technical writing, computer science, and poetry.

SRE teams having complete self-determination of workloads.

• The text provides professional backgrounds for key contributors to the Site Reliability Workbook, highlighting experience in telecommunications and cloud operations.
• Stephen Thorne's role focuses on Customer Reliability Engineering, bridging the gap between Google's SRE practices and external cloud customers.
• The book's cover features the green thornytail iguana, a small arboreal lizard native to the Amazon rainforest canopy.
• The iguana is characterized by its vivid green skin, black stripes, and mysterious rows of spikes on its tail.
• O'Reilly Media uses its cover art to highlight endangered or important animals, encouraging readers to support wildlife conservation.
• Technical details regarding the book's production include specific font choices like Adobe Minion Pro and Ubuntu Mono.

The green thornytail lizard is rather small, at about 3.5 inches long, and is difficult to spot in the wild because it tends to live high in the forest canopy.

Narrow, Rigid Incentives Narrow Your Success

Promote Toil Reduction as a Feature

Balancing Release Velocity and Reliability

Case Study 2: Perceived Overload After Organizational and Workload Changes

I love that the language and the process create a dispassionate contract between operational considerations and delivering new functionality.

• SLOs are the primary tool for data-informed decisions about the opportunity cost of reliability work.
• SLO adoption requires an error budget policy that formalizes priorities when reliability targets are missed.

One could even claim that without SLOs, there is no need for SREs.

• A 100% reliability goal blocks necessary updates and improvements because change is the primary source of outages.
• The gap between 100% and the chosen SLO creates an error budget that balances feature velocity with stability.

If you do manage to create an experience that is 100% reliable for your customers, and want to maintain that level of reliability, you can never update or improve your service.

• Effective SLOs require consensus among product managers, developers, and SREs so targets are user-satisfying and defensible.
• An error budget policy defines specific actions and responsibilities when reliability thresholds are breached.

The team responsible for the production environment who are tasked with defending this SLO have agreed that it is defensible without Herculean effort, excessive toil, and burnout.

• An SLO’s quality is measured by how well it correlates with real user pain.
• If error budget loss and support ticket spikes diverge, SLO coverage is lacking and the SLO or SLI needs adjustment.

It’s pointless to improve the recall of your system if you lower the precision such that the team must constantly respond to unimportant events.

• SLOs provide the highest-quality signal for when on-call engineers should intervene to protect the error budget.
• Alerting strategies should be evaluated by precision, recall, detection time, and reset time.

Precision is 100% if every alert corresponds to a significant event.

• Multi-burn-rate alerting prioritizes incidents by how quickly they exhaust the error budget.
• Multiwindow, multi-burn-rate alerting adds a short window to verify that an error spike is still active.

We can enhance the multi-burn-rate alerts in iteration 5 to notify us only when we’re still actively burning through the budget—thereby reducing the number of false positives.

• Persistent toil causes career stagnation, burnout-driven turnover, and morale loss by crowding out creative work.
• The true cost of toil is the high-value engineering work it displaces.

Toil can slowly deflate team morale. Time spent working on toil is generally time not spent thinking critically or expressing creativity.

• Effective automation should simplify and decompose human workflows into reusable software libraries, not merely transcribe them.
• Automation should fail safely by defaulting back to human operators when conditions are unsafe or ambiguous.

They tend to operate like a magical black box in that they “mostly work,” but few people understand how they work.

• Overanalysis can cause years of delay; imperfect automation is often better than manual processes.
• Defense in depth—secondary safety checks and hard limits—prevents automation from causing large-scale outages.

We spent nearly three years (2012–2015) collecting data on over 650 discrete memory error problems before realizing this exercise was probably overkill, or at least shouldn’t block our repair automation project.

• System complexity acts as an economic externality: maintainers often bear the cost of changes introduced elsewhere.
• SREs are well positioned to champion end-to-end simplicity because they see the whole production environment.

Frequently, the cost of complexity does not directly affect the individual, team, or role that introduces it—in economic terms, complexity is an externality.

• Google SRE teams preserve at least 50% project time to prevent burnout and address root causes.
• Target pager load is limited to about two incidents per 12-hour shift so engineers can follow up and recover.

At Google, the overall goal of being on-call is to provide coverage for critical services, while making sure that we never achieve reliability at the expense of an on-call engineer’s health.

• All paging alerts should be immediately actionable to prevent alert fatigue and preserve a high signal-to-noise ratio.
• New alerts should be reviewed like code and tested in production for false positives before paging humans.

Receiving a page creates a negative psychological impact. To minimize that impact, only introduce new paging alerts when you really need them.

• Incident management reduces chaos by letting teams focus on resolution instead of coordination during a crisis.
• Effective response relies on clear command lines, defined roles, continuous records, and early incident declaration.

ICS was established in 1968 by firefighters as a way to manage wildfires.

• A GKE outage was mitigated when an SRE used an existing GCR mirror to bypass a corrupted external DockerHub source.
• The postmortem emphasized generic mitigations—rollbacks or traffic redirection—that reduce user pain before root cause is known.

The outage was over. All that remained was cleanup, and writing a truly epic postmortem.

• A failed decommissioning automation erased disks globally because an empty list filter was interpreted as ‘no filter.’
• The incident shows why automated workflows need idempotency and sanity checks to prevent global unintended actions.

Within minutes, the disks of all satellite machines, globally, were erased.

• Postmortem action items should prevent recurrence by changing systems, not by asking humans to be less error-prone.
• Blameful narratives make people risk-averse and suppress the facts needed to prevent future failures.

Let’s plan for a future where we’re all as stupid as we are today.

• A load balancer routed more traffic to a failing region because it mistook load-shedding errors for efficient responses.
• Load balancing, shedding, and autoscaling must be treated as one system, not isolated tools.

As far as the load balancer system was concerned, each successive dropped request was a reduction in the per-request CPU cost.

• NALSD bridges whiteboard designs and real physical requirements such as CPU, RAM, and network constraints.
• The process is iterative, relying on sound reasoning and reasonable assumptions rather than perfect final calculations.

Google has learned (the hard way) that the people designing distributed systems need to develop and continuously exercise the muscle of turning a whiteboard design into concrete estimates of resources.

• Per-stage pipeline SLOs can over-alert and miss the actual user experience; end-to-end monitoring is essential.
• End-to-end monitoring can catch data corruption where each individual stage reports success but data is lost between components.

Both jobs think they are correct, but the user doesn’t see the data.

• Spotify decouples data collection from delivery with Google Cloud Pub/Sub, creating independent failure domains and resilience.
• Event streams are isolated by type so one stream cannot degrade another.

Once decoupled, data collection and delivery act as independent failure domains, which limits the impact of any production issues and results in a more resilient system.

• A single configuration change can have immediate, dramatic, and potentially catastrophic impact.
• Configuration is often changed under incident pressure, making usability central to reliability.

In contrast, changing a single configuration option can have dramatic changes on functionality—for example, one bad firewall configuration rule may lock you out of your own system.

• A common pitfall is failing to treat configuration design as a programming language problem.
• Reliable configuration systems need hermetic evaluation for predictable rollbacks and consistent replay across environments.

If you’re not intentionally designing a language, then it’s highly unlikely the “lan‐ guage” you’ll end up with is a good one.

• Canarying is a partial, time-limited deployment that evaluates changes before full rollout.
• Release engineering rests on reproducible builds, automated testing, and automated deployments.

The logic underpinning this approach is that usually the canary deployment is performed on a much smaller subset of production, or affects a much smaller subset of the user base than the control portion.

• Canary metrics should reflect user-visible problems such as latency and HTTP return codes, not ambiguous internal signals.
• Keep canary evaluation to a small set of high-signal metrics to avoid maintenance overhead and alert fatigue.

Too many metrics can bring diminishing returns, and at some point, the returns are outweighed by the cost of maintaining them, or the negative impact on trust in the release process if they are not maintained.

• Operational overload happens when urgent interrupts continually preempt priority work, reducing progress and increasing errors.
• Google SRE teams cap operational work at 50% so engineers have time to automate and reduce future toil.

A team is in a state of operational overload when it can’t make progress toward key priorities because urgent issues continually preempt project work.

• Early SRE involvement in architecture prevents costly redesigns by validating assumptions and establishing best practices.
• During development, SREs focus on productionization: capacity planning, redundancy, and sustainable monitoring.

Fixing architectural mistakes becomes more difficult later in the development cycle.

• The only reliability metric that truly matters is the user’s experience, regardless of internal dashboards.
• When a service offers an API, reliability becomes a partnership between the provider and the consumer’s implementation.

If your user is worried that your platform is responsible for instability they’re experiencing, then telling them “our monitoring looks fine; the problem must be on your end” won’t make them any less grumpy.