"97 Things Every SRE Should Know" book notes
Published at 2026-08-07T22:30:12+03:00
These are my personal book notes of Emil Stolarsky's and Jaime Woo's "97 Things Every SRE Should Know". They are for myself, but I hope they might be useful to you too.
Table of Contents
Introduction
Nothing is ever perfect, so don't aim for it. Be reliable just enough of the time, that's where the power is. And teams that operate only on the urgent side of the Eisenhower matrix are limited in what they can achieve.
- Why didn't it work like it did yesterday? What changed?
- It was as though production were a foreign land, and they needed me to accompany them as a translator.
- Any of us could see that it was slow; explaining why was next-level interesting.
- The harder and more subtle the bug, the more interested and energized they become.
- When we get together with other infrastructure engineers over a pint, we boast about the outages we have seen, the bugs we have found, and the "you-won't-believe-what-happened-last-holiday" stories.
Observability
Observing everything would swamp most observability systems with an obscene amount of storage and scale, and paying for a system capable of that would simply be impractical. Observability is not for debugging your code logic. It's for figuring out where in your systems to find the code you need to debug.
The ancient art of writing things down
When it comes to reliability, we're used to discussing new advances in the field, but one of the most powerful forces for reliability is also one of the oldest: the ancient art of writing things down.
- A culture of documenting our ideas helps us design, build, and maintain reliable systems.
- It lets us uncover misunderstandings before they lead to mistakes, and it can take critical minutes off outage resolution.
- A culture of writing things down reduces ambiguity and helps us make better decisions.
An SLO of 99.9% only tells you anything if you know what the service's owners consider "available" to mean. If there's an accompanying SLO definition document that explains that a one-second response is considered a success, and you were hoping for 10-millisecond latencies, you'll reevaluate whether this backend is the one for you.
- Writing shortens incidents too.
- Writing takes longer in the short term, but if you take a little extra time to describe what's happening, you'll help others save time by reading your mind.
The team's health
As we fine-tune SLOs and iterate on rotation design, it's equally important to keep in touch with the pulse of the team's health, and constantly ask: as a group, are we working in a way that is sustainable over the long haul?
- Emotional exhaustion: spending too much time caring too much.
- Depersonalization: feeling less empathy for others.
- Decreased sense of accomplishment.
"No one pays for generalists; you need to specialize." Now I'm an SRE. Burnout is a challenge. It will happen a few times, and each time you think, "I'll never fall for that again." Again, you will work too hard, too long, without reward or appreciation. It can permanently damage your health.
- The young and invincible assume it won't happen to them.
- Life is a marathon, not a sprint.
Dilemmas get easier when you ask, "In ten years, what will I wish I'd done?" Feeling financially trapped makes situations far worse. We work for managers, not companies. Ensure you are only 80% sure you can do the jobs you apply for, so you stretch yourself. Managers aren't your friends; they are your agents. Fire them if you don't like the community, work, or money they bring you.
The efforts and personal sacrifices of engineers are meaningless if they do not resonate at a strategic level. The Space Shuttle Challenger was approved for launch by NASA managers seeking to avoid delays in an already beleaguered schedule, despite known engineer concerns about the safety of the orbiter vehicle in subzero launch temperatures.
- When engineers engineer and leaders lead in isolated vacuums, trust and empathy cannot flourish.
- SRE offers a shared language for leveling the playing field between engineers and leaders.
- Measure, analyze, decide, act, reflect and repeat: that's site reliability engineering in six words.
Sharing responsibilities
"You build it, you run it": everyone shares responsibility for reliability.
- Through sharing the pain of running production services, opportunities to develop shared empathy and technical understanding necessary at scale are improved.
- You can't fix it all.
- Adding SRE to your company one task at a time and making things better.
- We're not aiming for perfection; we're just looking for better.
- Take small steps, with the understanding that when dealing with complex, unpredictable things, the plan can't specify everything.
The roles and the solo SRE
Three roles: incident manager, expert/operator, and communications, usually an incident commander, a technical lead, and a communications lead.
The most important point to remember in being a solo SRE is that although you can effect change within your organization, you cannot do it alone, so don't try to carry the weight of your organization's problems on your shoulders.
- SLOs must be able to evolve over time.
- SLIs, SLOs, and error budgets are the bedrock of site reliability engineering.
- Having a hard mandate about when to ship code probably doesn't make much sense in many situations, but using this data to help you figure out what your team should be focused on does.
- Use your error budget status to figure out when to experiment.
- Ensure you're not being more reliable than you advertise.
- At startups, SRE is often an afterthought behind shiny new features.
Being customer-focused
Understand the bottlenecks in your system and communicate them to stakeholders. There will be a strong push to ignore SRE work and focus on new features instead.
- If SLOs are not status quo in your organization, be prepared to invest a significant amount of time teaching stakeholders about the importance of SLOs.
- Textbook implementations of SRE rarely translate well in enterprises, given the diversity of businesses.
- Toil work measurement reduction from SLO improvements should always be quantifiable.
Don't have all the answers
There's pressure to feel like you have all the answers. In meetings, you see someone tap dancing nervously around an answer they don't have, especially when someone higher up the management chain asks. It's not our role as engineers and leaders to always have the answers.
- A simple tactic to get your work recognized: write a document listing your accomplishments.
Runbooks
- Runbooks (also known as playbooks) are not a silver bullet (nothing is). They share all of documentation's pitfalls: accuracy, quality, maintainability, drift.
- Runbooks are generally concerned with known unknowns, and we cannot anticipate every problem.
- Teams overinvest in runbooks, creating new sources of toil.
- Inaccurate or outdated runbooks can be more dangerous than no runbooks.
- Runbook creation, maintenance, and review should be a whole-team activity.
- Having too many runbooks is an anti-pattern.
- Playbooks: it's infeasible to assume that any playbook is absolutely complete, so expect it to be a tool that cannot fill the entire role of an SRE.
- Playbooks help an on-caller resolve issues but can contain too much or too little detail.
- Playbooks should ideally only contain the basics.
- The last anti-pattern is being too prescriptive.
Alerts per shift
- Severity and qualification of the user-visible impact.
- Alerts per shift: the maximum of 10 alerts per shift.
- On-call rotation: a minimum of eight people should be in the rotation, assuming week-long shifts and a primary/secondary setup.
- SRE happiness: a survey using an emoji rating is sent to SREs after each on call, aiming for an average of a smile. This is different from previous SLOs in that it is qualitative instead of quantitative.
In a transitory phase, people who are more often on call will get two mandatory consecutive days of recovery to prevent burnout.
- If the maximum number of alerts has been attained, the pager will be taken by someone else on the team to allow proper recovery time. Dealing with too much toil, having night shifts, and constantly being the first line of defense against outages can take a toll on SREs and the systems they work on. Prompt SREs to take time off when they encounter particularly stressful on calls.
Balancing velocity
As SREs, we see our job as balancing velocity with reliability.
You Don't Know for Sure Until It Runs in Production.
We often view production as a house of cards, like a fragile ecosystem that needs to be approached with care, silk gloves, or bunker gear. Incidents give us the space to zoom out and notice detrimental complexity, and incident reviews are the place to remove it.
Simpler systems that aren't perfect are usually better than complex ones. We often think of incidents in terms of TTx (time to x), like time to detect or time to mitigate, but these metrics provide little insight into what makes an incident interesting.
- If an engineer is a hero, there's a gap in the process, the infrastructure, or the tooling.
- Metrics are not SLIs (the measure-everything trap).
Metrics are raw numbers: how many items in a queue, how many days since the last failure. SLIs are combinations of metrics that tell a story, like if the queue keeps filling at the current rate.
- SLIs provide evidence of service efficiency and longevity.
- Important to revisit your SLIs constantly.
- When woken up in the night, will this metric help me or the team get the service back up faster?
- Will this metric be useful for alerting?
- Most metrics will never be looked at or read.
The power in knowing how to be self-sufficient
There is power in knowing how to be self-sufficient, in having the tools and the fearlessness to track answers down through layers of abstractions. SLOs are about quantifying delivered service, setting appropriate expectations, and changing tactics when things aren't going well.
- Time is the scarcest of resources in engineering.
- It starts with a commitment at the company level to enable engineers to consistently address reliability concerns on a project.
Prioritize towards the overall reliability goal
Part of the solution is to prioritize working on something small towards the overall reliability goals every day, rather than working on it for a week and then moving on, never to return.
- If SREs are constantly engaged with other teams, what about the SRE backlog?
- Adopt a shared-goals model to balance reducing the automation backlog and engaging with other teams.
- Requires a deep curiosity for how things work.
- Requires unrealistic expectations of complete knowledge from SREs.
- Organizations hire SREs assuming they code well, understand systems deeply, know monitoring and alerting, can run any service, debug production issues, and improve performance.
- Usually doesn't count on performance reviews and isn't recognized as delivering impact. Not included in the team's planning.
Mentoring others becomes part of this. It requires time, energy, dedication, and goodwill, so it is considered additional work.
- It is okay to accept an average solution that works and let the engineer improve it over time.
- Stepping back during an incident so others can learn and step up.
- Integrating mentoring into the team's day-to-day work is a building block that can make it more inclusive and help it thrive.
- When running services, we use baselines.
- Incident heroism can produce results but may also overshadow others and prevent them from gaining confidence.
The quiet time vs the burnout
Quiet time in the morning can be used to work on tasks with fewer interruptions. Remote ICs (individual contributors) have opportunities to be productive differently than before, like time-shifting work or breaking up their day.
- Problem-solving requires creativity, which requires free space.
- On the flip side from burnout, creativity thrives in semi-constrained spaces.
- Many insights result from detaching from a problem and finding insight elsewhere.
Create and maintain personal margin for your mental and physical health. Renewing activities counter environmental uncertainty: breaks, changes of scenery, and exercise. Incidents are unplanned investments in understanding systems. The learning budget is where you explore new, creative approaches.
Error budget as a learning budget
Also known as the error budget, this leftover part is where or when the service does not meet the objective. It's more helpful to think about this as the learning budget. Shouldn't we just be open that we're all committed to reliability and have leadership prioritize it? Sure, in a perfect world, but driving culture change means being passionate about the vision and patient enough to know folks need training wheels.
Focus not just on a single night; rather, lay the groundwork for creating an operationally mature organization. We are creatures of habit -- sudden changes of routine and operating outside our comfort zone attract doubt. Changing too much too quickly leads to confusion and skepticism.
Introducing SRE
Introducing SRE means overcoming inertia. It takes a lot of time to educate people and keep reinforcing the practices.
Change is hard, especially in large organizations. Focus on the most critical behaviors first.
- Identify culture carriers in your organization who empower others and build trust.
- A team of rock-star SREs doesn't guarantee success.
The book also discusses several sources of complexity. The biggest and hardest to deal with is state. State influences control flow, but the number of potential software states increases exponentially with variables. Separating the SRE team from development teams -- sometimes by creating a Center of Excellence -- causes problems rather than solving them. Elitism and knowledge constraints are issues.
- One solution can be embedding SREs into dev teams.
- Don't underestimate the power of documentation.
- Defining SLOs for your service, step by step.
- Two pages defining SLOs (high level).
- The biggest mistakes in engineering organizations often involve not creating well-structured and discoverable technical documentation.
- Others may doubt the maturity of the company in adopting SRE principles without proper documentation.
- Basic arguments for SLOs might conflict with existing goals, requiring patient explanation.
SLOs, SLIs, and error budgets will require convincing within the organization. Some may prioritize feature velocity over reliability work. Once engineering, operations, and product teams buy in, go to senior leadership and sell them on release velocity and early insight into the user experience.
Heroes and on-call practices
Heroes are necessary, but hero culture is not. A hero culture can easily form, but an SRE mindset helps combat this. If no action is required, tweak thresholds or delete alerts. Treat every page as an exceptional circumstance. Include on-call behaviors in developmental and career progression frameworks. On-callers should shadow experienced engineers to practice incident response. Trial by fire is not a prerequisite for being good on call. Best improvement ideas often come from the on-callers themselves.
Regular retrospectives improve on-call experiences. Successful teams frequently meet to improve processes and keep documentation up to date.
- Technical literacy and hands-on experience contribute to on-call satisfaction.
- Effective onboarding and training are essential.
- For clarity, ask, "Will this make sense if you've just been woken up?"
- Provide a clear escalation path with contact details and thresholds.
Prevent failures through improved system design
When a cascading failure occurs, many issues arise simultaneously, overwhelming systems. Even prepared teams struggle to mitigate without user impact. Better to prevent the failures through system design.
- SLIs, SLOs, and SLAs define service health.
- Availability and reliability are continuously measured.
- Postmortems focus on customer impact.
- Health checks quickly detect service failures.
On-call health and postmortems
On-call health matters. Postmortems should look at alert noise and at what can be automated. Action items from retrospectives need to be done on time.
- Link SLAs to on-call health to get a full picture of service quality.
- Error budgets concern not just availability but the quality of that availability.
- Performance budgets set limits on various performance metrics.
- Observability tools are designed for high cardinality data queries.
- Important tasks are prioritized; unimportant tasks are delegated or ignored.
- A roadmap helps avoid being trapped by immediate tasks.
Time management and cultural considerations
- SREs traditionally spend no more than 50% on ops work, with the rest coding.
- Over time, "at least 50% code" shifted to "at most 50% ops."
- Fifty percent ops work sounds viable, but not fifty percent toil.
Toil reduction is a goal for all engineering disciplines, not only SRE. Reliability needs planning, not only fixes. An SRE team should make systems need less human intervention. Make SRE contributions visible. You can't count prevented incidents, but the prevention work is what matters.
- In a complex world, avoid attributing issues solely to human error.
- Recognize tooling, operational, and resource gaps.
- An SRE mindset will be key in hiring for every engineering role.
- All engineers can incorporate SRE practices without needing dedicated SRE teams.
- SRE adoption is cultural, not merely about automating operations.
Things will always break, and that can lead to burnout. Mental health comes first. Error budgets give you data to decide with. For incidents outside SRE's control, only a culture change helps in the long run.
Building a good team in a large enterprise is hard. A culture of knowledge sharing and preparation beats runbooks alone.
- Mitigation tooling helps in incident management.
- Identify escalation paths: developers, back-end teams, or dedicated incident teams.
- Use consoles, logs, and inspection tools for problem-solving.
SREs protect critical systems. That's exciting and a burnout risk. Reliable systems need quick improvements and no processes that slow everything down. Modernize incrementally with small, frequent deployments.
A solid SRE culture is what makes it last. Documentation shouldn't go through the same review process as code.
- Happy, healthy on-call rotations lead to better results.
- Incentivize, reduce pain points, mentor, and iterate rapidly.
Alert volume vs effectiveness
How many alerts there are matters less than how well they get handled. Successful teams are built on trust, ownership and communication. Test your systems regularly, like a fire drill.
- Prioritize long-term impacts over daily distractions.
- SREs need to set limits on toil to mature as a discipline.
- Engineers must communicate risks clearly and prepare for future gaps exposed by incidents.
- Individuals understand only parts of complex systems.
SRE courses at universities would be a big step for the field.
Other book notes of mine are:
2026-08-07 '97 Things Every SRE Should Know' book notes (You are currently reading this)
2026-08-05 'Eat That Frog' book notes
2025-11-02 'The Courage To Be Disliked' book notes
2025-06-07 'A Monk's Guide to Happiness' book notes
2025-04-19 'When: The Scientific Secrets of Perfect Timing' book notes
2024-10-24 'Staff Engineer' book notes
2024-07-07 'The Stoic Challenge' book notes
2024-05-01 'Slow Productivity' book notes
2023-11-11 'Mind Management' book notes
2023-07-17 'Software Developers Career Guide and Soft Skills' book notes
2023-05-06 'The Obstacle is the Way' book notes
2023-04-01 'Never split the difference' book notes
2023-03-16 'The Pragmatic Programmer' book notes
E-Mail your comments to paul@nospam.buetow.org :-)
Back to the main site