Tech hiring Cloud, DevOps & security
Hire site reliability engineers who keep you up.
A site reliability engineer is a software engineer whose product is the reliability of yours. They define how available and fast a service must be, measure whether it is, lead the response when it is not, and write the code that stops the same failure happening twice.
01 The role
What site reliability engineers do all week, and what comes out of it.
A site reliability engineer applies software engineering to the problem of keeping production healthy. They agree service-level objectives with product and engineering, instrument systems so those objectives can be measured, design alerting that pages a person only when users are affected, run incident response, plan capacity, and automate the repetitive operational work that would otherwise grow with every new customer.
The technical work is demanding: finding why the slowest one per cent of requests take four seconds, or why a retry storm turned a small fault into an outage. The harder part is organisational. An error budget only works if product managers accept that a service which has spent it stops shipping features until it is stable. Blameless reviews only work if their actions get done. An SRE needs the standing to hold those lines.
It is easy to confuse the role with its neighbours. A DevOps engineer is measured on how safely and quickly change reaches production. A cloud engineer is measured on the design and cost of the platform. An SRE is measured on what users experience. The role has also widened in recent years: OpenTelemetry has become the common instrumentation standard, and reliability work now covers model-serving and data pipelines as well as web services.
What they ship
- Service-level indicators, objectives and error-budget policies
- Metrics, logs and traces wired through services with OpenTelemetry
- Alerting based on SLO burn rate, with a runbook behind every page
- On-call rotas, escalation paths and an incident process people follow
- Load tests, capacity models and autoscaling policies
- Automation that replaces manual operational tasks
- Post-incident reviews with owned, tracked follow-up actions
02 Skills and stack
What site reliability engineers work with, and what is optional.
Tools matter less here than in most infrastructure roles. An SRE who understands burn-rate alerting will learn your monitoring vendor in a week; the reverse does not hold.
Core
The role cannot be done without these.
- SLIs, SLOs and error budgets
- Observability: metrics, logs, distributed tracing
- Incident command and post-incident review
- Distributed systems failure modes
- Linux, networking and performance analysis
- Programming in Go or Python
- Capacity planning and load testing
- Reducing toil through automation
Tooling
Varies from team to team.
- Prometheus, Alertmanager and Grafana
- OpenTelemetry
- Datadog, New Relic, Honeycomb or Grafana Cloud
- Loki, Elasticsearch or OpenSearch for logs
- PagerDuty, Opsgenie, incident.io or Grafana OnCall
- Kubernetes and Helm
- Terraform or OpenTofu
- k6, Locust or Gatling for load testing
- Argo Rollouts or Flagger for progressive delivery
- eBPF-based profiling and tracing tools
Adjacent
Useful, not required.
- Chaos engineering and game days
- Database reliability: replication, failover, backups
- CI/CD and release engineering
- Cloud architecture and cost
- Security incident response
- Reliability of ML model serving and data pipelines
03 When to hire one
When a site reliability engineer is the right hire, and when not.
Most hiring mistakes are made before the job description is written. Read both lists before you brief us.
Hire a site reliability engineer when
-
Customers tell you about outages before your alerts do
Monitoring watches servers, not what users experience. An SRE starts by measuring the journeys that matter and alerting on those.
-
On-call is burning out your developers
Pages at night for things that fix themselves, alerts nobody understands, and the same two people carrying the rota. This is a design problem with known remedies.
-
The same incident keeps coming back
Each outage ends with a restart and a promise. Nobody owns the follow-up, so root causes stay in place until they recur at a worse moment.
-
You have signed an uptime commitment in a contract
An SLA with penalties needs an internal objective stricter than the promise, evidence that you are meeting it, and a plan for when you are not.
-
Traffic is about to grow by a multiple
A launch, a large customer or a seasonal peak is coming and nobody knows where the system breaks. Load testing and capacity planning answer that beforehand.
Look elsewhere when
-
You have no automated deployment yet
Reliability practice sits on top of a working delivery pipeline. If releases are manual, fix that first; it removes a large share of incidents by itself.
Hire DevOps engineers -
The infrastructure is one server someone set up by hand
There is nothing to tune until there is an architecture that can tolerate failure. Start with the foundations.
Hire cloud engineers -
The product is early and has few users
Before product-market fit, an hour of downtime costs little and an SRE has too little traffic to measure. Developers on call with basic monitoring are enough.
Hire full-stack developers -
Outages come from bugs in application code
An SRE can show you where requests fail, but cannot fix a codebase with no tests. Spend the budget on engineers who will add them.
Hire software developers -
Every incident is the database
Slow queries, lock contention and failed failovers on one large database call for a specialist in that engine.
Hire database administrators
04 How we assess
How we assess site reliability engineers, before you meet one.
A CV says what someone claims. We test what they can do, then send you the evidence with the shortlist.
-
Stage 1
Profile review
We look for production systems the candidate was on call for: their scale, the objectives they were held to, and incidents they led. We read any post-incident reviews, automation or tooling they can share, and discount experience that was only dashboard-watching.
-
Stage 2
Technical test
A timed test covering Linux, networking, distributed systems and coding, with scenario tasks such as diagnosing a fault from graphs and logs, defining an objective for a given service or writing a small automation script.
-
Stage 3
Conversation with our team
Our engineers ask the candidate to take us through a serious incident from first alert to completed follow-up: what they saw, what they ruled out, how they communicated and what changed afterwards. We listen for structured reasoning and for blame being absent.
-
Stage 4
Review and levelling
Reviewers score accuracy, speed, problem-solving, communication and teamwork, then assign a level from our DevOps framework. For SRE roles the notes say whether the person has defined reliability targets and led incidents, or mainly responded to them.
-
Then
Your shortlist
Assessed candidates, with notes and scores.
The engine
Evalia
Assessments run on Evalia, the assessment platform we built ourselves. Across the ProDevs network, more than 100,000 assessments have been completed.
The levels we assign are public. Read the DevOps framework, or see how a hire runs from brief to offer.
05 Seniority
Three levels. What each one can own.
Years of experience are a weak guide. Ask what the person has owned, and who was checking their work.
Junior
Owns runbooks and dashboards
- Builds dashboards and alerts to an agreed design and keeps runbooks current
- Handles routine pages by following the runbook, and escalates in good time
- Automates small manual tasks in Python, Go or Bash
- Shadows on-call before carrying it; true junior SREs are uncommon
Mid-level
Owns reliability for a service
- Defines indicators and objectives for a service with its developers
- Carries on-call, leads ordinary incidents and writes the review
- Traces latency and error problems across several services
- Runs load tests and turns the results into capacity changes
Senior
Owns reliability across the company
- Sets the SLO framework and gets product leadership to honour error budgets
- Commands major incidents and coaches others to do so
- Reviews architecture for failure modes before systems are built
- Decides where reliability investment pays back and where it is excess
06 Interview questions
Interviewing a site reliability engineer? Ask these.
None of these has one right answer. The way someone answers tells you how they work.
-
Pick a service you ran. What were its SLIs and SLO, and how did you choose the numbers?
A good answer shows: Indicators measured where the user feels them, such as successful requests and latency at a high percentile, and a target negotiated from what users need and what the system had achieved. “Five nines” quoted without a cost or a reason is a weak answer.
-
What happens on your team when a service has spent its error budget?
A good answer shows: A policy agreed in advance: feature releases slow or stop and effort moves to reliability until the budget recovers. Listen for a real instance of this being enforced, and how they handled the argument with product.
-
Walk me through the worst incident you led, minute by minute.
A good answer shows: A clear timeline: detection, declaring the incident, roles assigned, mitigation before diagnosis, updates to stakeholders, then follow-up. They should describe their own mistakes plainly. A story in which they simply spotted the bug and fixed it is not incident leadership.
-
How do you decide whether an alert should page someone at 3am?
A good answer shows: It must mean users are affected now and that a person can do something about it. Everything else is a ticket or a dashboard. Strong candidates describe burn-rate alerts and say how many pages per shift they consider acceptable.
-
p50 latency is fine, p99 is terrible. How do you investigate?
A good answer shows: They look at what the slow requests share: an endpoint, a customer, a host, a downstream call. They use traces to see where the time goes, and consider garbage collection, lock contention, cold starts and retries. Averages are never mentioned as evidence.
-
A dependency slows down and your service falls over. What should have been in place?
A good answer shows: Timeouts shorter than the caller’s, retries with backoff, jitter and a budget, circuit breakers, and load shedding so the service degrades instead of collapsing. They should recognise a retry storm from the description.
-
What toil have you removed, and how did you decide it was worth automating?
A good answer shows: A manual, repetitive task they measured in hours, then eliminated or automated, with the before and after. Good candidates also name something they chose not to automate because it was rare or about to disappear.
07 Ways to engage
Four ways to bring a site reliability engineer in.
A permanent hire is the usual route, not the only one. Choose by how long the work lasts and who should employ the person.
- Placement fee, of first-year salary. One-time.
- 10–15%
- Replacement guarantee on direct hires.
- 2 weeks
- To a first shortlist, for most roles.
- 48h
-
Direct hire
A permanent employee on your contract and your payroll. One-time placement fee of 10–15% of first-year salary.
Tech hiring -
Dedicated team
A site reliability engineer inside a team we assemble and run for you: product, design and engineering on your roadmap, for a monthly retainer.
Dedicated teams -
Outsourcing & payroll
A full-time person in a country where you have no legal entity. ProDevs employs and pays them: 20% of salary if we find the person for you, or 5% of salary, capped at $500 a month, for payroll and compliance on someone you have already chosen.
Outsourcing -
Freelance project
A defined piece of work with a clear end. You pay the agreed price per milestone through escrow, plus a 5% client fee. Free to post.
Freelancers
08 Around the role
A site reliability engineer is one seat. These are the others.
SREs either sit inside product teams as the reliability specialist or form a central team that sets standards and consults. In both arrangements they depend on influence more than authority, because the developers who wrote a service are usually the ones who must change it.
- Software developers
- Agrees objectives for each service, reviews designs for failure modes and shares the on-call rota.
- Product manager
- Negotiates reliability targets and what happens to the roadmap when an error budget runs out.
- DevOps engineer
- Adds canary analysis and automatic rollback to the pipeline, using the signals the SRE defines.
- Cloud engineer
- Feeds capacity forecasts and failure findings back into the architecture of the platform.
- Customer support
- Runs status communication during incidents and learns from support which failures users notice most.
09 Client testimonials
Hear it first-hand, from companies that hired.
Founders, operators and executives describe working with ProDevs, in their own words.
Play video: Client testimonial: Olufemi Seyi, COO, Casafina development company
Play video: Client testimonial: Aproko Doctor, Building AwaDoc
Play video: Client testimonial: JT Liddell, Promenade CEO
10 Questions
Before you hire a site reliability engineer: the usual questions.
Something else? Book a call (opens in a new tab) and ask us directly.
-
What is the difference between SRE and DevOps?
DevOps is a way of working in which development and operations share responsibility for delivery; a DevOps engineer usually owns pipelines and environments. SRE is a specific role with a specific method: service-level objectives, error budgets, incident practice and automation of operational work. A DevOps engineer gets change to production safely. An SRE is accountable for how production behaves afterwards.
-
When does a company need its first SRE?
When downtime has a clear cost, in lost revenue or in contract penalties, and incidents are frequent enough that developers lose real time to them. For most companies that comes after automated deployment and basic monitoring are in place. Earlier than that, a DevOps engineer or a senior developer with operational experience is the better hire.
-
Do SREs write code?
They should. The idea behind the role is that operations problems are solved with software: automation, tooling, and changes to the services themselves. An SRE who only configures dashboards and answers pages is an operations engineer under a new title. Ask candidates what they have written and whether it is still in use.
-
What is an error budget?
It is the amount of unreliability a service is allowed. If the objective is 99.9% of requests succeeding in a month, the remaining 0.1% is the budget. While budget remains, teams release freely. When it is spent, effort shifts to stability. It gives product and engineering a shared number in place of an argument.
-
Will an SRE take over on-call from our developers?
Not entirely, and it is healthier if they do not. An SRE improves alert quality, runbooks and the rota, and carries on-call for shared infrastructure. Developers who stay on call for their own services build more reliable software, because they feel the consequences of their designs. Expect the load to fall, not to move.
-
How do you assess site reliability engineers?
In four stages. We review the production systems and incidents a candidate has been responsible for, set a timed technical test, hold a conversation with our engineering team about an incident they led, then review the results together and assign a level from our public DevOps framework. The assessments run on Evalia.
Talk to ProDevs
Tell us about the role.
Tell us the level and the stack. We will tell you how quickly we can put assessed site reliability engineers in front of you.
Project work
Need it for a project instead?
Not every job needs a full-time hire. Describe the work, get proposals from vetted freelance site reliability engineers, and pay per milestone through escrow. Free to post.