Tech hiring Data & AI
Hire data engineers who make data trustworthy.
A data engineer moves data from the systems that create it to the people and models that use it. They build the pipelines, the warehouse and the tested tables in between, so that a number on a dashboard means the same thing tomorrow as it did today.
01 The role
What a data engineer does, and what they ship.
A data engineer collects data from your application databases, payment provider, CRM and event streams, lands it in a warehouse or lakehouse, and shapes it into tables that analysts, finance and machine learning models can query directly. They own the schedule, the tests, the documentation and the bill for all of it.
Writing a pipeline that works once is simple. The job is the second year: a source team renames a column, an API returns duplicates, a late-arriving event changes last month’s revenue, a backfill triples the warehouse cost. Strong data engineers design for reruns, test the data as well as the code, and can trace any figure back to its source.
Current practice favours loading raw data first and transforming it inside the warehouse with SQL under version control, usually with dbt, with an orchestrator such as Airflow or Dagster running the schedule. Streaming is used where minutes matter and avoided where they do not. The overlap with analytics engineering is large; the difference from a DBA is that a DBA runs the operational databases this data comes from.
What they ship
- Ingestion from databases, SaaS tools and APIs, including change data capture
- A cloud warehouse or lakehouse with clear layers and ownership
- Tested, documented dbt models that define each business metric once
- Orchestrated workflows with retries, alerting and safe backfills
- Streaming pipelines for events that cannot wait for a nightly batch
- Data quality checks, freshness monitoring and lineage
- Reverse ETL and feature tables that feed products and ML models
02 Skills and stack
What data engineers work with, and what is optional.
Warehouse skills transfer well between Snowflake, BigQuery, Redshift and Databricks, so do not screen people out on the vendor alone. SQL depth and modelling judgement are harder to find than experience of one platform.
Core
The role cannot be done without these.
- SQL, including window functions and query tuning
- Python, written as tested software
- Data modelling: dimensional models, slowly changing data
- Batch pipelines: idempotency, incremental loads, backfills
- A cloud data warehouse or lakehouse
- Orchestration and scheduling
- Data quality testing and observability
- Access control, privacy and cost awareness
Tooling
Varies from team to team.
- Snowflake, BigQuery, Redshift or Databricks
- dbt
- Airflow, Dagster or Prefect
- Fivetran or Airbyte for managed ingestion
- Apache Spark; DuckDB or Polars for smaller workloads
- Kafka, Kinesis or Pub/Sub; Flink for stream processing
- Debezium for change data capture
- Apache Iceberg or Delta Lake, with Parquet on object storage
- Great Expectations, Soda or dbt tests
- Terraform, Docker and CI/CD
Adjacent
Useful, not required.
- Analytics engineering and semantic layers
- BI tools: Looker, Power BI, Tableau, Metabase
- Feature pipelines and MLOps
- Operational database administration
- Data catalogues and governance: DataHub, Unity Catalog
- Vector stores and document pipelines for LLM retrieval
- GDPR and data retention requirements
03 When to hire one
Hire a data engineer for the right problem, or hire someone else.
Most hiring mistakes are made before the job description is written. Read both lists before you brief us.
Hire a data engineer when
-
Two dashboards give two different revenue figures
Each team has written its own query against raw tables. You need one tested definition of each metric, in one place, that everyone reads from.
-
Analysts spend most of their week cleaning data
Expensive analytical time goes on joining exports and fixing formats by hand. A modelled warehouse gives that time back to analysis.
-
Reports run against the production database
Heavy analytical queries slow the application for customers, and history disappears when rows are updated. Analytics needs its own copy, with history kept.
-
Scheduled scripts fail and nobody notices for days
Cron jobs and notebooks grew into a pipeline nobody designed. It needs orchestration, alerts and the ability to rerun a day safely.
-
You are preparing for machine learning or AI features
Models and retrieval systems need clean, current, access-controlled data. Building that foundation first is cheaper than debugging models trained on bad inputs.
Look elsewhere when
-
You want dashboards and answers, and the data is already tidy
If the tables are reliable and the need is reporting and explanation, that is analysis. A data engineer would be building plumbing you already have.
Hire data analysts -
You want forecasts, experiments or a predictive model
Statistical questions need statistical training. Data engineers supply the inputs; they do not usually design the experiment or the model.
Hire data scientists -
The production database itself is slow or fragile
Query tuning, replication and backups on the operational database are a DBA’s work, not a pipeline problem.
Hire database administrators -
A trained model has to be deployed and monitored
Serving, drift monitoring and retraining are machine learning engineering. The two roles share tools and meet at the feature tables.
Hire machine learning engineers -
You have three data sources and one analyst
A managed connector, a small warehouse and a few dbt models can be set up as a short project. A full-time engineer may be premature.
Post a freelance project
04 How we assess
Four stages, before a data engineer reaches your shortlist.
A CV says what someone claims. We test what they can do, then send you the evidence with the shortlist.
-
Stage 1
Profile review
We look for pipelines and warehouses the candidate built and then lived with: the sources, the volumes, the tools and who used the output. We note whether they owned data models and testing, or only moved files from one place to another.
-
Stage 2
Technical test
A timed test covering SQL in depth, Python and software fundamentals, data modelling and pipeline concepts such as incremental loading and idempotency, with practical exercises that include writing and debugging transformations.
-
Stage 3
Conversation with our team
Our engineers ask the candidate to walk through a platform they worked on: how data arrived, how it was modelled, what broke and how they found out. We listen for attention to correctness, reruns and cost, and for how they dealt with the teams upstream.
-
Stage 4
Review and levelling
Reviewers score accuracy, speed, problem-solving, communication and teamwork, then assign a level from our engineering framework. For data engineers the notes separate people who have designed a platform from those who have maintained one.
-
Then
Your shortlist
Assessed candidates, with notes and scores.
The engine
Evalia
Assessments run on Evalia, the assessment platform we built ourselves. Across the ProDevs network, more than 100,000 assessments have been completed.
The levels we assign are public. Read the engineering framework, or see how a hire runs from brief to offer.
05 Seniority
Three levels. What each one can own.
Levels differ in scope more than in speed. Match the level to how much direction you can give.
Junior
Owns a pipeline or a model
- Adds a source or a dbt model inside an existing project, following its conventions
- Writes correct SQL and Python, with tests, and fixes issues raised in review
- Investigates a failed run and reports what happened accurately
- Needs guidance on modelling choices, performance and backfills
Mid-level
Owns a data domain end to end
- Takes a domain such as billing or product events from raw source to trusted tables
- Designs incremental, rerunnable pipelines and knows what a late record will do
- Agrees metric definitions with analysts and changes with source-system owners
- Keeps an eye on warehouse cost and fixes the queries that drive it
Senior
Owns the data platform
- Chooses the warehouse, orchestration and ingestion approach, and can defend the cost
- Sets modelling standards, testing rules and data contracts across teams
- Decides where streaming is justified and where batch is enough
- Plans migrations and governance, and is trusted by finance with the numbers
Indicative salary
Published band for data and ML engineering. Annual, in US dollars, for remote roles with US, UK and EU companies.
- Mid-level
- $35–65k
- Senior
- $65–100k
- Typical US equivalent
- $140–220k
Indicative market ranges as of August 2026, based on ProDevs placement experience and published market data. The exact figure depends on stack, seniority and time-zone requirements. See the full cost breakdown.
06 Interview questions
Seven questions, worth asking a data engineer.
For your own interviews, after ours. Each question comes with what a strong answer tends to show.
-
A pipeline run fails halfway. How do you make it safe to run again?
A good answer shows: Idempotent design: partition overwrites or merges keyed on a stable identifier, so a rerun replaces data instead of duplicating it. They distinguish this from simply appending rows, and mention checkpoints for long jobs. “Delete the bad rows by hand” is the weak answer.
-
Finance says last month’s revenue changed after the books closed. How do you investigate?
A good answer shows: A methodical trace through lineage: late-arriving or corrected source records, a changed model definition, a backfill, time-zone boundaries. Strong candidates talk about snapshots and keeping history so that a past figure can be reproduced exactly.
-
How do you load a large table incrementally when rows can be updated or deleted at source?
A good answer shows: Change data capture or a reliable updated-at watermark, with a merge into the target and a plan for hard deletes. They know the failure modes: clock skew, missed updates, and why an occasional full reconciliation is worth scheduling.
-
When would you choose streaming over a batch job?
A good answer shows: Only when a decision depends on data that is minutes old, such as fraud checks or live operations. They are candid about the extra cost: ordering, late events, exactly-once semantics, on-call. Proposing Kafka for a daily report shows poor judgement.
-
How would you model orders, customers and subscriptions for analysts to query?
A good answer shows: A clear grain for each fact table, conformed dimensions, and a view on how to handle history, for example a customer changing plan. They explain trade-offs between a star schema and wide tables without treating either as doctrine.
-
What do you test in a data pipeline, and what happens when a test fails?
A good answer shows: Uniqueness, nulls, referential integrity, freshness and volume, plus business rules such as non-negative amounts. Equally important is the response: block the bad data from reaching users, alert the owner, and avoid so many warnings that all are ignored.
-
The warehouse bill doubled this quarter. Where do you start?
A good answer shows: Query history sorted by cost. Usual causes are full-table rebuilds that should be incremental, missing partition filters, dashboards refreshing too often and oversized compute left running. They fix the largest items first and add budgets or alerts.
07 Ways to engage
Four ways to bring a data engineer in.
A permanent hire is the usual route, not the only one. Choose by how long the work lasts and who should employ the person.
- Placement fee, of first-year salary. One-time.
- 10–15%
- Replacement guarantee on direct hires.
- 2 weeks
- To a first shortlist, for most roles.
- 48h
-
Direct hire
A permanent employee on your contract and your payroll. One-time placement fee of 10–15% of first-year salary.
Tech hiring -
Dedicated team
A data engineer inside a team we assemble and run for you: product, design and engineering on your roadmap, for a monthly retainer.
Dedicated teams -
Outsourcing & payroll
A full-time person in a country where you have no legal entity. ProDevs employs and pays them: 20% of salary if we find the person for you, or 5% of salary, capped at $500 a month, for payroll and compliance on someone you have already chosen.
Outsourcing -
Freelance project
A defined piece of work with a clear end. You pay the agreed price per milestone through escrow, plus a 5% client fee. Free to post.
Freelancers
08 Around the role
Who data engineers depend on, and who depends on them.
A data engineer serves other teams, which makes the role half engineering and half negotiation. Upstream are the developers whose systems produce the data; downstream are the analysts, scientists and finance staff who notice first when it is wrong.
- Data analyst
- Agrees metric definitions and builds the tables behind their dashboards.
- Data scientist
- Provides clean historical datasets and turns ad hoc feature queries into scheduled jobs.
- Machine learning engineer
- Supplies feature tables and event streams with agreed freshness.
- Backend developers
- Agrees event schemas and is warned of changes to source tables before they ship.
- Database administrator
- Reads from replicas or change streams so pipelines do not load the primary database.
09 Client testimonials
Hear it first-hand, from companies that hired.
Founders, operators and executives describe working with ProDevs, in their own words.
Play video: Client testimonial: Olufemi Seyi, COO, Casafina development company
Play video: Client testimonial: Aproko Doctor, Building AwaDoc
Play video: Client testimonial: JT Liddell, Promenade CEO
10 Questions
Before you hire a data engineer: the usual questions.
Something else? Book a call (opens in a new tab) and ask us directly.
-
What is the difference between a data engineer, a data analyst and a data scientist?
A data engineer builds and runs the pipelines and warehouse that make data available and reliable. A data analyst uses that data to produce reports and answer business questions. A data scientist applies statistics, experiments and models to predict or explain. The engineer usually comes first, because the other two depend on trustworthy data.
-
Should we hire a data engineer or an analytics engineer?
An analytics engineer concentrates on the transformation layer: dbt models, metric definitions and documentation, working closely with analysts. A data engineer also covers ingestion, orchestration, infrastructure and streaming. If managed connectors already land your data in a warehouse, an analytics engineer may be enough. If you are building the platform, you need a data engineer.
-
Which tools should a data engineer know in 2026?
Strong SQL and Python first. Then one cloud warehouse or lakehouse such as Snowflake, BigQuery or Databricks, dbt for transformations, an orchestrator such as Airflow or Dagster, and a managed ingestion tool. Spark, Kafka and open table formats such as Iceberg matter at larger scale. Skills carry over between vendors more easily than job adverts imply.
-
Do we need a data engineer before we hire for AI or machine learning?
Usually. Models and retrieval systems are only as good as the data they are given, and most stalled AI projects stall on access, quality or freshness of data. If your data already sits in a well-modelled warehouse you may not need one first. If it lives in exports and production tables, start here.
-
What does a data engineer in Africa cost?
For remote data and ML engineering roles with US, UK and EU companies, indicative annual salary ranges are $35,000–$65,000 at mid-level and $65,000–$100,000 at senior level. A direct hire through ProDevs also carries a one-time placement fee of 10–15% of first-year salary, with a two-week replacement guarantee.
-
How quickly can I hire a data engineer through ProDevs?
For most roles you see a first shortlist within 48 hours of agreeing the brief, and most companies find their hire within two to three weeks. It helps to tell us your warehouse, your main data sources and whether the work is mostly new build or maintenance, so the shortlist matches the real job.
Talk to ProDevs
Tell us about the role.
Tell us the level and the stack. We will tell you how quickly we can put assessed data engineers in front of you.
Project work
Need it for a project instead?
Not every job needs a full-time hire. Describe the work, get proposals from vetted freelance data engineers, and pay per milestone through escrow. Free to post.