Data Engineering & ETL Pipelines
Build reliable pipelines that ingest, validate, transform and deliver data for analytics, with orchestration, monitoring and clear documentation.
- Planned 30 days
- About 2 h per day
- 8 planned modules
This course is in development and not yet available for purchase. The planned curriculum below is a draft and may change before launch.
Overview
A full program in data engineering. You learn what data engineers do, how to ingest data from files, APIs and databases, full and incremental loads, cleaning and transforming data in SQL and Python, ETL and ELT, data warehouses and dimensional modelling, data quality checks and quarantining bad records, orchestration with dependencies, retries and backfills, batch and streaming processing, and monitoring and troubleshooting pipelines. The focus is moving and transforming data reliably rather than analysing it. The capstone is a documented pipeline that ingests a dataset, validates it, transforms it and produces a clean output.
What you will be able to do
- Design pipelines from source to serving layer
- Ingest data with full and incremental, idempotent loads
- Transform data with SQL and Python
- Choose between ETL and ELT and model data for analysis
- Validate data quality and handle bad records
- Orchestrate, monitor and troubleshoot pipelines
Before you start
- Python basics (Python Programming or equivalent)
- SQL basics (Database & SQL or equivalent)
Planned curriculum
A draft outline of the modules being written. Lessons are not available yet, and the outline may change after expert review.
Availability
This course is not on sale yet. It becomes available only after its lessons, assessments and resources pass Yumeora's review. When published, it is planned to be included in Yumeora All-Access. Price to be announced.