ETL

Extract-Transform-Load · Data integration

Tags:processingintegration

Examples

Architecture of a modern ETL pipeline: extraction from a data lake (Amazon S3), transformation with Apache Spark, and loading into a warehouse (Redshift / MongoDB), orchestrated with Apache Airflow.

Description

A component that brings information from sources outside the information system into it, adapting it to the internal data model. Its function is to ensure that the incoming data is consistent, up to date, and compatible with the structure of the system. To do so, it performs three tasks: it extracts the data from its original sources, transforms it according to business rules or intended uses, and loads it into the destination store.

Mandatory capabilities

Mandatory capabilities

• Includes the three stages of the ETL process: data extraction, transformation according to the system's rules, and loading into the central repository
• Ensures the consistency and compatibility of the imported information with the internal data model
• Operates without user intervention once triggered, integrating data autonomously
• Allows the automated integration of large volumes of data from heterogeneous sources

Additional capabilities

Additional capabilities

• Periodic scheduling: runs the ETL at defined intervals (hourly, daily, etc.) through a scheduler, without requiring an explicit trigger
• Manual trigger: lets an authorized user start the ETL on demand through an explicit action
• Activity logging: records each run (errors, durations, volume of data processed) for auditing and monitoring
• Automatic retries: retries failed stages (typically extraction) when facing transient network errors or temporarily unavailable sources

Boundaries

What it is not

• Does not define access policies or permissions over the data; it only brings the data into the system
• Does not replace the internal processing that derives, calculates, or analyzes information already inside the system (Scheduled Process or Reactive Process): the ETL brings data in from outside; it does not operate on the data already inside
• Does not provide an interface visible to the user, although it can be monitored or configured from a console or external service

Information flow

Input

Data from external sources (APIs, structured files, external databases) together with the transformation rules

→

Output

Data transformed and loaded into the central repository of the system, consistent with its internal data model

Variants

• Batch ETL: processes large volumes of data in scheduled batches (daily, weekly)
• Real-time (streaming) ETL: processes data continuously as it arrives, with low latency
• ELT (Extract-Load-Transform): loads the data untransformed and applies the transformations afterward, inside the destination repository

Typical dependencies

• External data source (APIs, servers, sensors, files)
• Destination repository (the system's Database)
• A scheduling mechanism or scheduler for running it, when that capability is present
• Business rules that define how the data is handled

Mapping to the Digital Board

Manual UpdateNo

Automatic UpdateAlways

Data ConceptsAlways

On-Demand OutputsNo

Automatic OutputsNo

Autonomous ProcessesAlways

Section	Relationship	Notes
Manual Update	No	The manual trigger only starts the run; the data enters the system automatically, not through direct writing by the operator.
Automatic Update	Always	The incorporation of data from external sources into the system.
Data Concepts	Always	The entities loaded into the system's repository after the transformation.
On-Demand Outputs	No	—
Automatic Outputs	No	—
Autonomous Processes	Always	The transformation of the data according to the system's rules. Logging and automatic retries, when those capabilities are present.

Digital Board example

Manual Update

—

Data Concepts

Analytical Data

On-Demand Outputs

—

Automatic Update

Scheduled Extraction from S3

Autonomous Processes

Transformation with Spark

Run Logs

Automatic Outputs

—