Thomas BrochardDevelopment · Data · AI
Let’s talk
Back to selected work

Data engineering

Case study / 03

Weather ELT Pipeline

A consistent grain across different sources.

Normalize heterogeneous weather feeds into traceable analytical observations with layered SQL and explicit quality rules.

Technologies & concepts

  • PostgreSQL
  • dbt
  • SQL
  • Docker
  • Airbyte ingestion design

Technical demonstration / selected source

Private repository

01 / The context

The problem behind the design.

Weather feeds use different station identifiers and field structures. Before observations can be compared, the analytical model needs a stable grain, source provenance and clear quality expectations.

02 / Engineering decisions

Choices made explicit.

Define the observation grain

One observation is identified by station, timestamp and source system. A deterministic key preserves this distinction rather than silently combining sources.

Keep transformations inspectable

Separate raw, staging, intermediate and mart layers normalize two source contracts and connect observations to station metadata.

Put quality rules beside the model

Uniqueness, non-null, relationship, allowed-value, range and temporal checks express the conditions needed for analysis. Duplicate observations are detected rather than silently removed.

03 / System view

Follow the flow.

Weather ELT design: raw observations pass through staging contracts and unified records into a station dimension and observation fact table.

Illustrative architecture from the project presentation. It describes the selected design; it does not imply a deployed service or a measured operational result.

View the full-size diagram
  1. Raw observations

  2. Staging contracts

  3. Unified records

  4. Station dimension

  5. Observation facts

04 / Selected artifacts

What the repository contains.

  • dbt/models/staging/
  • dbt/models/intermediate/
  • dbt/models/marts/
  • dbt/macros/generic_tests/
  • dbt/tests/
  • docs/model-design.md
  • docs/data-quality.md

05 / Scope & limitations

How to read this project.

Demonstration and source showcase using synthetic station metadata and generated observations. No live ingestion, deployment, execution results, throughput figures or test pass rates are claimed. Airbyte is an ingestion boundary; connections and operational scheduling require separate configuration. The six-station check is fixture-specific, and time-based checks depend on the snapshot.

Explore the source with access.

The GitHub repository is private. Its link works for authorized collaborators; other visitors can contact me to discuss the project.

Next case study / 04

CSV to MongoDB Migration

Data engineering

Explore the collection

The next conversation

Let’s talk about your project.

↗
A need, an idea, a technical question.
LinkedIn ↗GitHub ↗