Thomas BrochardDevelopment · Data · AI
Let’s talk
Back to selected work

Data engineering & automation

Case study / 01

BottleNeck Data Pipeline

From fragmented catalogs to a traceable revenue view.

Three spreadsheet sources. One explicit data flow. SQL reconciliation, quality checks and statistical price flags, connected through a Kestra workflow.

Technologies & concepts

  • Python
  • SQL
  • DuckDB
  • pandas
  • NumPy
  • Excel
  • Kestra

Technical demonstration / selected source

Private repository

Illustration of reconciliation and checks in the BottleNeck pipeline.
Design illustration · no operational results represented

01 / The context

The problem behind the design.

Product prices, inventory identifiers and web sales often live in separate files. Inconsistent keys and duplicated website records make it difficult to assemble a coherent revenue view. This demonstration explores how explicit cleaning rules, traceable joins and checks between stages make that work inspectable.

02 / Engineering decisions

Choices made explicit.

Reconcile before reporting

Stage each source separately, filter identifiers and use SQL window functions to deduplicate records. An explicit mapping connects ERP products to web sales before any revenue calculation.

Make checks part of the flow

Check key uniqueness, compare derived revenue with joined records and review segment coverage. The Kestra source illustrates dependencies, retries and independent export branches.

Give statistical flags a definition

Price scoring uses population standard deviation and a z-score threshold above two. This is a flag for review; it does not establish product quality or business value.

03 / System view

Follow the flow.

Six-stage pipeline: ERP, web and mapping inputs, CSV staging, SQL reconciliation, revenue calculation, price scoring and segment exports.

Illustrative architecture from the project presentation. It describes the selected design; it does not imply a deployed service or a measured operational result.

View the full-size diagram
  1. ERP + web + mapping

  2. CSV staging

  3. SQL reconciliation

  4. Revenue

  5. Price scoring

  6. Segment exports

04 / Selected artifacts

What the repository contains.

  • Five SQL cleaning, reconciliation and revenue transformations
  • Python ingestion, price scoring, exports and quality checks
  • Kestra workflow describing dependencies and retries
  • Architecture visual and static technical presentation

05 / Scope & limitations

How to read this project.

Demonstration and selected-source showcase, without original inputs or generated business reports. Revenue uses matched quantity × unit price; historical pricing, discounts, tax and returns are outside this example. Inner joins omit unmatched records. ERP and mapping deduplication have no business tie-breaker. Complete deployment validation, unmatched-record reporting and operational monitoring would need further work.

Explore the source with access.

The GitHub repository is private. Its link works for authorized collaborators; other visitors can contact me to discuss the project.

Next case study / 02

Fly Drone Tool

Geospatial systems & interface design

Explore the collection

The next conversation

Let’s talk about your project.

↗
A need, an idea, a technical question.
LinkedIn ↗GitHub ↗