Thomas BrochardDevelopment · Data · AI
Let’s talk
Back to selected work

Data engineering

Case study / 04

CSV to MongoDB Migration

Make the input contract explicit before writing.

Validate flat files, map typed documents and track bounded batch imports with checkpointed progress.

Technologies & concepts

  • Python
  • pandas
  • PyMongo
  • MongoDB
  • pytest

Technical demonstration / selected source

Private repository

01 / The context

The problem behind the design.

Moving a flat file into a document database involves more than insertion. Missing values, duplicate identifiers and invalid numeric fields need checks before conversion, while larger imports need bounded batches and visible progress.

02 / Engineering decisions

Choices made explicit.

Validate before conversion

Schema, required values, unique identifiers, dates and numeric ranges are checked before tabular fields become typed MongoDB documents.

Separate mapping from batch mechanics

The migration sequence owns validation and document construction. A configurable batch processor handles insertion, progress statistics and JSON checkpoints.

Expose intended write behavior

An explicit empty-target guard avoids deleting existing records. Synthetic orders and a mock collection illustrate mapping, batch boundaries and retry behavior for review.

03 / System view

Follow the flow.

CSV migration design: synthetic input is validated, mapped to typed documents and inserted in bounded batches with checkpoints and progress statistics.

Illustrative architecture from the project presentation. It describes the selected design; it does not imply a deployed service or a measured operational result.

View the full-size diagram
  1. Synthetic CSV

  2. Input validation

  3. Document mapping

  4. Batch insertion

  5. Checkpoint + progress

04 / Selected artifacts

What the repository contains.

  • migrate_to_mongodb.py
  • batch_processor.py
  • examples/synthetic_orders.csv
  • tests/test_batch_processor.py

05 / Scope & limitations

How to read this project.

Demonstration and source showcase with three synthetic order records. No database, credentials, generated exports, deployed migration service or successful test run is included. Checkpointing illustrates progress tracking without transactional or exactly-once guarantees; partial-write reconciliation, restart handling and source fingerprinting require further design.

Explore the source with access.

The GitHub repository is private. Its link works for authorized collaborators; other visitors can contact me to discuss the project.

Next case study / 05

Sales Warehouse Audit

Data engineering

Explore the collection

The next conversation

Let’s talk about your project.

↗
A need, an idea, a technical question.
LinkedIn ↗GitHub ↗