← Writing
data-engineering

Stop Saving CSVs to S3

tl;dr Stop writing CSVs to S3. Use Parquet, size your row groups to ~128MB, organise by partition key, and let your agent carry the convention forward.

Somewhere in your S3 bucket, there’s a folder full of CSVs.

Maybe it started as one file: a quick export mid-analysis, a checkpoint before a long run. Then it became a pattern. Every notebook drops another file. Every pipeline stage writes another export. Six months later you have three thousand CSVs and a query that needs to touch all of them.

You fire up pandas. You write the glob. You wait.

The schema is usually fine. The column names are there. But the types aren’t. Everything came back as object, the dates are strings, the integers have a stray decimal somewhere, and now you’re debugging a dtype issue before you’ve even started the actual analysis.

CSV is not a storage format. It’s a clipboard. And a lot of teams have been filing their work in a clipboard drawer for years.

The format problem

CSV is row-oriented. To read two columns from a 24GB file, you scan all 24GB. Every time. The format has no concept of skipping irrelevant data.

Parquet is columnar. Read two columns, touch two columns. It’s also binary, so there’s no parsing text into types: the floats are already floats on disk. And it compresses well. A datetime column that takes 1GB in CSV can drop to 140MB in Parquet. Categorical columns (status fields, region codes, labels) can shrink by 10×.

Matthew Rocklin benchmarked this in 2015 and the numbers are still embarrassing for CSV: reading a 24GB dataset with pandas takes around four minutes. That’s an interactive analysis problem. Four minutes between “I wonder if…” and seeing the answer is too long to hold a thought.

While we’re here: stop pickling too

Some teams reach for pickle when CSV gets painful. Smaller files, faster reads, preserves Python types. It feels like a reasonable trade.

It isn’t.

Pickle is a Python-to-Python protocol, not a storage format. Nothing else reads it: no Spark, no DuckDB, no downstream tool that isn’t Python. It offers no random access. And unpickling data from an untrusted source is arbitrary code execution. The docs say so explicitly.

If the file ever leaves your machine, touches S3, or gets read by anything you didn’t write, pickle is the wrong choice. The performance gains aren’t worth the fragility.

Parquet gives you type preservation, compact storage, and fast reads without the lock-in or the security risk.

Row groups: where Parquet earns its keep

A Parquet file is divided into row groups: horizontal chunks of the data, each storing its columns independently. On S3, each row group can be fetched with a range request. That means you can process a large Parquet file in parallel without splitting it into separate files first - one task per row group, fanned out across workers, results collected at the end.

With something like Flyte, this becomes a clean pattern:

  1. Read the Parquet metadata to get row group offsets - no data loaded yet
  2. Launch one task per row group
  3. Each task fetches only its range from S3, processes it, writes a result
  4. A final task collects and merges

The file stays whole. The parallelism is free. The IO each task sees is proportional to its chunk, not the full dataset.

A few sizing rules:

  • Target ~128MB per row group. Too small and you pay task overhead for nothing. Too large and you lose parallelism granularity.
  • Organise files in S3 by a meaningful key (date, region, entity type) so queries can skip entire files before opening one. This is about directory structure, not internal file layout - row groups don’t control partitioning.
  • Don’t create thousands of tiny Parquet files. You’ll recreate the CSV graveyard, just with a different extension.

This shouldn’t be your problem every time

Data scientists focus on analysis. Storage optimisation is infrastructure, and infrastructure decisions tend to follow whatever the SRE team documented two years ago - which usually says nothing about Parquet row group sizing.

So the pattern repeats: a new project starts, someone exports a CSV because that’s what’s familiar, and six months later there’s another graveyard.

If you’re using an LLM-based agent for EDA (and more teams are), this is exactly the kind of convention that belongs in its context. Not as a one-time fix, but as a standing rule the agent carries into every session:

  • Write Parquet, not CSV
  • Target 128MB row groups
  • Organise S3 paths by a key that matches your query patterns
  • Never pickle intermediate results

Storage hygiene stops being something individuals have to remember and becomes something the tooling just knows. The data scientist doesn’t need to have read a Rocklin benchmark post to make the right call - the agent already has.

What comes next

Parquet on S3 with row group parallelism gets you a long way. It handles the “too much IO, not enough warehouse” middle ground that most DS teams actually live in.

When your workflows grow beyond ad hoc EDA - when you’re running recurring queries across a stable dataset, joining across files, or need SQL semantics without standing up infrastructure - that’s when DuckDB becomes the conversation. But that’s a separate post.

For now: stop saving CSVs. Your future self, six months into a project, will notice.