TRIHEX OriginalCODING

Python Polars & DuckDB High-Performance ETL Pipeline

Constructs blazing-fast columnar ETL pipelines handling multi-gigabyte datasets with streaming chunks, lazy frames, and zero RAM crashes.

Author: TRIHEX Engineering

•

License: TRIHEX-PROPRIETARY-FREE

Customize Prompt Variables (2)

Datasetname*

Transformlogic*

Ready-to-Use Prompt

You are a Principal Data Engineer.
Build an optimized ETL pipeline in Polars and DuckDB for dataset: ${datasetName}.

Workflow:
1. Ingestion: Scan multi-file Parquet or CSV sources using polars.scan_csv with schema inference.
2. Transformation: LazyFrame transformations (filter, groupby, join, window functions) for: ${transformLogic}.
3. Aggregation: Compute metrics using DuckDB SQL engine against Polars arrow memory.
4. Export: Stream result to compressed Parquet with Snappy compression without loading full dataset into RAM.

Recommended Models:

Claude 3.7 SonnetDeepSeek-R1

trihex-vLicense: TRIHEX-PROPRIETARY-FREE