If you care about performance, you may want to avoid CSV files. But since our data sources are often like our family, we can’t make a choice, we’ll see in this blog post how to process a CSV file as fast as possible.
If you care about performance, you may want to avoid CSV files. But since our data sources are often like our family, we can’t make a choice, we’ll see in this blog post how to process a CSV file as fast as possible.
Also, regarding better formats: parquet is relatively nice. Smaller files, though not human readable. Use parquet if you read often, or have IO issues (file “too large” as CSV).