looks like a cool project, but id say keep working on it since there seems to be some confusion on why someone would want to use this: no benchmarks and overall pretty vibe-codey (which id personally be very hesitant to use in production)
another comment already mentioned comparison to vortex, which is the same compression ratio and same speeds as youre claiming - but your compression is half of parquet. and if speed is the main goal youre going for, python is an interesting choice. no hate, but def keep working on it, and would love to see more concrete benchmarks with various columnar store types
A binary file format built from first principles for modern data systems. Parse 100MB 50x faster than JSON, with 50-70% better compression. Full language support (Python, Java, JavaScript, Go, C#, Ruby). Includes a VS Code extension for viewing .kore files. 3 years of production testing before open source release.
looks like a cool project, but id say keep working on it since there seems to be some confusion on why someone would want to use this: no benchmarks and overall pretty vibe-codey (which id personally be very hesitant to use in production)
another comment already mentioned comparison to vortex, which is the same compression ratio and same speeds as youre claiming - but your compression is half of parquet. and if speed is the main goal youre going for, python is an interesting choice. no hate, but def keep working on it, and would love to see more concrete benchmarks with various columnar store types
What does it do better than parquet? The compression ratio quoted is lower than parquet but I expect higher to be better in this context
> The fastest, most compressed columnar format for big data
How large a dataset can it tackle? I work with Parquet files spanning 300million+ records (~800MB files) using DuckDB and it works within seconds.
I might be interested to see benchmarks against Parquet and Vortex. A DuckDB extension would be great as well.
A binary file format built from first principles for modern data systems. Parse 100MB 50x faster than JSON, with 50-70% better compression. Full language support (Python, Java, JavaScript, Go, C#, Ruby). Includes a VS Code extension for viewing .kore files. 3 years of production testing before open source release.
Curious what you see as key differentiators over parquet / iceberg formats with snappy or similar compression schemes?