It is not a chicken and egg problem, it is just a requirement to have an RDBMS a... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		matt123456789 38 days ago \| parent \| context \| favorite \| on: 650GB of Data (Delta Lake on S3). Polars vs. DuckD... It is not a chicken and egg problem, it is just a requirement to have an RDBMS available for systems like DuckLake and Hive to store their catalogs in. Metadata is relatively small and needs to provide ACID r/w => great RDBMS use case.

dsp_person 38 days ago [–]

What about file-based catalogs with Iceberg? Found one that puts it in a single json file: https://github.com/boringdata/boring-catalog

saxenaabhi 38 days ago | [–]

Then concurrency suffers since you have to have locks when you update files.

That's also why ducklake performs better than others.

For many use cases this trade-off is worth it.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact