Description
DataChain should work natively on databases by allowing datachain operations on any tables, not only on our custom datasets. It will open use cases for "structured-teams" who use document-like use cases and even structured data. Otherwise we force people copy data to our datasets which they don't like. Think DBT but from Python.
Description
DataChain should work natively on databases by allowing datachain operations on any tables, not only on our custom datasets. It will open use cases for "structured-teams" who use document-like use cases and even structured data. Otherwise we force people copy data to our datasets which they don't like. Think DBT but from Python.
to_table("my_tbl")- save dataset to a table. No data copy! Versioning is not supported. They types have to be derived from the table schema. All operations should work downstream from these tables - select, filter, join, mutate.-
Open question: should these table be visible as datasets (datasets(),read_dataset())? Probably not for now - I don't see why they should. Instead we probably should show tables as a separate tree in UI and API.- Yes, the dataset should be visible in the dataset table but as an external.
read_table("my_tbl")