how to use duckdbneo?

[r2evans]

I've come here from researching an issue elsewhere: I have a connection to an arrow object (from open_dataset) that has already gone through a couple dplyr changes, so the class of the object is now "arrow_dplyr_query" and not one of the arrow-native classes like "ArrowTabular". When I try to arrow::to_duckdb(arr), I get an error ! Failed to collect lazy table. I suspect I'm stretching the intent or capability of to_duckdb().

I came here to duckdbneo b/c of a past comment from you, duckdb/duckdb-r#162 (comment), hoping to find another way to approach the problem.

I get the idea (without reading its code) that the package is not meant to replace duckdb so much as to change how interaction is done internally, using ADBC and/or more things. Should I be "activating" something or registering any backends?

In case it matters, the underlying thing is that I have a 5M row lazy-arrow table and want to join on a range. Clearly apache/arrow#29841 is not ready yet, so we cannot use left_join(arr, ranges, join_by(between(time, start, end))). The second table ranges can be betwen 50 and 1K rows, so doing things manually is onerous and inefficient, and most other workarounds for range-joins tend to be iterative/inefficient at scale or heuristics with corner-cases I hit frequently enough to be painful.

With native-arrow lazy objects I'm able to do join_by(between(..)), but that's not currently how we get the data. Is duckdbneo something I should continue investigating for this use-case?

Thanks!

[krlmlr]

Thanks. This is a very rough draft today, need to set up a bit of infrastructure to evaluate if and where this is flying. Please don't hold your breath, but happy to review if you want to spend time/tokens to adapt this for your use case.