Decode compound values stored in JSON shared data

[iamnivash10]

Shared data stores values as . The type encoding is recursive and variable-length, so a single-byte lookup cannot resolve it, and serializeBinary differs from the native column format for compound types. Adds a recursive decoder for the single-value format and routes the non-scalar branch of _decode_variant through it.

Summary

Fixes #897 .

Compound JSON values (arrays, nested objects, maps) stored in shared data
are returned to callers as raw bytes instead of decoded Python objects.
Scalars in the same shared data decode correctly, and the same compound value
decodes correctly when the server stores the path as a dynamic subcolumn — so
the Python type a caller sees depends on an internal server storage decision
they cannot observe or control.

Root cause

Shared data stores each value as <binary type encoding><serializeBinary value>.
The current decoder resolves the type with a single-byte lookup in
STANDARD_DISCRIMINATOR_TYPES and returns the bytes untouched on a miss. Two
properties of the format defeat that:

1. The type encoding is recursive and variable-length. 0x1E does not mean
"Array" — it means "an array of whatever type encoding follows", and that
encoding can itself be compound. The bytes from the linked issue decode as:

1e 30 00 00 10 00 00 00 | 01 01 01 6b 15 01 76
└──────── type ────────┘ └────── value ──────┘

1e Array
30 JSON (the array's element type)
00 JSON serialization version
00 max_dynamic_paths
10 max_dynamic_types (16)
00 00 00 leb128 counts: typed paths, SKIP, SKIP REGEXP

01 element count (varuint)
01 path count
01 6b path "k"
15 String (dynamic path, so the value is self-describing)
01 76 "v"

matching the server's own report of the type:
arr: Array(JSON(max_dynamic_types=16, max_dynamic_paths=0)).

The type occupies eight bytes, not one; resolving it needs a recursive descent.

2. serializeBinary differs from the native column format for compound
types. The single-value format prefixes element counts with a varuint and
keeps elements self-contained; the native column format uses UInt64 cumulative
offsets with bulk-serialized elements. For scalars, String and Bool the two
are byte-identical — which is exactly why the existing scalar path is correct in
borrowing the native column reader. For Array and every other compound type
they diverge at the first byte.

So the missing piece isn't more entries in the discriminator table; it's an
implementation of the single-value format.

Changes

• New clickhouse_connect/datatypes/binary_value.py: a recursive decoder
  (_read_encoded_type + _read_binary_value) for the serializeBinary
  single-value format, covering Array, Nullable, LowCardinality, Tuple,
  NamedTuple, Map, Dynamic and nested JSON.
• The non-scalar branch of _decode_variant routes through it.
• Trailing-byte assertion: the decoder verifies it consumed exactly the
  input and raises otherwise, so a desynced parser fails loudly instead of
  returning plausible-but-wrong data. On any failure the caller falls back to
  raw bytes — today's behaviour — which makes the broad except safe rather
  than sloppy.
• Values decode through the driver's regular response buffer, compiled Cython when available.
• Scalar coverage extends to Date, Date32, DateTime and DateTime64 with time zones, UUID, IPv4, IPv6, BFloat16, Time and Time64, which default JSON type inference produces. Enum, Decimal, FixedString, Interval and Variant are deliberately unsupported because the JSON type normalizes them away before they reach shared data.
• Element counts are validated against the input length, with a cumulative budget for zero width elements, so corrupt or truncated buffers fail prior to allocating large results.
• Null is produced only for the Dynamic null encoding. A Nothing value in any other position is treated as corrupt input rather than fabricating None.

Scope / limitations

• Format coupling. The decoder re-implements part of the server's binary
  type encoding, so new type indices or a bumped serialization version need
  matching updates here. Inherent to supporting JSON rather than introduced by
  this PR, but a real maintenance obligation worth stating.

-JSON shared data only, enforced in code. Compound decoding is gated behind _decode_variant(..., decode_compound=True), which only decode_shared_data_value passes. The Dynamic SharedVariant path can also carry compound types with discriminators ≥ 0x20 (NamedTuple 0x20, Map 0x27, Dynamic 0x2B, JSON 0x30), and continues to return raw bytes for them. I have no reproduction for that path, so extending it belongs in a separate PR.

Tests

Added tests/unit_tests/test_binary_value.py.

Added to tests/integration_tests/test_dynamic.py, using max_dynamic_paths=0
to force shared data deterministically. Verified against ClickHouse 25.6.13.41;
each case fails on unpatched code and passes with the fix.

document	decoded
{"s": "hello", "arr": [{"k": "v"}]}	'hello', [{'k': 'v'}]
{"obj": {"a": "1"}}	{'a': '1'}
{"nums": [1, 2, 3]}	[1, 2, 3]
{"nested": [[1, 2], [3]]}	[[1, 2], [3]]
{"nullable": [1, null, 3]}	[1, None, 3]
{"m": {"x": "1", "y": "2"}}	{'x': '1', 'y': '2'}

The scalar case confirms no regression on the existing path.

Checklist

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG

---

This is my first open-source contribution — feedback of any kind is very welcome.

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

This PR fixes JSON shared-data decoding so compound values (Array, nested objects, maps, etc) come back as decoded Python objects instead of raw bytes, by adding a recursive decoder for ClickHouse’s single-value <encoded type><serializeBinary value> format and wiring it into variant decoding.

Changes:

• Added a recursive binary decoder (read_encoded_type + read_binary_value) for the shared-data single-value serializeBinary encoding.
• Routed the non-scalar/unknown-discriminator branch of _decode_variant through the new decoder with a safe fallback to raw bytes.
• Added integration tests covering compound JSON values forced into shared data via max_dynamic_paths=0.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
clickhouse_connect/datatypes/binary_value.py	New recursive decoder for <encoded type><value> in serializeBinary single-value format (arrays, tuples, nullable, map, dynamic, nested JSON).
clickhouse_connect/datatypes/dynamic.py	Uses the new decoder for previously “unknown discriminator” variant payloads (compound/shared-data cases).
tests/integration_tests/test_dynamic.py	Adds regression coverage for compound JSON values stored in shared data.

Comments suppressed due to low confidence (1)

tests/integration_tests/test_dynamic.py:708

• Same as the previous test: using test_client here makes this regression test sync-only, so it will not catch async decoding regressions. Please use param_client with the call fixture to exercise both sync and async clients, consistent with the rest of this module.

def test_json_shared_data_map(test_client: Client, table_context: Callable):
    type_available(test_client, "json")
    if not test_client.min_version("24.10"):
        pytest.skip("JSON shared data decoding requires 24.10+")

[joe-clickhouse]

Hey @iamnivash10 I've got a new native decoder in the works that may fix this exact issue. I'm not in front of my machine right now so I can't check but when I'm able I'll post a note back here. Thanks!

[iamnivash10]

Thanks @joe-clickhouse , that's good to hear — happy to have this closed by the better fix if the new decoder covers it.

One thing that might be worth checking against your work: shared-data values aren't in the native column format. They're stored as , and the two formats diverge for compound types — serializeBinary writes a var_uint element count with self-contained elements, while the native column format writes UInt64 offsets with bulk elements. They're byte-identical for scalars and String, which is why the current scalar-only path works. So a native-format decoder may still return raw bytes for compound values in shared data unless the single-value format is handled separately.

Either way, the integration tests here use max_dynamic_paths=0 to force shared data deterministically, so they'd work as regression coverage for your decoder too if that's useful — happy to strip this down to just the tests if that's the more useful contribution.

No rush at all, and thanks for taking a look.

[joe-clickhouse]

@iamnivash10 thanks for the work! I took a look and my yet-to-be-released work does NOT cover this. So this is definitely welcomed. I'm going to be mostly out of the office next week but I'll review as I'm able! Will keeep you posted. Thanks again.

[joe-clickhouse]

@iamnivash10 i'm taking a look at this again and will have it reviewed shortly. In the mean time if you're able to sign the CLA, that'd be great. thanks!

[iamnivash10]

Thanks @joe-clickhouse , I actually signed the CLA already — the check was failing because my commits were authored with an email not linked to my GitHub account. I've rewritten the commits with my linked email and force-pushed, so the CLA check should pass now. Let me know if anything else is needed!

[joe-clickhouse]

Thanks! I pushed a few changes on top of your branch to get this to merge state.

1. I merged current main
2. I fixed decoding under the compiled Cython buffer which raised TypeError for scalar values because the optimized column readers require the actual ResponseBuffer type.
3. I bounded element counts against the input length so corrupt buffers fail before building huge results, with a shared budget for zero width elements across nested containers.
4. I extended scalar coverage to Date, Date32, DateTime and DateTime64 with time zones, UUID, IPv4, IPv6, BFloat16, Time and Time64, which default JSON type inference reaches.
5. Null is now produced only for the Dynamic null encoding instead of being fabricated for Nothing values.
6. I added unit tests for the decoder and a changelog entry.
7. I updated the PR description to accurately capture what actually in the code.

In general your format analysis and overall approach looks good and is unchanged, so thanks for the work!