Describe the bug
When a JSON path is stored by the server in shared data, compound values
(arrays, nested objects, maps) are returned to the caller as raw bytes instead
of decoded Python objects. Scalar values in the same shared data decode
correctly, and the same compound value decodes correctly when the server stores
the path as a dynamic subcolumn instead.
The caller has no control over, or visibility into, which storage location the
server picks, so the same logical value can come back as either a list or
bytes depending on an internal storage decision.
The failure is silent inside the driver — it surfaces later in the application's
serialization layer (FastAPI/Pydantic in my case) as
Object of type bytes is not JSON serializable or UnicodeDecodeError, which
is hard to trace back to its origin.
Steps to reproduce
- Create a JSON column with
max_dynamic_paths=0, which forces every path into
shared data on insert (deterministic, no dependence on merge history).
- Insert one document containing a scalar and an array of objects.
SELECT the whole JSON column and inspect the Python types.
Expected behaviour
<class 'str'> 'hello'
<class 'list'> [{'k': 'v'}]
i.e. the same value that SELECT c.arr FROM t returns today.
Observed instead:
<class 'str'> 'hello'
<class 'bytes'> b'\x1e0\x00\x00\x10\x00\x00\x00\x01\x01\x01k\x15\x01v'
The scalar decodes; the array does not. The returned bytes are the complete
shared-data entry, <binary type encoding><value>:
1e 30 00 00 10 00 00 00 | 01 01 01 6b 15 01 76
└──────── type ────────┘ └────── value ──────┘
1e Array
30 JSON (the array's element type)
00 JSON serialization version
00 max_dynamic_paths
10 max_dynamic_types (16)
00 00 00 leb128 counts: typed paths, SKIP, SKIP REGEXP
which matches the type the server reports for the path:
SELECT JSONAllPathsWithTypes(c) FROM t;
-- {'arr':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=0))','s':'String'}Code example
import clickhouse_connect
client = clickhouse_connect.get_client()
client.command("DROP TABLE IF EXISTS t")
client.command(
"CREATE TABLE t (c JSON(max_dynamic_paths=0)) ENGINE = MergeTree ORDER BY tuple()"
)
client.command("""INSERT INTO t VALUES ('{"s": "hello", "arr": [{"k": "v"}]}')""")
row = client.query("SELECT c FROM t").result_rows[0][0]
print(type(row["s"]), repr(row["s"])) # <class 'str'> 'hello'
print(type(row["arr"]), repr(row["arr"])) # <class 'bytes'> <-- expected listConfiguration
Environment
- clickhouse-connect version: 1.6.0 (reproduced on
main @ cf0d49c)
- Python version: 3.11.13
- Operating system: macOS (arm64)
ClickHouse server
- ClickHouse Server version: 25.6.13.41
- ClickHouse Server non-default settings, if any: none (stock
clickhouse/clickhouse-server from the repo's docker-compose.yml)
CREATE TABLE statements for tables involved: see the code example above- Sample data for these tables: see the
INSERT in the code example — a single
synthetic row, no obfuscation needed
I have a fix and regression tests ready and am happy to open a PR.
Describe the bug
When a JSON path is stored by the server in shared data, compound values
(arrays, nested objects, maps) are returned to the caller as raw
bytesinsteadof decoded Python objects. Scalar values in the same shared data decode
correctly, and the same compound value decodes correctly when the server stores
the path as a dynamic subcolumn instead.
The caller has no control over, or visibility into, which storage location the
server picks, so the same logical value can come back as either a
listorbytesdepending on an internal storage decision.The failure is silent inside the driver — it surfaces later in the application's
serialization layer (FastAPI/Pydantic in my case) as
Object of type bytes is not JSON serializableorUnicodeDecodeError, whichis hard to trace back to its origin.
Steps to reproduce
max_dynamic_paths=0, which forces every path intoshared data on insert (deterministic, no dependence on merge history).
SELECTthe whole JSON column and inspect the Python types.Expected behaviour
<class 'str'> 'hello'
<class 'list'> [{'k': 'v'}]
i.e. the same value that
SELECT c.arr FROM treturns today.Observed instead:
<class 'str'> 'hello'
<class 'bytes'> b'\x1e0\x00\x00\x10\x00\x00\x00\x01\x01\x01k\x15\x01v'
The scalar decodes; the array does not. The returned bytes are the complete
shared-data entry,
<binary type encoding><value>:1e 30 00 00 10 00 00 00 | 01 01 01 6b 15 01 76
└──────── type ────────┘ └────── value ──────┘
1e Array
30 JSON (the array's element type)
00 JSON serialization version
00 max_dynamic_paths
10 max_dynamic_types (16)
00 00 00 leb128 counts: typed paths, SKIP, SKIP REGEXP
which matches the type the server reports for the path:
Code example
Configuration
Environment
main@ cf0d49c)ClickHouse server
clickhouse/clickhouse-serverfrom the repo'sdocker-compose.yml)CREATE TABLEstatements for tables involved: see the code example aboveINSERTin the code example — a singlesynthetic row, no obfuscation needed
I have a fix and regression tests ready and am happy to open a PR.