Feature/json rpc api improvements - #793
Conversation
13bd7c4 to
31e82e1
Compare
31e82e1 to
fc36dae
Compare
There was a problem hiding this comment.
Pull request overview
Expands the JSON-RPC API with batching, additional queries, richer filtering, updated schemas, and corresponding CLI support.
Changes:
- Adds batch dispatching with response budgeting and centralized error handling.
- Adds node information, virtual epoch lookup, output counts, ranges, and multi-value filters.
- Updates OpenRPC documentation, CLI commands, repository queries, and tests.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
internal/repository/repotest/report_test_cases.go |
Tests report index ranges. |
internal/repository/repotest/output_test_cases.go |
Tests output ranges and filters. |
internal/repository/repotest/input_test_cases.go |
Tests input index ranges. |
internal/repository/repotest/epoch_test_cases.go |
Tests epoch index ranges. |
internal/repository/repository.go |
Extends repository filters. |
internal/repository/postgres/report.go |
Implements report range filtering. |
internal/repository/postgres/output.go |
Implements output filtering and counts. |
internal/repository/postgres/input.go |
Implements input range filtering. |
internal/repository/postgres/epoch.go |
Implements epoch range filtering. |
internal/jsonrpc/util_test.go |
Closes test repositories. |
internal/jsonrpc/types.go |
Refactors JSON-RPC response encoding. |
internal/jsonrpc/limitedwriter.go |
Adds batch response budgeting. |
internal/jsonrpc/jsonrpc.go |
Adds methods, filters, and batch dispatch. |
internal/jsonrpc/jsonrpc-discover.json |
Updates the OpenRPC specification. |
internal/jsonrpc/jsonrpc_test.go |
Tests new API behavior. |
internal/jsonrpc/batchcalls_test.go |
Tests batch request handling. |
internal/jsonrpc/api/response.go |
Defines node information responses. |
internal/jsonrpc/api/params.go |
Adds parameters and positional decoding. |
internal/jsonrpc/api/params_test.go |
Tests parameter decoding. |
cmd/cartesi-rollups-cli/root/read/service/types.go |
Updates read-service parameter types. |
cmd/cartesi-rollups-cli/root/read/service/repository.go |
Supports multi-value repository filters. |
cmd/cartesi-rollups-cli/root/read/service/jsonrpc.go |
Updates JSON-RPC client validation. |
cmd/cartesi-rollups-cli/root/read/outputs/outputs.go |
Adds output filter flags. |
cmd/cartesi-rollups-cli/root/read/matchadvances/matchadvances.go |
Uses renamed match-advance parameters. |
cmd/cartesi-rollups-cli/root/read/epochs/epochs.go |
Adds repeated status filters. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
0f77172 to
ea9ccde
Compare
3a737ab to
2cbf0a5
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 67 out of 67 changed files in this pull request and generated no new comments.
Suppressed comments (4)
internal/jsonrpc/service.go:45
discoverSpecis decoded throughany, so JSON numbers becomefloat64. The newly documented offset maximum9223372036854775807is not exactly representable andrpc.discoverre-encodes it as a different value. Store the parsed document asjson.RawMessage(whichjson.Unmarshalvalidates and preserves) or decode withUseNumber.
internal/jsonrpc/limitedwriter.go:13- This budget only covers each encoded RPC object. Batch delimiters and every
-31003replacement response are written directly to the underlyinghttp.ResponseWriter, so the complete HTTP body can exceedMAX_RESPONSE_SIZEdespite the new documented 10 MiB per-request cap. Route the envelope and fallback responses through the request-level budget, or reserve their bytes before committing entries.
internal/jsonrpc/types.go:47 - Changing this helper to
io.Writerremoved itsContent-Typeassignment. In the single-object parse-error path, the error is written beforehandleRPCsets any header, so malformed{...requests now return JSON with an auto-detected non-JSON content type. Setapplication/jsonbefore parsing/switching on the body (and add a regression assertion).
internal/repository/postgres/schema/migrations/000001_create_initial_schema.up.sql:447 - Adding these indexes to migration
000001does not install them on databases already recorded at schema version 1:Upgrade()sees no newer migration and returns no change (schema.go:24,77-81). Those deployments will still scan the full output history for the new count/filter queries. Add a new migration version for the indexes and bumpExpectedVersionrather than editing only the already-applied migration.
|
I added two commits addressing the first and third issues of the Copilot review above. I’m deferring the second issue for now. The 10 MiB response budget is intended as a practical memory bound rather than a byte-exact HTTP body limit. Batch delimiters and replacement error responses may add a small, bounded overhead, which we consider negligible for the current purpose. I’m also deferring the fourth issue. We have decided not to introduce a new database migration at this time, based on our current deployment and usage patterns. Fresh databases will include the indexes, but existing schema-version-1 databases will not receive them automatically. We can add a versioned migration later if supporting those upgrades becomes necessary. |
c6b2f9e to
d8921d1
Compare
mpolitzer
left a comment
There was a problem hiding this comment.
Found a couple nits, will finish the review monday
- Omitted params - Top-level params: null - Empty positional arrays - Positional over-arity - Struct fields marked json:"-"
… a batch
The JSON-RPC server now:
- Calculates 'sum(limit)' for every list operation before dispatching a batch.
- Normalizes limits consistently with handlers:
- Omitted or zero → 50
- Above 10,000 → 10,000
- Allows a cumulative limit of exactly 10,000.
- Rejects totals above 10,000 before any handler or database query runs.
- Returns one batch-level -31003 error: Batch list item limit exceeded.
- Supports both named and positional parameters across every list method.
- Includes a registry coverage test so future list methods cannot silently
bypass the budget.
The client contract is documented in 'jsonrpc-discover.json', and operator
guidance—including the residual unbounded 'COUNT(*)' cost—is documented in
'docs/http-posture.md'.
…imeout to avoid incomplete responses - Added a 25-second JSON-RPC dispatch timeout, leaving five seconds before the HTTP 30-second WriteTimeout. - Applied the deadline at the top of handleRPC, propagating it through request dispatch. - Added an integration-style regression test through the actual server handler, verifying remaining batch entries receive -32070 responses. - Focused JSON-RPC regression tests pass.
- Added shared validation rejecting list offsets above math.MaxInt64 with JSON-RPC -32602 / "Invalid offset". - Validation occurs before repository access, preventing negative int64 conversion and erroneous operator alarms. - Covers all ten list methods plus named and positional parameters. - Updated every OpenRPC offset schema with maximum: 9223372036854775807.
…ed clients - Added a shared repository-error handler that passes wrapped context.Canceled errors through without Error logging. - Updated all JSON-RPC repository failure paths to use it. - Request dispatch now silently stops on cancellation without writing an internal-error response. - context.DeadlineExceeded retains existing logging and response behavior. - Added regression coverage for cancellation during an active repository operation.
- RPC method names are now capped at 64 bytes in every log path. - Truncation preserves valid UTF-8. - Single-request method visibility remains at Info. - Batch-entry method logging remains at Debug. - Updated and added tests verifying long method names are truncated and never logged in full.
… batch - Added per-entry panic recovery around dispatchOneRequest. - Panics are logged at Error with the method, panic value, and stack trace. - The affected entry receives JSON-RPC -32603. - Its private response buffer is discarded. - Remaining batch entries continue normally. - Added regression coverage proving the batch remains valid after a middle-entry panic.
- List input/output decoding failures now log per-row details at Debug instead of Error.
- Each list operation emits one aggregate Warn containing:
- Application
- Failure count
- First failing index
- Malformed rows remain in responses as partial decoded structures.
- Added regression coverage verifying four malformed rows produce four Debug logs, two aggregate Warns, and no Error logs.
…errors
- Request and response IDs now use json.RawMessage.
- Numeric and string IDs are echoed without decoding or precision loss.
- Validation still accepts omitted, null, string, and numeric IDs.
- Boolean, array, and object IDs remain invalid and receive a null response ID.
- Updated existing batch assertions for raw IDs.
- Added exact round-trip coverage for:
- 9007199254740993
- Values beyond uint64
- String IDs
- Both success and error responses
…large offsets - Updated docs/http-posture.md to clarify that the batch budget does not meter offset traversal. - Updated the OpenRPC x-batch-list-work-budget description with the same caveat. - Documented that traversal cost is bounded by the filtered set size, not the numeric offset, while PostgreSQL may still scan and discard matching rows.
- Added partial index output_executed_idx on input_epoch_application_id. - The index contains only rows where execution_transaction_hash IS NOT NULL. - Added the corresponding down-migration statement.
The memory model now documents: - 64 MiB of JSON-RPC request buffers. - Up to 640 MiB of response buffers. - Approximately 704 MiB combined. - Additional unbounded working memory from repository rows and decoded objects materialized before response-size enforcement.
…ne another
- Added a PostgreSQL advisory lock held for the full test-process lifetime.
- Applied it to all three schema-resetting packages:
- internal/jsonrpc
- internal/repository/postgres
- test/validator
- JSON-RPC services now clone the handler dispatch table.
- Test handlers modify only their service instance, not the package-global map.
- Added a regression test proving handler overrides do not leak across services.
`revive`'s `var-naming` rule reports the package-name violation only once per package, but the diagnostic location is not tied to one stable source file. When we added `//nolint:revive` to the reported package declaration, that file was excluded from the analysis and `revive` emitted the same package-level warning against another file's `package api` or `package jsonrpc` declaration. Meanwhile, `nolintlint` saw no diagnostic on the original line and reported the directive as unused. So a source directive caused this cycle: 1. Suppress the warning in one file. 2. revive reports it against another file in the same package. 3. nolintlint reports the first suppression as unused. The `.golangci.yml` exclusions instead match the diagnostic across the entire relevant package path and only for the two precise warning texts. Other `revive` checks remain enabled: - `internal/jsonrpc/api`: permits the established `api` name. - `internal/jsonrpc`: permits the established `jsonrpc` name despite its collision with a standard-library package name. Renaming the packages would also remove the warnings, but that would require a broad, unnecessary API and import change solely to satisfy a naming preference.
…nses - Sets Content-Type: application/json before switching/parsing the request body. - Removes redundant branch-specific assignments. - Adds a regression test for malformed single-object input.
…scover' - Changed Service.discoverSpec from any to json.RawMessage. - Retained json.Unmarshal, validating and copying the embedded JSON without converting numbers to float64. - Added a regression test asserting rpc.discover returns the exact literal 9223372036854775807.
- Added specific messages for invalid IDs and unsupported JSON-RPC versions. - Preserved empty-method validation as -32600. - Normalized standard messages to Parse error and Invalid Request. - Added regression assertions for the new messages.
e0942ca to
7558af6
Compare
Mirror the node API changes of cartesi/rollups-node#793 in @cartesi/rpc, and bubble them down to @cartesi/client and @cartesi/react. New methods: - cartesi_getEpochByVirtualIndex, fetching an epoch by its dense insertion rank (getEpochByVirtualIndex / useEpochByVirtualIndex) - cartesi_getExecutedOutputCount and cartesi_getPendingExecutableOutputCount (getExecutedOutputCount / useExecutedOutputCount, getPendingExecutableOutputCount / usePendingExecutableOutputCount) - cartesi_getNodeInfo, returning the chain id, the node version and the node's default block tag in one call (getNodeInfo / useNodeInfo). It replaces cartesi_getChainId and cartesi_getNodeVersion, which the node deprecated and which are now marked @deprecated here too. New listing filters: - from/to inclusive index ranges on listEpochs, listInputs, listOutputs and listReports - a list of statuses on listEpochs, and a list of output types plus the new executed flag on listOutputs Breaking changes: - cartesi_getMatchAdvanced is now cartesi_getMatchAdvance, so the getMatchAdvanced action is getMatchAdvance, the useMatchAdvanced hook is useMatchAdvance and the GetMatchAdvanced* types are GetMatchAdvance* - the node's application-level error codes moved out of the JSON-RPC reserved range (-31001/-31002 instead of -32001/-32002); they are now exported from @cartesi/rpc as errorCodes, along with the new batch, timeout and response-size-limit codes Also reformats four @cartesi/react hooks that biome was already reporting as unformatted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJpa3KXxvUarBGpHvW8Qdb
Mirror the JSON-RPC API changes of the Cartesi node in @cartesi/rpc, and bubble them down to @cartesi/client and @cartesi/react. Derived from cartesi/rollups-node#793, which has since merged into next/2.0 as 7558af6, so this is no longer a forward-looking port. It was verified against the merged jsonrpc-discover.json rather than the pull request diff: @cartesi/rpc declares exactly the 28 methods the merged specification defines, and every schema and parameter change between the pre-793 base (9bdd988) and merged next/2.0 is reflected here. Cross-checked against internal/jsonrpc/api/params.go, internal/jsonrpc/jsonrpc.go and internal/model/models.go. New methods: - cartesi_getEpochByVirtualIndex, fetching an epoch by its dense insertion rank (getEpochByVirtualIndex / useEpochByVirtualIndex) - cartesi_getExecutedOutputCount and cartesi_getPendingExecutableOutputCount (getExecutedOutputCount / useExecutedOutputCount, getPendingExecutableOutputCount / usePendingExecutableOutputCount). The executed count is monotone and meant to be polled for change detection; the pending count is a gauge and is not. - cartesi_getNodeInfo, returning the chain id, the node version and the node's default block tag in one call (getNodeInfo / useNodeInfo). It replaces cartesi_getChainId and cartesi_getNodeVersion, which the node deprecated and which are now marked @deprecated here too. New listing filters: - from/to inclusive index ranges on listEpochs, listInputs, listOutputs and listReports - a list of statuses on listEpochs, and a list of output types plus the new executed flag on listOutputs The node rejects an empty status or output_type list with invalid params, so the list-valued filters are typed as NonEmptyArray<T> rather than T[]: `status: []` and `outputType: []` are compile errors instead of failed requests. listOutputs maps the output types to selectors through the head of the list separately, so the result stays non-empty for the type checker, which Array.prototype.map would widen back to Hex[]. The constraint is guarded by a *.test-d.ts suite in @cartesi/client, which needed vitest type testing enabled there — CI runs `pnpm test` but no `tsc --noEmit` over the test files, so without it the constraint would go unchecked. Input completion changes, which landed on next/2.0 alongside but not as part of #793: - InputStatus loses its resource-limit members. The node collapsed OUTPUTS_LIMIT_EXCEEDED, REPORTS_LIMIT_EXCEEDED, CYCLE_LIMIT_EXCEEDED, TIME_LIMIT_EXCEEDED and PAYLOAD_LENGTH_LIMIT_EXCEEDED into the remaining outcomes, leaving NONE, ACCEPTED, REJECTED, EXCEPTION and MACHINE_HALTED. waitForInput listed four of them under rejectErrors; it now aborts on EXCEPTION, MACHINE_HALTED and REJECTED, which is every terminal status other than ACCEPTED — and no longer omits a failure status the way the old list omitted REPORTS_LIMIT_EXCEEDED. - Input gains exception_data / exceptionData, the raw guest-provided CMIO exception payload, non-null only when the status is EXCEPTION and empty-encoded as 0x. The bytes are passed through undecoded, matching how raw_data is handled. Breaking changes: - cartesi_getMatchAdvanced is now cartesi_getMatchAdvance, so the getMatchAdvanced action is getMatchAdvance, the useMatchAdvanced hook is useMatchAdvance and the GetMatchAdvanced* types are GetMatchAdvance* - the node's application-level error codes moved out of the JSON-RPC reserved range (-31001/-31002 instead of -32001/-32002); they are now exported from @cartesi/rpc as errorCodes, along with the new batch (-32040), timeout (-32070), response-size-limit (-31003) and batch-list-work (-31004) codes, plus the maxBatchSize (100), maxBatchListWork (10000) and defaultListLimit (50) constants that bound a batch - InputStatus shrank, as described above Batch requests needed no transport change — the underlying json-rpc-2.0 client already batches — so they are documented rather than implemented, including the two budgets that apply beyond the entry count: the response-size budget closes once exhausted, so every later entry gets -31003 even if its response would still have fit, and the list-work budget rejects the whole batch with a single -31004 before dispatching anything. Neither meters the COUNT queries behind pagination nor offset traversal, so a deep offset over a broad filter can still make the database scan and discard rows before the requested page. Two OpenRPC typing fixes needed no code change: prev_randao and voucher value are now UnsignedInteger256 in the specification, and both were already 256-bit-safe here. Also reformats four @cartesi/react hooks that biome was already reporting as unformatted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJpa3KXxvUarBGpHvW8Qdb
Mirror the JSON-RPC API changes of the Cartesi node in @cartesi/rpc, and bubble them down to @cartesi/client and @cartesi/react. Derived from cartesi/rollups-node#793, which has since merged into next/2.0 as 7558af6, so this is no longer a forward-looking port. It was verified against the merged jsonrpc-discover.json rather than the pull request diff: @cartesi/rpc declares exactly the 28 methods the merged specification defines, and every schema and parameter change between the pre-793 base (9bdd988) and merged next/2.0 is reflected here. Cross-checked against internal/jsonrpc/api/params.go, internal/jsonrpc/jsonrpc.go and internal/model/models.go. New methods: - cartesi_getEpochByVirtualIndex, fetching an epoch by its dense insertion rank (getEpochByVirtualIndex / useEpochByVirtualIndex) - cartesi_getExecutedOutputCount and cartesi_getPendingExecutableOutputCount (getExecutedOutputCount / useExecutedOutputCount, getPendingExecutableOutputCount / usePendingExecutableOutputCount). The executed count is monotone and meant to be polled for change detection; the pending count is a gauge and is not. - cartesi_getNodeInfo, returning the chain id, the node version and the node's default block tag in one call (getNodeInfo / useNodeInfo). It replaces cartesi_getChainId and cartesi_getNodeVersion, which the node deprecated and which are now marked @deprecated here too. New listing filters: - from/to inclusive index ranges on listEpochs, listInputs, listOutputs and listReports - a list of statuses on listEpochs, and a list of output types plus the new executed flag on listOutputs The node rejects an empty status or output_type list with invalid params, so the list-valued filters are typed as NonEmptyArray<T> rather than T[]: `status: []` and `outputType: []` are compile errors instead of failed requests. listOutputs maps the output types to selectors through the head of the list separately, so the result stays non-empty for the type checker, which Array.prototype.map would widen back to Hex[]. The constraint is guarded by a *.test-d.ts suite in @cartesi/client, which needed vitest type testing enabled there — CI runs `pnpm test` but no `tsc --noEmit` over the test files, so without it the constraint would go unchecked. Input completion changes, which landed on next/2.0 alongside but not as part of #793: - InputStatus loses its resource-limit members. The node collapsed OUTPUTS_LIMIT_EXCEEDED, REPORTS_LIMIT_EXCEEDED, CYCLE_LIMIT_EXCEEDED, TIME_LIMIT_EXCEEDED and PAYLOAD_LENGTH_LIMIT_EXCEEDED into the remaining outcomes, leaving NONE, ACCEPTED, REJECTED, EXCEPTION and MACHINE_HALTED. waitForInput listed four of them under rejectErrors; it now aborts on EXCEPTION, MACHINE_HALTED and REJECTED, which is every terminal status other than ACCEPTED — and no longer omits a failure status the way the old list omitted REPORTS_LIMIT_EXCEEDED. - Input gains exception_data / exceptionData, the raw guest-provided CMIO exception payload, non-null only when the status is EXCEPTION and empty-encoded as 0x. The bytes are passed through undecoded, matching how raw_data is handled. Breaking changes: - cartesi_getMatchAdvanced is now cartesi_getMatchAdvance, so the getMatchAdvanced action is getMatchAdvance, the useMatchAdvanced hook is useMatchAdvance and the GetMatchAdvanced* types are GetMatchAdvance* - the node's application-level error codes moved out of the JSON-RPC reserved range (-31001/-31002 instead of -32001/-32002); they are now exported from @cartesi/rpc as errorCodes, along with the new batch (-32040), timeout (-32070), response-size-limit (-31003) and batch-list-work (-31004) codes, plus the maxBatchSize (100), maxBatchListWork (10000) and defaultListLimit (50) constants that bound a batch - InputStatus shrank, as described above Batch requests needed no transport change — the underlying json-rpc-2.0 client already batches — so they are documented rather than implemented, including the two budgets that apply beyond the entry count: the response-size budget closes once exhausted, so every later entry gets -31003 even if its response would still have fit, and the list-work budget rejects the whole batch with a single -31004 before dispatching anything. Neither meters the COUNT queries behind pagination nor offset traversal, so a deep offset over a broad filter can still make the database scan and discard rows before the requested page. Two OpenRPC typing fixes needed no code change: prev_randao and voucher value are now UnsignedInteger256 in the specification, and both were already 256-bit-safe here. Also reformats four @cartesi/react hooks that biome was already reporting as unformatted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJpa3KXxvUarBGpHvW8Qdb
JSON-RPC API Improvements
This PR implements nine improvements identified as F1 to F9 as described here, and other improvements described in reviews here and here.
Divergences from Proposed Fixes
COUNT(*)rather thanErrNotFoundfor an unknown application.Infolevel toDebuglevel, but the original behavior was kept because oneInfoline was allowed per batch in the description of feature F1.Commit relation to Features and Improvements
The table below lists the commits, and column
IDindicates which improvement is introduced.