Spark 4.1: bind parameters in IcebergSparkSqlExtensionsParser#16626
Open
j1wonpark wants to merge 1 commit into
Open
Spark 4.1: bind parameters in IcebergSparkSqlExtensionsParser#16626j1wonpark wants to merge 1 commit into
j1wonpark wants to merge 1 commit into
Conversation
Spark 4.1 routes parameterized queries through ParserInterface.parsePlanWithParameters, whose default implementation drops the parameter context and falls back to parsePlan. The Iceberg extensions parser delegates non-Iceberg SQL to the Spark parser but did not override parsePlanWithParameters, so positional and named parameters were left unbound (UNBOUND_SQL_PARAMETER) for any SQL routed through the Iceberg session extensions on Spark Connect. Override parsePlanWithParameters to delegate with the parameter context intact, mirroring parsePlan. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16625.
Root cause
Spark 4.1 routes parameterized queries through
ParserInterface.parsePlanWithParameters(sqlText, parameterContext)(SPARK-53573). The trait's default implementation ignores theParameterContextand falls back toparsePlan.IcebergSparkSqlExtensionsParseroverridesparsePlanbut notparsePlanWithParameters, so with the extensions installed the context is dropped and parameter markers are never bound. Only 4.1 is affected — the method andParameterContextare new in 4.1; 3.5 and 4.0 bind parameters post-parse.Fix
Override
parsePlanWithParametersto mirrorparsePlan: Iceberg commands keep the existing path, everything else delegates to the wrapped Spark parser with theParameterContextintact.Testing
Added regression tests in
TestExtendedParser(all fail without the fix, pass with it):testParsePlanWithParametersDelegatesForNonIcebergSql— non-Iceberg SQL delegates with the context preserved.testParsePlanWithParametersBindsPositionalParameter—SELECT ? AS idbinds end-to-end.testParsePlanWithParametersBindsNamedParameter—SELECT :id AS idbinds end-to-end.spotlessCheck,scalastyle*Check, andcheckstyle*all pass.