Skip to content

feat: Add Spark SQL parser dialect config#22529

Open
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:fix/add-spark-dialect
Open

feat: Add Spark SQL parser dialect config#22529
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:fix/add-spark-dialect

Conversation

@kumarUjjawal
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

This lets users combine Spark-compatible functions with Spark SQL parsing when they choose the dialect.

What changes are included in this PR?

  • Add Spark to the SQL parser dialect config enum.
  • Accept spark and sparksql as dialect config values.
  • Use Spark dialect for Spark sqllogictests.
  • Keep with_spark_features() config-neutral.
  • Update dialect docs and expected config output.
  • Add tests for Spark dialect parsing and Spark SQL execution with Spark functions.

Are these changes tested?

Yes

Are there any user-facing changes?

Users can now set datafusion.sql_parser.dialect to spark or sparksql.

@github-actions github-actions Bot added documentation Improvements or additions to documentation core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate spark labels May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for Spark SQL dialect

1 participant