Skip to content

[BUG] Fused MMyyyy parser misses EXCEPTION-policy disagreement #15977

Description

@gerashegalov

Describe the bug

Under spark.sql.legacy.timeParserPolicy=EXCEPTION, Spark compares CORRECTED and LEGACY parsing when corrected parsing fails. The fused GPU timestamp parser does not currently preserve that disagreement for the packed MMyyyy format.

For input 12024 with format MMyyyy, Spark CPU throws SparkUpgradeException because CORRECTED parsing fails while LEGACY parses a value. The GPU fused parser returns null instead. Until this is fixed, MMyyyy must remain excluded from the explicit EXCEPTION-policy acceleration allowlist.

Steps/Code to reproduce bug

spark.conf.set("spark.sql.legacy.timeParserPolicy", "EXCEPTION")
spark.createDataFrame([("12024",)], "value string") \
    .selectExpr("to_date(value, 'MMyyyy')") \
    .collect()

Compare CPU execution with GPU execution using the fused timestamp parser.

Expected behavior

The GPU path should reproduce Spark EXCEPTION-policy behavior and throw when CORRECTED and LEGACY parsing disagree. Once the JNI parser exposes that behavior, add MMyyyy back to EXCEPTION_COMPATIBLE_FORMATS with coverage for both ordinary valid values and disagreement inputs.

Environment details (please complete the following information)

  • Environment location: Standalone
  • Spark versions confirmed: 3.5.7 and 4.0.1
  • Relevant settings: spark.sql.legacy.timeParserPolicy=EXCEPTION

Additional context

Found while validating the compatibility hardening in #15820. The same audit found no disagreement for the other explicitly allowlisted formats.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions