Describe the bug
Under spark.sql.legacy.timeParserPolicy=EXCEPTION, Spark compares CORRECTED and LEGACY parsing when corrected parsing fails. The fused GPU timestamp parser does not currently preserve that disagreement for the packed MMyyyy format.
For input 12024 with format MMyyyy, Spark CPU throws SparkUpgradeException because CORRECTED parsing fails while LEGACY parses a value. The GPU fused parser returns null instead. Until this is fixed, MMyyyy must remain excluded from the explicit EXCEPTION-policy acceleration allowlist.
Steps/Code to reproduce bug
spark.conf.set("spark.sql.legacy.timeParserPolicy", "EXCEPTION")
spark.createDataFrame([("12024",)], "value string") \
.selectExpr("to_date(value, 'MMyyyy')") \
.collect()
Compare CPU execution with GPU execution using the fused timestamp parser.
Expected behavior
The GPU path should reproduce Spark EXCEPTION-policy behavior and throw when CORRECTED and LEGACY parsing disagree. Once the JNI parser exposes that behavior, add MMyyyy back to EXCEPTION_COMPATIBLE_FORMATS with coverage for both ordinary valid values and disagreement inputs.
Environment details (please complete the following information)
- Environment location: Standalone
- Spark versions confirmed: 3.5.7 and 4.0.1
- Relevant settings:
spark.sql.legacy.timeParserPolicy=EXCEPTION
Additional context
Found while validating the compatibility hardening in #15820. The same audit found no disagreement for the other explicitly allowlisted formats.
Describe the bug
Under
spark.sql.legacy.timeParserPolicy=EXCEPTION, Spark compares CORRECTED and LEGACY parsing when corrected parsing fails. The fused GPU timestamp parser does not currently preserve that disagreement for the packedMMyyyyformat.For input
12024with formatMMyyyy, Spark CPU throwsSparkUpgradeExceptionbecause CORRECTED parsing fails while LEGACY parses a value. The GPU fused parser returns null instead. Until this is fixed,MMyyyymust remain excluded from the explicit EXCEPTION-policy acceleration allowlist.Steps/Code to reproduce bug
Compare CPU execution with GPU execution using the fused timestamp parser.
Expected behavior
The GPU path should reproduce Spark EXCEPTION-policy behavior and throw when CORRECTED and LEGACY parsing disagree. Once the JNI parser exposes that behavior, add
MMyyyyback toEXCEPTION_COMPATIBLE_FORMATSwith coverage for both ordinary valid values and disagreement inputs.Environment details (please complete the following information)
spark.sql.legacy.timeParserPolicy=EXCEPTIONAdditional context
Found while validating the compatibility hardening in #15820. The same audit found no disagreement for the other explicitly allowlisted formats.