Skip to content

[BUG] UNRESOLVED_COLUMN when table contains columns with dots #267

@SamyCoenen

Description

@SamyCoenen

Describe the bug
When row_dq rules are configured for a table containing dots, Spark Expectations will automatically resolve all columns, even when it's not in the row_dq rules itself.

To Reproduce
Steps to reproduce the behavior:

  1. Go to your rules table
  2. Append a rule with type row_dq and target a table containing dots in the column names

Expected behavior
A clear and concise description of what you expected to happen.
It should automatically escape the column names using backticks.

Logs

error occurred while processing spark expectations error occurred 
while executing func_process error occurred 
while taking action on given rules [UNRESOLVED_COLUMN.WITH_SUGGESTION] 
A column or function parameter
with name `event`.`actionCode` cannot be resolved.

Additional context
Spark Expectations version: v2.6.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions