MySQL Source Versioning V2 - reorganized boogaloo - #36333
Conversation
ublubu
left a comment
ublubu
left a comment
There was a problem hiding this comment.
I took a first pass at the "decode binlog rows by column name" commit. I'll look at the tests more carefully tomorrow.
martykulma
left a comment
martykulma
left a comment
There was a problem hiding this comment.
Looking good! There are some additional improvements to make; partly my fault - i didn't think through the strict vs. lenient checks in purification (sorry!).
|
I just realized that |
478081e to
090552c
Compare
Refactors `pack_mysql_row()` to accept `gtid_set` and `binlog_full_metadata` parameters. When `binlog_full_metadata=true`, columns are matched by name from the wire row to the table descriptor (safe under reordering). When false, falls back to position-based matching (original behavior, required for `binlog_row_metadata=MINIMAL`). Adds diagnostic helpers `decode_error()` and `describe_row_shape()` for richer error context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l_metadata=true Adds a `full_metadata: bool` parameter to `MySqlTableDesc::determine_compatibility()`. When true, columns are matched by name (allowing upstream reordering and safe addition of new columns). When false, uses the original positional prefix check. `verify_schemas()` now passes `output.binlog_full_metadata` to drive the choice. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
090552c to
bbce9d4
Compare
Ah this is actually the test that is flaky because of the DROP-> re-CREATE race condition, which isn't fixed quite yet, so it should stay disabled for now |
c0a6864 to
10b4c82
Compare
10b4c82 to
725768c
Compare
| Err(DataflowError::from(DefiniteError::ValueDecodeError( | ||
| format!( | ||
| "Table {0} was created with binlog_row_metadata=FULL but binlog_row_metadata has since been set to a different value, meaning we cannot reliably decode the columns", | ||
| output.table_name | ||
| ), |
There was a problem hiding this comment.
We never mark the export(s) as borked (ctx.errored_outputs), which means we will continue to emit errors as long as the setting is incorrect. Once the customer corrects it, there isn't a way to recover for MZ, but we will try!
Consider we have source will full metadata, a row with a value A that sees some updates and is deleted, and interleaved someone accidentally changes row metadata:
A -> B .... A:-1 B:+1
------------------------------------ metadata: FULL -> MINIMAL
B -> C .... Err(B):-1 Err(C):+1
------------------------------------ metadata: MINIMAL -> FULL
C -> NULL .... C:-1
There was a problem hiding this comment.
This seems like something that is fundamentally unrecoverable, no? You mean that by adding it to ctx.errored_outputs we and ensuring that it cannot be recovered, and also that the source is more visibly broken, I assume?
There was a problem hiding this comment.
yes, by adding it - we stop processing events for it. The last thing appended would be the error. I believe we do this for the MySQL DDL errors, and you should also find it in PG.
|
triggering a nightly before merging, fingers crossed |
5 participants
This PR effectively re-implements source versioning for mysql after we reverted the initial implementation last week due to decoding problems with exclude columns resulting in an incident. In doing so, it fixes a number of issues that existed with the first implementation, including:
https://github.com/MaterializeInc/database-issues/issues/11312
https://github.com/MaterializeInc/database-issues/issues/11313
https://github.com/MaterializeInc/database-issues/issues/11315
And provides a safer mechanism for handling schema changes and changes to the
binlog_row_metadataMySQL system variable.The first commit is roughly the changes in #36253 which should be merged first, and is required for the other changes.
Second commit updates the decoding logic based on the binlog metadata setting at source creation
Third commit updates the logic to verify mysql schemas with the schemas in the upstream, allowing for certain types of schema changes when binlog_row_metadata is FULL.
Fourth commit contains docs for how to make schema changes to your mysql source without downtime in materialize.