Skip to content

fix array_repeat scalar path overflows total repeated-value count#22274

Open
xiedeyantu wants to merge 3 commits into
apache:mainfrom
xiedeyantu:array_repeat
Open

fix array_repeat scalar path overflows total repeated-value count#22274
xiedeyantu wants to merge 3 commits into
apache:mainfrom
xiedeyantu:array_repeat

Conversation

@xiedeyantu
Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

The scalar path of array_repeat can overflow while computing the total number of repeated values. Previously, the implementation used unchecked accumulation for these totals, which could lead to overflow behavior that was not explicitly handled. This change makes the capacity and offset calculations overflow-safe and returns a clear execution error when the total output size exceeds usize.

What changes are included in this PR?

  • Adds overflow checks for the total repeated value count in the scalar array_repeat path.
  • Adds overflow checks for outer and inner total size calculations in the list repeat path.
  • Preserves the existing behavior where non-positive counts are treated as zero.
  • Adds sqllogictest coverage for the overflow error case.

Are these changes tested?

  • Yes. A new sqllogictest covers the scalar overflow failure case for array_repeat.
  • Existing related tests continue to pass.

Are there any user-facing changes?

  • Yes. For extremely large inputs, array_repeat now returns a clear execution error instead of relying on implicit overflow behavior.
  • Normal inputs are unchanged.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels May 16, 2026
@xiedeyantu xiedeyantu changed the title array_repeat scalar path overflows total repeated-value count fix array_repeat scalar path overflows total repeated-value count May 16, 2026
Copy link
Copy Markdown
Contributor

@kosiew kosiew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiedeyantu
Thanks for tightening up the overflow handling here. I found two remaining paths where we can still hit unchecked allocation or Arrow offset construction before returning the intended execution error.

"array_repeat: total repeated values overflowed usize",
)?;

let mut take_indices = Vec::with_capacity(total_repeated_values);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scalar/non-list path still allocates take_indices before confirming that the result fits in the returned List offset type.

For example, on the default List<i32> path, a count like 2147483648 does not overflow usize, so checked_sum_counts succeeds. After that, Vec::with_capacity(total_repeated_values) can try to reserve around 16 GiB before the later O::from_usize(running_offset) error is reached.

Could we validate O::from_usize(total_repeated_values) first, and return the existing offset error before doing any capacity reservation or extension?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list-input path still has a couple of unchecked capacity and offset construction spots.

outer_total + 1 can overflow when the checked sum is exactly usize::MAX, and the outer offsets are later rebuilt with OffsetBuffer::<O>::from_lengths(...), which can panic on usize or offset overflow in Arrow.

Since this function is aiming to make capacity and offset calculations overflow-safe, could we avoid the unchecked + 1 and avoid from_lengths for untrusted counts? It looks like we can reuse the already checked totals and build or validate the outer offsets with checked_add plus O::from_usize before allocating.

@xiedeyantu
Copy link
Copy Markdown
Member Author

@xiedeyantu Thanks for tightening up the overflow handling here. I found two remaining paths where we can still hit unchecked allocation or Arrow offset construction before returning the intended execution error.

@kosiew Thanks for the review! I’ll wait until #22305 is merged before making adjustments to this PR, as I’m concerned about potential merge conflicts.

Copy link
Copy Markdown
Contributor

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#22305 is now merged, are we waiting on anything else for this PR

Comment on lines +243 to +245
DataFusionError::Execution(
"array_repeat: outer total overflowed usize".to_string(),
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DataFusionError::Execution(
"array_repeat: outer total overflowed usize".to_string(),
)
exec_datafusion_err!("array_repeat: running_offset overflowed usize")

Same for others

}
}

fn checked_sum_counts(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably just inline this since its used only once

@xiedeyantu
Copy link
Copy Markdown
Member Author

#22305 is now merged, are we waiting on anything else for this PR

@Jefffrey Thank you for your attention to this PR. There are no longer any blockages, and I will complete the PR within a few hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic: array_repeat scalar path overflows total repeated-value count

3 participants