ak.to_packed does not remove OptionType if there are no Nones

[lauridsj]

Version of Awkward Array

2.9.0

Description and code to reproduce

ak.to_packed does not seem to remove OptionType from an array of records even if there are no more Nones in the array. This is a problem e.g. when writing to TTree with uproot, since TTree has no concept of option types.

Minimal reproducer:

# jagged array of records where a record is None
arr = ak.Array([[
    {"pt": 100., "eta": 0.5},
    {"pt": 200., "eta": 1.0},
    {"pt": 300., "eta": 1.5}
], [
    {"pt": 150., "eta": -0.5},
    {"pt": 250., "eta": -1.0},
    None
]])

# type is ListType(OptionType(...)) since there is a None
print("arr type:", repr(arr.type.content))

# Remove the event where there is a None
arr = arr[:1]
print("has nones:", ak.is_none(arr))

# type still has OptionType
print("arr type:", repr(arr.type.content))

# to_packed does not fix it
arr = ak.to_packed(arr)
print("arr type:", repr(arr.type.content))

# it is fixed when we explicitly drop nones from the array
arr = ak.drop_none(arr)
print("arr type:", repr(arr.type.content))

[ikrommyd]

Although I do agree that such fake option types should probably be dropped, it looks very intentional in the codebase at the moment because even UnmaskedArray which implements exactly such a fake option type with no missing values still returns an UnmaskedArray when to_packed is called on it

awkward/src/awkward/contents/unmaskedarray.py

Lines 604 to 608 in 7499387

	def _to_packed(self, recursive: bool = True) -> Self:
	return UnmaskedArray(
	self._content.to_packed(True) if recursive else self._content,
	parameters=self._parameters,
	)

Also the to_packed docstring says that IndexedOptionArray will remain an IndexedOptionArray.
We need to check what the implications of such a change are

[ariostas]

even UnmaskedArray which implements exactly such a fake option type with no missing values still returns an UnmaskedArray when to_packed is called on it

I think this is only needed because unions with optional types end up being stored as one single optional type, along with the other types wrapped by UnmaskedArray, so they can't be dropped in this case. But I'd say that it does make sense to fully drop the option type in to_packed when there are no nones.

[ianna]

even UnmaskedArray which implements exactly such a fake option type with no missing values still returns an UnmaskedArray when to_packed is called on it

I think this is only needed because unions with optional types end up being stored as one single optional type, along with the other types wrapped by UnmaskedArray, so they can't be dropped in this case. But I'd say that it does make sense to fully drop the option type in to_packed when there are no nones.

I think we already do it when simplifying an UnmaskedArray

[ikrommyd]

When simplifying an unmasked array yes. But to_packed does not call simplified. At the same time, simplified of an IndexedOptionArray remains an IndexedOptionArray even when no Nones are present.