fix(glue-alpha): race condition when creating mutiple indices

[Abogical]

Issue # (if applicable)

Closes #24813.

Reason for this change

Creating multiple partition indexes on a Glue table fails with:

│ "Index index1 is in CREATING state. Only 1 index can be created or deleted simultaneously per table."

This happens because:

1. The dependency direction between partition index custom resources was backwards — CloudFormation created
   them in reverse order.
2. AwsCustomResource reports success immediately after the createPartitionIndex API returns HTTP 200, while
   the index is still in CREATING state. CloudFormation then starts the next index before the previous one
   finishes.

Description of changes

The main change was the creation of a new custom resource handler to replace the existing one, based on AwsCustomResource. This new handler is asynchronous; it has an isComplete method that polls the service to check whether the requested operation is complete. This ensures that the creation requests are serialized. There are some edge cases worth mentioning, though:

• When the onEvent handler receives a "Create" event, it checks whether a partition index with those exact keys already exists. If it does, the handler does nothing and returns the same physical ID, to indicate that the operation was successful.
• The Glue API does not have an update operation. If the user changes an explicitly named index, the handler has to perform the update itself, by deleting the existing index and creating a new one. The order is exactly this: delete-then-create, which goes against the usual pattern create-then-delete. This is because tables can have at most 3 partition indexes. So a create-then-delete would risk trying to create a 4th one and would fail.
• Another departure from customary custom resource practices is that the isComplete has side effects. The update flow works like this: the onEvent, upon deciding that an update is necessary, sends a delete message to service. and returns. Then, on each subsequent poll, isComplete checks whether the index has been deleted. If the index is not there anymore, it creates the new index (see the sequence diagram below). It has to be done this way for the same reason that motivates this PR: the API is eventually consistent, so it may take an indefinite amount of time for the delete to be committed.
• A special case for the update flow is when what changed in the custom resource is the table name or database name. This can happen if the table is renamed or the index starts pointing to another table, for example. Either way, the onCreate handler won't delete anything. If the index needs to be deleted (for example, the table was replaced), then CloudFormation itself will delete the table and, with it, the indexes.

In addition, this PR fixes the dependency direction so each new partition index custom resource depends on the previous one (was backwards before).

Describe any new or updated permissions being added

The partition index handler Lambda roles require:

• glue:CreatePartitionIndex — to create indexes
• glue:DeletePartitionIndex — to clean up on stack deletion
• glue:GetPartitionIndexes — to poll index status in the isComplete handler
• glue:GetTable — required by the GetPartitionIndexes API
• glue:UpdateTable — required by the CreatePartitionIndex API

These are scoped to the specific table, database, and catalog ARNs for each table that uses partition indexes.

Custom resource sequence diagram

  sequenceDiagram
      participant CFN as CloudFormation
      participant OE as onEvent
      participant IC as isComplete (loop)
      participant Glue
  
      CFN->>OE: Update (keys changed)
      OE->>Glue: DeletePartitionIndex (old)
      OE-->>CFN: ack (async delete kicked off)
      loop every 10s until ACTIVE / timeout
          CFN->>IC: poll
          IC->>Glue: GetPartitionIndexes
          alt old index still present (ACTIVE-old or DELETING)
              IC-->>CFN: IsComplete=false
          else index gone
              IC->>Glue: CreatePartitionIndex (new keys)
              IC-->>CFN: IsComplete=false
          else new index present & CREATING
              IC-->>CFN: IsComplete=false
          else new index ACTIVE
              IC-->>CFN: IsComplete=true
          end
      end

Loading

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

BREAKING CHANGE: Partition index custom resources will be replaced on stack update due to implementation change from AwsCustomResource to Provider-based CustomResource. The CloudFormation resource type changes from Custom::AWS to Custom::GluePartitionIndex. Partition indexes are not stateful so no data is lost, but users will see resource replacement during the first update after upgrading.

BREAKING CHANGE: Indexes with tokenized key names must have an explicitly provided index name now.

[alvazjor]

Before this goes in, can we confirm the upgrade path was actually tested? Deploy an app on the current released aws-glue-alpha with a couple of partition indexes, then deploy this branch on top, and confirm the indexes survive and the stack updates cleanly.

What I specifically want covered is the create-before-delete overlap: the resource type changes from Custom::AWS to Custom::GluePartitionIndex, so CloudFormation creates the new resource (which adopts the existing index on AlreadyExists here) and then deletes the old Custom::AWS resource. I want to be sure that trailing delete of the old resource does not remove the index the new one just adopted. Every existing user hits this on their first upgrade, so the blast radius is large if it is wrong.

I do not think we need this as a permanent integ test, but I do want the manual test done and a short note of the result in the PR.

[alvazjor]

Nit: why 2 hours here? Partition index creation usually finishes in minutes, and a 2 hour totalTimeout means a stuck index keeps a deploy polling for a long time before it fails, which also surprises anyone with a shorter CI/CD step timeout. If there is a real backfill case that needs this long, a one line comment saying so would help. Otherwise something like 30-60 minutes seems closer to reality.