Benchmarking pipeline improvements

[rahulgurnani]

What?

Add retries with timeout to the verify installation and wait for all resources to be ready step.
Add checks for HealthCheckPolicy, BackendPolicy as part of the benchmarking pipeline at when the pipeline waits for resources all resources to be ready.

Why?
To make the benchmarking pipeline more reliable. Presently, it fails sometimes. This should help with avoiding regressions.

[k8s-ci-robot]

@rahulgurnani: GitHub didn't allow me to assign the following users: rlakhtakia.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @rlakhtakia

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rahulgurnani]

/assign

[rahulgurnani]

Also, the flags try to depend on GPUs,

gateway-api-inference-extension/.github/workflows/e2e-prefix-cache-aware-gke.yaml

Line 298 in 6cb1201

--set-string 'job.resources.limits.nvidia\.com/gpu=1'

This causes stockout (GPUs are scarce)
I think we don't really need this flag.

[rahulgurnani]

/unassign

[rahulgurnani]

I realized, I don't have bandwidth for a bit but would add some sample failures for folks to investigate further etc: https://github.com/kubernetes-sigs/gateway-api-inference-extension/actions/runs/23908983255/job/69725586261

[rlakhtakia]

/assign