Client side throttling in litmus checker

[Ghost-B0t]

We are facing throtlling for API discoveries for litmus checker, resulting in much delayed start of the experiments.

Currently, NewDeferredDiscoveryRESTMapper is called every minute which in turn discovers every installed api groups of the cluster. This becomes a bottleneck for clusters with high number of API groups and api-resources.

Logs:

1 request.go:668]  Waited for 1.156624404s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/source.toolkit.fluxcd.io/v1beta2?timeout=32s\\nI0213 10:51:40.090698
 1 request.go:668] Waited for 11.355498416s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/kafka.strimzi.io/v1alpha1?timeout=32s\\nI0213 10:51:50.289611
1 request.go:668] Waited for 21.554249198s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/crd.k8s.amazonaws.com/v1alpha1?timeout=32s\\n

Suggestion:

• Initialize the mapper once and reuse it instead of creating a new one every time.
• Allow setting up QPS and Burst for k8s api calls via ENV variables, the default value for go-client is (QPS: 5, Burst: 10)

Open to discussing other suggestions for a better fix, else I can work on this implementation if this looks good.