Resource API discovery results do not appear to come from a cache

[jessesuen]

It was discovered in #449 (comment) that controller was making 1M api calls to perform discovery.

The client-go does seem to provide a cached discovery option, but based on evidence of API calls, we may not be taking advantage of it

https://pkg.go.dev/k8s.io/client-go/discovery#CachedDiscoveryInterface

[crenshaw-dev]

This is a partial fix: https://github.com/argoproj/gitops-engine/pull/448/files

KubectlCmd calls that require GVK info instantiate and use a one-off discovery client. Fixing that will require a bigger refactor, so I left it for later.

[crenshaw-dev]

^ that fix pays no attention to the cluster changes which should invalidate the cache, so we probably shouldn't merge it as-is.