Which component are you using?:
cluster-autoscaler
What version of the component are you using?:
Component version:
Observed on current master at 49f227731.
What k8s version are you using (kubectl version)?:
Not cluster-version dependent. This looks like a client-side request construction bug in the Kamatera provider.
What environment is this in?:
Kamatera cloud provider.
What did you expect to happen?:
When the Kamatera REST client retries a POST request, each retry should send the same JSON request body as the first attempt.
What happened instead?:
request() JSON-encodes the body into a single bytes.Buffer before the retry loop, then passes that same buffer to every http.NewRequestWithContext call:
cluster-autoscaler/cloudprovider/kamatera/kamatera_request.go:45-61
After the first http.DefaultClient.Do(req), that buffer has been consumed. Later retry attempts reuse the same exhausted reader, so they send an empty body.
This affects retrying POST call sites such as:
DeleteServer() poweroff / terminate: cluster-autoscaler/cloudprovider/kamatera/kamatera_api_client_rest.go:123-156getServerTags(): cluster-autoscaler/cloudprovider/kamatera/kamatera_api_client_rest.go:244-254
There is a second issue in the same loop: res.Body.Close() is deferred inside the retry loop, so response bodies stay open until the function returns instead of being closed before the next attempt.
How to reproduce it (as minimally and precisely as possible):
One minimal way to reproduce is to make a Kamatera API endpoint fail once and then succeed on retry while asserting on the request body for both attempts:
- Use a test HTTP server that returns a retriable failure on the first POST.
- Retry the same Kamatera POST request (
/service/server/terminate, /service/server/poweroff, or /server/tags).
- Inspect the request body received by the server for each attempt.
- The first attempt contains the expected JSON body, while later attempts receive an empty body.
This behavior appears to have been introduced by 3c187264c (add retry mechanism), which moved request sending into a retry loop without recreating the request body reader for each attempt.
Anything else we need to know?:
A straightforward fix would be to encode once to []byte, then create a fresh reader per attempt, e.g.:
bodyBytes := buf.Bytes()
...
req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(bodyBytes))
and close res.Body before continuing to the next retry instead of deferring it inside the loop.
Which component are you using?:
cluster-autoscaler
What version of the component are you using?:
Component version:
Observed on current
masterat49f227731.What k8s version are you using (
kubectl version)?:Not cluster-version dependent. This looks like a client-side request construction bug in the Kamatera provider.
What environment is this in?:
Kamatera cloud provider.
What did you expect to happen?:
When the Kamatera REST client retries a POST request, each retry should send the same JSON request body as the first attempt.
What happened instead?:
request()JSON-encodes the body into a singlebytes.Bufferbefore the retry loop, then passes that same buffer to everyhttp.NewRequestWithContextcall:cluster-autoscaler/cloudprovider/kamatera/kamatera_request.go:45-61After the first
http.DefaultClient.Do(req), that buffer has been consumed. Later retry attempts reuse the same exhausted reader, so they send an empty body.This affects retrying POST call sites such as:
DeleteServer()poweroff / terminate:cluster-autoscaler/cloudprovider/kamatera/kamatera_api_client_rest.go:123-156getServerTags():cluster-autoscaler/cloudprovider/kamatera/kamatera_api_client_rest.go:244-254There is a second issue in the same loop:
res.Body.Close()is deferred inside the retry loop, so response bodies stay open until the function returns instead of being closed before the next attempt.How to reproduce it (as minimally and precisely as possible):
One minimal way to reproduce is to make a Kamatera API endpoint fail once and then succeed on retry while asserting on the request body for both attempts:
/service/server/terminate,/service/server/poweroff, or/server/tags).This behavior appears to have been introduced by
3c187264c(add retry mechanism), which moved request sending into a retry loop without recreating the request body reader for each attempt.Anything else we need to know?:
A straightforward fix would be to encode once to
[]byte, then create a fresh reader per attempt, e.g.:and close
res.Bodybefore continuing to the next retry instead of deferring it inside the loop.