Reusing http connections in Go

I'm currently struggling to find a way to reuse connections when making HTTP posts in Go.

I've created a transport and client like so:

// Create a new transport and HTTP client
tr := &http.Transport{}
client := &http.Client{Transport: tr}

I'm then passing this client pointer into a goroutine which is making multiple posts to the same endpoint like so:

r, err := client.Post(url, "application/json", post)

Looking at netstat this appears to be resulting in a new connection for every post resulting in a large number of concurrent connections being open.

What is the correct way to reuse connections in this case?

Solution 1:

Ensure that you read until the response is complete AND call Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

Again... To ensure http.Client connection reuse be sure to:

• Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
• Call Body.Close()

Solution 2:

If anyone is still finding answers on how to do it, this is how I am doing it.

package main

import (
  "bytes"
  "io/ioutil"
  "log"
  "net/http"
  "time"
)

func httpClient() *http.Client {
    client := &http.Client{
        Transport: &http.Transport{
            MaxIdleConnsPerHost: 20,
        },
        Timeout: 10 * time.Second,
    }

    return client
}

func sendRequest(client *http.Client, method string) []byte {
    endpoint := "https://httpbin.org/post"
    req, err := http.NewRequest(method, endpoint, bytes.NewBuffer([]byte("Post this data")))
    if err != nil {
        log.Fatalf("Error Occured. %+v", err)
    }

    response, err := client.Do(req)
    if err != nil {
        log.Fatalf("Error sending request to API endpoint. %+v", err)
    }

    // Close the connection to reuse it
    defer response.Body.Close()

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Fatalf("Couldn't parse response body. %+v", err)
    }

    return body
}

func main() {
    c := httpClient()
    response := sendRequest(c, http.MethodPost)
    log.Println("Response Body:", string(response))
}

Go Playground: https://play.golang.org/p/cYWdFu0r62e

In summary, I am creating a different method to create an HTTP client and assigning it to a variable, and then using it to make requests. Note the

defer response.Body.Close() 

This will close the connection after the request is complete at the end of the function execution and you can reuse the client as many times.

If you want to send a request in a loop call the function that sends the request in a loop.

If you want to change anything in the client transport configuration, like add proxy config, make a change in the client config.

Hope this will help someone.

Solution 3:

Edit: This is more of a note for people that construct a Transport and Client for every request.

Edit2: Changed link to godoc.

Transport is the struct that holds connections for re-use; see https://godoc.org/net/http#Transport ("By default, Transport caches connections for future re-use.")

So if you create a new Transport for each request, it will create new connections each time. In this case the solution is to share the one Transport instance between clients.

Solution 4:

IIRC, the default client does reuse connections. Are you closing the response?

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.