Why does uploading files ~2,5 MiB or larger cause a connection reset?

Question 1

We are trying to implement image uploading through POST requests. We also want to limit the images to ~1,0 MiB. It works fine on smaller images, but anything ~2,5 MiB or larger causes the connection to reset. It also seems to send multiple requests after the first to the same handler.

main.go:

package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", uploadHandler)
    http.ListenAndServe("localhost:8080", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method == "POST" {
        postHandler(w, r)
        return
    } else {
        http.ServeFile(w, r, "index.html")
    }
}

func postHandler(w http.ResponseWriter, r *http.Request) {
    // Send an error if the request is larger than 1 MiB
    if r.ContentLength > 1<<20 {
        // if larger than ~2,5 MiB, this will print 2 or more times
        log.Println("File too large")
        // And this error will never arrive, instead a Connection reset
        http.Error(w, "response too large", http.StatusRequestEntityTooLarge)
        return
    }
    return
}

index.html:

<!DOCTYPE html>
<html lang="">
  <head>
    <meta charset="utf-8">
    <title></title>
  </head>
  <body>
    <form method="POST" enctype="multipart/form-data">
      <input type="file" accept="image/*" name="profile-picture"><br>
      <button type="submit" >Upload</button>
  </form>
  </body>
</html>

Output when uploading a ~2,4 MiB file

$ go run main.go
2021/11/23 22:00:14 File too large

It also shows "request too large" in the browser

Output when uploading ~2,5 MiB file

$ go run main.go
2021/11/23 22:03:25 File too large
2021/11/23 22:03:25 File too large

The browser now shows that the connection was reset

Question 2

The client is trying to send data to the server. The server is not reading the data, it's just looking at the headers and closing the connection. The client is interpreting this as "connection was reset". This is out of your control.

Instead of checking the header, the header can lie, use http.MaxBytesReader to read the actual content, but error if it is too large.

const MAX_UPLOAD_SIZE = 1<<20

func postHandler(w http.ResponseWriter, r *http.Request) {
    // Wrap the body in a reader that will error at MAX_UPLOAD_SIZE
    r.Body = http.MaxBytesReader(w, r.Body, MAX_UPLOAD_SIZE)

    // Read the body as normal. Check for an error.
    if err := r.ParseMultipartForm(MAX_UPLOAD_SIZE); err != nil {
        // We're assuming it errored because the body is too large.
        // There are other reasons it could error, you'll have to
        // look at err to figure that out.
        log.Println("File too large")
        http.Error(w, "Your file is too powerful", http.StatusRequestEntityTooLarge)
        return
    }

    fmt.Fprintf(w, "Upload successful")
}

See How to process file uploads in Go for more detail.

Schwern · Answer 1 · 2021-11-23T20:55:11

The client is trying to send data to the server. The server is not reading the data, it's just looking at the headers and closing the connection. The client is interpreting this as "connection was reset". This is out of your control.

Instead of checking the header, the header can lie, use http.MaxBytesReader to read the actual content, but error if it is too large.

const MAX_UPLOAD_SIZE = 1<<20

func postHandler(w http.ResponseWriter, r *http.Request) {
    // Wrap the body in a reader that will error at MAX_UPLOAD_SIZE
    r.Body = http.MaxBytesReader(w, r.Body, MAX_UPLOAD_SIZE)

    // Read the body as normal. Check for an error.
    if err := r.ParseMultipartForm(MAX_UPLOAD_SIZE); err != nil {
        // We're assuming it errored because the body is too large.
        // There are other reasons it could error, you'll have to
        // look at err to figure that out.
        log.Println("File too large")
        http.Error(w, "Your file is too powerful", http.StatusRequestEntityTooLarge)
        return
    }

    fmt.Fprintf(w, "Upload successful")
}

See How to process file uploads in Go for more detail.

This seems to work to an extend and actually suits to our needs. However it still causes a connection reset with a large enough image like this, so for example, if I wanted to raise the MAX_UPLOAD_SIZE to 20 << 20 for whatever reason I wouldn't be able to upload anything of that size.
I also thought the r.ContentLength could be used as a quick check before any files are even uploaded at all, even though I know that it could be spoofed. I guess you could implement this in the client-side of things
I think I figured out why the connection reset from my above comment, you need to actually start using the data (for example with r.FormFile), otherwise it will just stop and return, and close the connection once it starts getting annoyed by the constant data being sent by the client.

Why does uploading files ~2,5 MiB or larger cause a connection reset?

In other languages

This page is in other languages

Popular in the category