Resumable file uploader: Creating http handlers

calendar Jun 18, 2021 · 9 min read  ·
Share on: linkedin copy

Welcome to tutorial no. 3 in our Resumable file uploader series.

The previous tutorials provided an introduction about the tus protocol and we also created the DB CRUD methods.

In this tutorial, we will create the http handlers to support the POST, PATCH and HEAD http methods.

This tutorial has the following sections

  • POST http handler
  • HEAD http handler
  • PATCH http handler
  • File validation
  • Upload complete validation
  • Upload offset validation
  • Content length validation
  • File patch

POST http handler

Before we create the POST handler, we need a directory to store the files. For simplicity, we are going to create a directory named fileserver inside the home directory to store the files.

 1const dirName="fileserver"
 2
 3func createFileDir() (string, error) {
 4	u, err := user.Current()
 5	if err != nil {
 6		log.Println("Error while fetching user home directory", err)
 7		return "", err
 8	}
 9	home := u.HomeDir
10	dirPath := path.Join(home, dirName)
11	err = os.MkdirAll(dirPath, 0744)
12	if err != nil {
13		log.Println("Error while creating file server directory", err)
14		return "", err
15	}
16	return dirPath, nil
17}

In the above function, we get the current user’s name and home directory and append dirName constant to create the directory. This function will return the path of the newly created directory or errors if any.

This function will be called from main and the dirPath returned from this function will be used by the POST file handler to create the file.

Now that we have the directory ready, let’s move to the POST http handler. We will name this handler createFileHandler. The POST http handler is used to create a new file and return the location of the newly created file in the Location header. It is mandatory for the request to contain a Upload-Length header indicating the entire file size.

 1func (fh fileHandler) createFileHandler(w http.ResponseWriter, r *http.Request) {
 2	ul, err := strconv.Atoi(r.Header.Get("Upload-Length"))
 3	if err != nil {
 4		e := "Improper upload length"
 5		log.Printf("%s %s", e, err)
 6		w.WriteHeader(http.StatusBadRequest)
 7		w.Write([]byte(e))
 8		return
 9	}
10	log.Printf("upload length %d\n", ul)
11	io := 0
12	uc := false
13	f := file{
14		offset:         &io,
15		uploadLength:   ul,
16		uploadComplete: &uc,
17	}
18	fileID, err := fh.createFile(f)
19	if err != nil {
20		e := "Error creating file in DB"
21		log.Printf("%s %s\n", e, err)
22		w.WriteHeader(http.StatusInternalServerError)
23		return
24	}
25
26	filePath := path.Join(fh.dirPath, fileID)
27	file, err := os.Create(filePath)
28	if err != nil {
29		e := "Error creating file in filesystem"
30		log.Printf("%s %s\n", e, err)
31		w.WriteHeader(http.StatusInternalServerError)
32		return
33	}
34	defer file.Close()
35	w.Header().Set("Location", fmt.Sprintf("localhost:8080/files/%s", fileID))
36	w.WriteHeader(http.StatusCreated)
37	return
38}

In line no. 2 we check whether the Upload-Length header is valid. If not we return a Bad Request response.

If the Upload-Length is valid, we create a file in the DB with the provided upload length and with initial offset 0 and upload complete false. Then we create the file in the filesystem and return the location of the file in the Location http header and a 201 created response code.

The dirPath field containing the path to store the file should be added to the fileHandler struct. This field will be updated with the dirPath returned from createFileDir() function later from main(). The updated fileHandler struct is provided below.

type fileHandler struct {
	db      *sql.DB
	dirPath string
}

HEAD http handler

When a HEAD request is received, we are supposed to return the offset of the file if it exists. If the file does not exist, then we should return a 404 not found response. We will name this handler as fileDetailsHandler.

 1func (fh fileHandler) fileDetailsHandler(w http.ResponseWriter, r *http.Request) {
 2	vars := mux.Vars(r)
 3	fID := vars["fileID"]
 4	file, err := fh.File(fID)
 5	if err != nil {
 6		w.WriteHeader(http.StatusNotFound)
 7		return
 8	}
 9	log.Println("going to write upload offset to output")
10	w.Header().Set("Upload-Offset", strconv.Itoa(*file.offset))
11	w.WriteHeader(http.StatusOK)
12	return
13}

We will use mux router to route the http requests. Please run the command go get github.com/gorilla/mux to fetch the mux router from github.

In line no 3. we get the fileID from the request URL using mux router.

For the purpose of understanding, I have provided the code which will call the above fileDetailsHandler. We will be writing the below line in the main function later.

r.HandleFunc("/files/{fileID:[0-9]+}", fh.fileDetailsHandler).Methods("HEAD")

This handler will be called when the URL has a valid integer fileID. [0-9]+ is a regular expression that matches one or more digits. If the fileID is valid, it will be stored with the key fileID in a map of type map[string]string . This map can be retrieved by calling the Vars function of the mux router. This is how we get the fileID in line no. 3.

After getting the fileID, we check whether the file exists by calling the File method in line no. 4. Remember we wrote this File Method in the last tutorial. If the file is valid, we return the response with the Upload-Offset header. If not we return a http.StatusNotFound response.

PATCH http handler

The only remaining handler is the PATCH http handler. There are a few validations to be done in the PATCH request before we move to the actual file patching. Let’s do them first.

File validation

The first step is to make sure the file trying to be uploaded actually exists.

 1func (fh fileHandler) filePatchHandler(w http.ResponseWriter, r *http.Request) {
 2log.Println("going to patch file")
 3	vars := mux.Vars(r)
 4	fID := vars["fileID"]
 5	file, err := fh.File(fID)
 6	if err != nil {
 7		w.WriteHeader(http.StatusNotFound)
 8		return
 9	}
10}

The above code is similar to the one we wrote in the head http handler. It validates whether the file exists.

Upload complete validation

The next step is to check whether the file has already been uploaded completely.

1if *file.uploadComplete == true {
2		e := "Upload already completed" //change to string
3		w.WriteHeader(http.StatusUnprocessableEntity)
4		w.Write([]byte(e))
5		return
6	}

If the upload is already complete, we return a StatusUnprocessableEntity status.

Upload offset validation

Each patch request should contain a Upload-Offset header field indicating the current offset of the data and the actual data to be patched to the file should be present in the message body.

 1off, err := strconv.Atoi(r.Header.Get("Upload-Offset"))
 2if err != nil {
 3	log.Println("Improper upload offset", err)
 4	w.WriteHeader(http.StatusBadRequest)
 5	return
 6}
 7log.Printf("Upload offset %d\n", off)
 8if *file.offset != off {
 9	e := fmt.Sprintf("Expected Offset %d got offset %d", *file.offset, off) 
10	w.WriteHeader(http.StatusConflict)
11	w.Write([]byte(e))
12	return
13}

In the above code, we first check whether the Upload-Offset in the request header is valid. If it is not, we return a StatusBadRequest.

In line no. 8, we compare the offset in the table *file.Offset with the one present in the header off. They are expected to be equal. Let’s take the example of a file with upload length 250 bytes. If 100 bytes are already uploaded, the upload offset in the database will be 100. Now the server will expect a request with Upload-offset header 100. If they are not equal, we return a StatusConflict header.

Content length validation

The next step is validating the content-length.

 1clh := r.Header.Get("Content-Length")
 2cl, err := strconv.Atoi(clh)
 3if err != nil {
 4	log.Println("unknown content length")
 5	w.WriteHeader(http.StatusInternalServerError)
 6	return
 7}
 8
 9if cl != (file.uploadLength - *file.offset) {
10	e := fmt.Sprintf("Content length doesn't not match upload length.Expected content length %d got %d", file.uploadLength-*file.offset, cl)
11	log.Println(e)
12	w.WriteHeader(http.StatusBadRequest)
13	w.Write([]byte(e))
14	return
15}

Let’s say a file is 250 bytes length and the current offset is 150. This indicates that there are 100 more bytes to be uploaded. Hence the Content-Length of the patch request should be exactly 100. This validation is done in line no. 9 of the above code.

File patch

Now comes the fun part. We have done all our validations and ready to patch the file.

 1body, err := ioutil.ReadAll(r.Body) 
 2if err != nil {
 3	log.Printf("Received file partially %s\n", err)
 4	log.Println("Size of received file ", len(body))
 5}
 6fp := fmt.Sprintf("%s/%s", fh.dirPath, fID)
 7f, err := os.OpenFile(fp, os.O_APPEND|os.O_WRONLY, 0644)
 8if err != nil {
 9	log.Printf("unable to open file %s\n", err)
10	w.WriteHeader(http.StatusInternalServerError)
11	return
12}
13defer f.Close()
14
15n, err := f.WriteAt(body, int64(off))
16if err != nil {
17	log.Printf("unable to write %s", err)
18	w.WriteHeader(http.StatusInternalServerError)
19	return
20}
21log.Println("number of bytes written ", n)
22no := *file.offset + n
23file.offset = &no
24
25uo := strconv.Itoa(*file.offset)
26w.Header().Set("Upload-Offset", uo)
27if *file.offset == file.uploadLength {
28	log.Println("upload completed successfully")
29	*file.uploadComplete = true
30}
31
32err = fh.updateFile(file)
33if err != nil {
34	log.Println("Error while updating file", err)
35	w.WriteHeader(http.StatusInternalServerError)
36	return
37}
38log.Println("going to send succesfully uploaded response")
39w.WriteHeader(http.StatusNoContent)

We start reading the message body in line no.1 of the above code. The ReadAll function returns the data it has read until EOF or there is an error. EOF is not considered as an error as ReadAll is expected to read from the source until EOF.

Let’s say the patch request disconnects before it is complete. When this happens, ReadAll will return a unexpected EOF error. Usually generic web servers will discard the request if it is incomplete. But we are creating a resumable file uploader and we shouldn’t do it. We should patch the file with the data we have received till now.

The length of data received is printed in line no. 4.

In line no. 7 we open the file in append mode if it already exists or create a new file if it doesn’t exist.

In line no. 15 we write the request body to the file at the offset provided in the request header. In line no. 23 we update the offset of the file by adding the number of bytes written. In line no. 26 we write the updated offset to the response header.

In line no. 27 we check whether the current offset is equal to the upload lenght. If this is the case then the upload has completed. We set the uploadComplete flag to true.

Finally in line no. 32 we write the updated file details to the database and return a StatusNoContent header indicating that the request is successful.

The entire code along with the main function available in github at https://github.com/golangbot/tusserver. We will need the Postgres driver to run the code. Please fetch the postgres driver running the command go get github.com/lib/pq in the terminal before running the program.

That’s about it. We have a working resumable file uploader. In the next tutorial, we will test this uploader using curl and dd commands and also discuss the possible enhancements.

Please leave your feedback and comments. Please consider sharing this tutorial tutorial on twitter and LinkedIn.

Next tutorial - Testing the server using curl and dd commands