Resumable file uploader: Understanding tus protocol
Welcome to tutorial no. 1 in our Resumable file uploader series.
How many times have you tried to upload a large file only to know that it failed because of a network issue! When you re-upload the file again, the upload starts from the beginning :(. Not cool at all. This is where resumable file uploaders come in handy.
Resumable file uploaders allow the file upload to start right from the point where it stopped instead of uploading the whole file again.
In this tutorial series we will learn how to create a resumable file upload server and client in Go using the tus protocol. This tutorial is not an exact implementation of the tus protocol, but rather a simplified version. This tutorial is self-sufficient to create a resumable file uploader using Go. We will keep improving this uploader in the upcoming tutorials and make it full tus compatible.
This tutorial has the following sections
- Tus protocol
- POST request to create the file
- PATCH request to update the file
- HEAD request to get the current file offset
Tus protocol
The tus protocol is quite simple and the best selling point of tus is that it works on top of HTTP. Let’s first understand how tus protocol works.
Tus protocol needs three http methods namely POST, PATCH and HEAD. It’s best to understand the tus protocol using an example.
Let’s take the example of uploading a file of size 250 bytes. The upcoming sections explain the sequence of http calls required to upload a file using tus protocol.
POST request to create the file
This is the first step. The client sends a POST request with the file’s upload length(size) to the server. The server creates a new file and responds with the file’s location.
Request
POST /files HTTP/1.1
Host: localhost:8080
Content-Length: 0
Upload-Length: 250
In the above request, we send a POST request to the URL localhost:8080/files to create a file with Upload-length 250 bytes. The Upload-length represents the size of the entire file. Since the request does not have a message body, the Content-Length field is zero.
The server creates the file and returns the following response.
Response
HTTP/1.1 201 Created
Location: localhost:8080/files/12
The Location header provides the location of the created file. In our case, it is localhost:8080/files/12
PATCH request to update the file
Patch request is used to write bytes to the file at offset Upload-Offset. Each patch request should contain a Upload-Offset field indicating the current offset of the file data being uploaded.
In our case, since we just created a new file and starting to upload data to the file, the client sends a PATCH request with Upload-Offset as 0. Please note that file offsets are zero based. The first byte of the file is at offset 0.
Request
PATCH /files/12 HTTP/1.1
Host: localhost:8080
Content-Length: 250
Upload-Offset: 0
[250 bytes of the file]
In the above request, the Content-Length field is 250 since we are uploading a file of size 250 bytes. The Upload-Offset is 0 indicating that the server should write the contents of the request at byte 1 of the file.
The server will respond with a 204 No Content header indicating the request is successful. Response to the PATCH request should contain the Upload-Offset field indicating the next byte to be uploaded. In this case, the Upload-Offset field will be 250 indicating that the server has received the entire file and the upload is complete.
Response
HTTP/1.1 204 No Content
Upload-Offset: 250
The above response from the server indicates that the upload has completed successfully since the Upload-Offset is equal to the Upload-Length 250.
HEAD request to get the current file offset
The patch request above was completed successfully without any network problems and the file was uploaded completely.
What if there was a network issue while the file was being uploaded and the upload failed in the middle. The client should not upload the entire file again but rather start uploading the file from the failed byte. This is where the HEAD request helps.
Let’s say the file upload request disconnected after uploading 100 bytes. The client needs to send a HEAD request to the server to get the current Upload-Offset of the file to know how many bytes have been uploaded and how much is still left to be uploaded.
Request
HEAD /files/12 HTTP/1.1
Host: localhost:8080
Response
HTTP/1.1 200 OK
Upload-Offset: 100
The server responds with the upload offset 100 indicating that the client has to start uploading again from the offset 100. Note that the response to a head request does not contain a message body. It only contains a header.
The client sends a PATCH request with this upload offset and request body containing the remaining 150 bytes
250(file size) - 100(upload offset) = 150 remaining bytes
Request
PATCH /files/12 HTTP/1.1
Host: localhost:8080
Content-Length: 150
Upload-Offset: 100
[Remaining 150 bytes]
Response
HTTP/1.1 204 No Content
Upload-Offset: 250
The server responds with a 204 status and Upload-Offset: 250 equal to Upload-Length indicating the file upload has been uploaded completely.
In case the request again fails in the middle during upload, the client should send a HEAD request followed by PATCH.
The gist is to keep calling HEAD to know the current Upload-Offset followed by PATCH until the server responds with a Upload-Offset equal to Upload-Length.
This brings us to the end of this tutorial. In the next tutorial, we will create the data model for the tus server. Have a good day.
Please leave your feedback and comments. Please consider sharing this tutorial on twitter or LinkedIn.
If you would like to advertise on this website, hire me, or if you have any other software development needs please email me at naveen[at]golangbot[dot]com.
Next tutorial - Implementing DB CRUD methods
