Resumable file uploader: Understanding tus protocol
Welcome to tutorial no. 1 in our Resumable file uploader series.
How many times have you tried to upload a large file only to know that it failed because of a network issue! When you re-upload the file again, the upload starts from the beginning :(. Not cool at all. This is where resumable file uploaders come in handy.
Resumable file uploaders allow the file upload to start right from the point where it stopped instead of uploading the whole file again.
In this tutorial series we will learn how to create a resumable file upload server and client in Go using the tus protocol. This tutorial is not an exact implementation of the tus protocol, but rather a simplified version. This tutorial is self-sufficient to create a resumable file uploader using Go. We will keep improving this uploader in the upcoming tutorials and make it full tus compatible.
This tutorial has the following sections
- Tus protocol
- POST request to create the file
- PATCH request to update the file
- HEAD request to get the current file offset
The tus protocol is quite simple and the best selling point of tus is that it works on top of HTTP. Let's first understand how tus protocol works.
Tus protocol needs three http methods namely POST, PATCH and HEAD. It's best to understand the tus protocol using an example.
Let's take the example of uploading a file of size 250 bytes. The upcoming sections explain the sequence of http calls required to upload a file using tus protocol.
POST request to create the file
This is the first step. The client sends a
POST request with the file's upload length(size) to the server. The server creates a new file and responds with the file's location.
POST /files HTTP/1.1 Host: localhost:8080 Content-Length: 0 Upload-Length: 250
In the above request, we send a POST request to the URL
localhost:8080/files to create a file with
Upload-length 250 bytes. The
Upload-length represents the size of the entire file. Since the request does not have a message body, the
Content-Length field is zero.
The server creates the file and returns the following response.
HTTP/1.1 201 Created Location: localhost:8080/files/12
Location header provides the location of the created file. In our case, it is
PATCH request to update the file
Patch request is used to write bytes to the file at offset
Upload-Offset. Each patch request should contain a
Upload-Offset field indicating the current offset of the file data being uploaded.
In our case, since we just created a new file and starting to upload data to the file, the client sends a
PATCH request with
Upload-Offset as 0. Please note that file offsets are zero based. The first byte of the file is at offset 0.
PATCH /files/12 HTTP/1.1 Host: localhost:8080 Content-Length: 250 Upload-Offset: 0 [250 bytes of the file]
In the above request, the
Content-Length field is 250 since we are uploading a file of size 250 bytes. The
Upload-Offset is 0 indicating that the server should write the contents of the request at byte 1 of the file.
The server will respond with a
204 No Content header indicating the request is successful. Response to the
PATCH request should contain the
Upload-Offset field indicating the next byte to be uploaded. In this case, the
Upload-Offset field will be
250 indicating that the server has received the entire file and the upload is complete.
HTTP/1.1 204 No Content Upload-Offset: 250
The above response from the server indicates that the upload has completed successfully since the
Upload-Offset is equal to the
HEAD request to get the current file offset
The patch request above was completed successfully without any network problems and the file was uploaded completely.
What if there was a network issue while the file was being uploaded and the upload failed in the middle. The client should not upload the entire file again but rather start uploading the file from the failed byte. This is where the HEAD request helps.
Let's say the file upload request disconnected after uploading
100 bytes. The client needs to send a
HEAD request to the server to get the current
Upload-Offset of the file to know how many bytes have been uploaded and how much is still left to be uploaded.
HEAD /files/12 HTTP/1.1 Host: localhost:8080
HTTP/1.1 200 OK Upload-Offset: 100
The server responds with the upload offset 100 indicating that the client has to start uploading again from the offset 100. Note that the response to a head request does not contain a message body. It only contains a header.
The client sends a PATCH request with this upload offset and request body containing the remaining 150 bytes
250(file size) - 100(upload offset) = 150 remaining bytes
PATCH /files/12 HTTP/1.1 Host: localhost:8080 Content-Length: 150 Upload-Offset: 100 [Remaining 150 bytes]
HTTP/1.1 204 No Content Upload-Offset: 250
The server responds with a
204 status and
Upload-Offset: 250 equal to
Upload-Length indicating the file upload has been uploaded completely.
In case the request again fails in the middle during upload, the client should send a
HEAD request followed by
The gist is to keep calling
HEAD to know the current
Upload-Offset followed by
PATCH until the server responds with a
Upload-Offset equal to
This brings us to the end of this tutorial. In the next tutorial, we will create the data model for the tus server. Have a good day.
Next tutorial - Implementing DB CRUD methods
Like my tutorials? Please show your support by donating. Your donations will help me create more awesome tutorials.