Resumable file uploader: Testing the server using curl and dd commands
Welcome to tutorial no. 4 in our Resumable file uploader series.
In the previous tutorials, we coded our resumable tus server using Go. In this tutorial we will use curl and dd commands to test the tus server.
Testing
We have the resumable file upload tus server ready but we have not tested it yet. We need a tus client to test the tus server. We will create the Go client in the upcoming tutorials. For now, we will use the curl command to test the tus server.
Let’s run the server first. Run the following commands in the terminal to fetch the code from github and then to run it.
go get github.com/golangbot/tusserver
go install github.com/golangbot/tusserver
tusserver
After running the above commands, the server will be up and running.
2019/03/30 18:01:41 Connection established successfully
2019/03/30 18:01:41 TUS Server started
2019/03/30 18:01:41 Directory created successfully
2019/03/30 18:01:41 table create successfully
We need a file to test the tus server. I have made a collage video of my pet and it is available at https://www.dropbox.com/s/evchz5hsuvtrvuu/mypet.mov?dl=0. Please feel free to use it :). I have downloaded the video to my ~/Downloads directory.
Let’s send a post request and create a new file. We need to specify the Upload-Length of the entire file in the post request. This is nothing but the size of the file. We can use the ls command to find the size of the file
ls -al ~/Downloads/mypet.mov
The above command returns the following output.
-rw-rw-r-- 1 naveen naveen 11743398 Mar 31 11:11 /home/naveen/Downloads/mypet.mov
11743398 is the size of the file. Now that we know the Upload-Length, let’s create the file by sending a post request.
curl --request POST localhost:8080/files --header "Upload-Length: 11743398" -i
The above command creates the file. The -i argument in the end is used to display the response headers. The above command will return the following result.
HTTP/1.1 201 Created
Location: localhost:8080/files/1
Date: Sun, 31 Mar 2019 07:47:33 GMT
Content-Length: 0
The file creation has been created successfully.
Now comes the tricky part. How do we test the tus server by simulating a network disconnection? If we send a patch request to the file URL using curl, the request will be completed immediately since the server is running locally and we will not be able to test whether the server is able to handle resumable uploads.
This is where the --limit-rate argument of curl helps us. This argument can be used to rate limit the patch file request.
curl --request PATCH --data-binary "@/home/naveen/Downloads/mypet.mov" localhost:8080/files/1 --header "Upload-Offset: 0" --header "Expect:" -i --limit-rate 200K
In the curl request above, we are sending a patch request to the file at location localhost:8080/files/1 and Upload-Offset: 0 and we are rate-limiting the request to 200KB/Sec. The contents of mypet.mov is added to the request body. The --header "Expect:" header is needed to prevent curl from sending Expect: 100-continue header. Please read https://gms.tf/when-curl-sends-100-continue.html to know why this is needed.
After issuing the above patch request, the file will be transferred at 200KB/S. Let the request run for a few seconds, say 10 seconds. After approximately 10 seconds, please stop the request by pressing ctrl + c. Now we have terminated the patch request in the middle. The server should have stored the bytes transferred till now. Let’s check whether it has done it.
Move to the server logs and you will be able to see the following in the log,
2019/03/31 13:36:00 Received file partially unexpected EOF
2019/03/31 13:36:00 Size of received file 1589248
2019/03/31 13:36:00 number of bytes written 1589248
The size of the received file may be different for you. The above is my output.
hmm looks like it has saved the bytes received till now. But how do we verify it? Well, let’s check the size of the uploaded file.
ls -al ~/fileserver/1
Running the above command outputs
-rw-r--r-- 1 naveen naveen 1589248 Mar 31 13:36 /home/naveen/fileserver/1
The size of the file matches the server output. Now we can be 100% sure that the server has saved the bytes it has received. If you try to play the video now, it won’t play since the file is still not completely uploaded yet.
The next step is to continue the patch request from where it stopped. We first needed to know the Upload-Offset so that we can issue the next patch request. This is where the head request comes in handy.
curl --head localhost:8080/files/1 -i
The above curl command will return the Upload-Offset
HTTP/1.1 200 OK
Upload-Offset: 1589248
Date: Sun, 31 Mar 2019 08:17:28 GMT
Note that the offset matches the server logs and the file size.
Now we need to send a PATCH request with the above upload offset. One more concern is we need to send the file data(bytes of the file) from this offset only, not the entire file.
This is where the dd command helps us.
dd if=/home/naveen/Downloads/mypet.mov skip=1589248 bs=1 | curl --request PATCH --data-binary @- localhost:8080/files/1 --header "Upload-Offset: 1589248" --header "Expect:" -i
In the above command, we use if to specify the input file and skip is used to skip 1589248 bytes. 1589248 is our Upload-Offset. bs specifies that we read one byte at a time. We pipe the output of dd to curl command. After running the above command, we will get the output
HTTP/1.1 204 No Content
Upload-Offset: 11743398
Date: Sun, 31 Mar 2019 08:25:10 GMT
A 204 No Content indicates that the patch was successful. To know whether the file upload is complete, we can again issue a head request and the upload offset should match the upload length(size) of the file.
curl --head localhost:8080/files/1 -i
The above command will output
HTTP/1.1 200 OK
Upload-Offset: 11743398
Date: Sun, 31 Mar 2019 08:30:54 GMT
The upload offset matches the upload length and we are sure that the file has been uploaded completely. Now if you try to issue a patch request again, the server will complain saying that the upload is already complete.
Let’s again check the file size now.
ls -al ~/fileserver/1
Running the above command outputs
-rw-r--r-- 1 naveen naveen 11743398 Mar 31 13:55 /home/naveen/fileserver/1
The file size in the output matches the upload length and this confirms that the file has been uploaded completely. You can go ahead and play the video and it will play now :)
Our resumable tus server is ready :)
Enhancements
Although the file uploader works, this code needs to refactored further. It currently doesn’t handle concurrency. For instance, we might hit a race condition when multiple clients send a concurrent patch request for the same file and at the same offset.
This code also doesn’t handle DB transactions well. There is a chance that a POST request to create a file might end up creating the file in the DB but not creating the actual file in the file system. For example, what happens when there is no space left in the file system.
All the code is currently present in the main package in a single file and this approach is not extensible. The code has to be refactored into usable packages. One approach to structure code in Go is using Domain Driven Design.
We also don’t have a tus client ready yet :).
All these will be addressed in the upcoming tutorials. I hope you enjoyed reading. Please leave your comments. Please consider sharing this tutorial on twitter and LinkedIn. Have a good day.
If you would like to advertise on this website, hire me, or if you have any other software development needs please email me at naveen[at]golangbot[dot]com.
