Upload video from Amazon EC2 to Youtube – it’s fast

Following on from this post from a few weeks ago, here is some further work on improving my youtube upload workflow.

I tend to work in batches of files – where there are a number of videos from a day long event. I’m aiming for a process that requires as little supervision as possible. So here is the point I’ve now reached.

The setup described below allows me to copy a bunch of files to my S3 bucket (I have the Cyberduck client because I like the name). I drop them in a folder in my bucket called “youtube-upload” and then run the following script on my Amazon EC2 instance.

This code is also on Github if you would like to improve it.

This moves the files from S3 bucket to my EC2 instance, into a folder called “video”.

aws s3 sync s3://mybucket/youtube-upload ~/video

To run this, you’ll need to install the aws-cli library which you can find out about here.

I can then run a script to upload this batch of videos to youtube (youtube-library library here) – changing the variables to match the event that the videos are from. I make a new copy of this script for each Youtube account I’m working on.

Here is a sample script I used to upload files from the recent Drupalgov conference in Canberra. Once each file is uploaded, the video file is moved to the Uploaded-videos so that its not included in future runs of the script, but still on the server in case something goes wrong.

for f in $FILES
  echo "Processing file " ${f##*/}
# take action on each file. $f stores current file name
  youtube-upload --email=$YOUTUBEEMAIL --password=$YOUTUBEPASS  "$f" --unlisted --title="${f##*/}" --category=Tech --description="Drupalgov, Canberra August 22, 2014 at the National Museum of Australia" --keywords="Drupal, Open Source, Conference, Drupalgov, Australia, Web, CMS, Content Management System"
  mv "$f" ~/Uploaded-videos

I’ve found the app specific passwords work really well with this script and they mean I can have safer access to accounts that are owned by other people.

I’ve found the speed of this process is much better than the typical upload to Youtube via my web browser. Sure – if you are only doing occasional uploads its fine, but I’m often uploading a few hours of video a week.

Here’s the read out from the terminal:

gavin@myserver:~$ ./upload-drupalgov 
Processing /home/gavin/video/3-Kim-Pepper.mov file... 3-Kim-Pepper.mov
Login to Youtube API: email='drupalemail@gmail.com', password='****************'
Start upload using a HTTP post: /home/gavin/video/3-Kim-Pepper.mov -> https://uploads.gdata.youtube.com/action/FormDataUpload/AIwbFAR0rRKDIibHvoOIpfte_46PoaP7f4-cadeLg
100% |#####################################| Time: 00:09:23   2.86 M/s

As you can see, the script took 9:23 to upload a 1.6Gig video file from my EC2 instance to YouTube. I think the upload to S3 using Cyberduck was about 15 minutes – so about 25 minutes all up. In the next few days, I’ll upload the same file via my browser to see how long it takes and compare.

Not to be sneezed at is the bonus of setting the meta data for batch of files at once. This saves me a few minutes per video if I’m doing things the old fashioned way. Though I’m not sure yet how to set advanced settings such as Creative Commons, location, recording date etc.

At some point, it would be good to take a look at why the browser based upload is slower. I suspect its due to both the method the browser uses to move the file and the network path between me and youtube. The files transfer method I use to move the file to S3 is probably more efficient.

Videoconferencing with Zoom

Disclaimer: AARNET were kind enough to provide a free Zoom account for the Govhack organisers to use this year – which is I think is worth about $10 per month. Apparently the free version works just as well but has a 40 minute time limit. This is a brief write up of my experience using it.

One of the jobs I have for Govhack is to wrangle the technology that we use to coordinate the communication between the various organisers – mainly the video conferencing tools.

In the past we have made a lot of use of Google Hangouts to do this. We liked the way it integrated with Google Calendar, it was mostly easy to use and it is free.

For about three months leading up to Govhack, we have a weekly meeting with some people meeting in at the Link Digital Office in Canberra, and the other people joining via video conference.

This year, we added another layer to the organisation with 11 sites running local Govhacks as part of the weekend of hacking. So each week after the national organiser meeting, we held a meeting for the organisers from each site.

We pretty quickly hit the 10 person limit on the free version of Google Hangouts – so it was opportune that I caught up with a contact at AARNET who suggested I look at Zoom – a new video conference system they were trialling, and finding to be quite good.

I ran a couple of small test calls with people and found the quality to be really good. There are a couple of key differences from Google Hangouts.

  • up to 25 people in the meeting
  • no need for an account (unless you are organising the meeting)
  • uses a native client, not browser based
  • option to dial in via phone
  • no masks, funny effects or other extensions
  • no option to broadcast (like hangouts on air)

We decided to give it a test with the national team, and pretty quickly determined that we’d replace Google Hangouts. The most striking improvement was with audio. Much clearer, and fewer problems with echo and echo cancellation. We also have a range of connection speeds between the people involved in these meetings. Zoom seemed to manage this a lot better – and the quality for people with good connections was excellent. Exceeding what we normally saw with Hangouts. For people with poor connections, the option to dial in via phone was really useful. If you can reliably access the audio of the meeting, you can still effectively participate in the meeting.

The AARNET instance of Zoom is running on hardware in their network – which might explain a good deal of the improvement. Improved latency and better contention rates. Its hard for me to say without going into a lot of testing that is not really my domain. I can say, that the experience was better all round.

Outside the quality of the calls – the experience of setting up meetings is also easy. There is a tradeoff in using Google Hangouts in the way it requires a Google Account for each person. Some people don’t have these, and if they aren’t tech savvy, it can be an obstacle that takes time to deal with.

When you create a Zoom meeting, either in the software client or website, it generates text that you can email to your meeting participants with simple instructions on how to join the meeting and a link to click. The text also includes the phone details to dial in. Pretty simple to use I think.

I created a weekly recurring meeting and shared the details with my team. All they had to do at the agreed time was click the link on their computer/device to trigger the client and then join the meeting.

As the meeting host, I have the option to mute participants and a few other controls. Its also possible to record the video of the meeting for later use (though we didn’t use this).

Govhack Organiser meeting in Zoom

There are a few display options that you can choose during the meeting. There is a mode similar to Google Hangouts where the person speaking automatically fills the bulk of the screen. I preferred to use the gallery style view where all the people in the meeting are visible (as shown in the image above). This allows me to pick up on visual cues from everyone in the meeting – much as you might if you were in the same room.

I think there is still a place for other video chat / conference systems that I have used recently (Hangouts, Skype, Appear.in, Jabber) but is great to have another tool to choose. I think its going to be my first choice for organising groups and committees.