Upload video from Amazon EC2 to Youtube – it’s fast

Following on from this post from a few weeks ago, here is some further work on improving my youtube upload workflow.

I tend to work in batches of files – where there are a number of videos from a day long event. I’m aiming for a process that requires as little supervision as possible. So here is the point I’ve now reached.

The setup described below allows me to copy a bunch of files to my S3 bucket (I have the Cyberduck client because I like the name). I drop them in a folder in my bucket called “youtube-upload” and then run the following script on my Amazon EC2 instance.

This code is also on Github if you would like to improve it.

This moves the files from S3 bucket to my EC2 instance, into a folder called “video”.

aws s3 sync s3://mybucket/youtube-upload ~/video

To run this, you’ll need to install the aws-cli library which you can find out about here.

I can then run a script to upload this batch of videos to youtube (youtube-library library here) – changing the variables to match the event that the videos are from. I make a new copy of this script for each Youtube account I’m working on.

Here is a sample script I used to upload files from the recent Drupalgov conference in Canberra. Once each file is uploaded, the video file is moved to the Uploaded-videos so that its not included in future runs of the script, but still on the server in case something goes wrong.

for f in $FILES
  echo "Processing file " ${f##*/}
# take action on each file. $f stores current file name
  youtube-upload --email=$YOUTUBEEMAIL --password=$YOUTUBEPASS  "$f" --unlisted --title="${f##*/}" --category=Tech --description="Drupalgov, Canberra August 22, 2014 at the National Museum of Australia" --keywords="Drupal, Open Source, Conference, Drupalgov, Australia, Web, CMS, Content Management System"
  mv "$f" ~/Uploaded-videos

I’ve found the app specific passwords work really well with this script and they mean I can have safer access to accounts that are owned by other people.

I’ve found the speed of this process is much better than the typical upload to Youtube via my web browser. Sure – if you are only doing occasional uploads its fine, but I’m often uploading a few hours of video a week.

Here’s the read out from the terminal:

gavin@myserver:~$ ./upload-drupalgov
Processing /home/gavin/video/3-Kim-Pepper.mov file... 3-Kim-Pepper.mov
Login to Youtube API: email='drupalemail@gmail.com', password='****************'
Start upload using a HTTP post: /home/gavin/video/3-Kim-Pepper.mov -> https://uploads.gdata.youtube.com/action/FormDataUpload/AIwbFAR0rRKDIibHvoOIpfte_46PoaP7f4-cadeLg
100% |#####################################| Time: 00:09:23   2.86 M/s
.. http://www.youtube.com/watch?v=10NU3bhlt1I&

As you can see, the script took 9:23 to upload a 1.6Gig video file from my EC2 instance to YouTube. I think the upload to S3 using Cyberduck was about 15 minutes – so about 25 minutes all up. In the next few days, I’ll upload the same file via my browser to see how long it takes and compare.

Not to be sneezed at is the bonus of setting the meta data for batch of files at once. This saves me a few minutes per video if I’m doing things the old fashioned way. Though I’m not sure yet how to set advanced settings such as Creative Commons, location, recording date etc.

At some point, it would be good to take a look at why the browser based upload is slower. I suspect its due to both the method the browser uses to move the file and the network path between me and youtube. The files transfer method I use to move the file to S3 is probably more efficient.

Videoconferencing with Zoom

Disclaimer: AARNET were kind enough to provide a free Zoom account for the Govhack organisers to use this year – which is I think is worth about $10 per month. Apparently the free version works just as well but has a 40 minute time limit. This is a brief write up of my experience using it.

One of the jobs I have for Govhack is to wrangle the technology that we use to coordinate the communication between the various organisers – mainly the video conferencing tools.

In the past we have made a lot of use of Google Hangouts to do this. We liked the way it integrated with Google Calendar, it was mostly easy to use and it is free.

For about three months leading up to Govhack, we have a weekly meeting with some people meeting in at the Link Digital Office in Canberra, and the other people joining via video conference.

This year, we added another layer to the organisation with 11 sites running local Govhacks as part of the weekend of hacking. So each week after the national organiser meeting, we held a meeting for the organisers from each site.

We pretty quickly hit the 10 person limit on the free version of Google Hangouts – so it was opportune that I caught up with a contact at AARNET who suggested I look at Zoom – a new video conference system they were trialling, and finding to be quite good.

I ran a couple of small test calls with people and found the quality to be really good. There are a couple of key differences from Google Hangouts.

  • up to 25 people in the meeting
  • no need for an account (unless you are organising the meeting)
  • uses a native client, not browser based
  • option to dial in via phone
  • no masks, funny effects or other extensions
  • no option to broadcast (like hangouts on air)

We decided to give it a test with the national team, and pretty quickly determined that we’d replace Google Hangouts. The most striking improvement was with audio. Much clearer, and fewer problems with echo and echo cancellation. We also have a range of connection speeds between the people involved in these meetings. Zoom seemed to manage this a lot better – and the quality for people with good connections was excellent. Exceeding what we normally saw with Hangouts. For people with poor connections, the option to dial in via phone was really useful. If you can reliably access the audio of the meeting, you can still effectively participate in the meeting.

The AARNET instance of Zoom is running on hardware in their network – which might explain a good deal of the improvement. Improved latency and better contention rates. Its hard for me to say without going into a lot of testing that is not really my domain. I can say, that the experience was better all round.

Outside the quality of the calls – the experience of setting up meetings is also easy. There is a tradeoff in using Google Hangouts in the way it requires a Google Account for each person. Some people don’t have these, and if they aren’t tech savvy, it can be an obstacle that takes time to deal with.

When you create a Zoom meeting, either in the software client or website, it generates text that you can email to your meeting participants with simple instructions on how to join the meeting and a link to click. The text also includes the phone details to dial in. Pretty simple to use I think.

I created a weekly recurring meeting and shared the details with my team. All they had to do at the agreed time was click the link on their computer/device to trigger the client and then join the meeting.

As the meeting host, I have the option to mute participants and a few other controls. Its also possible to record the video of the meeting for later use (though we didn’t use this).

Govhack Organiser meeting in Zoom

There are a few display options that you can choose during the meeting. There is a mode similar to Google Hangouts where the person speaking automatically fills the bulk of the screen. I preferred to use the gallery style view where all the people in the meeting are visible (as shown in the image above). This allows me to pick up on visual cues from everyone in the meeting – much as you might if you were in the same room.

I think there is still a place for other video chat / conference systems that I have used recently (Hangouts, Skype, Appear.in, Jabber) but is great to have another tool to choose. I think its going to be my first choice for organising groups and committees.

Make a multi-camera video using Quicktime

Did you know you can make a multi-camera video using Quicktime as one of the cameras?

Tomorrow I’ll be filming some presentations to a community group. There will be a few speakers – but the format will generally be someone speaking from a lectern to a seated audience with some slides projected on a screen at the front of the room.

Pretty standard stuff really.

The same technique can be used to make a demo or pitch video, like the ones we asked teams to make for Govhack this year.

There are lots of ways to film this sort of thing – but I tend to try to do it with as small an amount of equipment to carry as I can manage.

I’ve got a couple of tricks that can improve your finished video, and make shooting and editing little easier.
Sketch of room and equipment layout

Let’s assume you have one camera (we’ll get to multi camera in a bit). Sit the camera at the back of the room so that you capture the speaker across the heads of the audience. This will have your speaker facing the camera most of the time. Frame the shot so that you have the head and shoulders of the person in the shot most of the time – a fairly tight shot. Don’t worry about getting the slides in the shot.


If the laptop that is running the projector is a Mac, open Quicktime and do File -> New Screen Recording.
Make a multi-camera video using QuicktimeClick the small white down arrow, and make sure ‘Built in mic’ is selected at the audio source.Screen Shot 2014-07-23 at 8.56.54 pm

Hit the record button, and set it to record the whole screen (whichever screen the slides will be displayed on). You are now recording the slides, just as the are presented – including transitions, animations video playback – the whole thing.

If you are using a Windows PC, try CamStudio which is free software. I’m sure there are good alternatives for *nix systems (post a comment if you know of one).

Record the presentation. At the end, stop the screen recording on the laptop by clicking on the small ‘stop’ button in the bar at the top of the screen.

Save the screen recording. This will leave you with a Quicktime file that will easily drop into iMovie or Final Cut.

Using a second camera

If you have a 2nd camera I’d suggest you set it capture a shot of the speaker and projector screen together – so you can use footage of the speaker pointing and interacting with the screen if needed. The other useful spot can be near the front of the room – possible over the presenters shoulder in a fairly wide shot. This will give you a view of the room – which can provide a context for the viewers of your video. It may also provide an angle you can use if people ask questions. Either way – you’ll probably want to use this second camera angle sparingly.


When you are ready to edit the video of the presentation – you can take the screen recording and treat it like camera footage. If you are using Final Cut, import the camera footage and the screencast, select these clips and then click File -> New -> Multicam clip. The standard options should work ok.

Once its done you’ll have each cameras footage and the screen recording synched via the audio. Do Shift+Command+7 in Final Cut to open the Angle Viewer. Before you start playback, make sure you have the angle with the best audio set to provide the audio track – then use the ‘video only switching’ tool to choose which angle you want to show at any point in the video.

If you need to sync the camera footage with the Quicktime file manually, roughly line the files up on your timeline and then use the audio waveform to get them exactly in sync. If the camera and Quicktime file were recorded at different frame rates, they can drift out of sync over time. This is a pain, and can be time consuming to fix. You may need to split one of the videos into a number of parts and sync each one. Use the one you are showing less of. The video will still be out of sync much of the time, but you can use the splits to adjust it back in sync once it gets really bad.

Good audio is important – if you’re new to this, go and look for some tutorials on recording audio. You can record an audio track to a dedicated recorder like a Zoom H4N and include it in your multitrack, and set it as the audio track for your video.

Multi camera video editing is easier after you’ve seen some demos. Search on Youtube for examples.