I participated in a poster session at the AGU 2013 conference in San Francisco. I presented on a fog monitoring study I’ve been conducting in collaboration with the San Francisco Public Utilities Commission, and as part of the study we recorded 5-minute time-lapse imagery in conjunction with some soil moisture and leaf wetness sensor data. I’d seen some poster presentations bring laptops to show videos at AGU the previous year, and I decided to do the same. My goal was to create a video from the timelapse imagery that included a ‘moving plot’ of my sensor data that was synchronized with the time-lapse images. In an earlier post, I showed how to create a series of plots using ggplot2 which could be used as frames for a moving plot video. In this post, I’ll go over how to work with videos in Matlab and merge the plot images with each video frame.

The decision to work with Matlab was pragmatic. While it would have been better in the long run to use Python’s openCV module, I was already somewhat familiar with Matlab’s video processing capabilities. I also found this absolutely fantastic tutorial which shows you how to read and write videos from images step by step, with some extra tricks for working with windows directories. This meant that I could create my process in a matter of hours, rather than days if I were teaching myself openCV from scratch.

The first step is to trim the timelapse video to the range of the data. In my case, this was relatively easy as my sensors are operating at the same interval as time-lapse imagery (5 minutes). Therefore, as long as there are no breaks in the sensor data or timelapse imagery I can simply pick a window of time when both the sensors and the images are recording. The times of the video and the sensor readings probably won’t line up exactly (e.g. images start at 11:05 but the sensor starts at 11:07), but that’s fine for my needs. The first step is to trim the video down to the appropriate window. In my case, the timelapse video has a timestamp on the bottom. So I’ll simply extract all the timestamps to images, and name them by their frame position:

This creates a folder with the timestamps as images, named with their frame index number. I then just have to find the start and end frames corresponding to my time frame and note their index numbers.

The next step is to load the plot images. Assuming they are numbered sequentially, you can use regular expressions to load them all:

sortednames is a cell array containing the filenames or paths of the plot images, in ascending order. Next, we create the output video file:

I should say here that profiles in Matlab are pretty important. you can type VideoWriter.getProfiles to see a list of available profiles on your machine. Making sure a video can play outside of Matlab means you need to be careful about what video profile you choose and what resolution your output frames are. In my case, the cameras I was using record in 720p, so I simply expanded to 1080p and created plot images with the correct resolution to simply stick on the bottom of each timelapse frame. This also means that I can output in .mp4 format, which keeps the filesize manageable. To write the video, I simply loop through each frame of the video in my specified range, and merge it with the plot image. Since images in Matlab are loaded as a NxMx3 matrix (or NxMx4 if the image supports transparency), I can simply use vertcat() to concatenate the image matrices and create the new frame:

And you can end up with something like this:

The procedure I outlined here isn’t exactly what I did; I ended up creating a number of general functions for manipulating videos in Matlab. You can check them out here.