Video,Where Is Your Metadata?

　　Metadata can change video storytelling, through automated shot selection, image enhancement and presentation in a bigger way than EXIF did for digital photos.

　　Every photo you take, on any camera, is likely filled with metadata. This data often contains location, camera model, lens type and general information about how the image was taken. It is this data that makes your photo library easier to browse, automatically groups events, and auto-corrects exposures. All of this is possible because the metadata for photos is standardized and free.

　　Video, where is your metadata? While photo metadata is widespread using the EXIF standard, there is no equivalent for video. Yet video has the same needs, if not greater, than still images.

　　Metadata should answer: how was this file made? In what environment? At what location? If, when and how was the camera moving? As video needs to support temporally changing metadata, it can’t use EXIF.

　　At GoPro, we’ve had the ambition to solve these questions, but it has turned out to be treading new ground. The popular consumer video file format MP4 had no clear provision to store telemetry or any time varying metadata.

　　Yet we must store telemetry within the video, not in a separate file, so consumers can’t misplace side-car files, or be expected to upload them separately. Metadata can’t be lost because the camera didn’t correctly close a file (through sudden power loss, camera drop, etc.), so the metadata must be regularly interleaved with video and audio, not simply added to the end upon a file close.

　　In addition, it must store all of today’s and tomorrow’s wide range of sensor data, and be easily retrieved in more than GoPro’s own utilities, without technical or legal hurdles. We’ve needed a new standard, so GoPro’s created one, and made it completely free (open source linked below.)

　　The first release of this new MP4 embedded metadata came with the HERO5. The HERO5 has extensive metadata for photos and video, adding sensors for accelerometer, gyroscope and high frequency GPS location.

　　Running the GPS 18 times faster than your average smartphone and combining that with IMU data, allows every bike jump, wave drop-in, race turn, and top-speed reached to be accurately measured and displayed. You can display a lot of this today using GoPro's free Quik for Desktop app.

　　However, a rendered display of your best lap or highest jump is only scratching the surface for the use of telemetry and the broader category of metadata. The HERO5 stores much more than location and camera motion in its video files; it stores its own temperature, image exposure, shutter speed, the very precise GPS time, detailed flight telemetry from the Karma drone, and data from GoPro-enabled third-party Bluetooth devices. The amount and range of metadata will continue to expand over GoPro’s future.

　　General Purpose Metadata Framework (GPMF) is a new track within an MP4 file, working just like video and audio which have their own tracks. The timing relationship between metadata collection and the video and audio captured is stored using existing MP4 mechanisms, so if a developer can handle video and audio within an MP4, metadata is very similar.

　　GPMF handles the timing issues resulting from independently clocked sensor data streams, allowing low and high frequency sources to be interleaved within a single payload. The multiplexing of a range of sensor data streams within a single MP4 track is done in a self-describing way.

　　Sensor precision and complex data structures are described within the data stream. The self-describing nature is critical because it allows future sensors to be added without updating reader applications or registering the sensor's data format with a GPMF managing entity.

　　The big plus is that this is all completely free for anyone to use. A GPMF-parser was recently made available as an open source project for developers to easily include video metadata within their application.

　　GoPro's release of GPMF enables other camera systems, drones and smartphones, to easily collect, process and archive sensor data, as an enhancement to consumer-friendly video capture. Metadata can change video storytelling, through automated shot selection, image enhancement and presentation in a bigger way than EXIF did for digital photos.