Frame Interpolation

Frame Interpolation

Ever wished that you could refine time and space? Well here’s a tool that can get your footage there!

Frame Interpolation, is the process of synthesizing new frames from a given set of frames to create more fluid movement or to slow down the speed of a video.

Frame Rate

Frame Rates are essentially the number of frames that are displayed consecutively per a calculated length of time. The widely accepted identifier is “Frames Per Second” (FPS). The amount of frames that are shown per second plays a large role on how we perceive a video in not just the speed, but in creating a more comfortable experience or emoting the sense of a nostalgic era of film to name a few.

Common Frame Rates:

• 8mm: Generally shot at 16 FPS
• Super 8mm: Generally shot at 18 FPS
• 16mm (Sound): Generally shot at 24 FPS
• 16mm (Silent): Generally shot around 16-18 FPS
• 35mm (Sound): Generally shot at 24 FPS
• Cinematic: 24 FPS
• PAL: Generally shot at 25 FPS
• NTSC: Generally shot at 30 FPS or 29.97
• Video Blog: Generally shot at 30 FPS
• Cinematic Action Sequences: Generally shot around 60-120 FPS

Change Frame Rates:

Frame rates can be changed by selecting the caret and navigated down the selection of an existing frame rate.

Custom Frame Rates:

If you want to make modify the existing frame rates and make a custom one, you can simply type in a desired framerate.

• Click and Drag to highlight existing FPS
• Type New Value in it’s place
• Hit Enter to Save

Slow Motion

Post or Digital Slow Motion is the process of interpreting the original frames and duration of a video to algorithmically exceed the current length by a determined value, from the user, in order to produce a longer and slower video iteration of the original input. In other words, you can digitally slow down your original footage to create a result that was not recorded in reality.

Slow Motion options:

• Choose a slow motion factor, or to keep the speed the same, select None.
• Replace Duplicate Frames will replace duplicate frames with interpolated frames.
• Sensitivity can be raised to decrease the threshold for removing duplicate frames. A higher setting may remove frames that are not duplicates but look very similar.

Apollo

Specializes in non-linear motion and slightly blurry inputs.

• Generates 8 interpolated frames per run, and then creates the selected output framerate from this result.
• Faster than Chronos.
• Use when you need exactly 2x, 4x, or 8x frame count multiple. For example, converting from 30 → 60fps at 200% slow motion would require exactly 4x the number of frames.

Apollo worked wonderfully for this (slow motion 4x). This is a handheld, but nearly locked shot.

On the original footage, the water is rushing very quickly. Using this filter produced a wonderful result of the churning white water. The additional movement of the wind pushing through the leaves and plants resulted nicely, but the water is the true star of this shot.

Apollo Fast

Typically interpolates runs faster than Apollo, however, may produce a lesser quality output than Apollo when working with large amount of motion.

• Recommended for use for any non-double conversions. For example, 24fps -> 29.97 or 60.
• Use when processing speed is a factor.

*Both AI Models are using Slow Motion 4x .

Though the quality is slightly reduced compared to the standard Apollo, for a busy shot like this, the difference is not crucial to the intended effect of the filter.

It may be worth testing Apollo Fast when rendering long sequences. The result overall is still quite good, and could save significant amounts of rendering time.

Chronos

General frame interpolation model suitable for FPS conversion or slow-mo. Processing speed decreases linearly with the number of generated frames.

• If render speed is a factor for slow motion beyond 8x this is a good option if the Aion model is too time and system-intensive.
• Gives good results for 4x slow motion or less.

Being that Chronos is one of the app's earlier models, it is also one of the more general models that handles footage relatively well. For this shot, the boats are rocking in the water, there are reflections under the boats, and there are thin ropes to each mast. The results were pretty good even with something as drastic as slow motion 10x.

Chronos Fast

Specializes in fast motion with larger changes between frames.

Can process faster than Chronos, but can be very similar at times as well. Worth doing a small test for render time savings.

Using the same shot as Chronos, there are a few visual differences on how these work. The area of interest will be this small section of the water.

At a very close comparison the differences are quite noticeable.

In the “red” rectangular portion, Chronos Fast is treating the footage like scanlines or fields to quickly get the result. Chronos overall looks better in this area. However there are spots in the Chronos were it also has these very thin vertical scanlines. It’s worth noting that we are very zoomed into the footage to be able to see these differences.

Aion

High-precision interpolation for large media

• More accurate than Apollo and Chronos, but more system-intensive.
• Useful when working with large amounts of motion across the entire frame of an input video.

Here we have a simple panning shot to the left, handheld, but nearly locked off. There are several levels of parallax and Aion was able to handle this very well.

The original footage of the city moved abruptly and there were only a few moments of really stellar footage. With the Aion model, the few moments were expanded by 8 times. This is a very wide shot of New York City behind a glass wall. Even with the glass in front, the result was still great. All the tiny details were preserved.

Here we are looking at natively shot stereoscopic 180' footage for VR. Aion was able run the footage at 12 times and since it is paramount that both eyes are addressed in a similar way for reduced stereo artifacts, this model was able to provide the slowdown in equal quality in both eyes.

A Few Things to Consider

The slow motion tools and adjusting frame rates work very well in most cases. However, there are a few things to consider when filming and then using this tool to get the best possible results.

Remember, Frame Interpolation relies on motion estimation, so if there are drastic changes between frames Video AI will not “create” motion steps between those frames.

There are moments with some footage with water, in which there are high contrast/reflective surfaces with extreme differences from frame to frame, that will cause a strobing effect with bright and “clamped-like” colors when slowed down.

Caustics over heavily patterned moving subjects/objects can distort at times. Caustics over backgrounds and non moving subjects/objects are usually handled well. Essentially, cases where the patterned object is going in one direction while the other pattern is moving in the opposite direction could cause artifacting.

Slow Motion can handle parallax relatively well, however, in extreme cases of many thin layers of parallax on a turning curve, causes a minor wobble every once in a while to the parallaxed objects.

Refine Time

Whether you are just setting a new frame rate or slowing down some footage, these AI models can help you retime almost any footage. Have fun slowing down the world around you!