4 mins read

Comparing Image-to-Video tools: Runway’s Gen-2 vs. Stable Video Diffusion


AI generated Videos

Together with AI-generated images, progress has been made with AI video generations, a topic which is still emerging and not yet as popular as language or image generation. That’s probably due to the added complexity of video generation, considering that it requires a significant amount of resources for training and execution. Moreover, whereas with images you can sometimes get results that are almost perfect, with videos it’s easy to create artifacts or distortions, depending on the length of the video and the subjects.

However, initially, image generation was also very limited, producing low-quality results, so video generation will catch up. And with the right tools, it is already possible to generate interesting animations starting from an image. You can animate any image using different tools, including Runway Gen 2 and the most recent Stable Video Diffusion by Stability AI. In this article, I will show an example using Runway Gen 2 and Stable Video Diffusion, as well as some concepts on how camera control works in both.

Runway Gen 2

Runway AI (or RunwayML), one of the companies behind the creation of the original Stable Diffusion model, is focusing on image-to-video generation, providing a service that can be used to animate any image with discrete control over the movements. It also offers text-to-image, which essentially creates an image first and then animates it. Let’s try the Image to Video Gen 2.

Choose an initial image

ai generated image input of st peters

I created this image with Stable Diffusion, choosing a wide format to give it more of a cinematic feel. You can use images of any format in terms of height and width. After that, simply drag the image into the UI of RunwayML, from where it is possible to control the camera movements.

Use the motion brush

The motion brush is a powerful functionality which allows you to select specific areas of the input image and control their movements. In this example, I would like the fire and smoke to move more than the whole image, going up and closer.

motion brush for runway ml

After selecting the area (you don’t need to be precise), you can adjust the Horizontal, Vertical, and proximity movements of the camera: in this case, I want the smoke and fire to go up and come closer, so I set vertical movement to 1.0 and the Proximity to 1.1. The higher the number, the more pronounced the movement will be, but it will easily create artifacts or warping effects, so I prefer to keep it low, also to give a cinematic effect. Finally, you can save and adjust the last parameters.

Choose camera movements

It is also possible to adjust the camera movements for the whole picture. In this case, there are three more camera movements: pan, roll, and tilt.

camera motion for runway gen 2

I’d like my video to have a slow zoom-in effect, while slightly panning to the right, so I adjust the parameters accordingly. Everything is now set up; note that all of these parameters are not mandatory, you can also let Gen 2 do the animation freely.

The result is definitely nice. The pan effect looks a bit ‘flat’, like turning a 2D image, but overall it followed my instructions and produced a smooth animation.


Comparison with Stable Video Diffusion

Let’s see how Stable Video Diffusion performs, with the same initial image. I will be using ComfyUI and this simple workflow for SVD. The controls here are much more limited compared to Runway Gen 2, and definitely less predictable. The input picture is the main factor, together with the seed and motion_bucket_id, which, after some trials, might give the right panning. Here the results:

Original

With an optimized workflow to make each frame smoother during transitions.

svd optimized video

It took six minutes to generate this video on my Nvidia 3060 with 12 GB of VRAM, and I think I obtained nice results, considering the tools that I have available and the fact that Stable Video Diffusion is very new. However, Gen 2 gave me more control and a smoother final animation. All in all, considering that I have to pay for Runway Gen 2, I am definitely motivated to see improvements with SVD and try to get the most out of it.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.