While AI has grabbed the world’s attention in recent months with its rapid public advancements, VFX artists have been quietly using machine learning tools to transform workflows for years. It makes sense since VFX has always been a hub for technological innovation! Now, with the rise of generative AI (GenAI), the industry is poised for yet another transformative leap akin to the idea of using CG Dinosaurs taking the screen for the first time in the movie Jurassic Park instead of traditional stop motion. At Groove Jones, we’ve been riding this exponential curve of AI development, using these advancements to enhance our studio offerings and tackling long-standing challenges while staying true to the artistry at the core of VFX. In short, we’re using AI to help bring those WOW moments like we’ve always done faster and more efficiently!

These new tools, collectively called GenVFX, combine the power of Generative AI (GenAI) and Machine Learning (ML) with tried and true battle-tested VFX workflows, offering complementary yet distinctively new capabilities. GenAI excels in rapidly creating new visuals, while ML provides the backbone for automating repetitive tasks and enhancing production efficiency essential to a VFX studio. Together, they are poised to drive innovation across the entire VFX industry and redefine what’s possible in visual storytelling.

Understanding the AI Creative Spectrum: GenAI vs. Machine Learning

GenVFX lives in a continuum of capabilities within the “AI Creative Spectrum” where Generative AI (GenAI) and Machine Learning (ML) complement each other. Together, they create a cohesive new ecosystem that transforms how VFX is produced and delivered.

Generative AI (GenAI): This class of tools generates new content, such as images, animations, or video. GenAI operates like a creative partner, offering:

• Generating Stock Plates: AI can create ideal green or blue screen elements that are ready for traditional keying and compositing. These kinds of element plates are familiar to VFX artists. But we can go even further with GenVFX! For example, on one of our recent projects, we used “generative extras” animated with prompt-based idle animations in our VR experiences to populate a scene in the Apple Vision Pro and even included hyper-realistic 3D profile pictures with subtle facial expressions driven by filmed footage to connect with the user on a deeper level.
• Hybrid Background Plate Creation: GenAI enables the creation of generative background plates like Matte Paintings or new scene elements not present in shot footage, that integrate seamlessly with simple Compositing workflows.
• Neural Diffused Render Engines (NDREs): Simple depth maps or gray box renders can now drive final shots with exceptional control and iteration capabilities. We’ve been applying open-source AI methods in our stills and look development workflows for a while now and are actively expanding these processes to include video and rendered elements. [Insert image here: Grayscale depth map evolving into a photorealistic render.] This emerging capability has the potential to redefine how CG elements are rendered and integrated.
• Style to Video Transformations: With Style to Video, we can completely redefine every visual aspect of a video, creating entirely new scenes and expanding creative possibilities. From altering environments to transforming characters, this cutting-edge capability unlocks a new level of post-production freedom. [Insert image here: Example of a lush forest transformed into a snowy landscape using Style to Video transformations.]
• Relighting: AI-driven relighting enables precise adjustments to the lighting of source footage while preserving all other visual details. This includes seamless changes in the time of day, enhancing creative flexibility, and reducing the need for reshoots. [Insert image here: Example of a daytime scene transformed into nighttime using relighting techniques.] Whether altering a scene’s mood or adapting it to a new narrative context, the potential of relighting is game-changing.
• Machine Learning (ML): ML tools automate and enhance repetitive tasks in VFX production. Unlike GenAI, which generates content, ML refines and processes existing material to aid in new tasks.

At Groove Jones, we use ML for:

• Denoising: Eliminating unwanted noise from images and renders, improving clarity, and saving time. [Insert image here: Before-and-after stills of a grainy image processed through ML-driven denoising.] For example, ML-driven denoising quickly removed grain from low-light render sequences in a recent CG project, delivering a polished final product while reducing manual effort.
• Optical Flow Object Tracking: Motion tracking for complex 3D objects used to take hours, if not days, but utilizing these new ML models allows us to enhance VFX workflows involved in beauty work and precision edits to footage.
• Retiming: Producing smooth motion with minimal artifacts, such as creating slow-motion effects in post, was once an unpredictable hurdle for post-production and ran the risk of costly reshoots. But by utilizing ML-driven retiming, we can now deliver high-quality results every time!
• UpRes: Whether it’s for restoring low-resolution archival footage or simply to save on render time for CG projects, the ML ability to resize images while preserving critical features allows us to easily take 1k images and process them into 4k assets. We’ve even explored automatically bringing historical footage to life with rich color using machine learning models and even animation.
• Streamlining Rotoscoping and Tracking: Tedious tasks like rotoscoping are now automated with tools like SAM2 from Facebook, an advanced segmentation model that can isolate subjects into masks.

How Groove Jones Uses GenVFX in Production

We are using GenFX to do some amazing things that years ago would have taken a large team of artists and engineers years to accomplish. For example, In “Forrest Gump” (1994), Robert Zemeckis and the VFX team at ILM took over six months to create a 6-second shot of Tom Hanks interacting with JFK in the film. Ironically, the Academy Award-winning team (Best Visual Effects) made a huge mistake in this shot, resulting in the President’s tie changing colors between takes in the same scene.

Recently, Groove Jones has been using GenVFX to do similar tasks, which include audio synthesis and lip-syncing to existing video material. However, we are doing more in a fraction of the time and with a smaller team. Below is a BTS video highlighting the work we did with historian Bryan Rigg, a US-born historian who was inspired to tell the stories of German Jews who survived World War II by becoming members of the Nazi Party. Originally thought to be a handful, Rigg revealed that there were over 150,000 Jewish men who survived. This included decorated veterans, high-ranking officers, generals, and admirals. In the 1990s, Rigg traveled throughout Austria and Germany, capturing hundreds of hours of their stories on a VHS recorder. We started with authentic video interviews of Holocaust survivors telling their stories. We began working on an initiative to turn this historical media into an educational tool powered by AI and our background in interactive development and VFX.

GenFX BTS Video In Action – Transcribe, Translate, and Create Synthesized Voice and Lip-Sync Video

Other Groove Jones GenVFX Work

• Accelerating Iterations: Traditional VFX often requires significant time for revisions. GenAI allows us to iterate rapidly, refining visuals to meet client needs faster. For example, you can use open-source tools like StableDiffusion to generate storyboards quickly, providing clients with a visual roadmap in record time. This capability accelerates our studio’s creative process and ensures alignment with the client before diving into more complex production stages. Below is an example of the various mood boards and then eventual storyboards the Groove Jones creative team did for an anamorphic billboard we created for ReliaQuest.
  
• Enhancing Backgrounds and Assets: GenAI can generate new background plates and 3D assets, which our team then integrates with traditional renders to create seamless hybrid compositions. Art departments can now generate complex, photorealistic matte paintings rapidly, offering a creative jumpstart while traditional VFX techniques ensure refinement and integration into the broader scene.
• Generative Voice Overs: Generative Voice Overs are a transformative GenVFX tool, enabling us to clone a voice actor’s voice and generate new dialogue as needed. Interestingly, much like the visual fidelity challenges of GenAI, AI-generated voiceovers often lack the richness of real recordings. Issues such as accurate phonetics, dynamic range, and nuanced performance remain hurdles. Despite this, AI shines in enhancing or replacing specific lines in existing recordings, as seen in projects like the Invesco QQQ Innovation Arena or the American Heart Association productions. Tasks that once required days of rescheduling and rewrites can now be resolved almost instantly, streamlining production workflows significantly.
• Facial Expression Transfer: This exciting performance tool allows us to tweak an actor’s appearance or even change entire lines of dialogue for different languages. This technique, combined with Generated Voiceovers, aids in the process known as “re-localization,” changing an actor’s spoken language into another language they don’t actually speak. This enhances traditional dubbing processes in ways previously unimaginable with GenVFX tools.
• Virtual Try-Ons and Wardrobe Adjustments: Virtual Try-Ons, powered by advancements like LoRA flux training, offer new possibilities in production design. These tools can change an actor’s wardrobe for artistic reasons or practical fixes like removing or replacing logos on clothing. [Insert image here: Before-and-after shots of wardrobe adjustments, such as copyrighted logo removal or costume changes.] Previously, such adjustments required on-set solutions like gaffing tape or extensive post-production paint-outs. With GenVFX, these changes are seamless and efficient, providing unprecedented creative flexibility.

Why Balance Matters

While GenVFX represents the cutting edge of VFX production, it’s not a replacement for human creativity. At Groove Jones, we believe the best results come from:
Investing in Our Talent: Training our artists and production teams to use AI tools to maximize their output while emphasizing the artistry that defines VFX.
Adopting Hybrid Approaches: Combining AI tools with traditional VFX/CG techniques for optimal results like we have used in several of our Generative AI productions like the Method Coachella, Oracle, and the Keebler sELFie Studio photo booths.

GenVFX, powered by the dual forces of Generative AI and Machine Learning, is revolutionizing VFX production. At Groove Jones, we’re leveraging these tools to enhance creativity, tackle traditional pain points, and push the boundaries of visual storytelling to a new level. By understanding the unique strengths of GenAI and ML, and integrating them thoughtfully, we’re doing our part to shape what the next generation of VFX studio workflows look like. We see a future where technology amplifies our human imagination, enabling artists to truly create at the speed of thought!

Contact Groove Jones today, and let’s craft an unforgettable engagement for your audience – https://www.groovejones.com/contact!