Ty Audronis

I did a light comparison between Luma Dream Machine (including the enhanced version) and Runway Gen-3. Three different styles of text prompts and 10-second videos based on them. Here are the first clips produced by the applications. From these, it's clear that it's still quite a gamble (although Luma has the ability to animate from images, making the results less random). And since these are developing so rapidly, I guess we should have much better control by the end of the summer. PROMPT 1: "Hand held camera shot, chicken turns to egg, reverse transformation, educational video" Runway is the clear winner here, and even though I don't like the crossfade, the concept matches the prompt. Luma Enhanced understands the handheld camera best, but it creatively misunderstands the reverse part of the prompt, making the rooster (!) walk backward, totally without the transformation. As a fitting bonus for an educational video, Luma Enhanced offers subtitles — probably in some invented language (or does anyone recognize it?). The basic version of Luma is completely off. A roughly seven-fingered clay hand seems to be cooking eggs, with no sign of a chicken. PROMPT 2: "Drone shot, animation, Bill is sitting on a prison roof and throwing potatoes down, Italian 1960s style cartoon" Runway wins the second round by a landslide, although it didn't understand Bill as a person on the roof but wrote it as text instead. However, the style is captured very well, and the potato is flying. Neither version of Luma managed to create a cartoon look. Luma Enhanced ended up with some sort of CGI graphics, while the basic version is purely photorealistic. Luma Enhanced scores points for the nice camera movement, and both Luma versions captured the drone prompt well. It's hard to say what's actually on the roof, though. PROMPT 3: "Moving dolly camera on a track, cinematic action scene, SWAT team handling a hostage situation in a toy factory, criminal wears a pink furry costume" With the cinematic prompt, the results are more balanced. No video shows the hostage, but otherwise, the elements are captured fairly well, and Runway's location most closely resembles a toy factory. The basic version of Luma and Runway could be shots with a dolly camera, while Luma Enhanced seems to be handheld (which is nice for an action scene though). The basic version of Luma makes the most sense, clearly showing the police catching the criminal and leading him away. Luma Enhanced goes completely off the rails, and it's hard to understand what's happening as the characters are morphing. Runway's beginning looks promising, but it would be nice if the SWAT team reacted somehow to the pink bear. Maybe they just don't understand it's the criminal. :) That was today's test. Something different next time. Which camp are you in, by the way? Do you use both or something else entirely?

24

7 Comments

I've been playing a lot with Luma AI Dream Machine and ways to ensure consistency in both style and characters. I've found starting with Midjourney images, and using the original image as a style or character reference solves many of the issues. Also the Dream Machine keynote feature lets you improve motion for vehicles by offering the start and end point as an image. The character voice is from ElevenLabs using speech-to-speech and lip-sync is Character-1 from Hedra. I'm going to continue building out this series to experiment with different tools and techniques. Next step is upscaling the Hedra videos.

5

6 Comments

"Apple Vision Pro is magical until it's not...and then it's damn frustrating." "Everyone was hoping there would be a silver bullet and there would be one device to rule them all, but it's just not practical with today's technology." "Expectations are starting to shift, and I think there will be application-specific devices in vertical markets for a long time." AWE USA 2024 was the site of an excellent panel discussion I hosted between Karl Guttag, Jeri Ellsworth (Tilt Five), Adi Robert (The Verge), and Edward Tang (Avegant). This was one of the best attended and most engaged sessions at the show. A cleaned up, audio-only version of the discussion is now available on #theARShow podcast (and you can find the video on YouTube). https://lnkd.in/gBNJkaEV #smartglasses #spatialcomputing #hardware #videopassthrough #optics #augmentedreality #XR #AWE2024

44

12 Comments

Research on real-time animation and rendering of volumetric data. Can be used with volumetric video, depth cameras or skeleton tracking controlled avatars. Mesh is voxelized, voxels are animated using compute shaders, then an isosurface is extracted using marching cubes. All animation and rendering happens in real-time in the GPU using compute shaders. #creativecoding #unity3D #newmedia #motiongraphics

16