28
votes
AI video won't work in Hollywood, because it can't make small iterative changes, former Pixar animator says
Link information
This data is scraped automatically and may be incorrect.
- Title
- Former Pixar Animator Gives One Big Reason AI Video Won't Work in Hollywood
- Published
- Apr 26 2024
- Word count
- 817 words
That's kind of a strange take; it definitely can make small iterative changes. You do have to "manually" provide the previous iteration's output as input, which is a given for humans, but it's still very much doable.
While true I think the detail he’s going for is still a problem.
“The previous image with a better background” isn’t the same as literally just revising the shadows that you noticed are messed up or better textures on whatever.
From what I’ve seen that kind of detail work is mostly blind luck, and it’s not like you can sync the work across multiple employees
Definitely agreed. Like almost every opinion piece on AI in the workplace I've seen, though, it still falls into one of two camps: Either the opinion holder makes little to no effort to imagine what AI will be capable of doing, or they imagine AI will be capable of doing anything.
This one is in the first camp, like @unkz mentioned.
Well put. I’m constantly reading hot takes that are either hype-filled nonsense or weirdly dismissive of the whole field. The truth, as usual, is somewhere in the middle. But people seem to have a tendency toward the extremes with this particular subject.
I don’t think it’s fair to expect people to speculate about technologies that haven’t been invented yet. Can’t we just talk about the demos we can see?
People who want to hype AI can spin whatever scenarios they want, but until the software gets released, it’s just vaporware.
I’m not sure. Unlike an image, where it’s easy to let the AI just work on one area or have it give you a starting point, I’m not sure how it’s working with video.
Is it creating rigged models and a scene with lighting and texture maps and all that other stuff?
I could see the answer to that being no, that it’s just a progression of images, and that means you fundamentally can’t edit this stuff. There’s certainly not b footage from another camera or something to work with.
Could it eventually do that too? I don’t think so from my understanding of llms. Or at least not without various human steps. I’m still firmly in the “it’s an impressive tool but not replacing a ton of people” camp
Yes, the AI generators have to regenerate the entire image to address a change.
However, when you're talking about a well funded entity (such as a professional computer effects or animation company), the computers will be able to regenerate a desktop sized image in seconds. Well less than a minute. Even if you scale up the resolution and complexity and all that, it's still really fast. And that's today; what happens next year, or five years from now? With faster computers and better AI programs?
Even today though, how much faster is it than a human team? Because I read credits on animated movies. They have whole teams that do nothing but focus on light, others that do backgrounds, others that do textures or cloth, some focus only on characters, and so on. Lot of people laboring on little pieces, and a lot of the time the audience doesn't even really take full note of it.
Further, I think the most likely use of AI tech here will initially will be via the drafting and drawing board process. The AI kicks out options, and that acts as a spark for humans who look at it.
And further still, any image an animator or artist wants to change ... they can still change. A professional illustrator or artist or whoever can take any image, human or AI, and make modifications to it. Maybe the image is great except for some little things. They could spend months having the hand animated teams redo that stuff for a high level guy to then tweak and approve and play back-and-forth with them over, or they could have the computer run overnight and then the guy comes in next morning and has the day to fiddle with tweaks.
Change is scary. I get it. A lot of people who like what they do are worried about not being paid to do it anymore.
But Hollywood has always churned through people. They capitalize on the dreams of "working in Hollywood" to not pay what they might have to if, say, working in Hollywood was considered the same as staffing a remote arctic oil field or deep sea factory fishing trawler. Which is to say, for every anyone who gives up on Hollywood, more than just that departure is waiting to step in and have their shot.
With animation, it's always been incredibly labor intensive. The corporate biography of the Walt Disney company, for example, talks about how switching away from Walt's vision of all animation all the time saved the company. Because it was taking years to produce feature length animation films. They needed revenue, and live action gave them that in a timely manner because a feature could be shot in a couple of months, then edited and prepped and released in a couple more.
Why is no one doing feature length traditional hand drawn animation now? Cost. No one wants to pay scores of artists to draw in the traditional manner for several years to produce all the frames needed for a feature. Heck, no one really wants to pay for them to do it for a four minute cartoon, much less a feature. They all use CG animation because it's cheaper, and use CG production techniques even when they bring in hand animators for some aspects for the same reasons.
I miss Don Bluth. He had a dream, and it was to do feature length animation projects. He was pretty darned good at it. But his star faded because his projects didn't earn back. Which is a shit reason to write off a creative type, but the money pays for the art. That's pretty much how it's always been, and AI is not the reason for it.
AI could save it though. Maybe.
One of the ideas in the back of my head is some pro animation team taking an AI art program, and project training it on all their key frames and hero character art and all of that. The stuff the big names on the team would draw, while the scores of people listed in the parts of the credits no one ever really reads does the bulk of it. Grinding it out while those big names get to be creative.
They train the project AI on their best bits. The style and look of the project. Is it bright, is it gritty, how angular, all the seventeen million things that could tweak how art looks. Represented by the art they give it to emulate.
Then they keep doing that, while the program team starts having the AI churn out what they need for the project. The animator team reviews and tweaks and revises, sometimes altering, sometimes ordering regenerations.
I have a feeling, once AI programs are a bit more commercial and a bit less hobby/laboratory mode, we might get some projects that look more like that lauded lost yesteryear of glory animation. How quickly could a Pixar level bank of computers generate a Secret of Nimh kind of film? I bet way less than the ~30months Wikipedia lists Nimh having taken by hand.
If the computer churns for a year, and the team spends their time tweaking and finding the magic instead of cranking out iterative frames of background stuff and getting yelled at for taking too long, that's the kind of thing that could be economical. Which could bring back a lot of animation styles we don't get anymore because they're not economical.
Or, we could scream the sky is falling and bury our heads in the sand waiting to feel the thump on our asses.
As far as I know the AI generators currently produce flat images. A professional graphic or video artist will be working with multi-layered images where every element is discrete and editable, effects layers and FX nodes separate again. A single image might have tens or even hundreds of layers. That's what "editable" means in a professional environment. Because depending on the exacting requirements of your creative direction you're often changing single pixels or processes being done to single pixels. In video that might span tens, hundreds, even thousands of frames and require exact and consistent application over them all. 3D adds even more complexity and as such requires even better repeatability.
I'm not saying the sort of AI you're referring to won't be able to do that one day, but it's exactly the kind of context, and repeatability, that it's currently not very good at.
fwiw AI is absolutely being used in professional graphics already but it's not the whole-image generator toys - and they really are just toys - that we see on the web at the moment. AI is just another tool in a very large toolbox, for use by humans.
Worth mentioning that AI-powered masking is my currently most-used AI-assistive tool in Krita. I can click on a partially-obstructed $object, and 7,8 times out of 10 it will be properly masked, with minimal to no further editing required before I can make it its own layer, etc. Even the failures are time savers over manual masking. It is almost painful to think how much time I could have saved back in the day, were such things available!
Yeah this is the thing where I think AI is going to be a MASSIVE force multiplier, and basically required to compete already. It will 100% make it into the toolkit, but there's a very large difference between 'assisting with most steps' and 'creating editable content'. It's literally playing with images, not the uhh...projects?...that a professional level animator (especially 3d) is dealing with. Rigging, lighting, textures, masks, matts, etc (i have 0 professional knowledge of this but have always found it interesting and know some people tangential to the industry).
You're spot-on with the force multiplier take. If you want a fun example of what pros can do, Corridor made an 'animation' using AI to do the heavy lifting a year ago: https://www.youtube.com/watch?v=GVT3WUa-48Y
They have a detailed how-to on their main site, and technology has improved a lot since then.
Yeah i'm not surprised, but also it's a "devil is in the details/how far have we come" situation.
I'll have to glance through the guide later because I do think this stuff is fascinating, but I feel one of the strongest points to AI will be quicker prototyping/onboarding/low effortish material.
For those of us who have a neat idea but not a ton of talent it's going to be a great tool to suddenly make a joke or quick story or whatever, but I don't think it's going to compare to the likes of Pixar anytime soon. Still I'm sure even those animators are saving a shit ton of time using these new tools in smaller capacities to help them iterate and handle more tedious and obvious scenarios.
It can't make these kind of changes... yet... It's downright obtuse to think that it couldn't be programmed to be more useful in situations like this. AI generation has come ridiculously far in 2 years, and detractors keep coming up with milquetoast arguments about why it isn't very good at a certain small thing.
I've spent a lot of time and effort attempting to learn how these systems work, and more importantly, what their limitations are. If you have something valid to contribute that shows why these limitations aren't legit, then I would love to hear it.
As I understand it, you're already bottlenecked to some extent on the underlying matrix multiplication that powers these things and is why first GPUs, and possibly hybrid or analogue chips, are being used. Should analogue chips pan out that would help, but there's still very real limits.
Beyond that though there's limitations on what can and can't be computed reasonably from the LLM method. It, quite literally, can't create. It can combine, and that's oftentimes "good enough" or again, a good starting point, but from everything I understand you're still relying on the underlying dataset to get your output. I do not believe an LLM could have EVER come up with something like the recent Sony animated spiderman movies without already having that style in its database.
I believe this might create an industry of "easy for AI to make" content, and that might choke out other options within that industry, but it doesn't surprise me that someone at pixar, a company that is often pushing the edge of the medium, is unimpressed with the options it provides.
As I already alluded to in my previous comment, the way these make video currently makes them next to impossible to edit. I do think (hell am certain) you'll see AI tools to help with rigging, lighting, basically every step of the creation process, but I don't think you're going to get any good results from "take this scene and make the rigging and lightning needed for it" because i'm not sure that dataset exists, or can exist, in a meangingful way for a LLM to contribute.
It's far different to say, ask the AI to help rig your Lovecraftian monster model, than to just have it provide a scene of it moving around and doing whatever you need.
Further, NONE OF THIS, is cheap. We hit the edge of Moore's law awhile ago and barring some major breakthroughs (and the aforementioned chips), these processes are EXTREMELY expensive. It should be, as the "theory" is that they'll supplant lots of top level work that costs millions already, but that's not proven at all yet. Again I think it's far more likely we see a massive drop in the quality/creativity of the industry as AI work becomes "good enough" to justify the cost, and things that push the edge like Sony's spiderman just aren't made anymore (or are extremely rare exceptions).
So not sure how any of that qualifies as milquetoast, but if you want to contribute I am eager to learn more on this subject. It is immensely complicated and covered in hype and marketing. Bluntly the "line goes up" mindset has proven to be foolish over and over and over, but more than ever we've got entire industries (see the hyper loop or the current moon landing plans) that are based on just ignoring the very real limitations of groundbreaking tech.
The first humans started making boats supposedly in something like 8000 BC. The first known circumnavigation of the globe was in the 1500s AD. Tech moves in jolts and spurts and while AI will have an effect, I don't think there's any good evidence this is half as simple as you make it out to be.
Are they small and precise enough for a picky director? Could be
I think the article gave Craig Good more nuance than what I am seeing in the comments so wanted to address that. Good's argument is that AI right now cannot currently allow the iterative process animation studios need. Good never argues that AI will never replace humans, and Good states that it may get there, but right now it cannot and so studios will have to wait until iterative tools are developed. Secondly, it is a discussion of OpenAI Sora, which is not publicly released, and the article does state that we do not currently know all the features it supports. My reading of Good's comments were more "before AI can be used in the studio, it needs to able to do iterative workflow and I don't think it can yet, but it might get there in the future."
I have to imagine it isnt already being used as well, to spit out or enhance models humans can then play with, or build/enhance shaders or lighting models. It's a weird comment for someone in the know to make.
It doesn’t take much imagination to see how the next iteration will make this possible though.
I think that’s a good example of how to properly hedge when talking about the present. Imagining better software is feeding the hype.
People were already complaining about hands and fingers being an ongoing issue. Can't imagine what problems crop up next
This interview with the people behind the Sora made short film "Air Head" goes into some detail about how it is definitely much more manual work than simply writing a few prompts.
Most technology developments goes through a overestimation in the short term but underestimated in the long term, and I think we are still very much in the overestimation phase. It is anyone's guess when and how long the long term will be. But it is not a natural given than progress comes linearly fashion, sometimes thing hit a plateau and where the last 20% takes 80% of the effort or more. The current AI models are not their infancy, but the culmination of decades of research and many of the limitations are not like small bugs that needs to be fixed. There are fundamental limitations with the approach that isn't just fixable with a few more years of work. Like we have some NP-hard problems in computer science that have been known for several decades, that simply can't be solved with non-exponential algorithms. I think we are seeing some sort of plateau with the current tech of AI. The hype 2 years ago would have us believe that we by now should have amazing AGI that could do everything, because people just extrapolated half a years of progress and thought it would keep going like that. It is pretty clear that AI didn't just explode exponentially in capabilities.
Yet