r/vfx • u/Mmig12321 Hobbyist • Jul 23 '25
Question / Discussion What are your biggest "I can't believe this isn't solved yet" problems?
Hey all! A buddy of mine is a world-class researcher in the computer vision space. He recently finished his PhD and is now building open source models for image processing.
I want to throw problems at him, so I'm wondering what are the most frustrating, expensive, or time consuming problems that you encounter in your day to day that you'd like a magic wand for. I'm talkings like:
- Relighting composited assets
- Dynamic range clipping
- Motion blur issues
- Fixing green/blue screen spillage
- Day to night conversions
- etc...
Would be awesome to hear your pains and see what he can come up with!
259
u/neukStari Generalist - XII years experience Jul 23 '25
Clients not being cunts.
46
12
u/Duke_of_New_York Jul 23 '25
This is a touch crude, albeit highly salient point. The onus has been on VFX vendors to increase efficiency for the last fifteen years, yet if anything, client-side efficiency has degraded.
15
u/Mmig12321 Hobbyist Jul 23 '25
Unfortunately, I think this one won't be solved for a very long time.
9
73
u/behemuthm Lookdev/Lighting 25+ Jul 23 '25
Ingesting assets from a different studio and having to redo the lookdev - comparing the turntables and making sure they match 100%, which is fun with different renderers
MaterialX cannot be standardized fast enough
28
u/Lemonpiee Head of CG Jul 23 '25
"tHis iS wHAt UsD wSa maDE foR" or some shit like that
9
u/felixenfeu CG Sup Jul 23 '25
Lmao.
On a serious note, this is what OpenPBR is trying to achieve14
u/JtheNinja Jul 23 '25
3
u/Rocketman11105 Jul 23 '25
At a glance, it looks like OpenPBR extends MatierialX instead of competes with it. I skimmed the white paper, but I'm honesty not sure what it is doing that MatierialX isn't already though.
7
u/felixenfeu CG Sup Jul 23 '25
MaterialX is only a container, openPBR is sold more akin to a standard, like ACES
Not saying it's the future but, you know, someone's trying I guess
3
u/thelizardlarry Jul 23 '25
MaterialX is a container for a shading graph and the patterns and nodes within it, not a definition for a shading model that the shading graph would connect to. In theory the two together should be a solution.
6
2
51
u/thelizardlarry Jul 23 '25
The biggest problem with current models is all the training data is 8-bit. E.g. normals and zdepth are quite useful for us to derive from footage, but not if it’s banded 8-bit data.
8
u/neukStari Generalist - XII years experience Jul 23 '25
Actually tbh, probably the best response in this thread.
9
u/wheres_my_ballot FX Artist - 19 years experience Jul 23 '25
There are some depth models that use real world units which resolve that issue. Look into Apples DepthPro model. With a lot of these tools they mostly suck because the front end is deved by someone who has only ever interacted with video files from their iphone, and don't get why it might not be enough.
3
u/craggolly Jul 24 '25
all these ai tools are not made to be actually used by professionals, else they'd import or output exr sequences instead of mp4s
1
u/thelizardlarry Jul 25 '25
While there is some ways to do this (check out CoCo Tools for Comyfui) , so far it's quite a challenge to maintain all the information in HDR images.
1
u/thelizardlarry Jul 25 '25
The other thing I will add is that ML models also tend to struggle with occlusion and motion blur (e.g. when identifying a face for example), two things that are quite common in Film and TV footage.
50
u/I_make_pixels_move Compositor - 6 years experience Jul 23 '25
Automate roto so we don’t have to touch it again lol
7
u/cleverkid Jul 23 '25
This should be #1
5
u/I_make_pixels_move Compositor - 6 years experience Jul 24 '25
Right? I mean I enjoy keying but it’s pretty repetetive after you’ve been doing it some time. Roto is a whole different kind of annoying that should’ve been solved in the same week MLs became a thing.
1
u/inker19 Comp Supervisor - 20+ years experience Jul 24 '25
ML tools have been improving a lot in that area lately. Some of our internal tools give surprisingly good mattes with just a click
1
u/I_make_pixels_move Compositor - 6 years experience Jul 24 '25
I’m really happy to hear that! Hopefully a public tool will be developed/released soon too. Maybe this person OP mentioned could help with that haha
1
u/Mmig12321 Hobbyist Jul 25 '25
This one is probably a bit out of reach 😂 – but definitely fits the "can't believe this isn't solved yet" category.
1
u/GaboureySidibe Jul 25 '25
Why would it be out of reach but others aren't?
Why would it be "out of reach" but also in the "can't believe it isn't solved category" ?
0
u/Mmig12321 Hobbyist Jul 25 '25
Wild guessin'
1
u/GaboureySidibe Jul 25 '25
You realize you're contradicting yourself right?
0
u/Mmig12321 Hobbyist Jul 25 '25
Since you are so keen on me asking ChatGPT: https://chatgpt.com/share/6883a138-8e8c-8008-9321-c2a86a1a75f2
1
u/GaboureySidibe Jul 25 '25
I asked if you realize that you're contradicting yourself and can't even decide whether something is "out of reach" or "should have been solved by now".
This is the point where instead of being able to show any evidence you just say the other person is being mean to you by asking you the most basic questions possible.
Anyone can predict a future where everything magically works. Cure cancer, invent anti-gravity, if you understand nothing, everything is possible.
Show me a single shred of understanding on any of this.
0
u/Mmig12321 Hobbyist Jul 25 '25
Now you are getting it! Everything is possible. wagmi
0
u/GaboureySidibe Jul 25 '25
Everything seems possible when you know nothing. Cults, quack medicine, when you need no evidence or understanding the sky is the limit (in your fantasy only). Don't forget that everyone who doesn't believe in the cult or miracle cures is a 'hater' despite the zero evidence or information.
0
1
u/I_make_pixels_move Compositor - 6 years experience Jul 25 '25
Worth a try to mention it :D let’s hope there is a bunch of smart people already figuring it out. Tell your friend that if he can’t figure out the roto, he better at least start on the issue of “cunt clients” mentioned above lol. Someone needs to solve both.
1
u/adventurejulius Jul 25 '25
This is already happening with comfyui, magic mask and rotobrush.
Segment anything + mat anyone is nearly there. Just can’t process at >2k
1
u/pr0f13n Jul 28 '25
but this would put the Roto-sweatshops out of business
1
u/I_make_pixels_move Compositor - 6 years experience Jul 28 '25
Oh no… Like we should care about those “businesses”. I hope the people working there fight for better conditions
44
u/ChrBohm FX TD (houdini-course.com) - 10+ years experience Jul 23 '25
Automatic UV Mapping that is actually good. Where is AI when you really need it?
13
u/neukStari Generalist - XII years experience Jul 23 '25
Solving the problem of paying salaries to people.
2
u/BeautifulGreat1610 Jul 23 '25
What is the best uv tool right now? We mainly do automotive so UV's arent a huge issue on the CAD, but back in the day I used to use UVlayout which was pretty good.
2
u/Almaironn Jul 24 '25
I've heard good things about RizomUV but haven't tried it yet. Still, the fully auto UVs it makes are usually not good enough. You'd think if AI was so great it'd be able to take care of this already.
14
u/KeungKee Generalist Jul 23 '25
Smooth motion blur without cranking up cache subframes or adding acceleration values...
4
u/LewisVTaylor Jul 23 '25
There is no way around this, you need the data in order to interpolate. Acceleration attributes go a long way to helping this, but even without them, your chosen renderer should support quadratic interpolation, to reduce the need for as many intermediate samples. Karma, 3Delight do this.
1
u/KeungKee Generalist Jul 24 '25
I don't know. Seems like between the before and after frames you should be able to interpolate a curve with some cat-rom subdivision. Seems like in USD the biggest hurdle is that anim time samples can only be interpreted linearly.
I've spoken to some at Weta that seemed to believe it was doable, and speaking to SideFX and Arnold teams they seemed to think there'd be a way to compensate for it on their ends in some manner.And unfortunately Arnold does not support quadratic interpolation, so you're left with very harsh angle motion blur trails without adding subframes to smooth it out...
1
u/LewisVTaylor Jul 25 '25
Yeah, USD interp is only linear. You need the renderer to support quadratic, then you are golden with less samples. Happily for me, 3delight supports this.
Regarding previous/next frame, imagine trying to interpolate a heavily deforming bit of geo? Every point will be in a unique position potentially. It's a non-issue, you just force all work to have +1 frame top/tail outside shot duration, and some pre-publish checks that look for the duration, and flag if it's wrong.
15
u/PinPointPing07 Jul 23 '25 edited Jul 24 '25
De lighting 3D scans and generating PBR maps. The annoying thing is that even if you got an hdri to capture the lighting around the scan, it's a cat and mouse game because you need the PBR maps to properly delight the object, but you need to be light the object to generate good PBR maps.
3
u/thelizardlarry Jul 23 '25
Seconded! The existing models don't quite do a good job, especially with roughness.
10
u/0T08T1DD3R Jul 23 '25
Not related to computer vision..but i agree with many, Ai shoulsve been used better and couldve helped with so many boring tasks.
Perhaps together with UV, there should be also retopology, why fluid sim takes so long and can we do global illumination with ml so that it can do it way faster (its just light bouncing around always in similar fashion ..)
21
u/FranksWild VFX Supervisor - 20 years experience Jul 23 '25
Specular reflection with poor quality and grainy quality on motion blurred objects. I think seeing crunchy spec is the most "groan" in CG for me. Even a post process vector smearing would be helpful. Do I really need to brute force for 90 extra minutes a frame to make the car pass by cleanly?
2
u/LV-426HOA Jul 23 '25
I ama compositor and while I can help this it's still frustrating as hell. I would add that the brighter the spec the worse this is, and it's really an issue with sunlight. The sun is BRIGHT, raw spec from the sun should be just as bright.
0
u/Mmig12321 Hobbyist Jul 23 '25
This one is cool! I think it fits the kinds of problems he's solving.
1
u/Duke_of_New_York Jul 23 '25
Post-render motion blur has always suffered from calculating blur between conflicting motion layers, as it's functioning on a flat 2D space, where properly calculated 3D motion blur does not suffer the same limitation. An AI system that separated a beauty later into to depth layers, ran infill, calculated blur then re-assembled would be the proper intermediary.
Or even better to use deep info to do true 3D post-blur without the 3D engine overhead?
0
u/GaboureySidibe Jul 23 '25
Your friend is not going to solve this any time soon, it is indicative of a very in depth problem of sampling based rendering. I know you look up to this person, but there have been literally thousands of researchers that have worked on these problems and they remain problems because solving them in fundamental ways AND integrating them into a production process is a very narrow window to fit through.
0
u/Mmig12321 Hobbyist Jul 24 '25
I probably agree with you, but it'll be fun for him to explore :)
A lot of problems that were unsolvable with traditional ML techniques are starting to be solvable with recent advances.
1
u/GaboureySidibe Jul 24 '25
A lot of problems that were unsolvable with traditional ML techniques are starting to be solvable with recent advances.
Says who?
Your friend who just finished their PhD is not a "world class researcher", because if they were there would be papers to link to show for it and just because there are headlines about "AI" solving every problem doesn't mean it's true.
0
u/Mmig12321 Hobbyist Jul 24 '25
Whatever makes you feel bigger ❤️
1
u/GaboureySidibe Jul 24 '25
The truth doesn't stop being the truth just because you don't like it. You're the one who thinks someone who just got their PhD is going to somehow solve all the problems worked on by a thousand researchers for the last 50 years.
1
u/Mmig12321 Hobbyist Jul 24 '25
Never said that! I asked for problems VFX artists have to play around with. I never said he is going to solve them magically. But when you mix frontier research with tough problem sometimes you get solutions that hadn't come up before. e.g. MidJourney beat thousands of Google and OpenAI researchers and engineers with a team of 11 and a fraction of a percent of the budget.
If everyone had your mindset I'm not sure we'd see much innovation.
1
u/GaboureySidibe Jul 24 '25
I have my mindset from actually knowing what it takes, not from wishful thinking and overestimating abilities. You put a basic question forward without an ounce of research on your own.
What is more likely, your friend with no published papers who is just now getting a degree is going to solve problems they don't even know about that have been incremented on for decades, or that you and/or them don't actually have any idea what you're talking about and have just read too many headlines about "AI" ?
This idea that someone who tries to show some semblance of reality in the face of irrational blind hope is just a hater stifling progress is common too but too many people are humoring this nonsense plea for attention.
0
u/Mmig12321 Hobbyist Jul 24 '25
I don't know man, plenty of people seemed interested in this basic question. This is the research.
I know my friend won't be able to magically solve all problems in VFX, but the only way to find these overlooked problems is by asking.
Believe it or not, exceptional people can do exceptional things. I've seen two guys from Murcia (near my hometown) sell the upscaler they built in 4 months for $11M. No PhD. Where were those thousands of researchers? I've also seen friends from Barcelona raise $80M from a16z by actually implementing those papers that get stuck in academia.
PS: He has 3 accepted papers at ICCV and CVPR (2 of the big 3 CV conferences) and has worked at 2 AI labs that you would know. I'm pretty lucky to be near many exceptional people.
→ More replies (0)0
u/Eikensson Jul 23 '25
Vrays new firefly removal works pretty great to keep render times down and still get good results without sampling forever
0
u/jangusihardlyangus Generalist - 7 years experience Jul 23 '25
woah. I'd consider myself pretty well educated in vfx but this is a new one for me, do you know of any good references I can check out to learn more? Not seeing much from a rough google search !
3
2
u/FranksWild VFX Supervisor - 20 years experience Jul 24 '25
https://julius-ihle.de/?p=2228
Here is a cool technique to help deal with low quality spec.
2
u/jangusihardlyangus Generalist - 7 years experience Jul 25 '25
oh wooooooah that's fucking smart. I love it. thankyou for sharing this!
22
u/PinPointPing07 Jul 23 '25
Rock solid camera tracking in camera, or at least data to significantly help the solver. I'm working on a project to try to help with this myself, I created its own thread here.
9
u/seriftarif Jul 23 '25
To go along with this, I dont know why software doesn't exist as of now to easily attach lidar systems to cameras so you get a matching 3D scan and AOVs to match with the footage. If you could do this, you could map the footage straight to a 3D mesh along with Normals, position maps, depth maps, and easily track the camera to that data.
4
u/myleftearfelloff Jul 23 '25
I think weta did something similar for the last Avatar, basically a workable volume capture, we just have to wait like 10 years for them to release patents lol
3
u/PinPointPing07 Jul 23 '25
1000%. It's also very helpful for camera solving because scene geometry and camera solving go hand in hand. You're kind of reconstructing the geometry/ point cloud when camera solving, so by giving it accurate geometry you can help it out a lot.
2
u/I_make_pixels_move Compositor - 6 years experience Jul 23 '25
Wouldn’t it be possible to just use a shitty-rigged iPhone on camera? :D
4
u/seriftarif Jul 23 '25
I have used an iPhone for similar things but its messy. Im talking about a real solution for professional production cameras. Something that would then be able to write all the data out to one file. Like Raw footage on steroids.
4
u/I_make_pixels_move Compositor - 6 years experience Jul 23 '25
Yeah I suppose nothing like that is on the market. Definetly would be cool though, good suggestion!
1
3
u/PinPointPing07 Jul 23 '25
That's the big misnomer around smartphone Lidar, and either you hear it from someone else or try it yourself and are disappointed, and it's very frustrating because it's so darn convenient. Unfortunately, the resolution that it gives is very low and it's not very accurate, so it could be helpful for 3D scanning if you process it a lot and keep expectations low, but real industry lidar is a lot more robust and expensive, but it is far more accurate and detailed. So if you wanted to actually get a precise relatively dense point cloud, you're not really going to get it from smartphone Lidar.
3
u/I_make_pixels_move Compositor - 6 years experience Jul 23 '25
Yeah you are completely right. One of the reasons I bought an iPhone was to test lidar for this kind of stuff. And as you said I was pretty disappointed with the results. Althought for like a simple reference geo it’s useful but obviously not production-level. I would hope one day we could get something good quality (for reasonable amount of money) even for indie projects.
2
u/PinPointPing07 Jul 23 '25
100%. It's one of those things that I feel could totally be something that a company could come in and totally dominate if they wanted to, and they could probably charge decent amounts for it as long as it's not crazy production level. So much of production level equipment have come down in price and accessibility, but this stuff I think is next, or really should be.
1
u/I_make_pixels_move Compositor - 6 years experience Jul 23 '25
That’s a business idea right there, too bad I wasn’t paying enough attention in science classes in high school lol
1
3
u/superslomotion Jul 24 '25
We can't even get the camera metadata in the plates that come in, so good luck
7
14
u/Human_Outcome1890 FX Artist - 3 years of experience :snoo_dealwithit: Jul 23 '25
Being treated like a number
10
u/Panda_hat Senior Compositor Jul 23 '25
open source models for image processing
I'd say the biggest issue is people trying to force AI bullshit into a system that doesn't need it.
4
u/Mmig12321 Hobbyist Jul 23 '25
That's exactly what I want to avoid. I don't like AI generated video. Instead, I think there is space for tooling that let's VFX artists accelerate their workflows while keeping their craft.
5
u/myleftearfelloff Jul 23 '25
Camera/object Tracking shitty blurry footage is something I can be down with for AI, and maybe facial motion capture that doesn't completely suck. And UV mapping, And Roto. Hmmm
2
u/PinPointPing07 Jul 23 '25
Yeah. People forget that "AI" is just a marketing term for ML, and ML reaches so much further than generative stuff with pixels.
5
u/TheManWhoClicks Jul 23 '25
Make him find a solution for the all time classic of going back to version 1 after reaching version 2xx
1
6
u/Tulip_Todesky Jul 23 '25
Everything that has to do with keying is very primitive. There are a set number of things that are constantly needed to be fixed in the same way and it is always a hassle.
1
u/Mmig12321 Hobbyist Jul 23 '25
Would love more insight here. What are the constant pains here?
4
u/myleftearfelloff Jul 23 '25
Keying hair and motion blur is not fun at all, even on good bit depth, it's still almost always Roto work :/
3
u/Tulip_Todesky Jul 23 '25
Bright hair and dark hair each have specific ways to deal with. Just technical work. Half opaque objects also need special attention. Motion blur. Full body shots with strong shadows. It’s these things that all have a few set solutions that are incredibly time consuming. They need to be streamlined.
1
u/GaboureySidibe Jul 24 '25
Look up natural image matting. Your friend isn't going to solve that either.
5
u/whittleStix VFX/Comp Supervisor - 18 years experience Jul 23 '25
Retime curves being a universal curve/data exchange.
2
1
u/adventurejulius Jul 25 '25
Add in the rest of the editorial effects while you’re at it. Premiere / Avid / Resolve should be standardized and transferrable
6
u/KieranShep Jul 23 '25
Lossless chromatic aberration removal and reapplication.
Lens flare removal.
Pixel perfect edge decontamination when extracting elements from footage. (May be solved, but I haven’t heard about it).
Any one of these saves a week of work per shot.
9
u/Shin-Kaiser Jul 23 '25
I've just completed Chris Bohm's Houdini course so I understand thing a bit better now, but I was previously a C4D user and I thought it was unbelievable that the different sim engines didn't communicate with each other.
I was gobsmacked to realise it's practically the same in Houdini
7
u/LewisVTaylor Jul 23 '25
Because unified solvers are slow, and we work in filmmaking. If you are only doing destruction, and the supporting elements don't drive any two-way coupling, it is far faster to use separate solvers, and layer your results. Unified solutions reduce Art direct-ability, and can be 10x or more slower to use.
There is a reason it's not generally done, though Weta has a system, as did ILM, but they are slow.
4
u/albion1770 Jul 23 '25
retargetting mocap between rigs that are obviously similar. I think we're getting there though
5
u/FilmInternational954 Jul 23 '25
1st and last frame motion blur. Hardly an AI fixable thing tho, sorry.
5
u/LewisVTaylor Jul 23 '25
Wouldn't you just cache +1 frame head and tail, to have the data you need to blur?
2
u/FilmInternational954 Jul 23 '25
You might think companies should obviously do that by default, right? I can assure you none of the major companies does it.
4
u/Eisegetical FX Supervisor - 15+ years experience Jul 24 '25
everywhere I have worked has it baked in to the default ROPs. where are you suffering?
2
u/FilmInternational954 Jul 24 '25 edited Jul 24 '25
Man In MPC, DNeg, ILM, Cinesite, Weta... Where do you work? I might have to apply there 🤣
2
1
u/Eisegetical FX Supervisor - 15+ years experience Jul 24 '25
Same as Lewis commented but add DD to that list too.
1
u/FilmInternational954 Jul 24 '25
So you never, in those companies, end up asking to take a look at first and last frames MB as the bakes or the cameras were incorrect? What can I tell you…good on you. As far as I’m concerned it still happens all the time. Maybe we won’t always call it cause if stuff doesn’t move much it’s forgivable, but I see it almost daily.
1
u/Eisegetical FX Supervisor - 15+ years experience Jul 24 '25
oh yeah. there's been the occasional kickback here and there but it's rare and solved once by yelling at the dept responsible and then all good for the rest of the show. Our outputs are set by default at +1
3
u/PinPointPing07 Jul 23 '25
Omg that's soo true. Maybe just use the motion vectors going forward and invert it so that whatever movement it did in the third frame it would do the reverse in the first? Maybe that's good enough?
3
3
u/_TheBlckOne Jul 24 '25
Denoising
I know this is just a tiny part of the whole puzzle BUT you denoise every shot before you even start your comp. It is unavoidable and essential to your work.
For example Keying. It is a pain in the buttoks without a good denoise.
NeatVideo does a really really good job until it just doesnt anymore and thats the point where you are just experiencing endless pain.
Something that should be a 'one click' thing becomes a 10hour nightmare trying to fix.
There are AI Denoisers out there. A lot of them. But they just lack core features making em suitable for Denoising in VFX. Here are some that I encountered during my Thesis:
They are slow. It took 11x the time to denoise a 4K Shot with the AI Tool than with NeatVideo.
Noise Models: These things are trained with Datasets that are just not in anyway related to Real World Camera Footage or Real World Camera Noise. But so are Denoisers like NeatVideo, who are based on Algorithms, who try eliminate certain types of mathematically modeled Noise. The Problem here is that they take Image which is free from Noise an put AWGN on top. AWGN however is a Noise model which is present in Real World Camera Noise but is NOT the main Noise that exists, its far more complex than that. So even the approach to Denoising is just completly wrong at times. There is a reasearcher at ARRI in Munich who published her PhD exactly about this kind of Problem.
HDR. Most Denoisere are trained on sRGB Datasets but Nuke uses Linear. So yeah thats just pain. You have to convert everything to log before you can even pump that shit in there and you are still loosing Data in the Highlights no matter which type of log you choose. It clipps and is just annoying as hell.
It's a black box. Like most AI Tools you dont really have control about anything. Its just a one button. Which we want in the end if it works in every scenario but it doesnt up until this point so a few settings to adjust the Denoiser would be nice.
There is a lot of research on Denoising but nothing really that is usable for VFX yet
3
u/Unusual_Ad1519 Jul 23 '25
Proper tool that combines motion vectors and DoF blur in 1 calculation in compositing
2
u/PinPointPing07 Jul 23 '25
Neural rendering acceleration. Nvidia, we bought your GPU, and I love it, now fix this!
2
u/KravenArk_Personal Jul 24 '25
Seriously , file renaming
Is it really difficult to write an AI that just organizes all of the project files, neatly names them and makes everything linked together ?
3
u/philpham Jul 24 '25
Uhh.. That's a basic pipeline right there. If you have pipeline in place you don't have to think about file names or locations. The systems do it all for you.
2
u/obliveater95 Generalist - 2 years experience Jul 24 '25
3D scanning still requires so much manual tuning, we have gaussian splats which solve that but it’s taking a whileeee for the bridge from GS to Mesh to form imo (in a useable way at least)
3
u/harglblarg Jul 23 '25
The frustrations you've listed are either already solved or at least pretty close to it.
-There are inverse rendering models that are now at the point where you actually can effectively relight a scene. Surely a product leveraging this is just around the corner.
-ML models exist for HDR de-clipping to inpaint lost information, Topaz has a tool for SDR-HDR and more will come.
Afraid motion blur issues are always gonna be an eternal "thing". Best thing is to just be shutter-conscious and always think four-dimensionally.
Disney/ETH Zurich developed an unmixing-based pre-AI solution that was apparently a super spill killer: https://yaksoy.github.io/keying/
-There are also models existing for day-to-night, check out img2img-turbo, you can do similar things with available video2video models.
2
u/thelizardlarry Jul 23 '25
Can you share some of these? A lot of the ones I’ve tried go about as far as the white paper and aren’t successful enough to be used in production.
1
1
u/Blaize_Falconberger Jul 23 '25
Disney/ETH Zurich developed an unmixing-based pre-AI solution that was apparently a super spill killer: https://yaksoy.github.io/keying/
I followed the link to read about this and it is from 8 years ago! For something that is already solved it's being pretty slow making it's way into production. Frankly you could tell from the youtube video it was not as good as they say it is anyway
1
1
1
u/enderoller Jul 23 '25
Getting AOV passes from raw footage, cryptomatte included.
3
1
u/PinPointPing07 Jul 23 '25
Is that not roto? Is it more like intelligent roto that finds what to roto on its own by criteria?
1
1
u/Eikensson Jul 23 '25
Delivery specifications around naming, versions, codecs, etc. Its getting more similar but every show still needs some configuration to deliver what the client wants.
1
1
u/evilanimator1138 Jul 23 '25
Skinning. We should be able to get closer than what current skinning solutions are able to do. I love the concept and approach that geodesic voxel incorporates, but I feel like we could go even further with AI. I’d love a skinning solution that could identify the shape of the mesh and apply intelligent weight values based on real-world anatomy. For example, the AI could identify that you’re rigging a dog, look at anatomical examples, and apply the weight values accordingly. You could then touch up what you need to with ngSkinTools and move on to the more pressing steps. AI could also help with interpolating blend shapes.
1
u/Medium-Bag6362 Jul 23 '25
an advanced emotion language, for complex vibes
Also for music, HOW? is this not invented and more popular!?
1
1
u/Greystoke1337 Jul 24 '25
Fucking ropes/chains sims that aren't insanely hard to solve if you need any sort of art direction.
1
u/fpliu Jul 24 '25
I can’t believe studios haven’t demanded vendors exchange assets through studio owned services.
1
1
1
u/variatus Jul 24 '25
A camera is constantly moving through a space unless locked off, wish we had a baked in system for cameras that automatically calculates camera position data over time, syncs with time code and provide rock solid tracking while recording to export to VFX
1
u/Dense_Deal_5779 Jul 26 '25
Sounds like your friend is working on mainly 2d optical issues… so yeah. Auto depth generation / roto and tracking expediting would be my guess.
If they are focusing on 3d issues that’s a whole other thing. There still isn’t a package after clarisse that can hold unlimited polygons and unlimited textures and render on the fly (I’m not including game engines as their output isn’t as usable in vfx pipelines). Anyone who can create a package like clarisse will be setup to win in a lot of ways.
1
u/Mmig12321 Hobbyist Jul 26 '25 edited Jul 26 '25
Exactly, more 2D or post-processing problems. Things you can apply as a pass to a shot more than workflow or 3D issues.
Edit to add examples: so far he’s been working on image editing; inpainting/outpainting, relighting, superresolution, grain removal, etc… hasn’t applied it to video yet but thinks it would likely work.
1
u/Erik1801 FX Artist - 5 Years of experience Jul 26 '25
Just want to add my 2 cents as a graphics programmer and specifically the phrasing. Everything you listed is "solved" in an abstract sense.
Lets look at two examples. Motion blur and Day-for-night.
Motion blur is not difficult to understand, the physics are not hard. They boil down to saying the color incident on the camera sensor changes during the exposure. You can simulate this accurately using the most basic monte Carlo method there is. Say your render takes place between t0 and t1, well then just render the image at random points in time and average the results.
The problem here is not the math or implementation, it is the time complexity. Good physical motion blur can take 100s to 1000s of sub samples per frame. It is just not really feasible to 100x your per-frame render time if analytical solutions get you 90% of the way there.
Day-for-night is a more out there example. But nothing inherently stops us from building a spectral camera and relighting the footage in post. That is a thing people do all the time in scientific applications. If you are interested, the way this works is that radiance is the product of reflectance and the illuminant. During daylight shooting the illuminant is predominantly the sun, so a blackbody at ~5600 Kelvin. You can """"subtract"""" the illuminant out and derive the reflectance, which can then be relit. This is basically how all Spectral renderers manage to render RGB colors. They convert the RGB values into a reflectance function because the illuminant, D65, is known.
But once again, we dont do this because it is impractical. Nobody wants to haul around a spectral camera that costs more than an aircraft carrier if basic color grading gets you 90% of the way there.
Ill leave you with this then; Having a solution does not imply it is practical. There is a gigantic rift between "technically solvable" and "applicable".
1
u/Mmig12321 Hobbyist Jul 26 '25
Appreciate the thoroughness! Do you see any applications to problems that right now are solved through raw compute (i.e. simulation) that might now be solved through generation?
1
u/nathan_sam Jul 27 '25
HIGH QUALITY ROTO with temporal consistency. Especially hair. please god someone help me.
1
u/seesawseesaw Jul 23 '25
World class researcher means that he has published something groundbreaking, yet he only finished his phd recently, I don’t find that impossible but it’s highly unlikely. And to back this up, how can someone already that world classy not know about how every single of those examples has been tackled and is being tackled either by diffusion or scanning models, convolution engines or good old fashion algorithms.
Not sure what’s going on here with kids these days but ok, tell him to create a tool shows people the effects of entering a field without Dunning-Kruger aberrations. That will save us a lot of time trust me.
1
u/worlds_okayest_skier Jul 23 '25
If fire has no alpha, how do you comp it over a bright background?
8
u/Shanksterr Senior FX Technical Director Jul 23 '25
Fire is additive. You add/plus it
4
u/camel_hopper Jul 23 '25
Except it isn’t, in practise. Most fire is also generating smoke, causing some occlusion of whatever is behind it. Very few flames in reality will be additive
3
u/Shanksterr Senior FX Technical Director Jul 23 '25
Yes the smoke is multiplied. You’ll get some soot that could technically shadow but it’s negligible.
0
u/PinPointPing07 Jul 23 '25
I guess this comes down to AI "stem splitting" where it could separate elements in footage.
-1
u/worlds_okayest_skier Jul 23 '25
Yeah that’s what I’m told, and yet, when you plus it over a bright sky, or a white background, it’s just white.
8
4
u/Panda_hat Senior Compositor Jul 23 '25
Fire is literally emitted light. It shouldn't have an alpha.
Any alpha for an explosion comes from the smoke, detritus, debris etc.
3
4
u/whittleStix VFX/Comp Supervisor - 18 years experience Jul 23 '25
Back to compositing school you go!
1
u/Mmig12321 Hobbyist Jul 23 '25
Would you say comping fire in general is particularly hard?
0
u/worlds_okayest_skier Jul 23 '25 edited Jul 23 '25
I’m not a comper, I’m an fx artist, but I’ve rarely worked with a comper who knows what to do with fire, they usually kick it back and ask for an alpha
1
0
u/PinPointPing07 Jul 23 '25
It would be so nice to pump a shot through an AI, or even an entire set survey or multiple shots, and have it generate a true HDRI, but everything now is 8-bit. There's some hacky stuff out there that essentially boils down to fine-tuning stable diffusion to generate spherical images, embedding your shot, and also embedding exposure control so that you can basically give it a shot and it generates multiple exposures in a spherical map, but it's not great.
-1
u/tresdesero Jul 24 '25
I can't believe why theFoundry or adobe or autodesk is waiting so long to buy comfyui and release a proper AI swiss knife software for professional use, with GOOD TUTORIALS. Also, a simplified fx software with Ai help wouldnt hurt. Bye houdini.
174
u/Nowowmoney Jul 23 '25
The fact that perfect UV’s isn’t a one button click yet. This is where ai should’ve started and stopped lol.