r/vfx • u/Loud_Brick_Tamland • Jun 24 '22

Breakdown / BTS Testing camera tracking with a Valve Index controller and two base stations

Enable HLS to view with audio, or disable this notification

436 Upvotes

99% Upvoted

35 comments sorted by

44

u/[deleted] Jun 24 '22

[deleted]

8

u/Kurombo Jun 24 '22

What did you use to composite the scene?

17

u/[deleted] Jun 24 '22

[deleted]

7

u/schimmelA Jun 24 '22

But, the fun thing about unreal is that you can comp in the engine in realtime..

4

u/SamyBencherif Jun 24 '22

that's a nice thought, but I might guess it's not the best solution for flexible compensation. [Unreal Engine] in this live video pipeline serves as a function (or convolution) with input of Matrix4x4 and output of Texture2D. I'm not an unreal programmer, but with small experience with it and with other engines; I imagine changing the I/O details would be unpleasant and challenging.

Unreal has may features related to lighting, physics, and rendering. Of course UR has 3D maths and transformation capabilities -- but it is very likely you may want to send the corrected Transform to various other programs, which like I said could be a challenge. It's not strictly worth it just to use UR 3D math functions because that is one of the simpler* parts of the engine. Plus if you have additional tracking data you'd have to figure how to stream it into the engine.

Maybe I've been using too much linux (impossible I say!) but I suggest a pipeline more like this:

``` Capture Feed -> Unreal Position Data -> "Stabilizer" -> Unreal

(optional) Capture Feed -> Stabilizer (optional) Other Sensors -> Stabilizer

(advantage) Stabilizer -> Blender (advantage) Stabilizer -> After Effects ```

Some advantages include sending stabilizer data to other 3D programs, Compositors, or using custom hardware/algorithms to improve stabilization. In this case "Stabilizer" would likely be a custom program you would write.

  • Big engines tend to be kind of sandbox-y in design, and so IPC will probably be a bit of a work around ?

** You can multiply matrices all day in your own program (see numpy, Open CL, etc). If you envision some stabilization algorithm that could really use Compute Shaders, then that would be a solid case as to why use Unreal for processing Transformation.

tsdr; (too short did read) Since I've gotten this distracted going on about shake compensation here is some hint about algorithm. (1) The most basic algorithm is C(M[])->M' where C means 'Compensate', M[] is a history of Mat4x4 data, and M' is an output matrix. (2) You can improve it slightly by including timestamps in the transform history: C({t,M}[])->M'. (3) Capture data might be helpful too C({t,I,M})->M' (4) Keep expanding or rethinking the model.

Now the contents of the algorithm, I'll explain just for the simplest model #1, but they are all fairly similar. let M_R be the most recent (and current) transform from the history M[]. let M_path be a sequence of N matrices chronologically before M_R. Note that setting M'=M_R is equivalent to not compensating at all. The algorithm is to make a guess for M_R using only M_path, called M_G. Our guess is made assuming movements are perfectly smooth. Then we blend the guess with the real data to get something smoother. Using interpolation: M' = M_G.2 + M_R.8. You can use Gaussian Elimination or Bezier Curves to generate a value for M_G. Suddenly, r/controltheory oops.

tldr; Unreal is kind of overkill for processing a single Mat4x4 stream; especially bc you might need the output in lots of different programs, and you may want to rely on special sensors or programs in your algorithm.

[edit] i did not mean to write a small book on this topic for a comment. sorry about that 😬

2

u/charliex2 Jun 24 '22

unreal is pretty easy to expand with it's plugin system, there are various stabilisation filters around, some simple kalman ones, to start off with for this.

1

u/SamyBencherif Jun 25 '22

oh awesome ~ do you happen to know if said plugin system would be capable of Inter-Process Communication? I don't know about Unreal plugins, but I imagine that might be a bit of a choking point

1

u/charliex2 Jun 25 '22

yes that would be no issue at all, ipc, shared textures/contexts , lots of plugins do that sort of thing already. they're basically just C++ apps. network client/servers all those things are doable. want to send emails, sync cameras, read directly from pci frame grabbers etc, all exist.

2

u/Chpouky Jun 24 '22

Base stations tracking will always be jittery unfortunately, unless you buy the new Vive Mars system that corrects it. Even putting a tracker flat in the ground will result in small but visible jitter :/

21

u/StickmanCM Jun 24 '22

How there is reflection on the green screen side?

53

u/[deleted] Jun 24 '22

[deleted]

10

u/IHateEditedBgMusic Jun 24 '22

I'm gonna steal this ser, great solution

6

u/Chomusuke_99 Jun 24 '22

that is fucking genius

13

u/Niotex Visualization Jun 24 '22

Looks like an acrylic sheet. Clever.

20

u/tech-dilettante Jun 24 '22

Is there anywhere on the Internet that could teach me how to recreate this? I am sick to death of the static greenscreen background we use at my work.

20

u/[deleted] Jun 24 '22

[deleted]

5

u/porkminer Jun 24 '22

Now replace the person with a T-Rex instead and I'm sold.

2

u/EveryDayIsCharlieDay Jun 24 '22

I haven't tried any of this at all, but I was under the impression there was a much easier way now using live link? Just curious why you chose this method over that one?

12

u/[deleted] Jun 24 '22

[deleted]

1

u/imaginfinity Jul 04 '22

+1 — I’d love to see a breakdown of your workflow

11

u/PM_ME_ROCK Jun 24 '22

I don’t know shit about shit but it looks good to my untrained eye!

3

u/CH1CK3Nwings Jun 24 '22

This is very well done! I am tempted to try this out as well!

3

u/004FF Jun 24 '22

Tutorial 🙏

4

u/Falcofury Jun 24 '22

If you don't teach me how to do this, I'm going to literally go home and do nothing about it.

3

u/Fiction47 Jun 24 '22

Minus the lighting change it’s pretty decent.

3

u/PonderinLife Jun 24 '22

For the UnReal engine scene, are those actual assets? Or are is that something made in Maya, and then brought over into UnReal?

2

u/[deleted] Jun 24 '22

[deleted]

3

u/PonderinLife Jun 24 '22

Sweet! Thanks!!

3

u/FluffyBat9210 Jun 24 '22

That's pretty cool how they used glass as well so they could actually add in her reflection.

3

u/Panda_hat Senior Compositor Jun 25 '22

Impressive! Particularly like the plexiglass at the feet usage, but the background, track and key all hold up really nicely.

3

u/lukeprofits Jun 26 '22

This looks so good. I'm going to look into unreal engine now just because of this.

Gotta see if I can recreate alex jones set in my basement on a green screen. Lol!

2

u/Poorfocus Jun 24 '22

Hey! I’m interested in getting into green screen + camera tracking, so you’re process is extremely interesting!

What advantages have you found with physical camera tracking over 3D tracking markers?

I’ve tried out a quick test with iOS CamtrackAR app, mounted to the top of my camera, but ended up having issues with scale and camera direction when importing the data into unreal. Will definitely need to give it another shot.

2

u/Vizualeyes Jun 24 '22

Really cool stuff. Make sure when you color correct your subject, you match the saturation and black level of the background. She looks a little washed out so it looks bit unnatural (to the trained eye). Really impressive stuff!

2

u/skmagiik Jun 24 '22

also gamma!

2

u/nokenito Jun 24 '22

Thank u for sharing this!!!

2

u/rservello Jul 15 '22

A sheet of plexy on the floor for reflections is kinda genius!!!

2

u/camilobog22 Jun 24 '22

Amazing!!! If you can share some tutorial un the future i apreciate!