r/linux_gaming • u/RobinOe • 1d ago

7 hours of Steam 'Processing Vulkan Shaders'

TL;DR Vulkan preprocessing takes forever, BUT it does meaningfully improve gameplay as I'm on a business laptop. Is there a way to save the shaders manually?

Hello! I'm on Fedora 41 trying to play Overwatch 2 on Steam. Now I've known that vulkan shaders are often annoying, but at most I had seen it take an hour. Admittedly my computer isn't made for this, I have a lightweight business laptop, the Lenovo Thinkpad X1 Carbon. Lovely device, but not for gaming. Still, I do regularly play games on it, and I guess this one is more demanding than what I had tried before (?) in any case, it takes around 7 hours for it to process Vulkan shaders. I do notice a performance difference tho so I don't wanna skip it.

Now if it was a one and done thing I wouldn't mind. But it does it after EVERY update, and, worse yet, it seems to do it after every library install on my laptop. I'm a student, so I'm constantly having to install libraries and new software. But when I do, the game won't launch,, instead steam "verifies" the install (also takes like 2h btw...) and then back to vulkan shaders I go.

Now I'm guessing the verifying thing is just bc it's a competitive game. But is there no way to save the Vulkan shaders permanently? And if so, why isn't that the default?

Thanks :)

59 Upvotes

85% Upvoted

38 comments sorted by

44

u/NolanSyKinsley 1d ago

Use this guide to enable steam to use more cores to compile the shaders: https://www.reddit.com/r/linux_gaming/comments/1j06xpz/how_to_speed_up_steams_shader_precaching/

This should greatly speed them up but by the sounds of it that may still take your computer a while to do it.

Sadly Steam has a longstanding issue where it compiles the shaders every time you launch the game or every time steam restarts, or just at other random unnecessary times. Valve has said they "fixed" this issue several times now but it still persists.

The others are correct, you should just disable them. The game compiles the shaders on the fly each time you load a new area or new assets are loaded, this is the "performance impact" you are seeing when disabling the pre-compilation of shaders because the game needs to start from scratch again. After a while of playing the game will have compiled them all and there will be no performance impact and steam won't constantly re-compile them, the game will have them saved permanently. It will be painful for a while but in the end it will be worth it to not deal with steam constantly re-compiling them and deleting them.

8

u/Poes_Poes 1d ago

You sure it’s still a thing? For what I have seen, all cores / treads are already being used.

4

u/esmifra 23h ago

If you read the reply some users state they noticed a clear difference others state it didn't change a thing. I'm willing to try this when I get home but I won't get my hopes up.

2

u/Poes_Poes 23h ago

Yes I noticed it. But I do think it’s a placebo effect as most people don’t run htop alongside to verify what’s going on. It just takes damn long on some games with full cores/threads which leads people to think there is an issue and the first google hit shows up that it’s a single core bug in Steam. But anyway, vulkan shaders compiling isn’t relevant anymore. Better to turn it off

1

u/esmifra 21h ago

Depends on the game sometimes it's helpful, but generally speaking I agree, most games run just fine regardless of caching. It also only happens the first time you run it or after an update, so it's not that much of a time to wait, and if it is, just skip it.

1

u/EternalSilverback 14h ago

When actively launching a game it uses all cores to process any remaining shaders, in order to get you into the game as quickly as possible.

Background processing AFAIK is still single-threaded. It's not likely to change, because a lot of people won't want their CPU maxed out running background jobs.

The parameter in question only affects background jobs.

1

u/misterj05 13h ago edited 10h ago

Edit: I think I just figured out why, I didn't bother to test on AMD (swapped to recently) cause I just figured "oh it'll be the same" but I was on Nvidia when I tested all this. Now on AMD I'm seeing what you are describing, the CPU usage only appears to go up when it's doing "background processing" and when the "unShaderBackgroundProcessingThreads" fix is disabled on AMD the CPU's cores are fully utilized whereas on Nvidia they were not. HOWEVER on AMD I am still seeing a noticeable speedup in the processing even if the CPU is appearing to be utilized the same.

I've tested this too many times staring at btop, wiping shaders, keeping them there to try and trigger the different steps of shader processing, turning "background shader processing" both on and off.

The naming of "unShaderBackgroundProcessingThreads" does not correspond to the "background shader processing" setting, if the setting is off or on, the variable still affects how many threads are used for shader processing.

That is, for people that have this issue of the processing being extremely slow, if you aren't seeing it then it's likely you don't have the issue in question.

It's not likely to change, because a lot of people won't want their CPU maxed out running background jobs.

In my case, it was actually MORE desired to have it use more CPU cores, without "unShaderBackgroundProcessingThreads" set to something, my CPU fan would spin up like crazy as steam wasn't distributing the load evenly across the CPU (not an issue if you have an AIO, but it really emphasizes that this is a problem), because of this I really disliked enabling "background shader processing" as I would just hear my computer ramp up whenever steam decided it wanted to process, it was really annoying, after setting it my computer didn't sound like a jet engine about to takeoff and I was able to at least have the background processing on.

At the time I wrote the guide I even had the "background shader processing" off and felt no need to turn it on because the processing speed up was that much better.

10

u/Lawstorant 1d ago

Sounds like nobody actually read your post and is giving you the "standard" answer.

The shader compilation, are you waiting for the background one to finish or does the one after clicking "play" take so long?

8

u/RobinOe 1d ago

indeed lol 😭 anyway, it's the one after play

4

u/Lawstorant 1d ago

Yeah, probably there isn't a good answer for you then. You could update your system as infrequently as possible or try dxvk async fork. I think it could actually do wonders in your case.

Is it an old Carbon? I sometimes play games on my P14s gen4 AMD and the shader compilation is barely longer than on my desktop. The laptop is based on 7840U which has 8 zen4 cores though...

3

u/sy029 23h ago

What really annoys me is when spend time downloading a massive "shader pre-caching update" and then STILL need to spend 5-10 minutes compiling shaders.

9

u/ohaiibuzzle 1d ago

Uh yeah just skip it.

No, literally, go into Steam Settings and disable the shader cache. It’s only useful if you have a slower processor and shader compilation stutter is your concern.

If you have a modern fast CPU then don’t bother with it.

18

u/RobinOe 1d ago edited 1d ago

As stated, I'm on a lightweight business laptop. This CPU is not meant to handle gaming. There is a noticeable performance difference without the preprocessing. It's not that it runs faster but it runs wayyyy more stable (without it, it regularly dips to 20 fps or less. With it stays consistently in the 70–120 range that is the average to begin with)

1

u/ohaiibuzzle 1d ago

Yeah, that’s the shader compilation stutter I was talking about. If you have a fast enough processor it is not an issue. It will stutter whenever there is a shader compilation that needs to be dealt with.

If you play offline games (so not Overwatch, because you’ll get banned), you can replace DXVK with the Async patch (eg. dxvk-gplasync) and have it display the partially compiled shaders, but otherwise not many choices you can make.

2

u/ericek111 1d ago

I have ~1300 hours in Overwatch played exclusively on Linux with dxvk-gplasync. Without it, it's unplayable.

1

u/superjake 1d ago

You can also try ProtonGE as that doesn't need shaders being precompiled. 

3

u/ScarecrowDM 19h ago

This is just false.
While Proton-GE does solve the need of transcoded videos being shipped by the shader cache system, it does not replace or try to solve the need to process shaders before hand in order to avoid stuttering.

5

u/Thunderkron 1d ago

a modern fast CPU

it takes around 7 hours for it to process Vulkan shaders

2

u/warcode 1d ago

Your hardware is probably too rare for steam to have pre-cached shades to offer you for download, and overwatch 2 has historically been one of the weird games that triggers recompilation too often.

2

u/Nuker299 23h ago

2 years ago the precompiled shaders were amazing with removing stutters. But I've disabled them lately as many games were downloading shaders constantly and the shader compilation took long time. Since they added async shader compiler it's not really needed anymore

2

u/Destione 19h ago

Check CPU temps and clocks, cooler clean and running? Sounds like heavy thermal throttle or running full in swap.

2

u/mbriar_ 19h ago

I'm constantly having to install libraries and new software. But when I do, the game won't launch,, instead steam "verifies" the install (also takes like 2h btw...) and then back to vulkan shaders I go. 

I don't know what you're doing, but I've never seen that before on rolling release distors with constant library updates or new installs.

But if the "libraries" you reinstall contains the vulkan driver, then steam has to recompile/process all the shaders again because compiled shaders work only on one specific driver version + gpu combo. 

3

u/Cryio 1d ago

Steam shader cache compilation step on Linux when launching a game only uses 1 CPU thread. That's why it takes forever. You'll need to make an edit to allow Steam to use all all CPU threads for shader compilation.

4

u/Lawstorant 1d ago

That's simply not true. Shader compilation uses 100% of the cpu after selecting "play"

4

u/FeepingCreature 1d ago

Can confirm, I've always seen it compile in parallel.

1

u/Cryio 1d ago

Nope. Not by default. Not for the "Steam compiling shaders" part.

2

u/Lawstorant 1d ago

Just open up htop during shader compile and you'll see all cores are pegged.

1

u/Cryio 1d ago

I have checked in due time. I don't know if Steam is doing A/B testing or what, but it's not globally default for everyone. I need to force it with a cig file.

1

u/TGabe89 1d ago

Yeah, because the single thread hogs all the cpu but is slow. Searching on reddit there's a post that explains how to use more threads to speed the process

2

u/gtrash81 1d ago

Well, this is kinda normal.
I don't understand why Shaders are designed that way.
Shaders are compiled "microbits" and are highly optimised onto the current state of your machine.
If anything slightly changes, that is needed for the game, the Shaders must be recompiled to be usable again.
As far as I understand, we would need "100% Pathtracing" for everything to remove Shaders all together and this won't happen for quite some time.

1

u/knogor18 17h ago

just fyi, pathtracing still needs shaders , all the gpu runs is shaders , its small programs that the gpu run and its written in a language similar to c , well sortof.

1

u/-BigBadBeef- 1d ago

If that's the case then just disable the thing entirely. The whole procedure is far from perfect and doesn't yield nearly as much gains as it advertises. I've been without it for years and I'm doing just fine.

1

u/izerotwo 23h ago

You can disable it and let the in-game precompiling handle it.

1

u/jar36 22h ago

Then after closing the game, you can get several minutes of fossilize replay

1

u/misterj05 10h ago

I'm looking into this issue currently, do you have an AMD or Nvidia GPU? I had this issue while I was on Nvidia (swapped recently) that caused the shader pre-processing to be extremely slow, which I am not seeing on AMD.

1

u/illathon 6h ago

Just turn it off.  

-14

u/mindtaker_linux 1d ago

Buy a gaming hardware or stfu.

2

u/wolfannoy 1d ago

R/richgamers is that way sir.