Mixed-reality VR Twitch streaming

It’s as cool as it sounds. We’ve started live streaming Fantastic Contraption on our Twitch stream every Thursday at noon PST. Here’s this past week’s stream:

Twitch isn’t just for e-sports and speedruns anymore; it’s getting downright mainstream as a marketing tool, a way for people to check games out before buying them, participate in events, and to obsess over games while at work / any moment they can’t be playing them (guilty!). But for months we’ve been asking: how the hell do you stream virtual reality games? Especially room-scale VR using the HTC Vive?

The standard picture-in-picture game footage + webcam technique doesn’t do VR justice. The first-person in-game feed from VR games gives at best a cropped, distorted view of what the player is actually seeing, and talking heads wearing VR headsets are even duller than regular talking heads. After an hour-long session with Youtuber Northernlion, we did some brainstorming.

Then we geared up:

Our livingroom has huge windows on two sides, so it was a challenge to keep the green screen lighting consistent (bedsheets and cardboard were involved). But we discovered that our webcam feed has considerably less lag during the day when all that natural light lowers exposure time.

Our first trials used OBS to combine three views. We stuck a webcam on a tripod and synced it’s position with two in-game 3rd-person cameras. One only saw foreground objects, and the other only saw the sky, ground, and objects behind the headset. We first tried using a clipping pane, then tried blipping game objects between two visibility layers.

We output the in-game cameras side-by-side then smushed the 3 feeds together in OBS:

It’s not half bad without the green screens too, if you overlay the background camera at 50% transparency.

But for our next stream we’re going to try piping the live webcam feed into Fantastic Contraption, so we can display it on a moving plane in the game. This should give us fewer blipping glitches and a higher output resolution. Thanks to Edwon for the suggestion and help!

We’ve got some in-game tools to use while streaming, like a floating Twitch comments feed that only the player can see, and director controls that let our “couchies” swap the view between various game cameras.

We’ll keep things fresh by bringing on special guests, and will be reaching out to local Vancouver Twitch streamers to come stream from our rad green screen studio (aka our livingroom). Stay tuned, Thursdays at noon PST!