Mixed-reality VR Twitch streaming

It’s as cool as it sounds. We’ve started live streaming Fantastic Contraption on our Twitch stream every Thursday at noon PST. Here’s this past week’s stream:

Twitch isn’t just for e-sports and speedruns anymore; it’s getting downright mainstream as a marketing tool, a way for people to check games out before buying them, participate in events, and to obsess over games while at work / any moment they can’t be playing them (guilty!). But for months we’ve been asking: how the hell do you stream virtual reality games? Especially room-scale VR using the HTC Vive?

The standard picture-in-picture game footage + webcam technique doesn’t do VR justice. The first-person in-game feed from VR games gives at best a cropped, distorted view of what the player is actually seeing, and talking heads wearing VR headsets are even duller than regular talking heads. After an hour-long session with Youtuber Northernlion, we did some brainstorming.

Then we geared up:

400sqft of green muslin blankets our livingroom
400sqft of green muslin + 2 layers of black, a 2000 watt light kit, Logitech C930e webcam and Blue Yeti mic, total: $1000 cad

Our livingroom has huge windows on two sides, so it was a challenge to keep the green screen lighting consistent (bedsheets and cardboard were involved). But we discovered that our webcam feed has considerably less lag during the day when all that natural light lowers exposure time.

Syncing: we move the in-game camera to line up real and virtual hand positions
Our first trials used OBS to combine three views. We stuck a webcam on a tripod and synced it’s position with two in-game 3rd-person cameras. One only saw foreground objects, and the other only saw the sky, ground, and objects behind the headset. We first tried using a clipping pane, then tried blipping game objects between two visibility layers.

We output the in-game cameras side-by-side then smushed the 3 feeds together in OBS:

streaming_obs_600

It’s not half bad without the green screens too, if you overlay the background camera at 50% transparency.

But for our next stream we’re going to try piping the live webcam feed into Fantastic Contraption, so we can display it on a moving plane in the game. This should give us fewer blipping glitches and a higher output resolution. Thanks to Edwon for the suggestion and help!

Lindsay builds, Colin and Gord advise
We’ve got some in-game tools to use while streaming, like a floating Twitch comments feed that only the player can see, and director controls that let our “couchies” swap the view between various game cameras.

We’ll keep things fresh by bringing on special guests, and will be reaching out to local Vancouver Twitch streamers to come stream from our rad green screen studio (aka our livingroom). Stay tuned, Thursdays at noon PST!

3 thoughts on “Mixed-reality VR Twitch streaming

  1. Hi,
    That’s very cool idea.
    By using a floating plane to display the video in the game, you are going to run into a problem where everything that’s not keyed out will move with the headset

    Maybe use a strongly connected components (aka floodfill) algorithm to isolate the person wearing the headset from the rest of the people/coaches on the set.
    Alternatively, use a Kinect2 (or LIDAR) to position of each video pixel in your game space after keying them.

    Sincerely,
    Ben

  2. This is great, was super impressed when I saw it. Your setup is certainly on the right track with how VR on twitch should be displayed!

  3. Hi
    I thought I would add a small suggestion if I may?
    For chatting with your viewers you could use StreamTalker.com it ‘Speaks’ the chat text and announces arrivals using your TTS voice.
    Blatant self promotion but thought you should check it out and who knows maybe it will be useful for your VR Streaming :)

Leave a Reply

Your email address will not be published. Required fields are marked *