Category: technical

  • Mixed Reality With Quartered Screens

    There are a couple of reasons you might not want to use the in-game webcam system (which is easier to set up). For technical reasons the in-game system only supports webcams so if you are using a camera with a capture card you can’t use it. This is unfortunate because webcams look terrible when the subject is more than a meter or two away from the camera. You might also be compositing the layers together after-the-fact in which case you will need to use this method to get the foreground/background split into two.

    1. Game Setup

    Game setup is pretty easy:

    • Start the game holding shift while opening the game in steam. This will open the resolution options. Set the game to open full screen at the resolution of your  monitor.
    • Open the “Settings” menu (you might have to hit the eye button to open the left-menu if it’s closed
    • Turn on “Enable director controlls”
    • Pick “Quatered Views” in the Camera dropdown
    • Close the settings menu

    2. OBS setup

    OBS (Open Broadcasting Software) is popular streaming software and is pleasently flexible. Weirdly OBS makes it kind of hard to export settings files so I’m going to walk you through setting it up for Fantastic Contraption. It’s good to know how this works anyway so you can make any changes you might decide you want to make. First download OBS if you don’t have it already (download the Studio version). Then open OBS.

    Note that since the game is broken into four quadrants your output resolution will be half of whatever your screen resolution is. I’m going to do this totorial with a 4k monitor which will let me stream at 1080 but you don’t need a 4k screen to stream the game. Just adjust the numbers to whatever resolution your monitor runs at.

    First let’s add the game background:

    • Hit the “+” under “Sources” and select “Window Capture”. Name the layer “Background”selectWindowCapture
    • Select “Fantastic Contraption” in the “Window” dropdown
    • Flip off “Capture Cursor”
    • Hit “ok”
    • Right click on “Background” and select “Filters”
    • Hit the “+” on the Filters window and select “Crop”
    • Set the “Right” to “1920” (or half of whatever your monitor width is)
    • Set the “Bottom” to “1080” (or half of whatever your monitor height is)cropValues
    • Hit close
    • Grab the little red circle at the bottom right and resize the layer to fill the whole canvas area

    Now let’s add the webcam

    • Hit the little “+” under “Sources” again and select “Video Capture Device”. Name the layer “RL Camera”
    • Under “Device” select your camera
    • Hit “ok”
    • Resize the camera layer to fill the whole canvas area
    • Right click on “RL Camera” and select “Filters”
    • Hit the “+” under “Effect Filters” and select “Color Key”. Name it “Green Screen”
    • Fiddle with “Similarity” and “Smoothness” until you get the desired effectcolourKey
    • Hit close

    Lastly, let’s add the game foreground:

    • Hit the “+” under “Sources” and select “Game Capture” name the layer “Foreground”
    • Unslect “Capture Any Foreground Application”
    • Select “Fantastic Contraption” in the “Window” dropdown.
    • Check “Allow Transparency”
    • Flip off “Capture Cursor”foregroundOptions
    • Hit “ok”
    • Right click on “Foreground” and select “Filters”
    • Hit the “+” on the Filters window and select “Crop”
    • Set the “Left” to “1920” (or half of whatever your monitor width is)
    • Set the “Bottom” to “1080” (or half of whatever your monitor height is)
    • Hit close
    • Grab the little red circle at the bottom right and resize the layer to fill the whole canvas area

    OBS setup Done!

    3. Camera Sync

    • The last thing you have to do is sync the in-game camera with the real world camera. This means moving the in-game camera to where the real-world camera is and setting the in-game camera Field of View to match the real-world camera. This is all done very manually.
    • Make sure the game and OBS are both open and the game has the Depth Cameras turned on
    • Now open the OBS file menu and select “Always On Top”
    • Give the game focus and use the wasd keys (as well as q & e for up & down) to fly the camera around. Use the arrow keys to change the angle the camera looks.
    • This will be much easier with another person to help you. To get a rough idea of where the camera should be have the other person go into VR. They will see an insect with a big eye and wings, that’s your camera. Have them direct you, move the in-game camera to where the real-world camera is.

      Using hand controller positions to align the cameras
      Using hand controller positions to align the cameras
    • Then have them pick up the two controllers and walk to the back of the play area. Get them to hold out their arms in a T-pose and then not move. Using the arrow keys change the angle of the camera until the controllers are as close to matching the real-world controllers as possible.  Note that the FOV is still off so you won’t get a good match.
    • To match the FOV make the OBS window small enough that you can use the game’s settings menu again. Open the settings menu
    • Drag the “3rd person FOV” slider until the in-game controllers are the same distance apart as your friends (very tired) arms.
    • After setting the position and FOV set the “Camera Delay” slider. You will notice the camera lags behind the game as the player moves around. This slider will delay the game so it can match the delay of the camera. Drag this slider until the game and the player are in sync.
    • You are done setup! Because of lighthouse drift and camera jostling you will probably have to re-tune this every time you stream.

     

     

    Now go back to the previous instructions to learn the Director Controls.

     

     

  • Streaming and Recording Mixed Reality

    Welcome! We’ve spent a lot of time on the streaming tools for Fantastic Contraption. We were the first people to use these Mixed Reality techniques to show VR and we’d love for you to come join us in the game.

    maxresdefault (1)

    The instructions are broken up into three sections. Start with our first Mixed Reality Streaming How To. If you’re using a webcam then that’s all you need to know. If you’re using a fancier camera or want to do your compositing in post check out our Mixed Reality with Quartered Screens post and if you want to go full-badass and stream with a handheld camera we have Hand Held Instructions.

    Happy streaming!

  • How to Mixed Reality: Handheld

    I previously gave detailed instructions on how to stream Fantastic Contraption in mixed reality with a stationary camera.

    People are starting to pick that up and it’s very exciting! Fantastic Contraption also works with a hand held camera and in this post I’ll show you how to use it. To record hand-held video you do need a third controller though, which is going to be a big barrier for most people.

    If you have a third controller and already have the stationary camera stream set up then hand-held setup is pretty simple:

    1. get a third Vive controller (this is the hard part)
    2. duck tape that controller to your camera and plug it into the USB on your computer
    3. know where to put the camera/controller when you start hand-held mode
    4. turn on mixed reality and then hand-held mode in Fantastic Contraption
    5. align the cameras

     

    cam1. Get a Third Controller

    OK, yeah, this is pretty hard right now. Valve has said that they’re working on a way to get a third controller and also working on tracking devices that are designed to be added to real-world objects to make them trackable. At the moment the only way I know of to get a third controller is to borrow one from a second Vive. I have no other suggestions, sorry.

    2. Duct Tape

    The third controller tells the game where your camera is so the first thing you have to do is meld your controller to your camera. Make the coupling super solid so it doesn’t drift over time. Use duck tape, zap straps, putty, glue, whatever makes it nice and solid. Also make sure the sensors on the controller will be visible to the light houses.

    Now plug the third controller into the USB of your computer. Third controllers can not be used wirelessly like the controllers that come with the Vive. It will only be recognised if you plug it into a USB port on your computer. You’re probably going to have to buy a USB extension cable.

    cam2

    3. Why is my Hand the Camera?

    The game knows you have three controllers but it doesn’t know which one is tied to a camera. To work around this I assume that the camera controller is the one that is most “Vive East” when handheld mode is turned on. Vive East is the direction AWAY from the pink goal in-game. If you did room setup like Valve suggested then it is in the direction to the right of your monitor.

    4. Turn on Hand Held

    Go to the “Settings” menu and turn on “Handheld mode”. This will take the mixed-reality cameras and move them to where the third controller is.

    5. Align the Cameras

    You need to align the in-game and real-world cameras like in the previous guide. Use the wasdqe and arrow keys. Note that you can “roll” the camera with shift arrow.

    It’s also very important to get the camera delay correct when filming hand-held or the whole world will be swimming around.

     

  • Mixed-reality VR Twitch streaming

    Mixed-reality VR Twitch streaming

    It’s as cool as it sounds. We’ve started live streaming Fantastic Contraption on our Twitch stream every Thursday at noon PST. Here’s this past week’s stream:

    Twitch isn’t just for e-sports and speedruns anymore; it’s getting downright mainstream as a marketing tool, a way for people to check games out before buying them, participate in events, and to obsess over games while at work / any moment they can’t be playing them (guilty!). But for months we’ve been asking: how the hell do you stream virtual reality games? Especially room-scale VR using the HTC Vive?

    The standard picture-in-picture game footage + webcam technique doesn’t do VR justice. The first-person in-game feed from VR games gives at best a cropped, distorted view of what the player is actually seeing, and talking heads wearing VR headsets are even duller than regular talking heads. After an hour-long session with Youtuber Northernlion, we did some brainstorming.

    Then we geared up:

    400sqft of green muslin blankets our livingroom
    400sqft of green muslin + 2 layers of black, a 2000 watt light kit, Logitech C930e webcam and Blue Yeti mic, total: $1000 cad

    Our livingroom has huge windows on two sides, so it was a challenge to keep the green screen lighting consistent (bedsheets and cardboard were involved). But we discovered that our webcam feed has considerably less lag during the day when all that natural light lowers exposure time.

    Syncing: we move the in-game camera to line up real and virtual hand positions
    Our first trials used OBS to combine three views. We stuck a webcam on a tripod and synced it’s position with two in-game 3rd-person cameras. One only saw foreground objects, and the other only saw the sky, ground, and objects behind the headset. We first tried using a clipping pane, then tried blipping game objects between two visibility layers.

    We output the in-game cameras side-by-side then smushed the 3 feeds together in OBS:

    streaming_obs_600

    It’s not half bad without the green screens too, if you overlay the background camera at 50% transparency.

    But for our next stream we’re going to try piping the live webcam feed into Fantastic Contraption, so we can display it on a moving plane in the game. This should give us fewer blipping glitches and a higher output resolution. Thanks to Edwon for the suggestion and help!

    Lindsay builds, Colin and Gord advise
    We’ve got some in-game tools to use while streaming, like a floating Twitch comments feed that only the player can see, and director controls that let our “couchies” swap the view between various game cameras.

    We’ll keep things fresh by bringing on special guests, and will be reaching out to local Vancouver Twitch streamers to come stream from our rad green screen studio (aka our livingroom). Stay tuned, Thursdays at noon PST!

  • Setting up Subversion on Amazon EC2 for free

    Remote backups are crazily important for Colin and I with our nomadic lifestyle. We consider ourselves very lucky that we haven’t had our laptops stolen or destroyed in 5 years of travel. We take precautions: we lock our doors, keep our laptops out of sight, and try to give off an air of “thrifty backpackers” while actually carrying around $6,000 worth of electronics. But if it should ever happen, we could be up and working again within 24 hours thanks to offsite backups.

    I use Dropbox for large files and Google Drive for documents (both free to a point), and for code I use offsite version control software. There are free hosting services for Git… but after working with it on several projects over many years I still don’t fully understand Git and consistently screw up commits. I am much more comfortable with older, simpler Subversion. But there are no free or even reasonably-priced hosting options for svn projects unless you make them open source.

    However, Amazon is offering free linux EC2 servers with 30gb ssd hard drive space. For a relatively small project like Rebuild with only a couple contributors and few branches, that’s more than enough, and the equivalent hosted option costs $50/month or more. The trick is configuring it. This took me a few hours, some false starts and useful tutorials. You do need to be familiar with Linux. I documented the process for my own uses, but here it is for you:

    Creating an EC2 instance and SSHing in

    Create a new EC2 instance
    They won’t charge you unless you go beyond their free tier
    I chose to put mine in Oregon (us-west-2)
    Check “only free” and pick a freebie amazon linux ami
    I used amzn-ami-hvm-2014.09.1.x86_64-ebs (ami-b5a7ea85)
    Next page, choose max free (30gb) ssd
    Next page, open HTTP port to all (0.0.0.0) – SSH is already open by default
    Complete and launch your instance

    Create a new key pair when prompted and download the .PPK file from amazon.
    Visit the EC2 dashboard, instance should be Running and status checks 2/2
    Record your instance IP (YOUR_INSTANCE_IP)
    Download Putty
    SSH into your instance using the PPK from amazon (guide)
    Default amazon AMI SSH username is ec2-user, no password
    ec2-user@YOUR_INSTANCE_IP

    You’re in! Putty hint: right-click to paste clipboard contents.

    Installing software

    Update pre-installed software:
    # sudo yum update -y

    Visit the public ip in your browser: http://YOUR_INSTANCE_IP
    should see Amazon Linux AMI Test Page if Apache is installed and running

    If Apache is not installed (guide):
    # sudo yum groupinstall -y "Web Server" "MySQL Database" "PHP Support"
    # sudo yum install -y php-mysql
    # sudo service httpd start

    Install subversion and mod_dav_svn (should see a long list of all changes):
    # sudo yum –y install mod_dav_svn
    # sudo yum –y install subversion

    Edit the Apache configuration file for subversion:
    # sudo vi /etc/httpd/conf.d/subversion.conf
    Replace any subversion.conf content with:

    LoadModule dav_svn_module     modules/mod_dav_svn.so
    LoadModule authz_svn_module   modules/mod_authz_svn.so
    <Location /repos>
       DAV svn
       SVNParentPath /var/www/svn
       # Limit write permission to list of valid users.
       AuthType Basic
       AuthName "Authorization Realm"
       AuthUserFile /var/www/svn-auth/passwd
       AuthzSVNAccessFile  /var/www/svn-auth/access
       Require valid-user
    </Location>
    

    Create the directory which will contain the subversion repository:
    # sudo mkdir /var/www/svn

    Create the directory which will contain the permissions files.
    # sudo mkdir /var/www/svn-auth

    Create the permission file:
    # sudo vi /var/www/svn-auth/access
    And fill it with (replace sarah, colin, guest with your usernames):

    [/]
    sarah = rw
    colin = rw
    guest = rw
    

    Create and add to the password file (use -c the first time to create)
    # sudo htpasswd -cb /var/www/svn-auth/passwd sarah SARAHSPASSWORD
    # sudo htpasswd -b /var/www/svn-auth/passwd colin COLINSPASSWORD
    # sudo htpasswd -b /var/www/svn-auth/passwd guest GUESTSPASSWORD

    Create a repository (REPONAME is the name of your repository eg rebuild):
    # cd /var/www/svn
    # sudo svnadmin create REPONAME

    Change files authorization (again after creating new repos too):
    # sudo chown -R apache.apache /var/www/svn /var/www/svn-auth
    # sudo chmod 600 /var/www/svn-auth/access /var/www/svn-auth/passwd

    Start apache web server:
    # sudo service httpd restart
    Will complain about determining domain and using 127.0.0.1, that’s ok

    To make sure apache always starts on boot:
    # sudo chkconfig httpd on
    # sudo chkconfig --list
    Should show 2:on 3:on 4:on 5:on for httpd

    Verify the subversion repo by opening in a browser:
    http://YOUR_INSTANCE_IP/repos/REPONAME

    You’re done! Connect via Tortoise or your fav svn client using the url above.

    Other operations

    To copy from an older repo including revisions:
    # sudo svnadmin dump /var/www/svn/REPONAME > /tmp/REPONAME.svn
    (copy the file to the new server then)
    # sudo svnadmin load /var/www/svn/REPONAME < /tmp/REPONAME.svn

    To connect a backup mirror on another (non-free) EC2 server with the same setup (guide)
    First make revisions editable in the mirror repo:
    # sudo echo '#!/bin/sh' > /var/www/svn/REPONAME/hooks/pre-revprop-change
    # sudo chmod 755 /var/www/svn/REPONAME/hooks/pre-revprop-change
    Then initialize the mirror from the old one:
    # sudo svnsync init file:///var/www/svn/REPONAME http://YOUR_INSTANCE_IP/repos/REPONAME
    Should see "Copied properties for revision 0."
    Then copy the data including all revisions:
    # sudo svnsync sync file:///var/www/svn/REPONAME
    Can use this to make nightly backups to another server