#MICROSOFT BACKGROUND SCREENING TEAM FULL#
#MICROSOFT BACKGROUND SCREENING TEAM SOFTWARE#

From the speaker/content creator side, you will need 1 or 2 Azure Kinect sensors (plus applicable hardware/software requirements as listed).How can you use this project for your own events/videos? We also refine through an unsupervised GAN to improve the results even more. So what we input to this first model is the sensor information (both the IR and silhouette as well as the video without any processing) and the background without the speaker (captured from the front-end app by the user) - giving us a more precise foreground and alpha estimation as an output. The first step is done by a deep network that estimates the foreground and alpha from input comprised of the original image, the background photo, and an automatically computed soft segmentation of the person in frame.īy combining this approach with the Azure Kinect API, we can replace the automatically computed soft segmentation of the person in frame with the more precise silhouette captured with our sensor. The UW approach proposes 2 steps: – first, extract the background based in supervised learning and second, refine the output in an unsupervised way through a GAN. This is where the capabilities of Azure Kinect have helped. Yet, that still doesn’t solve for the lack of precision when it comes to recognizing the legs. To solve the bounding box issue, we split the image in two.

#MICROSOFT BACKGROUND SCREENING TEAM FULL#

This limitation not only means that legs are not properly processed by the model due to lack of training data, but also that the neural net output is a square bounding box – perfect for just upper-body images but not ideal for a full body. One of the key challenges we tried to solve in our implementation is that the dataset used for training (Adobe Composition-1k dataset) contained only upper body images - and we wanted to capture our presenters in long shots. This process leverages the approach by the University of Washington team we added improvements so we can utilize the depth recording from the Azure Kinect sensor to remove the background more precisely. The output of this process is the video with just the presenter (the rest being a virtual green screen).

#MICROSOFT BACKGROUND SCREENING TEAM SOFTWARE#

Once the user provides the video files, we need to create the virtual green screen so that we can edit in the software of our choice. This makes the process easier for the post-production team to ensure the best possible results in a remote collaboration scenario. We have also included in the front-end app an easy wizard to allow the user to record the background without them, in order to solve for the foreground and alpha value of the matting process. The app allows the presenter to review multiple recordings and keep the best ones before sending to the back-end process to generate the virtual green screen.

It can record 3 video files from a single recording: up to two videos from Azure Kinect sensors and one separate file for the presenter’s slides with audio, to make post-production easier. The presenter uses the app to record themselves.

This project consists of 2 different elements:

Microsoft background screening team