Artificial Intelligence and Cloud to Process Videos and Remove Backgrounds

Sometimes we record a video or take a photograph and the background is very distracting, or we just simply prefer to replace it with another one.

Over the last years, there have been advances in replacing backgrounds with other images, especially during video calls, popularized by remote work. And the progress continues: we recently learned that Google has been working to improve the processing of video calls on its Google Meet service with technology based in the Cloud. According to the company, effects such as background blur and light adjustment are processed in the cloud if they cannot be run in real-time on certain devices. This helps save battery and up to 30% CPU, according to the company.

This is just the latest example of how the Cloud and Artificial Intelligence (AI), are changing video processing and the extraction or changing of image backgrounds. But there’s more.

How to Remove Backgrounds from Video: Artificial Intelligence to Improve Background Shading and Extraction

AI helps to shade and remove video backgrounds with professional quality for all types of virtual presentations.

In this sense, Virtual Stage is a tool that takes advantage of the latest advances in AI and Azure Kinect (a recording device with voice models, AI sensors, and artificial vision, linked to the cloud thanks to Azure Cognitive Services) to reproduce human figures with a different background without losing definition. Specifically, Virtual Stage makes it possible to record participants in a conference or webinar as if they were in a conventional auditorium or TV studio. The technology makes it possible to separate the human figure from the background and place it on another ‘set’ or add image effects, presentations…

To remove video backgrounds, Virtual Stage is based on two technologies: Speaker Recorder App, which records the speaker with Azure Kinect equipment, and Background Matting, which removes the original background thanks to artificial intelligence and information from the Kinect.

Plain Concepts has improved Background Matting processing on delicate edge areas such as hands, feet and hair. For this, we have used AI models from the Azure Body Tracking SDK, resulting in more accurate separation in similar and ambiguous color areas. On the other hand, the human body is separated into two square images that are processed separately; this helps the AI to check areas such as the shadow that is generated between the feet or to dwell more on places that need more precision, such as hair and hands.

Improving Video Processing with Cloud

With the popularization of video cameras in many industries (security, industrial buildings, sporting events…), intelligent video analysis in real-time is also becoming popular, providing data to improve processes or obtain new forms of business. The cloud is used to host this large amount of data. At the same time, artificial intelligence creates ways to manage it, thanks to services such as Kubernetes.

Similarly, cloud computing can be used when there is not enough local processing capacity or bandwidth for recording, playback, or broadcasting video. This could happen, for example, when recording or broadcasting in 4K, a type of resolution of millions of pixels with which content is already being created for television and video-on-demand platforms.

In addition, by relying on the Cloud, not only is there less chance of technical failure but video can be stored with multiple backups. At the same time, cloud technologies allow all members of a team to access the video from their workplaces, homes, or other locations.

Cloud computing is also essential for storing and transmitting content from the many streaming platforms that exist. Transferring it to companies that also have to host videos at a more ‘humble’ level (e.g. distance education portals), the cloud for hosting videos is scalable: storage capacity can grow or shrink according to the company’s needs, in a process that should not affect the quality of the transmission.

Why Background Extraction and Video Processing Are So Important Today

Removing or changing the background of a video call has become a very common resource with the popularization of remote work: we do not want to show what is behind us in the room where we work, or we prefer to use a corporate image, with the company logo or the image of a product.

But it’s not the only reason. The popularization of 5G networks, which allow video to be transmitted at higher speeds and with better processing, means that related video calling technologies will have to improve to keep up. The volume of traffic will increase in numerous industries, from entertainment to telemedicine, and technologies such as Augmented Reality or the use of live artificial intelligence algorithms can be applied.

The Azure team has been looking for such integration between this cloud technology and 5G for some time. In fact, in November 2021, Microsoft and telecommunications company AT&T introduced Azure Public MEC, which provides the low latency characteristic of 5G integrated with Azure on mobile networks. Thus, developers can create low-latency applications as if they were in the Azure public cloud.

If you want your employees to make video calls with customers without losing the definition of their figure; you are looking for a technology with which to broadcast digital events with speakers; or you want to process and manage video from the cloud, we can help you: at Plain Concepts, we have experience in implementing these projects that, with the emergence of 5G networks, become even more attractive. Contact us and we will support you in everything you need.