"Virtual Studio" Concept at KST Moschkau

KST has equipped a virtual test studio at the their site in Kreuzau/Stockheim, Germany. What is important here, in addition to the proof of concept, is that there is a central theme that is traceable throughout the workflow – from the image acquisition through the rendering process, up to and including the directing and final production. As a result, we have created the following article that looks to answer the following questions:

• What is a Virtual Studio?
• How large should a virtual studio be?
• Which systems and components are necessary to set up a Virtual Studio?
• How must a studio be set up and illuminated to ensure optimal keying?
• What workstations are required by the Virtual Studio workflow?

What is a Virtual Studio?

In a Virtual Studio, a large part of the environment is entirely digitally generated, thus it is only virtually present. The environment, which can later be  visualised, can thereby be completely computer generated. The moderator and all the real elements, which are located in front of the green area, are automatically cut out live in the so-called Keying process and can be inserted at any desired point in the virtual environment. In addition to the virtual reality elements, augmented reality elements are also used. The intention here is that the elements actually appear spatially correct and interactive, giving the impression that they truly exist in the real world.

How large should a virtual studio be?

The computer-designed virtual environments are nearly unlimited in both their size and  nature. There is freedom of movement within them and the cut-out real image content can be positioned anywhere within.The real Green Area must be at least as large as the silhouette of the real objects located within it. In order to ensure that the keying works well, the entire outline of an object must be surrounded by the green background. The actual minimum size of the studio is thus only limited by the field of motion of the camera and the area in which the moderator can move freely around.

Which systems and components are necessary to set up a Virtual Studio?

Assembling

Fundamentally, a suitable studio camera is necessary to capture the actual unaltered images. In our environment we use the AK-UB300 from Panasonic. When the camera is to be moved, this must be communicated to the rendering system precisely and in realtime and therefore a tracking system is required. In our environment we use an AR+ UR10 robot arm for directly handling the tracking in addition to precise camera movement.

Rendering

In addition to the real image, the virtual must be computed. A rendering engine, in this case the AVID HDVG+, first cuts out the background from the real image by means of keying. Next, the exact position and camera angle in the virtual space are computed using the tracking data. Aided by AVID Maestro, additional on-air graphics and clips for the virtual background screens are run. In conclusion, the rendering engine relays the finalized rendered image, i.e. the combination of real and virtual elements, to the Director.

Director/production

In the Director – the central nervous system of the studio – all image threads and events come together and are fed live into the image mixer, in this case  a Panasonic AV-HS6000, and then mixed. It is also the responsibility of the Director to issue instructions to the crew in order to properly control the sequence. Additionally, from here the virtual background screens can be run live together with content input.

How must a studio be set up and illuminated to ensure optimum keying?

It is essential that the green area has a unicoloured background and even illumination. In order to prevent errors, such as the cutting out of clothes for examplefrom occurring, we have selected a  special media shade of green as the background colour. This is one of the colours least likely to occur in natural environments, thus avoiding incorrect cutouts. The area must be illuminated as uniformly and continuously as possible. Additional spotlights help avoid shadow-casting, such as in the face of the moderator. The green area itself should not have any edges; they would lead to shadows.

What workstations result from the workflow in the Virtual Studio?

In our Virtual Studio, four new – in part original – work areas result from the workflow:

• The Green Area
• The Robotics Operator
• The Graphics Operator
• The Control Room

The Green Area

Here is where the actual perceptible performance in front of the camera happens. The moderator, and possibly additional guests, are active in front of the camera. Unlike in a real set, here the performing person(s) must adequately project both imagination and empathy. The virtual and augmented elements in the Green Area are not visible to viewers. They must have been presented or visualised by means of tricks. It often makes sense to position dummy elements, such as a table in the real environment, even if a different element will replace it virtually. In the hybrid mode you can also combine real and virtual elements.

The Robotics Operator

Here the cameras are controlled by operator. The operator has the option of using up to 16 robots with his or here console or touch display, controlling them either manually or by sequence. In the process he/she does not see the camera image but instead the finished rendered image. He or she can therefore look around in the final environment without having to be concerned at all about the actual background situation.Live motion sequences can be created by means of touch input which ensure the exactness and reproducibility of the tracking shots.

The Graphics Operator

The main work of the graphic artist takes place in the design of the set – before the live operation. This is where a major advantage of the Virtual Studio becomes evident: you can very quickly switch between sets, at the push of a button, without requiring costly and time-consuming conventional staging. During the show it is then possible, using AVID Maestro, to control the background clips in the virtual realm and to prepare, edit and play both animated elements and the on-air graphics. The operator can then, for example, also create or edit content such as statistics or banners during ongoing operation and then overlay them in the presentation.

The Control Room

All studio operations come together via  the Director. This is where the necessary instructions are issued to the crew and all camera signals and content input are monitored. A live mix of the material is possible here; the virtual background screens from the rendering engine can also be run.This way the director has absolute control, both over the content input feed and the completed production of the image output.