Imagine you enter a room and a true-to-life image of yourself is created on a computer in real time, without any delay. Your position in the room and every movement you make is recorded from several angles and stored in a kind of three-dimensional film you can watch as often as you like, for example, to analyze your own behaviour during a training session.

Such a room has recently been created at the University of Konstanz, more precisely in the university's GameLab. Benjamin Schäfer, the manager of the GameLab, has just installed a modern 3D setup in which up to ten special 3D cameras capture the centre of the room from different perspectives. A powerful computer bundles and synchronizes the information from the cameras and uses it to generate the 3D reconstruction of the room or volume and the people moving in it. 

For more information on the cameras, please click on the (+)

Not like in the cinema, but really 3D
The trick of the method, compared to ordinary video recordings or 3D films many know from the cinema is as follows: Whereas a normal film recording gives observers a fixed perspective from which they have to view the action, this perspective can be freely selected in the GameLab's volumetric 3D recordings.

To do so, the data from the 3D cameras is simply loaded into a game engine, i.e. software that is mainly used for game development. Thanks to the game engine, the viewer can then move through a digitally constructed room on the computer and view the hologram-like images from all sides. For a more intensive examination of the recording, just a few adjustment are necessary to experience it with a VR headset. What's more:

"The game engine allows us to play back the recording in any virtual environment, or to place several recordings next to each other for analysis. In post-production within the game engine, we are currently testing anonymization options."

GameLab manager Benjamin Schäfer

Next-level teacher training
There are already ideas for using the volumetric recording system in teaching and research. Led by Stefanie Findeisen from the Department of Economics, a very specific pilot project from the field of business education was launched in April 2024, focusing on the training of teachers. The first courses within the project begin in October 2024.

In the project, student teachers from the University of Göttingen and the University of Konstanz will jointly recreate teaching situations in the GameLab. One person from the group will take on the role of the teacher, while the other students play the pupils. The teacher is recorded three-dimensionally and the pupils audiovisually using a video camera. Afterwards, the group discusses and reflects on the situation. In teacher training, classic video recordings of teaching situations are a common tool and are made, for example, for lesson debriefing and self-reflection.

Decide who you want to be
"Classic video recording is an extremely reduced representation of reality. You have to always decide in advance exactly which aspect should be recorded", explains Schäfer. Are the teacher's facial expressions and gestures more interesting than the pupils? Then the camera has to front-face the teacher. If, on the other hand, the pupils' reactions to the lesson are of primary interest, the camera must be positioned exactly the other way around. In both cases, a large amount of information about the interaction between pupils and teachers is lost.

© GameLab, University of Konstanz

A person explains something in front of a whiteboard. In this snapshot, the scene is depicted as a three-dimensional point cloud.
 

The advantages of 3D recording with a freely selectable perspective are quite obvious: If teachers want to analyze and reflect on their own behaviour in the classroom, they are free to choose how they view the situation. "They can adopt their own perspective in order to keep an eye on the pupils' reactions. But, they can just as easily watch themselves from different angles to analyze their own posture and body language. They can even take on the perspective of the pupils", Schäfer explains. This means that much more information is retained and there is scope to shift the focus of a question or even change it completely.

A trial balloon with countless possibilities
"We are looking forward to the start of this pilot study, because even we can't yet foresee all the possibilities that the setup will offer", says Benjamin Schäfer. "First of all, the recordings will provide us with much more information about the filmed situation than conventional recording methods. We will therefore have to find out which of this information is really useful for our questions". Because: more information inevitably means more analysis work.

And yet there are already many other ideas for projects using the GameLab's 3D recording system, for example, analyzing social interactions while playing games. "In the GameLab, we are particularly interested in games and game situations. We believe that the setup is very well suited to studying interpersonal interactions while people play games", says Schäfer. Similar to reading a good book, players quickly tune out their surroundings as soon as a game is in full swing. They "forget" that they are under observation in the 3D recording studio, so to speak. The rather unusual environment of the studio should therefore hardly influence the players' behaviour and interactions.

Adaptable to different situations
To ensure that the 3D recordings function smoothly and that the information from the different cameras results in a bigger picture, the setup must be calibrated before use. And yet it is not intended that the setup is to be used permanently in its current, calibrated form. On the contrary: Flexible use of the studio was planned from the outset. For example, the studio ceiling is equipped with a system of crossbars along which the position of the cameras can be moved relatively freely. What is known as double-grip from film technology then ensures that the cameras can also be positioned at any height.

© GameLab, University of Konstanz

Markers are placed in the room to calibrate the cameras (left). The right-hand image shows the markers in the 3D view. During calibration, the software sets the camera images in relation to each other to create an overall spatial image.
 

The Scientific Engineering Services at the University of Konstanz adapted the system of crossbars and suspensions to meet the needs of the GameLab. "Even though we were able to purchase the camera suspensions, for example, they were of course originally intended for use in film and television studios – in other words, for very different conditions to those in our GameLab. We are very grateful for the support of the Scientific Engineering Services, which helped us to realize our recording studio in the GameLab rooms", says Schäfer.

For all disciplines
Even if the problem of camera placement and suspension may sound trivial, this is a decisive factor for the success of the recording, especially when making 3D recordings with several cameras. Thanks to the flexible system of crossbars and suspensions, the three-dimensional recording studio can be adapted to very different observation situations. This way, internal and external users from all disciplines can use this core facility of the University of Konstanz for their research.

"Our technical equipment is continuously maintained and expanded, and researchers receive technical support from us when implementing their projects. This means that it doesn't matter whether the research question is about a group of students recreating a classroom situation or a group of people gathered around a table playing a game together", explains Schäfer.

Contact:
GameLab Coordination team (PO box 157)
Universitaetsstr. 10
78457 Konstanz
GERMANY

Phone.: 07531 / 88 - 5643
Mail: gamelab@uni-konstanz.de