DFKI augmented vision researchers win 2 awards in the Object Pose Estimation challenge (BOP Challenge, ECCV 2022)
HumanTech's Project Coordinator Jason Rambach and Yongzhi Su and Praveen Nathan, scientists at our partner organisation DFKI Augmented Vision Research Unit, have received the first prize in the prestigious BOP Challenge 2022.
In particular, they have won two awards for their Object Pose Estimation method, ZebraPose, in the categories of Overall Best Segmentation Method and Best BlenderProc-Trained Segmentation Method.
The BOP Benchmark and Challenge addresses the problem of the 6-degree-of-freedom object pose estimation, which is crucial for many applications, such as robot grasping or augmented reality.
This year, the BOP competition was held in the "Recovering 6D Object Pose" workshop at the European Computer Vision Conference (ECCV) in Tel Aviv, Israel.
The winning method was based on the CVPR 2022 paper "ZebraPose". It was developed by a team led by DFKI AV, with the collaboration of researchers from the Technical University of Munich and Zhejiang University.
A total prize of $4000 was distributed among the winning teams of the challenge, donated by Meta Reality Labs and Niantic. The awards were received by HumanTech's Project Coordinator Dr Jason Rambach, who is also a Senior Researcher and Team Leader in Spatial Sensing and Machine Perception at DFKI, on behalf of the DFKI team, who also gave a brief presentation of the method.
Congratulations to all the professionals involved in this milestone!
Meet the HumanTech team: Jason Rambach, his passion and motivation guide our project
At HumanTech, we are working to create a safer, greener and more productive construction industry, as well as a rewarding and digital work environment for a new generation of highly skilled construction workers and engineers. To this end, we have set out to achieve significant advances in cutting-edge human-centred technologies that contribute to the sector's digitalisation and accelerate the transition to green construction.
This will only be possible thanks to the work of the HumanTech team, made up of 22 partners present in 10 countries. From leading research institutes and universities to highly acclaimed SMEs and large companies — with expertise in robotics, wearables, artificial intelligence (AI) and extended reality (XR).
Over the next few months, we will introduce our team members through the "Meet the HumanTech Team" interview series. We will ask them about their role in the project and the impact they want to achieve with it. Their vision for the future of construction and the evolution towards a safer and more sustainable industry. Their learnings, advice and aspirations.
Here we go!
For the first interview in this new series, we spoke with HumanTech’s Project Coordinator Jason Rambach, a Senior Researcher and Team Leader in Spatial Sensing and Machine Perception at the German Research Center for Artificial Intelligence (DFKI) in Kaiserslautern, Germany.
This is what he told us.
"I am very happy about the opportunity to lead HumanTech. I believe that, as a consortium, we are all very excited and motivated for the next 2.5 years of the project and the further development of our results."
Q: Jason, tell us a little about yourself. What motivates you? What are you focused on nowadays?
A: I enjoy working in research because of the freedom to experiment with new ideas, the variability of work and the openness of the research community. However, I think we should not forget the real-world applications of our research and how they can positively impact people.
My research focuses on enabling full spatial perception for autonomous systems by fusing different sensors with machine learning. Acquiring a deeper understanding of the environment at the geometric (where an object is) and semantic (what that object is) levels is key for intelligent applications in robotics and XR.
Q: How interesting! Please, tell us more about what DFKI does and what you have set out to achieve.
A: The German Research Center for Artificial Intelligence (DFKI) is one of the largest AI research institutes in Germany and the world. It has several sites and groups with expertise in different areas of AI, from computer vision to natural language processing and robotics. We perform cutting-edge research in the "Human-centric AI" field and share our work at leading AI conferences. Research is typically conducted in an application-related manner in various fields such as autonomous driving, healthcare, smart buildings and cities, or agriculture. This positions DFKI as an intermediary between basic research conducted in universities and fully applied research and development in the industry, making technology transfer one of DFKI's main goals.
Q: A much-needed work. Please, keep it up! Now, let us know about your role at HumanTech. What do you do, and what is the most rewarding thing about it?
A: I am the consortium coordinator of the HumanTech project. This makes me responsible for the overall technical coordination of the work and the management in terms of reporting and project deliverables. Although this is a new and quite challenging experience for me, I enjoy working with such a diverse, talented and passionate consortium that is determined to contribute to the construction industry's digital transformation. Working at HumanTech allows my team and me to work on research topics that interest us greatly. In addition, we believe in the positive effect HumanTech can have on the safety of people and the green transition of construction.
"Although this is a new and quite challenging experience for me, I enjoy working with such a diverse, talented and passionate consortium that is determined to contribute to the construction industry's digital transformation."
Q: Without a doubt, a challenging task. But your commitment and interest in the project will greatly help us achieve our goals! What two activities are you working on that you are most excited about?
A: Here, I would have to mention some research activities closer to my interests and experience. First, the dynamic semantic digital twin work in HumanTech aims to enable the creation of digital representations of a construction site in its current state in a very efficient way. This is a core component of HumanTech, facilitating applications such as automatic progress monitoring by comparison with construction plans (BIM). We are addressing this challenging topic by capturing data with different cameras and 3D sensors, applying machine learning techniques for semantic segmentation on 3D data, and subsequent comparison with a CAD database to build our digital twin models.
Another fascinating topic is perception for robotics, where we develop computer vision algorithms for the precise localization of objects and people in 3D space. Together with our HumanTech robotics partners, such as SINTEF, TUK and BAUBOT, we will use this to develop human-robot collaboration scenarios in construction, such as material pickup and delivery.
Q: Sounds amazing. We can't wait to learn more about the advances we make in developing these innovations. Can you tell us the positive impact you hope to generate through this project?
A: Digitalization is progressing quite slowly in the construction industry. One of HumanTech's main objectives is to raise awareness of the solutions we develop and shape them so they can be integrated into construction sites in the coming years.
HumanTech technologies aim to have a positive impact mainly on three key aspects of construction:
- The first is worker safety and health, which remains highly problematic in the sector. HumanTech presents a set of wearables (exoskeletons, inertial body sensors, XR glasses) that seek to actively assist workers and reduce the strain produced by repetitive tasks, help them optimize their body posture, and prevent accidents and injuries.
- Secondly, we aim for sustainability and a greener construction industry. We hope to achieve this by helping to avoid costly mistakes through progress monitoring and encouraging best practices through worker training with visualization technologies such as XR.
- Finally, we hope to increase the industry's efficiency through our digital twin and robotic assistance technologies and create a more attractive working environment for all.
"Digitalization is progressing quite slowly in the construction industry. One of HumanTech's main objectives is to raise awareness of the solutions we develop and shape them so they can be integrated into construction sites in the coming years."
Q: And what about the milestones you hope to achieve?
A: There are several milestones defined in our project. However, I think the most impactful and challenging will be our pilots. We plan to demonstrate HumanTech prototypes on real construction sites in 5 different locations, including scenarios such as bridge inspection, remote-controlled demolition, and human-robot collaborative brick-laying.
Q: That's right. Our pilots will be vital to deploy and validating HumanTech's technical approach. Before finishing, we would like to know your vision for the future of construction. Do you think achieving net zero emissions in the sector will be possible? If so, what are the biggest challenges to doing so?
A: This remains to be seen, but there is still a long way to go. For now, we hope to be able to contribute in that direction within HumanTech! I think the main challenges are accepting new technologies and moving on from outdated practices. Of course, what we propose in HumanTech is just one piece of the puzzle; for example, a lot of important research is being done in building materials (3D printing, sustainable materials, etc.).
Q: Finally, can you tell us about other projects in the construction sector that have inspired you?
A: When conceptualizing the HumanTech project, we were inspired by two other EU projects that are or were related to the same field. These are the BIMprove project, in the field of digital twins, and the BIONIC project, from the wearable sensors for health and safety perspective.
Curious to learn more about HumanTech and the team behind the project? Subscribe to our newsletter and follow us on LinkedIn and Twitter so you don't miss a thing!
5 objectives to understand what HumanTech is all about
HumanTech was born to address the most important challenges faced by the European construction industry today — making it safer, greener and more efficient. The cutting-edge, human-centred technologies we are developing will transform the sector by increasing the safety and well-being of its workers, improving their productivity and enhancing circular and efficient practices.
With our innovations, we aim at:
- a 10x increase in the adoption of exoskeletons in construction;
- a 30% decrease in construction worker injuries;
- a 50% decrease in safety inspection time;
- an identification rate of fall hazards at 90%;
- and a 20% minimum reduction in waste and CO2 emissions.
HumanTech's 5 scientific and technological objectives
From a new breed of digital twins to wearable solutions (an exoskeleton or XR glasses) or autonomous demolition and construction site robots, these are the five specific scientific and technological objectives we are targeting.
1. To generate and update a dynamic digital twin of construction sites based on an extended BIM standard
First, a digital twin is a virtual model of a physical object. And Building Information Modeling (BIM) refers to creating and managing information about a built asset throughout its life cycle, from planning and design to construction and operations.
At HumanTech, we will create a digital twin to represent a construction site accurately — both in terms of geometry (through 3D reconstruction) and semantic information (with object-level labels and materials information), focusing on a continuous dynamic update of the model.
The model will form the reference framework for any autonomous robotic solution or XR visualisation we will develop. XR (extended reality) is an umbrella term that rolls into VR (virtual reality), AR (augmented reality), and MR (mixed reality). In a nutshell, “XR technology is any tech that takes your display and makes it more immersive or interacts with your real-world surroundings in some way.”
To develop efficient and accurate methods to generate BIM models, different construction sites will be scanned with autonomous ground and aerial vehicles. We will also use novel hardware solutions, such as a 360-degree spherical camera provided by Ricoh, one of our consortium partners. To process this data, new machine learning methods for semantic segmentation and data alignment will be developed within HumanTech.
Moreover, we will create a new BIM extended Digital Twin (BIMxD) standardisation formulation. It will include additional information to support human-robot collaborative task planning and scheduling, as well as related to worker health and safety.
2. Integrated, lightweight, non-intrusive solutions to improve the safety and health of construction workers and assist them in their tasks
Construction workers often have to put their health at risk to do their job. They are exposed to all kinds of hazards and accidents, many of which are fatal. In 2019, more than a fifth of all fatal accidents at work in the EU took place within the construction sector, according to Eurostat.
One of our primary goals is to improve the safety and lives of construction employees, specifically by reducing their work-related injuries by 30%.
To do so, we will develop lightweight, easy-to-carry and easy-to-use solutions that monitor their health, prevent injuries and support them in dangerous tasks while providing them information with which they can make better decisions.
Our integrated wearable solutions will include:
- an exoskeleton to support workers in strenuous or repetitive tasks and reduce the negative impact they may have on their health;
- a body pose inertial sensor network to track workers’ posture, and guide/activate exoskeleton components;
- XR glasses that will help workers avoid errors by displaying BIM information about the current state of a building;
- and a miniature camera for precise 6DoF (six degrees of freedom) localisation of the XR visualisations for task assistance and to provide contextual information to the inertial sensors and exoskeleton.
3. Semi-autonomous demolition robots supported by remote workers
Demolition is one of the riskiest and unhealthy tasks faced by workers in the sector (for example, due to the inhalation of dust, asbestos or other hazardous materials) and for which human labour is required almost entirely.
At HumanTech, we will automate this activity as much as possible. By doing so, we hope to reduce workers' health and safety risks and increase productivity in this type of activity.
We will develop a semi-automatic demolition robot that will carry out tasks autonomously. If necessary, a human can operate it remotely in a safe environment and have detailed information to make efficient high-level decisions.
We will optimise demolition planning by using the digital twin as a reference, aiding the collection of reusable material. In addition, the remote operation of the robot will incorporate haptic feedback (referring to the physical response of a virtual action) to the user and demonstration learning principles to improve automation. We will also use specialised 360-degree RGB-D vision cameras on the robot to build an immersive environment for the remote user in a VR environment.
4. Autonomous construction site robots for human support and material transport
Autonomous robots that can easily navigate a construction site are used to support human workers, for example, transporting material or picking up debris. PS's current solution (BauBot) offers a modular system in its robots that can easily expand their functionalities by adding 3D and vision sensors and machine learning algorithms.
At HumanTech, we want to help construction workers with the most arduous tasks and prevent body postures that can cause long-term deformities (e.g. working on the floor or the ceiling). To this end, we will develop a new generation of autonomous mobile robotic platforms that can work alongside them and take on tasks autonomously.
Specifically, we will create:
- an autonomous and versatile ground robot with sensory systems suitable for safely navigating in dynamic environments;
- a robotic solution with a high payload capacity;
- and a human-machine interface for effectively communicating and identifying construction and demolition waste.
5. Training and usability assessment of our solutions to promote awareness, the acquisition of new skills and the acceptance of users in the construction industry
To achieve the impact we want, construction workers must embrace the HumanTech model. Therefore, we aim to train them from an innovative approach: including BIMxD data transmission and the automatic capture of expert knowledge with wearable cameras that will be transmitted to the trainees with AR tools.
Usability assessment, focusing primarily on wearables (including exoskeletons), will be performed with both traditional and objective methods employing the measurement of the user's psychophysiological signals (involving mental and bodily processes).
We will collect this knowledge and evaluation through virtual case studies, gamification, videos and on-site training (some of which will be made available as Open Educational Resources — OERs).
In addition, we will design our training resources to ensure greater knowledge about sustainability and circular economy practices applied to construction.
Take a look at our demonstrators to learn more about how we will use these innovations in real construction sites across Europe and Japan.
HumanTech arrives to develop human-centred construction technologies
The construction industry plays an essential role in the European economy. It contributes to around 9% of the region's GDP and provides 18 million direct jobs and solutions to social, climate and energy challenges. At HumanTech, we aim to help the sector become safer, greener and more competitive by advancing innovative human-centred technologies — from robotic devices and exoskeletons to a new breed of digital twins.
European construction faces challenges affecting its workers, the environment, and business competitiveness.
- Slow technology adoption. The sector is very slow to incorporate new technologies, even though they could help solve many of its problems.
- Workers' safety and productivity. Construction accidents can be severe due to the type of activities workers must carry out, affecting their productivity.
- Shortage of skilled workers, explained by the sector’s low attractiveness for young people and the growing need for specific skills, which education has difficulty satisfying.
- Inefficient use of resources. Buildings account for about 35% of all the EU's greenhouse emissions. The transition to a resource-efficient and low-carbon economy is vital to reduce its negative environmental impact.
How can technology help create practical solutions to these challenges? The 22 partner organizations that make up HumanTech, an ambitious initiative funded by the European Commission under the Horizon Europe programme, have started to work together to answer this question.
Technology as an ally to provide innovative solutions to construction challenges
HumanTech aims to to enable a safe, green, rewarding and digital work environment for a new generation of highly skilled construction workers and engineers.
These advances will include:
- Robotic devices equipped with vision and intelligence.
- Smart, unobtrusive workers' protection and support equipment.
- A new breed of Dynamic Semantic Digital Twins (DSDTs).
The project will also create resources to spread the knowledge needed to work with these technologies and thus increase the number of qualified workers in the sector.
- Open educational resources on technologies for workers' safety, well-being, and human-robot collaboration in construction.
- Ten training sessions with 200 participants and 1 ‘Train the Trainer’ session for 20 educators and trainers.
We also intend to increase the adoption of exoskeletons in construction by 10x five years after the end of the project, decrease construction worker injuries by 30%, and reduce waste and CO2 emissions from the sector by a minimum of 20%.
We will validate our technologies in in Europe and Japan. These will allow us to evaluate the project’s advances and feed its technical developments.
22 partners, 11 disciplines, 10 countries
started working together on this groundbreaking initiative in June 2022 and will continue to do so over the next three years.
They are leading research institutes and universities, high-tech SMEs, large companies, and construction groups with expertise in disciplines ranging from artificial intelligence to extended reality.
Their multidisciplinarity, complementarity of knowledge and objectives, and excellent track record in research and innovative product development make them the ideal team for this project.
Learn more about HumanTech in our , , follow us on LinkedIn and Twitter, and contact us to explore collaboration opportunities!
*Header image by rawpixel.com on Freepik.
We meet in Germany to launch HumanTech!
Our HumanTech team gathered in Kaiserslautern (Germany) to kick off this ambitious new project in a two-day hybrid meeting. Representatives from the twenty-two organizations working on this initiative — led by the Augmented Vision Department of the German Research Center for Artificial Intelligence (DFKI) — could get to know each other and learn more about the work we will be doing over the next three years.
About HumanTech
Our goal is to address the most critical challenges faced by the European construction industry today — making it safer, greener and more efficient.
How? By developing . Solutions such as wearables for worker safety and support and intelligent robotic devices that can co-exist with human workers while contributing to the green transition of the industry.
brings together entities from leading research institutes and universities to innovative SMEs and large companies, present in ten countries and with expertise in eleven different areas.
Throughout the project, we want to explore possible synergies with other initiatives, organizations and experts to support the advancements of HumanTech and create a better construction industry. So, if you are interested in collaborating, !
Learn more about HumanTech in our and keep up to date with our progress by following us on social media (LinkedIn and Twitter) and .