HumanTech deliverables

The HumanTech project comprises research on different technologies to obtain a digital twin of the construction site or existing buildings, and the application of wearable devices and advanced robots in construction. All HumanTech partners envision that the application of all these technologies will lead to an advance of construction towards a safer and greener construction industry. This document will describe this unified HumanTech vision. As the main challenges for the construction industry were identified: a growing demand in construction induced by a growing demand for energy efficient renovations, infrastructure investments and urbanisation, climate change, resource scarcity and a shortage of labour. The main research requirements are extending open BIM standards for data exchange and interoperability, methods to obtain Dynamic Semantic Digital Twins of construction sites and other assets, unobtrusive wearables with intelligent transparency, robotic interfaces and learning from demonstration, on- site safe human-robot-collaboration and workflow capturing with XR visualization. The impact of HumanTech is expected in the fields of worker’s health and safety and a transition towards green, climate-friendly, and resource-efficient construction by reducing errors, increasing accuracy and efficient robotic task automation.

This deliverable summarizes essential user requirements for interactive robots, ex- oskeletons and smart glasses. Likewise, an overview of working conditions of the European construction industry is given. Furthermore recommendations for the ethical handling of technologies and data are presented. The work package serves as basis for the application scenarios planned in the HumanTech project.

This deliverable presents the IDM (Information Delivery Manual) for the HumanTech BIMxD platform specification. It represents the outcome of task T2.1, which focuses on BIMxD Formats and Specifications.

In this Deliverable, open-source tools are presented to create BIMxD objects from reconstructed semantics and geometry, object filtering and information extraction, inheritance of semantic labels from IFC objects and BIMxD update are presented.

This document, deliverable 2.4 is the first results of the analysis of the openBIM standards for the proposal of an IFC extension using bSDD. It provides an analysis of the advancements in interoperability and openBIM standards within the construction industry within the domain of the HumanTech Project. It explores the critical role of interoperability in integrating diverse systems and the transformative impact of openBIM over traditional closed BIM methodologies. The document delves into buildingSMART’s openBIM standards, emphasizing the importance of standards such as IFC and bSDD in facilitating information exchange. It also examines the standardization process for interoperability and highlights the integration of innovative elements like robotics and safety protocols through the HumanTech project. The conclusion addresses the need for ongoing adoption and adaptation of evolving openBIM standards and the future steps of the task T2.4.

This deliverable, part of the HumanTech project, explores the integration of advanced technologies into BIM workflows through the extension of the Industry Foundation Classes (IFC) schema. The document focuses on enriching the buildingSMART Data Dictionary (bSDD) with new classes, properties, and relationships, with specific attention to dynamic and portable construction entities such as robots. It demonstrates mapping approaches to existing IFC entities and highlights gaps requiring future enhancements. The findings contribute to a more interoperable, flexible, and efficient digital construction ecosystem, aligning with the evolving demands of modern construction workflows.

This report encapsulates an exploration of hyperspectral imaging technologies for material identification processes in construction settings. The initial focus on sensor selection involved an in-depth investigation of the MicaSense RedEdge multispectral sensor. Unforeseen challenges led to a strategic shift, leveraging the available FX10 hyperspectral camera by Specim. The subsequent data acquisition phase covered the acquisition setup, software utilization, and the systematic creation of a dataset featuring diverse construction materials. Data analysis encompassed detailed preprocessing steps, including calibration and specific bands selection to replicate the data that would have been acquired with the initially considered RedEdge multispectral sensor. A Principal Component Analysis unveiled underlying patterns within the dataset, highlighting distinctive spectral signatures of different construction materials.

In this report, we review the benefits of artificial data in semantic point cloud segmentation to improve segmentation on real-life data. This could also be a response to the lack of annotated data on some fields, and more especially for BIM models. Annotating data by hand is a very tedious task and has to be supervised in some ways. The main point of focus is 3D interiors scans obtained with a LiDAR sensor. Our work will be divided into two main parts. The first part targets artificial data generation using a simulated LiDAR sensor inside Unreal Engine 5. There is a need to have more annotated data and generating artificial datasets will be proven to be a viable alternative. Our reference will be the Stanford dataset which consists of real-world interior scans (alongside their annotations, meshes, semantics, surface normal, materials and textures). The second part consists in the evaluation of such artificially generated data. In order to measure the benefits of using artificial data, we will use domain adaptation and finetuning on two pretrained models for semantic segmentation. We will also discuss the need of “good” artificial data, especially when it comes to complex tasks – even regardless of the model, and methods to generate them.

This report presents the implementation, training, and evaluation of machine learning algorithms for semantic segmentation of point clouds and panoramic RGB-D images in the context of construction. The developed methods not only meet the project’s key performance indicators but also achieve state-of-the-art results in single frame panoramic RGB-D image semantic segmentation. To support model training, a comprehensive annotation guideline for joint 2D-3D data was proposed and published, filling a gap in research standards. The trained models, leveraging a combination of HumanTech, public, and simulated data, will be integrated into the scan-to-BIM pipeline to generate the Semantic Digital Twin.

This deliverable reports the development of a complete Scan-to-BIMxD pipeline delivered by the HumanTech project. Initially, a semantic segmentation of the point cloud is performed followed by geometric reconstruction procedures. Open BIM authoring tools are then used to deliver BIM objects of wall, door and columns. Finally, results of our experiments will be reported.

This document describes the visual inertial sensor network which the workers wear in some use cases within the HumanTech project. The focus of this deliverable lies on the hardware properties as well as its calibration. As part of this hardware, the local processing device will be introduced as well. It is planned, that this powerful device will also host other applications in the context of the worker, such as the localization of Task 4.3 and the exoskeleton controller with intention prediction of Task 4.2 and will exchange data with them. The output of D4.1 is mainly used to predict the wearers intention in T4.2 but it may also interact with T4.3. It will be demonstrated within Pilot 1.

Work-related musculoskeletal disorders (WRMDs) are the most common occupational health problem in Europe. They have significant prevalence and impact in the construction sector. In recent years, exoskeletons have been proposed as the solution to all the problems associated with WRMDs. However, the promise of effortless work thanks has not yet been fulfilled due to the inherent limitations of the technologies currently used in exoskeletons. One of these barriers is their usability and acceptability. The aim of task 4.2 “Intention prediction and exoskeleton integration” is the design of an exoskeleton with automated activation based on the prediction of the intention of its wearer. Such an exoskeleton will minimize the cognitive and physical burden of interaction between it and its wearer. This deliverable presents the first prototype of this intelligent transparent exoskeleton.

This document summarizes the research results accomplished within Task 4.3. The target of this task has been to develop and assess methods to localize a camera in the digital twin of a construction site from a data stream consisting of video images and IMU data of a body worn sensor system, as the one described in D4.1.or system, as the one described in D4.1.

This document provides both an abstract and an executive overview of the Extended Reality (XR) prototype developed within Task T4.4 of the HumanTech project.

This report comprehensively outlines the efforts undertaken during Task 5.1, focusing on the development of a task planner for the automatic execution of demolition activities. At its core, it leverages a demolition ontology, an extension of ifcOWL, to establish a cognitive foundation. This ontology meticulously delineates the demolition environment, encompassing representations of walls, openings, and robots. Demolition task planning has a detailed focus on marking, drilling, and cutting operations. These tasks seamlessly unfold through the task planner, which meticulously assesses feasibility based on available resources.

The aim of task 5.2 “Remote interfaces for demolition is the development of a user- centric control console for the remote operation of robots in the hazardous context of demolition. This console will integrate advanced features in viewing and haptics which will increase the sense of telepresence while augmenting the precision and dexterity of the operator. This deliverable contains a review of studies and techniques oriented to improve the performance and safety in teleoperation tasks with special focus on visual and force feedback. Some of them have been integrated in the teleoperation framework developed in HumanTech which is described in the las point of this document.

This deliverable describes the development of a modular mobile robotic system. The focal point of the robotic system is to serve as a platform for developing diverse applications suited for the construction sector. The intent is to create a foundational structure for future mobile robotic solutions that can serve as a testing ground for diverse applications. The system architecture of the mobile robotic system is first described including the individual elements of the system. Subsequently, the external application interface for developing new applications on top of the mobile robotic platform is described. Finally, the Human Machine Interface and the safety elements of the system are described.

This deliverable describes the development of robotic perception algorithms for object pose estimation in HumanTech. Object Pose estimation based on camera images as input is a key-task for localizing objects for robotic grasping. We first provide an overview of the object pose estimation problem and the overall context of the task in the HumanTech project with respect to construction material robotic grasping. Subsequently, we describe the selected object pose estimation framework for the task, the state-of-the-art algorithm ZebraPose developed at DFKI. Finally, we describe the object pose estimation task on the HumanTech object of interest category, construction bricks. We detail the approach for generating and training our machine learning models exclusively on synthetic data and conclude with an evaluation of the brick grasping pose accuracy and the next steps for integration of the method on the human-tech robotic platform for real-time functionality.

The deliverable describes the innovative methods for “teaching by demonstration” to streamline the automation of complex tasks, eliminating the need for traditional, explicit robot programming. This approach leverages teleoperation, where human operators can control and program heavy-duty construction robots remotely, ensuring their safety while interacting with potentially dangerous environments. The core of this method lies in moving beyond simple recording of trajectories and forces during demonstrations. Instead, it emphasizes the extraction of control “policies” from multiple human demonstrations. These policies enable the robot to not only replicate specific tasks but also adapt to varying conditions and handle uncertainties inherent in real-world environments, such as changes in materials, terrain, or sensor reliability. This capability to generalize from learned experiences positions the robots to operate autonomously and efficiently in diverse scenarios, ultimately advancing the field of construction automation and making it more robust and flexible.

In this deliverable we present research and development results obtained in the task T5.6. More specifically, we present human-robot communication framework with four developed communication channels. The channels are speech, pose communication, as well as light-based robot response and glove tracking for robot tool control. The developed technology was demonstrated in a laboratory environment for robot control in a collaborative bricklaying process. As a part of the implementation, a robot-mounted sensor unit for communication was designed and assembled. The deliverable contains introduction background and motivation for the presented development. Further it contains presentation of the proposed framework, where development of all communication channels is described, and implementation is demonstrated. Consideration on implementation of a multimodal communication case is also given. Finally, the conclusions and further steps for future research are given.

This deliverable presents the HumanTech micro-learning unit descriptors and the steps taken to archive these results. Based on the work carried out in WP6 of the project, the descriptors of 12 modules, their content and the corresponding ULOs have been shared in this paper. These Micro-Learning Units will go on to be developed through the remainder of the HumanTech project and training will then be carried out with HEI, VET and industry professionals. This is a collaborative process, and the HumanTech partners and project results will heavily influence the resulting training material.

This deliverable summarises the sequential and ongoing evaluation of the HT wear- ables system and human-robot interactions. To identify the needs of the users at an early stage for incorporate them into the design process, an evaluation will be per- formed in a three-step approach. In this first deliverable the findings of a series of workshops held with workers and other type of stakeholders is presented. Based on the findings of Task 1.4, a lab-based approach was first carried out to evaluate usability, trust in automation and discomfort. At a further stage, the objective assessment will be carried out by evaluation of physiological sensor data collected in dedicated training sessions.

Document detailing the project dissemination, exploitation and communication plans, outlining the target groups and their segments.

This document details the activities of ecosystem building, dissemination, and communication carried out during the first year and a half of the HumanTech project as part of the master plan to maximise the project’s impact, outlining the schedule for the next period (M19-M36).

The efforts described in this report are directly linked to the execution of WP8 – Outreach, Exploitation and Collaboration, as described in the Description of Action (DoA). This document details the dissemination, and communication activities carried out during the second reporting period of the HumanTech project as part of the master plan to maximise the project’s impact.