It’s been a year since the start of HumanTech! And we are getting a feeling of great momentum building up in our project. Having invested a good amount of time in clearly defining tasks, dependencies and architectures during this period, we will focus even more on our technical developments in the coming year.

Since HumanTech began, we have already seen some promising first results — from scanning and semantic reconstruction to wearables integration and robotics & perception — while initiating our first round of user evaluations. Meanwhile, the scientific excellence of our consortium is already evident with three publications and technical challenge awards.

We have also been very active in communication, dissemination and community building for artificial intelligence (AI) in construction by organising two technical workshops with impressive attendance and having a very active online presence.

Below, discover our progress in this first part of the project and our plans for the period ahead.

Overall framework definitions

Our partners at the German Research Centre for Artificial Intelligence (DFKI) have led the implementation of the HumanTech cooperation, defining our overall vision and creating the first version of the project’s system architecture.

After many fruitful team discussions and collaboration with partners involved in the project’s technical development and pilots, they have defined a detailed architecture at the modular level — clearly paving a path from technical developments to integrated proof-of-concept implementations for our pilots.

Furthermore, they have been able to define a comprehensive collection of end-user requirements for our HumanTech technologies, which will be the basis for further human-centred technology evaluation activities and, above that, have guiding character for our technology developing partners.

Our team working on the project’s overall framework definitions will now closely follow the progress of the rest of the team to make the necessary updates towards the final version of the HumanTech system architecture.

BIMxD formats and standardisation

Our team focused on BIMxD (multidimensional building information modelling) formats and standardisation, led by the University of Padova, has worked on implementing standardised and openBIM methods in designing a BIMxD platform, in terms of technology implementation and processes.

Their main activities during our first project year could be summarised in these:

  • Analysis of the existing standards and openBIM implementation in semantic BIM modelling, use of BIM models and robotics in construction activities
  • Definition of the exchange requirements of the BIMxD platform
  • Design of the BIMxD platform for collaboration
  • Implementation of the BCF (BIM Collaboration Format) for the task
  • Initial work on open-source BIM software and framework for modelling and updating services of BIMxD representations

Regarding their main results achieved, they have advanced on the implementation of the IDM for the BIMxD platform and executed the first successful workflow in the BIMxD platform using an open-source framework.

Next, they will focus on implementing the BIMxD platform, the BSDD domain for the BMxD platform and required implementation, and the framework and software modules for the opensource BIM tool.

Dynamic semantic digital twin generation

The main activities of our team focused on dynamic semantic digital twin generation, led by Naska.AI, have been:

  • The development of hardware and software pipelines for unmanned ground vehicles (UGV) and unmanned aerial vehicles (UAV) data capture platforms
  • The development and testing of prototype RGB-D sensor
  • The development of an industrial 360° RGB (red, green, and blue) camera
  • The capture of several RGBD (red, green, blue depth) datasets, development of annotation workflow and tools for semantic segmentation
  • Initial work on synthetic data generation from BIM models
  • The definition of scan-to-BIM pipeline and algorithm development
  • The study of the state-of-the-art for the use of spectral sensors for building material classification and identification of suitable sensors

In this regard, we have achieved the following:

  • An initial hardware and software pipeline for UGV data capture
  • Promising initial results of RGB-D camera prototype
  • Multimodal data capture at a test site in Weingarten, Germany
  • Promising initial results on semantic segmentation from RGB-D data

Our work during the next phase of the project will be focused on:

  • The integration of the RICOH industrial 360° camera on UGV and UAV platforms
  • The development of path planning algorithm for autonomous UGV inspection
  • The testing of a multispectral sensor for building material classification and its integration on a UGV platform
  • Continuing the development of semantic segmentation algorithms using RGB-D data
  • Synthetic data generation from BIM models and domain adaptation
  • Multimodal data capture with final configurations of UGV and UAV platforms
  • The development of a scan-to-BIM pipeline

Wearable technologies for construction

Our partners working on wearable technologies, also led by the German Research Centre for Artificial Intelligence (DFKI), have been involved in discussions and workshops with other partners to help define a detailed architecture and specify the pilot use cases to which the components they are developing will contribute.

They have detailed specifications on visual body sensor network components and their interfaces, and designed and constructed a novel miniaturised 360° camera that is being integrated into the visual body sensor network.

In parallel, they have built test equipment with IMUs (inertial measurement units) and physical sensors to reproduce the laboratory’s pilot use case of the exoskeleton.

Based on a new 360° camera, they are developing a first setup of a visual-inertial tracking system. The concept of this visual body sensor network will be documented in a public report. Also, they will collect sensor data with the exoskeleton to develop an intention recognition algorithm for activating it.

In this next phase, the partners involved in these tasks will also work on localising the wearable camera in the digital twin and the BIMxD visualisation in extended reality (XR) glasses. They have already started to conceive and discuss concepts and have initiated workshops with other partners to define interfaces and benefit from existing BIM standards.

Their next step is to define suitable laboratory spaces, equip them accordingly, and generate BIM models from them as a basis for our research on the target functionalities.

Construction robotics and human-robot collaboration

Our main achievements in our work on construction robotics and human-robot collaboration, led by SINTEF, have been:

  • Establishing a common understanding for our pilots
  • Defining the robotic tasks for demolition, handling and mastic application
  • Doing a site visit to get a hands-on experience with construction processes
  • Initial research and development for the teleoperation of a robotic system
  • System design for robotic manipulation platform to be used in pilots
  • Design and prototyping of robotic grippers for pilots
  • Training of vision algorithms for the detection of bricks and humans
  • Initial testing and studies with human-robot communication interfaces (e.g., voice or gesture control)

Our first results in this field have been:

  • To develop the first draft of robotic construction ontology
  • To document and publish vision algorithms for brick/human detection
  • To test teaching algorithms for programming robots
  • To establish a common data repository for exchanging designs and plans
  • To design a robotic task execution framework

Next, a robotic platform will be ready to be tested on construction sites.

Human factors – Training and usability assessment

Our work on human factors and training and usability assessment, led by the Technological University of the Shannon (TUS), has focused on the following:

  • Preparing material for the subjective assessment of worker’s technology acceptance related to collaborative and interactive robots, exoskeletons and XR technologies
  • Organising three focus groups in Spain (2) and Ireland (1) with the main stakeholders of the construction sector (apprentices, workers and supervisors), where we have presented technology features and benefits
  • Making questionnaires about the technology acceptance, features required and ways of interaction with technology developments to participants

Currently, we are processing the answers from participants and analysing the data we have gathered. We will soon deliver our results and conclusions.

Continuing with the subjective assessment, we will organise more focus groups in Central Europe. Once our technologies are ready to test, we will proceed with the objective assessment, which we will carry out by evaluating physiological sensor data collected in dedicated training sessions.

Pilots, evaluation and validation

Our work on pilots, evaluation and validation, led by the University of Kaiserslautern-Landau (RPTU), has been dedicated to defining the use cases and interfaces to other partners.

In several on and off-site workshops, we have managed to define interesting use cases for human-robot collaborative bricklaying, robotic mastic application, including learning from demonstration, and robotic demolition.

We have held fruitful discussions that have led to good ideas about possible use cases for our digital twin pilots, including health and safety monitoring, construction progress tracking, dynamic semantic digital twin generation and BIMxD update.

Outreach and collaboration

We have focused our work on communication and dissemination, as well as on community building, led by Australo, on:

  • Establishing our visual identity and our internal and external communication channels
  • Sharing what our project is about, what the technologies we are developing consist of, how they will positively contribute to the construction industry and its workers, and the work and stories of the organisations and professionals involved in HumanTech
  • Giving visibility to our main achievements, including publishing our first scientific paper, receiving several technical challenge awards (BOP Challenge, ECCV 2022, EC³, CVPR 2023), and organising two technical workshops
  • Collaborating with our sister projects, BEEYONDERS and RoBétArmé, with which we have created the #Tech4EUconstruction cluster, and StandICT.eu

During the project’s next phase, we will continue strengthening our online and offline presence, engaging our stakeholders in our progress and raising awareness of our work through more events, publications and communication campaigns.

We look forward to the second year of our project to continue working towards a safer, greener and more efficient construction industry with our outstanding team of diverse skills and an impressive openness to collaboration.

Subscribe to our newsletter and follow us on LinkedIn and Twitter to learn more about our work at HumanTech, and stay updated on our advances!