The challenge

Recent, exploitable, research in robotics and the associated fields of computer vision and sensors covers the components that are needed in an automated system that in one pass can perform inspection and assessment of the civil infrastructure in general and transportation tunnel infrastructure in particular.


Yet, such an automated system is cruelly missing today. The goal of this work is to offer an automated system that includes, integrated, all the required components for the inspection and assessment of tunnels in one pass. Moreover, this solution can automatically adapt to different use cases. The work includes adaptation of the existing components to fit the requirements of the specific application. Field evaluation, under actual conditions, in a variety of cases, proves the exploitability of the proposed system in the short to medium term.


Leading end users (Halcrow, EUROTUNNEL, EOAS) in charge of maintenance of major rail tunnels (e.g., London Underground, EUROTUNNEL), and road tunnels (Egnatia Motorway) and leading producers of mobile inspection equipment for tunnels (VSH (Amberg)) are offering user requirements, access to their facilities to obtain datasets and field evaluation, while a large company (CAS(EADS)), in charge of exploitation of results, is considering the new market of inspection and assessment of the civil infrastructure via robots.

One of the greatest challenges facing engineers today is the inspection, assessment, maintenance1 and safe operation of the existing civil infrastructure such as, tunnels, bridges, roads, pipelines, and much more. Due to ageing, environmental factors, increased loading, change in use, damages caused by human/natural factors, inadequate or poor maintenance and deferred repairs, civil infrastructure is progressively deteriorating, urgently needing inspection, assessment and repair work. Nowhere is this need more apparent than in underground transportation tunnels, a large number of which have been in operation for more than half a century and there are widespread signs of deterioration, evidenced by an increase in the proportion of budgets spent on inspection and assessment. Things are bad to the point that there have been a number of failures resulting in collapses in tunnels in recent years which highlighted the need for better ways to inspect and assess tunnel stability of in service tunnels. One should add here that (a) the cost of new tunnel construction is very high and, thus, inspection, assessment and repair of the existing tunnel infrastructure is of utmost importance, (b) in the coming two decades, the rate of expansion of the transport infrastructure in the European Union (EU) will not keep pace with the increase in transport demand necessitating the maximization of the operational uptime of tunnels and, thus, maintenance should be proactive based on timely inspection and assessment, (c) the inspection and assessment should be speedy in order to minimise tunnel closures or partial closures, and (d) the engineering hours for tunnel inspection and assessment are severely limited.

Objectives

The specific scientific and technological objectives of this work are:

  1. To develop an automated and incrementally optimized robotic inspection and assessment system for tunnels that will include the computer vision system described in ‘2 to 5’, the sensor system in ‘6’ and laser equipment (Leica, TS30 or a similar/enhanced system available at the time of the project) able to measure radial deformation with an accuracy of 1 mm, while it will be integrated with the software in ‘7.’ It will be an autonomous system that will accept basic commands such as - advance, stop, take some or all of the measurements. The system will become incrementally optimized throughout the interaction with the environment and the computer vision system. A set of parametric behaviours which can be activated individually or simultaneously will be programmed:
    • In case of cast in place concrete linings stop when you detect a horizontal crack with a length of more than 1 m at the crown or sidewalls. In case of segmental concrete linings stop when you detect a horizontal joint opening with a length of more than 1 m at the crown or sidewalls.
    •  When you detect the above use the sensor system in ‘6’ to find the widest point or points. If this is less than 0.2 mm12, then advance with no more measurements. If the widest point is more than 0.2 mm, then (a) use the sensor system in ‘6’ to measure the crack depth at this point, (b) use TS30 to measure radial deformation at predefined points of the cross-section and (c) use the computer vision system to measure the distance between parallel cracks. Additionally, use the computer vision system to measure spalls, delaminations, rust stains, white deposits, exposed reinforcement. In segmental linings report the missing bolts, loose bolts and discoloured bolts.
    • Compare the above with previous measurements to determine the rate of change.
  2. To develop a computer vision system for tunnel inspection and assessment of the structural condition that will detect structural defects (e.g., cracking, spalling) and colour changes (evidence of material deterioration such as corrosion or efflorescence) at the inspected concrete lining intrados. This system, at a fast rate of about 1 m/sec, will acquire 2D images of the tunnel lining at a coarse level of detail applying fast object recognition techniques to identify areas of interest in the coarse 2D image and then, at a slower rate, concentrate the image acquisition on details of interest, thus allowing the higher resolution 3D sampling of these details. To increase the speed and reliability of the above system and to permit it to act as a controller to automate the way of inspecting the tunnels the partners will:
    • Apply hierarchical computer vision schemes so as to make the recognition accuracy just-in-time, and thus significantly reduce the time and the effort needed for such a visual inspection. This is important because it maximises the tunnel uptime.
    •  Extend state-of-the-art vision schemes related to Riemannian Manifolds geometry and constrained optimization methods for the purpose of extracting reliable, robust and precise 3D measurements (in millimetres accuracy) using either multi-view cameras or even monocular ones.
    •  Apply recent advances in active continuous learning to tunnels’ inspection mechanisms so as to achieve on-line understanding of the cracks as the system surveys the tunnels. That is, instead of transmitting the data to a central computer unit and then applying off-line learning strategies to improve the analysis, ROBINPECT continuously learn tunnels characteristics as it surveys certain specific tunnels’ surfaces.
    • Modify the velocity and the orientation of the robotic arm to improve the automation process of the inspection. This means that the vision system, apart from the inspection, it also acts as a controller to automate the way of inspecting the tunnels.
    • Incorporate recent state-of-the-art semi-supervised learning schemes towards detecting tunnel anomalies. Semi-supervision exploits a small set of labelled data to roughly train some initial classifiers that will be used to detect tunnel defects. Then, the abundant amount of unlabelled data will be exploited to re-adjust the classifier based on new knowledge that is currently collected from the robotic/control system. This is crucial for attaining an easy adaptation of the robotic system to different lining types and places, since it is practically impossible to label all the data captured from the tunnels due to time constraints and the huge financial cost required.
    • Allow socially-based feedback schemes exploiting common practices and civil engineering knowledge so that probable erroneous performances of the system are automatically adapted without requiring the users to acquire machine learning knowledge. The knowledge of civil engineers with different views on tunnels’ conditions should be taken into account at an automated way.
    • Allow gradual reconstruction of the rune surface, exploiting previous knowledge of the models so as to reduce the required time.
  3. To adopt specific methodologies for collecting data. Tunnels’ inspection is not a process in which many and free visual data are available. On the contrary, such type of information is in fact missing from the computer vision society. In this project new strategies for data collections will be developed from the early stages of the project and these strategies will be refined later on. Data collection is an important element in ROBO-SPECT since the quality of this dataset will affect the performance of the developed system.
  4. To extend already existing 3D reconstruction tools so that they can be able to precisely detect tunnels’ cracks and use these models at later stages so as to derive more robust knowledge on tunnels stresses. Computer vision tools and sensory data will be exploited towards this case.
  5. To promote a new evaluation framework through which it is possible to:
    • extract local knowledge from one type of tunnels that can be straight-forwardly extended to other types (compensating local and generic information)
    • incorporate social knowledge from several civil engineers
    • process data from diverse sources
    • compensate erroneous decisions by averaging the knowledge
    • improve the control mechanisms of the robots in terms of velocity of the inspection and the orientation based on the results of computer vision techniques.
  6. To develop a sensor system suitable for being installed on the robotic inspection system described at ‘1’ that will measure (a) the depth of cracks or the depth of the opening of joints of interest with an accuracy of 1 mm and (b) the width of these cracks and openings with an accuracy of 0.1 mm. The solution proposed for these measurements will be based on existing sensor technologies that will be exploited to assemble a sensor unit able to perform the required crack analysis. In particular, ultrasonic transducers will be employed for the generation and detection of ultrasonic waves whose scattering across the crack will be exploited to evaluate crack depth and width. In order to do that, the ultrasonic far and near field transmitted across the crack will be analyzed, using commercial piezoelectric sensors for ultrasound generation and for far field detection at a distance of few centimetres from the crack, whereas optical-acoustic, fibre-optic ultrasound sensors will be used to perform measurements on point areas of the ultrasonic near field scattering at a distance of few millimetres from the crack. The sensors will be integrated in the robotic platform on the moving arm, in order to be placed on cracks selected for measurement during tunnel inspection. The measurement procedure of the crack will be defined in order to be completed within 15 seconds on a single crack section and with a confidence (defined as the average number of measurements successfully performed on the total amount) of 90%. The fiber-optic sensors, based on ultrasound-sensitive, polymeric Fabry-Perot cavities on micromachined silicon frames, will be fabricated using a Micro-Opto-Mechanical technology developed by CNR in previous research projects (Belsito et al., 2011).

    Such sensors, operated with a telecommunication laser and a wavelength of 1520 nm, will allow for broadband detection of ultrasounds on extremely small areas (around 0.01 mm2), in close proximity to the crack, for ultrasonic near field detection. For the processing of the detected ultrasonic signal, both Time-Of-Flight (TOF) and Frequency Domain (FD) techniques will be tested, in order to reach the required goals in crack width/depth measurements, and the needed confidence level.
  7. To develop a quantitative structural assessment tool that based on input from the inspection with the system in ‘1’, construction information and information on the  operative environment, will automatically assess the structural condition and stability of the tunnel at the time of the inspection and at future times so that tunnel managers can decide on an immediate intervention or on the time for the next inspection.

    There can be a degradation of the lining as a function of time. Common material defects responsible for the latter degradation in concrete linings produce signs visible at the tunnel intrados (e.g., calcium leaching produces white deposits on the concrete surface and reinforcing steel corrosion produces brown/reddish staining of the surface). Once the computer vision system detects such material defects, quantitative predictive degradation models from the literature, modified to reflect the tunnel conditions, will be used to evaluate the change in the mechanical properties of the lining because of these defects as a function of time in terms of initial conditions (e.g., commissioning year) and identified influential operational parameters (e.g., volume of traffic). Successive inspection results with the computer vision system will be used to assess the rate of attack which will be used to update the above models. The quantitative models described above together with input from inspection on structural damage (e.g., cracks) will provide input for the assessment of the stiffness and resistance in lining sections of the tunnel cross-section under study. It will be used in quantitative, mechanical models that will be developed to assess the structural condition and safety of the tunnel lining at the time of the inspection based on measurements of the deformed shape of the tunnel cross section provided by the inspection in ‘1.’ Four cases will be covered: cast in place and segmental concrete linings where vertical loads are higher than horizontal loads (deep tunnels) and cast in place and segmental concrete linings where horizontal loads are higher than vertical loads. Lining deterioration as a function of time will cause the risk of structural failure to accelerate.
  8. To test, validate and benchmark the system in ‘1’ at the research infrastructure of tunnels of VSH early in the project and in real rail and road tunnel environments (at the London Post Office rail tunnels and Egnatia Motorways respectively) later in the project, in terms of its potential take up and operational deployment. The selected sections of the tunnels for the field inspections and assessments are needed until the external loads stabilise tests include a variety of damage, lighting and traffic conditions, different cross-section types and different space availability for tunnel inspection.

 

 

 

Expected results and impacts

1. Use case projects are geared towards opening potential new markets in the emerging service robot sector. Automated inspection of highway, railway and metro tunnels is a potential new market for the emerging service robot sector for several reasons including:

  • There is a need for speedy inspection. Inspection cannot take place when vehicles are using the tunnel and closing of some or all lanes for inspection can create traffic jams that in some cases have significant social and financial cost
  • There is a limited and decreasing labour force for tunnel inspection and assessment at a time when a significant fraction of the tunnel length is in need of inspection. Much of the tunnel infrastructure was constructed more than half a century ago and there is a wide spread evidence of deterioration. The facts that little is known about the long-term structural performance of tunnels and that inadequate tunnel structural safety can have grave consequences for a very large number of people (e.g., a metro line in a large city may transport 60,000 passengers/hour) make the need for inspection more pressing.
  • The working conditions for tunnel inspectors are unhealthy and unsafe.
  • The current employable technologies for tunnel inspection range from visual inspection that is slow, expensive and subjective, to the deployment of instruments for the more in depth assessment of sections of concern that is slow and expensive.

2. Higher use of robotics and stronger level of participation by EU companies.

  • Higher use of robotics in the emerging service robot sector: The civil infrastructure is ageing and there is a huge need for automated structural inspection and assessment. ROBO-SPECT, if successful, will make attractive the use of autonomous, integrated, robots by the transportation tunnel inspection industry, which is a new market for robots with high potential. Moreover, parts of this work can be easily modified to be useful for the inspection of the civil infrastructure in general, another new market for robots. For more, please see the response to expected impact 1 above.
  • Stronger level of participation by EU companies, including those not yet active in EU settings, in RTD in the emerging service robot market: CAS has been involved in projects involving robots for space and security. With this work they will be involved in RTD in the emerging service robot market. Moreover, EOAE, RISA, TECNIC and DBA are participating for the first time in an RTD project in the service robot market.
  • Industry and user driven RTD in the emerging service robot sector: Industry and users ((VSH, Halcrow, EOAE, EUROTUNNEL and CAS) are the driving force behind ROBO-SPECT.

3. Successful technology transfer in terms of volume and scale of innovative products and services in application areas that will include societal challenges (e.g., inspection) and professional services as well as new industry sectors which have not used robots so far.

Recent, exploitable, research in robotics and in the associated fields of computer vision and sensors covers the components that are needed in an automated system that in one pass can perform inspection and assessment of the civil infrastructure in general and transportation tunnel infrastructure in particular.
What is unique, and novel, in this work is that it offers an automated system that includes, integrated, all the required components for the inspection and assessment of tunnels in one pass. Moreover, this solution can automatically adapt to different use cases.
The work includes adaptation of existing components to fit the requirements of the specific application.

4. Increased visibility of the programme to the European citizen via traditional and new social media channels.

The partners will undertake a number of actions involving traditional and new social media channels to bring the project results to the attention of the public.

5. Robotic automation can play an important role in providing an answer to the major societal challenge of inspecting and assessing the ageing infrastructure.
Thus, the proposed inspection and assessment system will:

  • Permit proactive condition-based maintenance of tunnels. The ‘Corrective/Reactive Maintenance’ approach is presently the predominant method of tunnel maintenance: tunnels are rehabilitated when they fail. This method is being used despite the resulting high maintenance costs, reduced safety, reduced service level and environmental impact. ROBO-SPECT will permit optimisation of the timing of inspections (and thus, optimisation of the repair intervals) based on the probability of failure instead of the current schedule-based maintenance. This will allow maintenance to be performed prior to significant structural deterioration. The result is:
  • A significant decrease in life-cycle maintenance cost, because problems are less expensive to fix when they are first developing which is most important at a time of shrinking budgets.
  • An increase in tunnel safety
  • The elimination of the need for emergency repairs that are inelastic in terms of timing and require a lot of time to perform and often necessitate service interruptions. This will lead to an increased time capacity for freight and passengers at lower cost than providing additional infrastructure, to reduced environmental impact due to reduced construction activity and to reduced disturbance for industries depending on prompt deliveries. It will also lead to a reduced disturbance for passengers travelling by trains and to lower costs for the railway authorities when in case ofservice interruptions for unplanned tunnel repairs, they have to buss the passengers to their destinations or around the repair area. For road tunnels traffic deviations because of tunnel repairs can be lengthy. They often result in traffic jams, increased noise, vibration and vehicle emissions, with local health and environmental problems. Moreover, road safety is reduced on detours because people are driving on poorer, unfamiliar roads, generally in bad mood. Prolonged repair works can also affect local employment and cause business failures, because mobility is so important.
  •  Reduce tunnel closures or partial closures because of tunnel inspections. The impact of this will be significant for tunnels with heavy traffic volume (There are traffic jams in some such tunnels even without tunnel partial closures: For the Tauern tunnel for example, traffic jams of 100 km are not any more considered exceptional). For the results of traffic jams see discussion above.
  • Increase the residual lifetime of existing tunnels. This will preserve natural resources, result in significant cost savings25 and increase transportation capacity.
  • Minimise use of scarce tunnel inspectors. Presently, tunnel structural maintenance is largely based on biannual visual inspections (FHWA et al., 2005). Such inspections are labour intensive and subjective. The proposed automated system will provide objective results and minimise use of inspectors.
  • Improve the working conditions of tunnel inspectors that will not have to reach difficult to access tunnel parts and in severe climatic conditions.
  • Result in a dramatic decrease in the engineering time needed to assess the tunnel structural condition
  • Provide better quality, objective, timely data and an improved knowledge of the structural response to ambient disturbances of in-service tunnels over their lifespan. This will have an influence on the initial design phase as well making possible to avoid unnecessarily conservative stress analysis procedures.
  • Promote the use of robotics in the civil infrastructure inspection sector.
  • Increase the competitiveness of the European robotics industry through new high added value products.
  • Strengthen the global competitiveness of the European tunnel inspection industry
  • Create employment. A system like the proposed one does not exist in the market. Thus, it appears an opportunity for the partners to access the European and the international tunnel inspection market creating employment for the system production, use and maintenance.

This project is funded by the European Union