Automatic morphologic analysis of quasi-periodic masonry walls from LiDAR
Belén Riveiro*
Applied Geotechnologies Group, Department of Materials Engineering, Applied Mechanics and Construction, University of Vigo, C.P. 36208, Vigo, Spain.
Paulo B. Lourenço, Daniel V. Oliveira
ISISE, University of Minho, Department of Civil Engineering, Azurém, 4800-058 Guimarães, Portugal.
&
Higinio González-Jorge, Pedro Arias
Applied Geotechnologies Group, Department of Natural Resources and Environmental Engineering, University of Vigo, C.P.
36310, Vigo, Spain
Abstract: This article presents a novel segmentation algorithm that allows the automatic segmentation of masonry blocks from a 3D point cloud acquired with LiDAR technology, for both stationary and mobile devices.
The point cloud segmentation algorithm is based on a 2.5D approach that creates images based on the intensity attribute of LiDAR systems. Image processing algorithms based on an improvement of the marked-controlled watershed was successfully used to produce the automatic segmentation of the point cloud in the 3D space isolating each individual stone block. Finally, morphologic analysis in two case studies has been carried out. The morphologic analysis provides information about the assemblage of masonry pieces which is valuable for the structural evaluation of masonry buildings.
1 INTRODUCTION
Masonry is a key material along the history of construction. Nowadays many modern typologies of masonry are used in construction, but over several millennia natural stone, earth blocks and clay bricks were the basis for human shelter around the world. It is easy to demonstrate the robustness of masonry constructions through ancient structures, which, in many cases, are still in service after thousands or hundreds of years. These structures suffered from ageing effects, from more
demanding service conditions than those for which they were conceived, such as extreme events, and other actions.
Until the 20th century, this material was the most used in construction, being the structural basis of many existing buildings.
Studies demonstrate that Roman Engineers had geometrical rules when designing their structures, which were partially imitated and improved in later times.
Nowadays, advanced analytical and computational procedures allow conducting designs in a more accurate, economical and safe way, but sometimes they involve a great effort in terms of time and costs. The advanced analysis of masonry structures received considerable attention in the recent decades and a review is given in Lourenço (2002). Despite all the improvements, the challenges in the seismic assessment of these structures remain rather complex (Lourenço et al., 2011). Salient aspects in the analysis of historic structures are that: (a) geometric data is missing; (b) information about the inner core of the structural elements are also missing; (c) characterization of the mechanical properties of the materials used is difficult and expensive; (d) there is a large variability of mechanical properties, because of workmanship and the use of natural materials; (e) significant changes have occurred in the core and constitution of structural elements, associated with long construction periods; (f) the construction sequence is unknown; (g) existing damages in the structure is unknown;
(h) regulations and codes are hardly applicable. Modern recommendations for structural analysis (ICOMOS, 2003)
address the aspects of uncertainties and personal judgments, by recognizing the contribution of different sources of information using an historic approach (Knowing from history and using the building as true-scale experiment, real-scale information), a qualitative approach (an inductive procedure that compares and extrapolates from other buildings), an analytical approach (a deductive procedure based on structural analysis) and an experimental approach (experiments on individual components or the entire building). Still, the quantitative approach which uses structural analysis remains a major contribution for the safety assessment.
Masonry is a heterogeneous material that consists of units and joints. Such units include bricks, blocks, ashlars, adobes, irregular stones and others. Mortar can be clay, bitumen, chalk, lime/cement based mortar, glue, etc.
In general, masonry can be classified in two main categories: rubble masonry and ashlar masonry. Rubble masonry can be subdivided into uncoursed masonry (which is the form with poorest structural functions) and coursed masonry (mortared or dry joints). Ashlar masonry comprises blocks with regular dimensions that are carefully dressed. Uncoursed rubble masonry is principally used in small walls without structural relevance, whilst the other classes mentioned are those used for large constructions or buildings. As it will be explained later in this paper, uncoursed rubble masonry will not be studied in the present investigation.
The huge number of possible combinations generated by the geometry, nature and arrangement of units as well as the characteristics of mortars raises doubts about the accuracy of the term “masonry”. Still, masonry can be considered a composite material that consists of units and mortar joints.
However, due to its composite nature, masonry exhibits distinct directional properties due to the mortar joints which act as planes of weakness. In general, the approach towards its numerical representation can focus on the micro- modelling of the individual components, which are unit (brick, block, etc.) and mortar, or the macro-modelling of masonry as a composite). Depending on the level of accuracy and the simplicity desired, it is possible to use the following modelling strategies, see Figure 1: (a) Detailed micro-modelling - units and mortar in the joints are represented by continuum elements whereas the unit-mortar interface is represented by discontinuous elements; (b) Simplified micro-modelling - expanded units are represented by continuum elements whereas the behaviour of the mortar joints and unit-mortar interface is lumped in discontinuous elements; (c) Macro-modelling - units, mortar and unit-mortar interface are smeared out in the continuum. Macro-modelling is more practice oriented due to the reduced time and memory requirements as well as a user-friendly mesh generation. This type of modelling is most valuable when a compromise between accuracy and efficiency is needed. Still, much effort has been made
recently to incorporate the geometry of masonry in structural analysis, thereby creating a composite material by resorting to homogenization techniques, see Lourenço et al.
(2007) for a review. These techniques have been recently extended to random or quasi-periodic masonry, e.g. Cluni and Gusella (2004) or Milani and Lourenço (2010).
The present article aims at developing a fast and reliable technique for obtaining the morphologic characterization of blocks in quasi-periodic masonry walls, as a key aspect in masonry mechanics. The automation of this process coupled with advanced structural analysis software using either a micro-modelling strategy, in which all blocks are modelled individually, or homogenization approaches, in which an equivalent continuum is defined from the observed micro-structure. For this purpose, laser scanning data was used as a basic source to identify the geometry of the component blocks in masonry walls. Also, to optimize the use of this technique and avoid the tedious data processing tasks, an approach that automates the detection of single masonry units and its dimensions is developed.
This paper presents the methodologies for mobile and static terrestrial laser scanning platforms, including the results of the automatic processing. Also the results of the algorithms are validated at the end of this study.
2 LASER SCANNING AND IMAGE PROCESSING 2.1 Background
This technology was successfully applied to many different fields, from archaeological and architectural documentation (Rodríguez-Gonzálvez et al. 2012; Kang et al. 2013); industrial applications (Walsh et al. 2013), and structural applications (McInerney et al. 2012), among others. Specific applications to structural engineering can be seen in (Riveiro et al., 2011b) who developed a methodology that allows point clouds acquired by terrestrial laser scanning to be converted into metric images, which can be subsequently used for stability analysis of masonry arches. Truong-Hong et al, 2012 focused in the automatic processing of point clouds in order to automatically built geometric models suitable for structural purposes. Park et al. (2007) showed the application of LiDAR (Light
Figure 1 Modelling strategies for masonry structures:
(a) detailed micro-modelling; (b) simplified micro- modelling; (c) macro-modelling.
Detection and Ranging) data for accurately measuring deflection of loaded beams. Later, Lee & Park (2011) demonstrated how laser scanning might help to determine beam stresses in combination with Finite Element Modelling (FEM). In general, laser scanning is gaining popularity in the field of structural and civil engineering.
Also, the advances in automatic data processing are easing the implementation of this geomatic technology in many aspects for infrastructure management. Examples of mobile LiDAR advances are reported by Puente et al, 2013a and Varela-Gonzalez, et al. 2013.
Laser scanning is a geomatic method that allows obtaining 3D geometry of objects’ surface in an automatic way by using LiDAR technology. This technology allows distances to be remotely computed by measuring the time delay between the emission and return of a laser beam that travels from the instrument (a laser scanner) to the object.
Also, laser scanners contain mirrors to deflect the laser beam in different directions in order to efficiently cover the whole field of view of the instrument. In terrestrial laser scanners (TLS) (static scanners) it involves directing the laser in vertical directions (rotation around horizontal axis), while rotation around vertical axis may be done through another mirror or through a mechanism. As a result, a very dense point cloud is obtained. This point cloud basically contains the X, Y and Z coordinates of each surface point, and some other attributes such as amplitude of the laser beam or RGB data.
During the last years mobile LiDAR technology (on-the- fly) is being more and more available. The basic principle of measuring distances is the same as that of TLS, but the creation of the 3D point cloud slightly differs. A mobile LiDAR system is formed by two main subsystems (Graham 2010, Petri 2010). First, a navigation system for geo- referencing the instrument during its motion (normally mounted on vehicle-borne platform). The second subsystem is the laser ranging system. The main difference of this instrumentation compared to the terrestrial scanners described previously is due to the degrees of freedom of the deflection mechanism: laser scanners of mobile systems deflect the laser beam only around one axis so that the point cloud recorded is contained in a unique plane. The point cloud in the 3D space is obtained as the scanned profiles are fanned out along the trajectory due to the vehicle motion.
All the measuring units are vehicle oriented.
2.2 Intensity data
Laser scanners not only provide information about the geometric position of object surfaces but also information about the portion of energy reflected by the object’s surface, which depends on its reflectance characteristics.
The backscatter generated after the collision of the laser beam with the object surface is recorded by most of LiDAR instruments as a function of time (Höfle and Pfeiffer 2008).
Most systems directly provide a discrete number of signal echoes with their respective signal intensity (for further details see Stilla & Judzi, in Shan & Toth 2009).
Signal intensity depends on biconical reflectance (Höfle and Pfeiffer 2008; Wagner, 2010) which is defined as the ratio of reflected to incident flux (Nicodemus et al. 1977).
Materials with Lambertian characteristics may be detected with respect to the response of this property through the spectrum, and this is usually denoted by spectral curve.
Knowing the reflectance value for a particular wavelength would allow identifying the material being observed.
2.3 Instrumentation
The proposed methods include data collected through static laser scanners (TLS) as well as those using mobile technology (MLS).
2.3.1 TLS: Riegl LMS Z390i
For the case studies surveyed in this research, a static scanner manufactured by Riegl model LMS-Z390i was used (Riegl, 2013). This instrument exploits the time of flight principle for computing the range between the scanner and the object, for distances in a range of 1.5 to 400 m. The nominal accuracy of this instrument is 6 mm (one sigma) at 50 m in normal illumination and reflectivity conditions. The laser is infrared and has a wavelength of 1.54 μm. The viewing field has 80 degrees vertical amplitude and 360 degrees horizontal amplitude. The minimum and maximum angular resolutions are 0.2 and 0.002 degrees (equivalent to 4mm at a range of 25m), respectively, and the point acquisition rate ranges between 8,000 and 11,000 points per second. The beam divergence is 0.3 mrad giving a 30 mm diameter spot at 100 m range.
The beam divergence is 0.3 mrad, equivalent to 30 mm per 100 m range.
This scanner is able to measure only one echo and record the attribute of intensity with a radiometric resolution of 8- bit.
2.3.2 MLS: Lynx Mobile Mapping
The mobile laser scanning system used for the case study was the Lynx Mobile Mapper, released at the end of 2007 by OPTECH Inc. (Optech, 2012). The Lynx is composed of a LiDAR sub-system plus a navigation system. Two LiDAR sensor heads with a nominal accuracy of 8 mm (one sigma) are mounted forming an angle of 90º between their respective rotation axis, and 45º with respect to the trajectory of the vehicle. This configuration improves the field of view, thereby avoiding occlusions. These sensors allow collecting survey-grade LiDAR data at 500,000 measurements per second with a 360º FOV (each scanner).
The navigation system incorporates the POS LV 520 produced by Applanix, which integrates an IMU with a 2-
antenna heading measurement system (GAMS), providing 0.015º in heading, 0.005º in roll and pitch, 0.02 m in X,Y position and 0.05 m in Z position. All those values are determined by differential GPS post-processing after data collection using GPS base station data (Puente et al, 2013b).
The Optech LiDAR provides up to four echoes per pulse with their respective intensity attribute. The radiometric resolution of intensity provided by the systems is 12-bit.
2.4 Data processing
As explained in section 2.1, the process of formation of the 3D point cloud in static and on-the-fly scanning systems is not the same. While static scanners record the point cloud in a spherical coordinate system directly, mobile systems need pre-processing the trajectory registered by the navigation unit and the range data (LiDAR) separately. In any case, handling these preliminary files is not trivial but advanced knowledge in geomatics is required.
Static scanners have evolved significantly and most of the instruments already allow exporting the point cloud converted to a Cartesian coordinate system easily.
Nevertheless, automation of mobile LiDAR data is still pendent. In addition to these limitations, probably, the most important restriction is related to the use of specific software packages and their associated costs. More details related to pre-processing of static and mobile scanning data may be seen in (Riveiro et al, 2011b) and (Puente et al, 2013b) respectively.
The main challenge for non-experts in geomatics, when handling laser scanning data, is point cloud processing to produce exploitable product. The process chain may be summarized in the diagram shown in Fig. 2. Traditional processing normally needs high performance computers to accomplish the tasks involved.
Point cloud (for both TLS and MLS) normally comprises a tedious and costly processing task, and with a high concentration from the human operator. These aspects make the implementation of LiDAR technology in many fields difficult, because such implementation normally requires specific training in the technology. Additionally, even if LiDAR manufacturers are launching free software packages for point cloud visualization, a final product production requires a license purchase that may imply a significant cost in the global budget of the problem to be solved with the project. In conclusion, all these factors indicate that technicians outside the geomatic domain can hardly access LiDAR products.
2.5 Segmentation of masonry from imagery
As presented in the introduction section, there exist two wide classes of masonry: rubble masonry (uncoursed and coursed) and ashlar masonry. From the point of view of image segmentation, the perception of uncoursed rubble masonry and the rest of assembly classes may differ. This is motivated by the perception of contacts: in dressed masonry the contacts have a very small thickness compared to masonry pieces so they are hardly perceived at certain distances, however in uncoursed masonry the contacts between adjacent blocks represent a short portion of the perimeter so contact class and block class can be distinguish easier.
Even masonry has not been largely studied in terms of image processing there are some relevant works in this regards. Yue et al. (2003) have presented two segmentation methods applied to a cylindrical asphalt concrete sample that is scalable to uncoursed rubble masonry. The overall workflow consisted of the thresholding and subsequent contour detection by two different methods: a) using a region segmentation method, and b) using an edge detection method; where the second one offered more suitable results for their investigations. Sithole, G. (2008) presented an approach to detect bricks in masonry walls from TLS. He proposed a method based on weighted proximity segmentation under the assumption that the mortar channel is reasonably deep and wide, which is not always the case of coursed masonry walls. Also, point cloud data needed to be processed from each scanner position, and a triangulated mesh was required to be one of the input data together with the point cloud. Even the method was applied to both the uncoursed and coursed masonry, results are not fully satisfactory. Oses et al. (2014) present an interesting Figure 2 Point cloud processing to obtain metrical
data to use for morphologic analysis. Traditional workflow (left) and new 2.5D approach to automate the
point cloud processing (right).
approach for the delineation and classification of masonry walls. The purpose of their work is the automatic classification from imagery according to masonry assembly. They focus in the classification of the entire wall instead of individual blocks because they state that delineation of each individual block is infeasible. They propose a method of delineation, and then classification is performed using machine learning classifiers (K nearest neighbor (K-NN), support vector machines (SVMs), naive Bayes (NB), and classification trees (C4.5).
Even the mentioned works propose measures to geometrically characterize masonry walls in an automatic manner, the results found do not solve the problem of characterizing the walls based on the geometry of individual pieces, so the morphology of the wall cannot be quantified.
This paper focuses on the geometric characterization at the level of the individual block comprising the masonry wall in order to carry out subsequent structural analysis of the entire wall based on masonry assembly. Because of the differences between uncoursed masonry and course masonry mentioned above, this paper concentrated in coursed masonry walls (pieces in a given course are not restricted to have the same height) since it is the most common assembly typology presented in large constructions, and with more important structural missions.
3 PROPOSED METHODOLOGY
To ease the usage of LiDAR data in structural engineering applications, an algorithm focused on automatic processing of point clouds was developed using Matlab software®. This algorithm is based in a 2.5D approach as shown in Figure 2 by exploiting the attribute of the intensity of laser beam. This algorithm may be summarized in three main tasks: data preparation: raw data to point cloud; data pre-processing: conversion from 3D to 2D space; data processing: Morphological image processing and image segmentation.
3.1 Data preparation: LiDAR raw data to point cloud The first step consists of exporting the point cloud measured by the scanner to an ASCII file into a Cartesian coordinate system where each point is defined by orthogonal coordinates (X,Y,Z) and the attribute of intensity. The resolution of the intensity depends on each scanner model.
In the case of point clouds measured by static scanners (TLS), the point cloud is automatically converted from the spherical coordinate system centred in the scanner to the Cartesian one. The TLS Riegl instrument used in for the case studies of this paper is managed through the software Riscan Pro®.
In the case of mobile technology, the creation of the point cloud is not immediate and requires special processing. First, the smooth best estimated trajectory (SBET) is computed, by using the software POSPac MMS 5.4®, from the data gathered by the navigation system during the survey (the accuracy of the GNSS additional measurements from a base station in Braga located 20 km far from Guimarães). The variations in the measured GNSS location at the base station and its surveyed location are recorded and timed during the mission, and applied soon afterwards to the trajectory data. Then, using DASHMap software the raw LiDAR data measured by the Lynx system are processed together with the SBET and is used to calculate a geo-referenced point cloud in 3D space containing the intensity data.
3.2 Data pre-processing: 2.5D approach
The aim of this paper was to perform morphologic analysis of quasi-periodic masonry walls in an automatic way. For that purpose, geometric segmentation of LiDAR data is accomplished by a 2.5D approach. This approach consists of reducing the 3D point cloud coloured by intensity to the 2D domain to create 2D intensity images.
The core idea of the method consisted of latticing this planar point cloud of the masonry structure into an organized point cloud. Raster models are commonly used for geospatial operations due to their easy storage and processing, as well as versatility. Each pixel is coloured by averaging the intensity attribute of all corresponding points from the point cloud. As previously said, intensity attribute is directly related to the spectral characteristics of materials so the two principal objects in a masonry wall (stone and joint) may be distinguished.
The plane of projection was computed as the plane that best fits the point cloud in the masonry wall. This operation was computed by hyper-planar fitting using orthogonal regression. This model uses least squares where the errors are measured orthogonally to the proposed plane.
Once this plane is obtained, the orthogonal projection of the points forming the 3D point cloud is immediate. Once the point cloud is in the 2D space, the next task consists of converting this structure, composed of isolated points, into a raster image so the grid that acts as the pixel structure is defined by the resolution (pixel size) and boundary (calculated from the maximum and minimum coordinates of the point cloud). The user can fix the resolution of the image (pixel size) or estimating a value depending on point cloud density in the 2D space. As a consequence, a raster grid of m rows and n columns is built.
In continuation, the next step is defining the intensity attribute for each element in the raster structure. The intensity attribute of an area S of the masonry wall can be modelled as the random field {I(s): s ∈S ⊂ R2}. The set of points of the point cloud covering the element area can be considered as the collection of independent observations at
locations s = {s1, s2, … , sn} on the random field, and is denoted by the data vector I(s) = {I(s1), I(s2), …, I(sn)}.
According to the 2.5D approach explained above, the raster representation of the point cloud consists of latticing the continuous domain S and computing a value of intensity I for each raster element (pixel). For a given pixel defined by the region P and the corresponding area |P|, it is possible to estimate the intensity of the pixel averaging the random field in P (equation 1):
(1)
The value of the I(P) is computed by using the observed data (point cloud sample) contained in the region of the pixel. Consequently, the spatial resolution of the results of the classification proposed here will be constrained by the window size for I prediction: the pixel size. Figure 3 (top) presents the intensity image corresponding to a masonry wall. Digital numbers correspond to infrared spectrum (1550 nm in the Lynx Mobile Mapper and 1540 nm in the Riegl LMS Z-390i ).
3.3 Data processing: Geometrically-constrained marker controlled watershed segmentation
Once the raster structure is built and the value of intensity I of each pixel is computed, a segmentation strategy that allowed differentiating masonry blocks and joints (with mortar or dry joint) is developed. The core objective is to isolate masonry blocks to subsequently perform the morphological analysis for each individual stone unit.
The segmentation objective was to distinguish masonry from joints (with mortar or dry), however the micro-relief of the granite causes that many isolated pixels corresponding to masonry show similar intensity values than joints, and so, clustering does not provide useful results. Based on those unsuccessful results, an approach based on geometrically constrained marker-controlled watershed segmentation is proposed to obtain a correct image partition.
Watershed is an algorithm within the framework of mathematical morphology that allows partitioning an image on the basis of assuming that a graytone image can be viewed as a topographical surface where the graytone of each pixel represents the height of the surface (Beucher &
Lantuéjoul, 1979). Figure 3 (bottom) illustrates a 1D signal (corresponding to an image row of the intensity image).
Rigorous mathematical definitions of the watershed line were presented in (Meyer, 1993; Meyer, 1994). One of the most common problems encountered when performing a direct watershed transformation is over-segmentation due to noise. The intensity image of granite stone easily gives
incorrect results for the segmentation. According to (Gonzalez et al, 2009), an approach to avoid over- segmentation consists of using markers. Several publications present strategies to get an efficient marking map (Beare, 2006; Wang & Vallotton, 2010), but, for the case of masonry walls as analysed in this paper, the proposed marking do not provide proper results principally due to the short thickness of block contours. For this reason, a geometrically constrained marking process was developed.
The original intensity image was used to produce the markers. To reduce noise caused by the texture of the granite components in the intensity image, a median filter (medfilt2 function) followed by a lowpass Gaussian filter (using the fspecial function) is applied. The medfilt2 function computes the value of each pixel as the median value of a rectangular neighbourhood centered on the corresponding pixel in the original image. The fspecial function is used to obtain a rotationally symmetric Gaussian lowpass filter whose standard deviation is imposed.
Subsequently, this filter is applied to the image where the noise caused by the stone texture and missed pixels is significantly reduced.
Afterwards, directional derivatives to the image (using
“Sobel” operator) were computed along the principal directions of masonry joints: horizontal gradient image HGI and vertical gradient image VGI. The overall geometry of masonry courses was assumed to follow horizontal rows and the next step consisted of identifying the horizontal joints between adjacent masonry courses. The strategy involved computing the sum of I, R(i), of all those pixels corresponding to each row in the Horizontal Gradient Image HGI (equation 2).
Figure 3 1D image (bottom) extracted from an intensity image of a masonry wall (top), where vertical
axis represents the value of intensity (I) of the pixels and horizontal axis the pixels contained in the segment
line AB respectively.
(2)
where:
i=1,2,…,m, and m being the number of rows in the intensity image.
j is the column index, while n is the number of columns in the intensity image.
An example of intensity image for segmentation is presented in figure 4a. In order to easily identify the rows of pixels that correspond to joints, a gradient of the function R(i) was computed so the peaks of this function denote the presence of joints (figure 4b). To compute peaks, the findpeaks function was used constraining the number of events based on the minimum distance between adjacent peaks. This assumption comes from the fact that the masonry blocks have a minimum dimensions. This avoids to find two peaks at both of the edges of a joint, whose distance in between is less than few centimetres. The output of this step was a vector with the indexes R that serves for the partition of the original image in a discrete number of sub-images of corresponding masonry courses; r is the number of indexes detected.
The next step has to do with performing the previous operation again while focusing on detecting peaks that enables the identification of the joints between blocks for each sub-image in the vertical gradient image VGI (equation 3), whose number of image rows is the distance between peaks in the previous step: for k=1:r, the number of rows in the sub-image is R(k+1)-R(k), and the number of columns is n.
(3)
Figure 4c shows the gradient of this new function CR(j) for each masonry course k where peaks provide the indexes C for the pixels that represent joints between adjacent blocks in the same course. Peaks were computed as explained for the HGI image. c is number of joints found in each course.
As explained earlier in section 2.5, masonry pieces may present different heights in the same course, so for each of the block detected in the previous step, heights are again computed using equation 2, where the number of image columns is the distance between peaks in the previous step:
for l=1:c, the number of columns is C(l+1)-C(l), and the number of rows was fixed at 3·(R(k+1)-R(k)). The height is fixed at this value in order to account by stones with significant height differences into the same course as shown in figure 4b (blue rectangle). Figure 4d shows the detection of peaks for height for each stone piece.
Using the pixel coordinates provided by the previous steps, a wireframe image denoting the four corners of each stone block is built. These rectangles are eroded so that the markers for each stone are obtained. Figure 4e represents
Figure 4 Definition of markers based on geometry for the watershed segmentation. a) Intensity image; b) detection of joints between masonry courses in the gradient image; c) detection of joints between adjacent ashlars for each masonry course; d) wireframe denoting
the corners for the markers of each block; e) markers after erosion.
the markers used for the watershed segmentation which are superimposed on the graytone image of the masonry wall.
Finally, a watershed segmentation using the watershed function is carried out to improve the outlining of stone blocks according to the real contour of each stone.
Figure 5 illustrates the results found in a wall were courses have not constant height.
At the end of the process, the results of segmentation found in the 2D space are projected back to the 3D point cloud so length and height are computed for each ashlar, as well as its area and perimeter (Figure 6). Also, the labelling of each pixel after segmentation may be used to colour the original point cloud. These are possible because each point in the point cloud is geometrically related to each pixel of the intensity image, so the results of transformation are directly projectable to the original point cloud in the 3D space.
4 RESULTS FOR THE WALLS OF GUIMARÃES
4.1 Case studies: Guimarães Castle and Medieval Defence Wall of Guimarães
To validate the performance of the algorithm presented here, two masonry constructions in Guimarães (Portugal) were surveyed by laser scanning: walls of Guimarães Castle, and the Medieval Wall of Guimarães. The first one was surveyed using the terrestrial laser scanner in static mode. The second construction was measured using the mobile mapping system by using on-the-fly mode.
4.1.1. Guimarães Castle
Guimarães Castle (Figure 7a) shows a regular five-sided polygonal plan with the shape of a shield (Figure 7c). The castle wall is composed of wall sections reinforced through eight turrets which envelope a keep tower. The height of the castle wall varies between 6m and 8m in the interior faces and at the exterior faces the height variation is higher depending on the presence of granitic outcrops. The turrets have heights between 12m and 20 m.
Part of the walls still preserve the Romanesque architecture where ashlars were disposed and organized maintaining a similar height for the pieces of each course and slightly different height among different courses. The validation of the algorithm presented in this article was performed in the entire external face of the Castle Wall.
4.1.2. Medieval Wall
In the 13th century, during the reign of Afonso III, the defence walls were extended to the village of Guimarães thereby turning it into a walled village, so that the bailey moved outside the keep region, see Figure 7b. The Figure 6 Results of segmentation projected back to
the 3D space in a curved surface.
Figure 5 Markers and segmentation results in a wall where courses have stones with different heights. a) original intensity image; b) markers generated the for
watershed; c) results of segmentation.
Figure 7 Virtual reconstruction of the walls of Guimarães (bottom), containing castle (a) and mediaeval wall encircled in (b); c) plan view of castle
fortification walls; and d) shows the mediaeval wall encircled in the point cloud collected during the
survey.
validation of the algorithm presented in this article was performed over a stretch 230 m long of the wall with a straight direction. This wall contains battlements similarly to the Castle and the masonry has a regular course bond.
4.2 Laser scanning survey and segmentation results.
4.2.1. TLS in Guimarães Castle
The survey of the defence walls of Guimarães Castle was performed using the terrestrial laser scanner Riegl described in section 2.3.1. This survey required setting up 11 scanner positions to cover the scanning of the whole envelope of the masonry construction. The scanning protocol is the same as the one presented in (Riveiro et al., 2011b). Table 1 presents a summary of the TLS survey.
After registering the point cloud of the Castle walls, the next step is manual selection of the walls of interest. After exporting the point cloud from the software that controls the terrestrial laser scanner (RiscanPro®), the point cloud can be imported in any point cloud reader that permits surfing and performing customized point cloud viewers under free licenses terms. This operation needs to be manually controlled by a human operator but does not require any special training in geomatics.
Once the walls of interest were extracted from the global point cloud the developed algorithm could be run. From this case study, the Alcaçova wall is used to illustrate the results (Figure 8). Note that the entire curtain walls of the Castle were processed by the algorithm, but Alçacova wall is used to illustrate the morphologic results in this construction. For the entire curtain walls 12 manually selected point clouds were selected from the global point cloud, and the Alcaçova wall was completely included in one of the selected point clouds. In the case of the Alcaçova wall, once the intensity image was created, 8 frames were used to complete the morphologic analysis of the entire wall. If the window cavities are located at the same height in the wall, and with
the same window height they are classified as a stones.
However, if the windows are irregularly located in the wall, they might affect the results of segmentation, so they should not be included in the frames of analysis. This limitations force to divide the wall in a larger number of wall regions.
The intensity image of this wall was built for a pixel size of 2 cm, which is a spatial resolution that allows differentiating joints in the whole wall. Note that zones closer to the scanner contain more points that describe the wall morphology more clearly for the human eye, principally when frontal views are registered (figure 9a).
The further areas are not so clear for the human eye to see due to obliqueness (as shown in figure 9b), however they can be distinguished by the algorithm properly. Figure 9 shows the segmentation results in the two zones of the Alcaçova Wall with different image quality (a) close to scanner zone (frontal view) and (b) far from the scanner zone and with an important obliqueness.
After applying the algorithm to the entire wall, a morphologic analysis was performed. Figure 10 shows the histograms of length and height of masonry stones of the entire castle walls. In the case of evaluating the entire Alcaçova wall, the probability density function for length indicates that distribution fits to a lognormal distribution, however heights are randomly distributed. Average values and sigma for both dimensions are summarized in table 2.
Table 1
TLS survey of curtain walls in Guimarães Castle (external face).
Figure 8 Intensity image of the Alcaçova Wall.
4.2.2. MLS in the Wall of Guimarães
The survey of the mediaeval Wall of Guimarães was performed using the mobile laser scanning system Lynx Mobile Mapper described in section 2.3.2. The wall is located in a street with traffic circulation limited to speed of 30km/h. The survey was conducted in a cloudy day at midday with normal traffic condition (traffic was not stopped during survey). The scanning time in the wall was 237.03 seconds when the vehicle was driven with an average speed of 1.01m/s. The two LiDAR sensor heads were measured with an acquisition rate of 250 000 points per second, thereby a dense point cloud was registered.
Figure 6d (top) shows the point cloud collected in the street of the Wall, as well as in the surrounding area where building façades, urban objects and people are accurately identified. The point cloud of the wall was cropped from the global point cloud to ease the processing of the data.
Figure 6 (top) contains points used for the subsequent processing which corresponds to the point cloud collected by a single LiDAR sensor.
Figure 11 Sample window of an intensity image created from MLS data: original intensity image 8bit
(top); watershed segmentation results (middle);
superimposition of segmented image and original image (bottom).
Table 2
Average values and standard deviation for both dimensions of the masonry walls studied.
Figure 9 Result of segmentation in two zones of Alcaçova Wall with different image quality.
Figure 10 Histograms and lognormal probability distribution of masonry dimensions for Alcaçova Wall.
The incidence angle and range may affect the intensity attribute of each point. Consequently, this motivated the decision of using the data captured by both sensors separately, because each sensor is measuring the points of the wall from different positions and would provoke that, for example, two neighbour points show different intensity values (caused by different range or different incidence angle) even they belong to the same material. The point cloud corresponding to the masonry wall used for processing contained 7.8 million points.
It must be noted that the spatial resolution was significantly lower than in the static instrument. This caused intensity images with reduced quality as shown in Figure 11 (a). The intensity image of the wall was created with a pixel size of 3 cm. Figure 11 (b) show worse segmentation results than those obtained in the Castle walls, however the dimensions of blocks were found with enough accuracy for structural analysis purposes. The watershed segmentation of the point cloud can be seen in figure 11(c) which is very consistent.
Finally, the probability density function for block dimensions showed the results presented in figure 12.
Similarly, the length follows a lognormal distribution, but the heights are randomly distributed like that of Alcaçova wall. Mean values and sigma are shown in table 2.
4.3 Geometric validation of the segmentation
Before accepting the algorithm as a valid tool for the automatic geometry extraction of masonry walls, a validation of the measures was performed. For that purpose, both laser scanners were tested for the same wall stretch:
TLS with higher resolution and MLS with centimetre resolution. In the images built from MLS point clouds the pixel size was 3cm, while, for TLS images, the resolution was 1cm.
Once the survey images were selected, these sample windows were searched into the original point clouds. It should be noted that for large objects such as the walls shown in this paper, laser scanning is the most suitable method, whose metrological characteristic are presented in section 2.3. Validation was performed in a set of 200 granitic ashlars.
The checking of measures was performed by an operator with knowledge in masonry structures by manual selection of averaged length and height of each stone piece. Riscan Pro software was used for both types of point clouds. The selection of the most suitable distance was not an easy task since the measurements had to be performed in the 3D space. Therefore, three measurements for each distance were performed to provide a distance value, with about 1cm variability.
The differences between automatic measurement and manual measurement in the real geometry showed the distribution in Figure 12. Figure 13 (top) shows a normal distribution of differences (cm) for TLS sample, where the mean has a value of –0.19 cm and a standard deviation of 1.13 cm. When accounting for absolute values of distances, the mean has a value of 0.79 cm with a standard deviation of 0.82 cm. Figure 13 (bottom) presents the distribution of differences (cm) for the point cloud captured by the MLS with a Gaussian shape where mean is –2.0 cm and a standard deviation of 2.56 cm. If differences are measured in absolute value, the mean reaches 2.60 cm and a standard deviation of 1.93 cm. These differences are, again, not relevant from the structural point of view, with respect to the size of the blocks.
A normality test based on quantile-quantile plot (Q-Q plot) has been performed in order to verify that measures are normally distributed. The measures have been accounted using the Matlab Statistical Toolbox, whose results are shown in table 3. The set for one sigma and two sigma again denote the normal distribution of the values.
Note that the pixel size of images is 1cm for TLS and 3 cm for MLS.
Figure 12 Histograms and lognormal probability distributions for the Mediaeval Wall measured by MLS.
According to these results, the proposed algorithm may be accepted as valid since in both cases the averaged errors are smaller than the resolution of images (ps) whose interval is defined from –ps to + ps. This suggests that the quality of the results generated by the algorithm will depend on the density of the original point cloud.
In the comparison between manual and automatic measurements it was found that some of the pieces were incorrectly characterized. This was because the algorithm provokes over-segmentation when fractures appear in the masonry block. It should be noted that it was not easy for the human eye to distinguish masonry joints and fractures
during the manual measurements. For the case of MLS, 2.9% of the pieces were oversegmented, and for the case of TLS 5.7% were wrongly identified. To avoid distorting the metrological measures with incorrect pieces they were deleted from the sample used for the validation.
The results show the validity of the method for quasi- periodic masonry walls, being uncoursed rubble masonry excluded from this investigation. This is crucial for the creation of the geometrically-constrained markers, otherwise markers would be automatically created based on the intensity image producing incorrect results (oversegmentation).
When processing flat surfaces the rectangular coordinates are used to create the intensity images without modifying the scale of dimensions. However, to include in the method also curved surfaces, the distances obtained through the segmentation process need to be projected back to the original point cloud where final dimensions are calculated (this is the core idea of our 2.5D approach). Note that each pixel in the 2D image is linked by indexes to all those points of the original point cloud projected in that pixel.
Consequently, the precision of the segmented point cloud will be constrained by the pixel size used.
It should be noted that this procedure would also be valid for photographic images. However it would require a preprocessing to obtain metric information, so the perspective system of each single image and its external orientation must be computed. Consequently control points are required to pre-process images. The most simple photogrammetic approach, based on image rectification through a projective transformation, requires having auxiliary instrumentation containing control points or scale bars. Even the geometric results could be obtained through the additional processing steps mentioned above; results of segmentation would remain in the 2D space.
In the case of the laser scanning data, the original data (point cloud) and segmentation results are provided in the 3D space. Note that the 2D space is just used to compute the segmentation (this is the common processing to 3D point clouds and photographic images).
In the case of using photogrammetric networks to provide results in 3D, the complexity would increase significantly and the processing time would increase abruptly (see Riveiro et al. (2011a) for estimation of resources needed for the photogrammetric network in a medium-size masonry construction). Thus, the productivity during data processing in the photogrammetric case would not be appropriated for large structures such as those shown in the paper. Contrary, the role of LiDAR is justified because it collects 3D geometric and radiometric data in a massive manner, and the data can be automatically processed providing the results to the 3D space.
However, some of the main limitation of LiDAR data relates to the requirement of important resources for data storage and computational performance. This problem is Figure 13 Distribution of errors (cm): point cloud
captured with the TLS (upper), point cloud captured with the MLS (down).
Table 3
Distribution of error measures for both TLS and MLS data.
particularly important when surveying large structures by MLS because this technology is used to scan larger constructions, and increases with the spatial resolution of the scanner. Even so, after reducing the point clouds to raster images, the computational performance is enhanced.
5 CONCLUSIONS
A methodology for the surveying of quasi-periodic masonry walls and subsequent geometric segmentation was developed. An algorithm for the automatic processing of laser scanning data based on digital image processing tools was developed and successful results were obtained. The automatic processing includes the automatic metric analysis of each masonry bock. The methodology was tested in historic masonry walls located in Guimarães (Portugal) and a morphologic analysis of the masonry was performed.
The development of a new marking process based on geometric and radiometric data contributed to produce successful watershed segmentation for the case of coursed masonry walls. Also, the method can distinguish pieces with different height at each course.
The method proposed in this paper have solved the problem of characterizing each block of the wall individually, which was not achieved previously in the literature as stated in Oses et al. (2014).
One of the main advantages of the methods is the 3D nature of laser scanning data. The 2.5D approach developed permits to have the results of segmentation back in the 3D space so the application of the method is not only restricted to planar walls, but also curved construction can be accurately evaluated.
The resolution of the point cloud is a key parameter when building the intensity images because high-resolution point cloud produce high quality images. This high quality images are formed by pixels of smaller sizes so they describe the details of the walls clearer than using point clouds collected by the mobile systems. In any case, the algorithm developed was not significantly sensitive to low quality images because acceptable segmentation results were found in cases where blocks could not be visually segmented. The present algorithm is only valid for periodic or quasi-periodic masonry walls.
The validation of the algorithm through two case studies demonstrated that the errors are mainly due to the resolution of the images created by the algorithm. This indicates the robustness of the algorithm, and also shows that the inaccuracies are motivated by the density of the original point cloud. It is expected that the advances in laser scanning technology guarantee the proper density of point clouds since nowadays MLS is still limited to produce high density point clouds.
Nowadays a terrestrial laser scanner can collect data at extremely rapid rates (accuracy reach 2mm at 25m
distance), instruments are controlled by operator through easy to use software, the registration of different scanner positions can be made automatically, are portable and their cost has significantly decreased in the last times (see Faro Focus3D X330 for reference (Faro, 2014)). Even photographic cameras are cheaper, more portable and easy to use than scanners, the data acquisition require more time to collect information about a large construction if the time required to compose the mosaic of images is computed.
Also, photographic cameras require auxiliary data (scale bars) to produce geometric results, and the method would be restricted to planar structures.
During the process of validation, it was observed, as expected, that the algorithm is not smart enough to differentiate fractures from masonry joints, so further developments in this aspect, by removing outliers, would improve the final application of the algorithm.
Due to the inner characteristics of the laser scanning data, all the results are geo-referenced. This allows that each masonry block and their dimensions after processing may be located in its real position in the 3D space, and subsequent structural analysis can be performed. It shows an appropriate way to document historical structures, due to its non-destructive character as well as a quantitative and accurate tool. The results may also be used to perform stability analysis of the structure.
ACKNOWLEDGMENTS
This work has been partially supported by the Spanish Ministry of Economy and Competitiveness (Ref.: TIN2013- 46801-C4-4-R) and by Xunta de Galicia (Grant No.
CN2012/269 and Grant No. EM2013/005).
Authors want to give thanks to the reviewers for their constructive comments that contributed to improve both the method and the presentation of results.
REFERENCES
Beare, R. (2006). A locally constrained watershed transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28 (7):1063-1074).
Beucher, S and Lantuéjoul, C. (1979). Use of watershed in contour detection, Proc. Workshop on Image Processing, CCETT/IRISA, Tennes, France, 2.1-2.12.
Cluni F, Gusella V (2004). Homogenization of non- periodic masonry structures. Int J Solids Struct 41: 1911–
1923.
FARO (2914). http://www.faro.com (last accessed 20.10.14).
Gonzalez, R.C., Woods, R.E., Eddins, S.L. (2009).
Digital Image Processing using MATLAB. Gatesmark, LLC.
Graham L. Mobile mapping systems overview. (2010).
Photogrammetric Engineering and Remote Sensing, 76(3):
222 – 228.
Höfle, B., Pfeiffer, N. (2007). Correction of laser scanning intensity data: Data and model-driven approaches.
ISPRS Journal of Photogrammetry & Remote Sensing, 62:
415–433.
ICOMOS. (2003). Recommendations for the Analysis and Restoration of Historical Structures, ISCARSAH.
Kang, Z, Zhang, L., Yue, H, Lindenbergh, R. (2013).
Range image techniques for fast detection and quantification of changes in repeatedly scanned buildings.
Photogrammetric Engineering and Remote Sensing, 79 (8):
695-708.
Lee H.M., and Park, H.S. (2011). Gage-Free Stress Estimation of a Beam-like Structure Based on Terrestrial Laser Scanning. Computer-Aided Civil And Infrastructure Engineering, 26 (8): 647–658.
Lourenço, P.B., (2002). Computations of historical masonry constructions, Progress in Structural Engineering and Materials, 4(3): 301-319.
Lourenço, P.B., Milani, G., Tralli, A., Zucchini, A., (2007). Analysis of masonry structures: review of and recent trends of homogenisation techniques, Canadian Journal of Civil Engineering, 34 (11): 1443-1457.
Lourenço, P.B., Mendes, N., Ramos, L.F., Oliveira, D.V., (2011). Analysis of masonry structures without box behavior, International Journal of Architectural Heritage, 5: 369-382.
McInerney J, Trzcinski I, DeJong MJ. (2012). Discrete element modelling of masonry using laser scanning data, 8th Int Conf on Structural Analysis of Historical Constructions (SAHC), Wroclaw, Poland.
Meyer, F. Integrals, gradients and whatershed lines, (1993). Proc. Mathematical morphology and its applications to signal processing. Barcelona, May 1993: 200-211.
Meyer, F. (1994). Topographic distance and watershed lines. Signal Processing, 38(1): 113 – 125.
Milani, G., Lourenço, P.B., (2010). Monte Carlo homogenized limit analysis model for randomly assembled blocks in-plane loaded, Computational Mechanics, 46 (6):
827-849
Nicodemus, F.E., Richmond, J.C., Hsia, J.J., Ginsberg, I.W., Limperis, T., (1977). Geometrical Considerations and Nomenclature for Reflectance. US Department of Commerce, National Bureau of Standards.
OPTECH, www.optech.ca, 2013. (last accessed 16.02.14).
Oses, N., Dornaika, F., Moujahid, A. (2014). Image- Based Delineation and Classification of Built Heritage Masonry. Remote Sensing. 6, 1863-1889.
Park, H.S. Lee, H.M., Adeli, H. and Lee, I., A New Approach for Health Monitoring of Structures: Terrestrial Laser Scanning, Computer-Aided Civil and Infrastructure Engineering, Vol. 22, No. 1, 2007, pp. 19-30.
Petri, G. (2010). Mobile mapping systems: an introduction to the technology. Geoinformatics 13 (1): 32 – 43.
Puente I, González-Jorge H, Martínez-Sánchez J, Arias P, (2013a). Review of mobile mapping and surveying technologies, Measurement, 46: 2127 – 2145.
Puente , I., González-Jorge, H., Riveiro, B., Arias, P.
(2013b). Accuracy verification of the Lynx Mobile Mapper system. Optics & Laser Technology, 45: 578–586.
Riegl Laser Measurement Systems, 2013.
http://www.riegl.com/nc/products/terrestrial-scanning/ (last accesed 16.02.2014).
Riveiro, B., Caamaño, J.C., Arias, P., Sanz, E. (2011).
Photogrammetric 3D modeling and mechanical analysis of masonry arches: an approach based on a discontinuous model of voussoirs. Automation in Construction, 20 (4), 380-388.
Riveiro, B., Morer, P., Arias, P., De Arteaga, I. (2011).
Terrestrial laser scanning and limit analysis of masonry arch bridges. Construction and Building Materials, 25 (4):
1726–1735.
Rodríguez-Gonzálvez, P., Muñoz-Nieto, A., Gozalo- Sanza, I., Mancera-Taboada, J., González-Aguilera, D., Carrasco-Morillo, P. (2014). Geomatics and Geophysics Synergies to Evaluate Underground Wine Cellars, International Journal of Architectural Heritage:
Conservation, Analysis, and Restoration, 8 (4): 537-555.
Sithole, G. (2008). Detection of Bricks in a Masonry Wall, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.
XXXVII. Part B5. Beijing 2008.
Stilla, U. and Jutzi, B. (2009). Waveform Analysis for Small-Footprint Pulsed Laser Systems, in Shan, J. and Toth, C.K. (eds). Topographic Laser Ranging and Scanning: Principles and Processing. CRC Press, Boca Raton, FL, USA: 215-234.
Truong-Hong, L., Laefer, D.F., Hinks, T., Carr, H.
(2013). Combining an Angle Criterion with Voxelization and the Flying Voxel Method in Reconstructing Building Models from LiDAR Data. Computer-Aided Civil and Infrastructure Engineering, 28 (2): 112–129.
Varela-González, M., González-Jorge, H., Riveiro, B., Arias, P. (2013). Performance testing of LiDAR exploitation software. Computers and Geosciences, 54:
122-129.
Wagner, W. (2010). Radiometric calibration of small- footprint full-waveform airborne laser scanner measurements: Basic physical concepts. ISPRS Journal of Photogrammetry and Remote Sensing, 65: 505–513.
Walsh, S. B., Borello, D. J., Guldur, B. and Hajjar, J. F.
(2013). Data Processing of Point Clouds for Object Detection for Structural Engineering Applications.
Computer-Aided Civil and Infrastructure Engineering, 28:
495–508.
Wang, D. and Vallotton, P. (2010). Improved marker- controlled watershed segmentation with local boundary priors. 25th International Conference of Image and Vision Computing, New Zealand (IVCNZ). Queenstown, New Zealand, November 2010.
Yue, Z.Q., Chen, S., Tham, L.G. (2003). Finite element modeling of geomaterials using digital image processing.
Computers and Geotechnics, 30: 375–397.
View publication stats View publication stats
