1 Introduction

The sewerage infrastructure is an essential part of modern society consisting of huge pipe networks hidden under the ground. This system is normally not a concern for the general public, but it is indispensable. To reduce maintenance costs, regular inspection of the entire sewer system is important. With today’s methods this is not feasible, as inspection of sewer systems is a laborious task, carried out manually by skilled operators [3]. This is done by sending a remote-controlled platform with a camera through a section of the sewer system while an operator monitors the live video feed from this camera. As a result, pipes are often replaced prematurely to avoid older pipes causing issues. Pipes with an expected lifetime of 75 years are replaced after just 50 years, causing several hundred million DKK in expenses for the Danish state [5]. To improve this lack of inspection, an ongoing research project, the Automated Sewer Inspection Robot (ASIR) project, intends to develop an autonomous robot, that will roam the sewers and utilise machine learning to identify defects [5]. For a machine learning algorithm to properly identify defects and landmarks such as joints and branch pipes in a sewer network, the algorithm that is utilised by the robot requires training, and for this, an annotated data set is desired. The intended plan of collecting data from the sewers is using depth- and RGB cameras. Currently, only databases with RGB images from CCTV inceptions are available, and a data set consisting of depth information is needed. As capturing data from real sewers can be a huge and expensive task, an alternative could be generating synthetic data.

Different types of depth sensors can be utilised for capturing depth information, such as Time of Flight (ToF) [7, 11], stereo cameras [8], and structured light [21]. The ToF sensor Pico Flexx by Camboard is chosen as the sensor to be implemented as part of the proposed system. The sensor is chosen because of its small size, high resolution and low power consumption. It is also shown by [9] that ToF sensors are ideal for use in inspection of hazardous and hard to reach places. The Pico Flexx outputs point clouds, which offers more depth information that can help identify defects in the sewer network. This sensor will be implemented and will be the focus of this paper.

Contributions. Our contribution is a novel system, making the first foray into generating Synthetic Point Clouds (SPC) in the sewer domain. This system consists of two parts;

  • Generate virtual sewer networks which are based on the method Structured Domain Randomization (SDR) method.

  • Generate SPCs, approximating the Pico Flexx output, that can be used for training machine learning algorithms.

2 Related Work

We will be looking at the state of the art for both the sewer generation and ToF simulation.

Depth Sensors: Bulczak et al. [2] suggests a method for simulating amplitude modulated continuous wave (AMCW) ToF sensors including the artefacts caused by multi-path interference (MPI). MPI is of particular interest for simulating sewer inspection in textured pipes, which could be of interest even in smoother plastic pipes, as it could model reflections. This method increases computational costs, however, it is specifically designed for GPU execution to allow fast processing.

Sarbolandi et al. [18] compares structured light and ToF depth sensing in detail, specifically comparing the two different versions of the Microsoft Kinect sensors that employ these different technologies. They found that the ToF variant performed better at rejecting background illumination, motion, highly translucent objects, and at large incident angles.

Sarker et al. [19] explores the use of a stereo vision based depth camera for the use of crack detection in concrete, showing the ability to detect and classify variants of cracks.

Synthetic Generation of Environments: In recent years the interest in generating annotated synthetic data has been steadily increasing, as to meet the demands of more complex deep learning architectures in an inexpensive manner [1, 6, 10, 16]. Tobin et al. [20] presents a method for computer generation of synthetic data, called Domain Randomization (DR), which is achieved by generating structured variations of a chosen scenario, such that a machine learning algorithm would process real images as if they were another variation of the scenario. The work was further developed by Prakash et al.  [14], into an expanded version of DR: Structured Domain Randomisation (SDR). Instead of placing objects by a uniform random distribution with DR, context of the scene is considered. This structures the randomisation by a set of rules that corresponds to the intended environment. This allows machine learning algorithms to train not only on the virtual objects placed in the scene, but also on the context of where these objects are positioned in relation to each other. The SDR method is found to fit the focus of our solution, and it will be used for generating the environment that will later be used for the 3D data acquisition.

3 Methods and Material

The proposed solution is implemented in Unity, which consists of two parts: Environment Generator and Data Generator. The overall structure of the system is shown in Fig. 1.

Fig. 1.
figure 1

Structure of the proposed solution.

Full size image

The Environment Generator dynamically generates a virtual sewer network consisting of splines, control points and pipe meshes based on given parameters. These are passed to the Data Generator that generates point clouds based on them and two additional parameters: folder path and number of point clouds. Folder path: defines the output folder for point cloud data. Number of point clouds: defines how many point clouds to create. The camera moves forward until this parameter is satisfied.

3.1 Pico Flexx

Camboard Pico Flexx, is a joint production by PMD Technologies and Infineon. It is based on PMD ToF technology (AMCW-ToF), which uses Near Infrared (NIR) laser to determine the distance from the sensor to the impact points of objects. The small size and the low power consumption enhances the number of possible applications. To mimic the Pico Flexx, its parameters should be implemented for the synthetic sensor. Because the information available from PMD [13] only shows camera characteristics such as field of view and aspect ratio, a complex ToF simulation can be difficult to be implemented as more information is required. The sensor approximation in this paper is therefore estimated using ray casting.

3.2 Environment Generator

The gap between the real- and the virtual domain can be reduced using 3D models that mimic physical models and utilize SDR to generate the environment. 3D models and environment generator will be explored in the upcoming sections.

Pipes and Defects: To mimic the physical environment, 3D representations of physical pipes and defects are utilised. The system generates points along the network, where handmade defects can spawn.

Generator: To enhance the generator, the structure of how objects are generated should be acknowledged by utalizing SDR. As the sewer system domain differs to the domain in the SDR paper, other contexts such as rubber rings mostly occurring at pipe displacements should be considered. SDR proposes a taxonomy which covers the following four principles. Scenario: determines general parameters to generate the domain such as length of the sewer network and defect probability. Global Parameters: generates contextual splines based on parameters from the scenario. Context Splines: instantiate objects based on given probabilities and context. Objects: contain a transform, 3D mesh, collider, and defect- or fine tag.

3.3 Data Generator

Without extensive information about the Pico Flexx, an approximated virtual camera in Unity is set up to mimic the output of the Pico Flexx. The virtual camera utilises the same resolution and focal length. Moreover, to detect depth in a scene, rays are cast up to the distance of the detectable depth of the Pico Flexx. These rays are cast from each pixel of the virtual camera in the direction of the viewport in normalized coordinates. Figure 2 shows a comparison of 2D depth images from the data generator and Pico Flexx. However, to evaluate the data generation, comparing point clouds is preferred to avoid dimensionality reduction.

In a simulated environment, a physics engine can utilise ray casting to determine impact positions at 3D meshes. This provides additional information for each ray which can label the points within the point cloud. For each ray, random noise from a Gaussian distribution is added with a range of \(\pm 1\%\) from the true value, based on the datasheet from PMD [13].

Fig. 2.
figure 2

Output image from the image generator, compared to the Pico Flexx, where white indicates missing points.

Full size image

4 Results

To evaluate the solution, point clouds from both Pico Flexx and the system using similar setups are acquired.

4.1 Data Gathering from Pico Flexx

To gather Pico Flexx data, a controlled environment was set up, and an already built robot with a mounted Pico Flexx was used, as shown in Fig. 3a. It is preferred to get data around pipe connections, as displaced pipes are the most common defectFootnote 1. Physical pipes were set up inside a windowless room, to avoid external light sources disturbing the Pico Flexx sensor that was attached to a small remote controlled mobile platform. Based on the setup in Fig. 3b, two scenarios were arranged; the first setup without displacement and second with displacement. In the first scenario (S1), the robot was placed at the start of the pipe and programmed to move forward through the pipe. In the second scenario (S2), the robot was placed 50 cm from both the first pipe connection and a misplaced rubber ring, and programmed to move through the connection. The outcome from these scenarios was point cloud data sequences split by time stamps. In order to evaluate the Pico Flexx data, the sewer system and scenarios of the physical setup were mimicked for the virtual setup, as shown in Fig. 3c where the Pico Flexx’s orientations were mimicked by approximately transforming the virtual camera. Using these virtual scenarios, SPCs were extracted to be compared to the Physical Point Clouds (PPC).

Fig. 3.
figure 3

The figures represent the robot’s placement, the physical- and virtual setup, respectively. Note: The branch pipe is not within the sensor’s FoV.

Full size image

4.2 Point Cloud Comparison

To evaluate the simulation performance, differences and similarities between the PPCs and SPCs were compared, which allows a per point distance calculation. For this, the widely used software CloudCompare [4], which has been tested by several studies [12, 15], is utilised. The PPC will be used as the ground truth whilst the SPC will be used to compare with. Before comparing the point clouds, they are aligned using the fine registration method, Iterative Closest Point (ICP) [17], which aligns the desired point clouds to the ground truth, by minimizing the distance error between them. As missing points exist in the PPCs, local modelling is utilised to approximate planes based on least square to estimate the pipe geometry. This minimizes the distance between point clouds, as approximated planes can cover holes in the PPCs.

Fig. 4.
figure 4

This figure represents the two scenarios, where the white point clouds are PPCs and the colored point clouds are SPCs. Moreover, the legend indicates the absolute distance error value for the gradient colors in the SPCs. (Color figure online)

Full size image

The absolute distances distribution of the compared point clouds for the two scenarios is presented in Fig. 5. The results show a difference of 5.78 ± 8.92 mm for S1 and 7.58 ± 8.68 mm for S2.

4.3 Accuracy of Point Clouds

To ensure both the synthetic- and physical data give correct measurements of the real pipe, the diameter error is calculated. This is done on S1 as this is the scenario with the longest straight section. Each point cloud is sliced into overlapping segments of 100 mm, using a step size of 10 mm. At each segment, the distance from each point to its most distant point is calculated, and the mean of these distances are then used as the diameter for that section.

5 Discussion

In the point cloud comparison, some of the error described by the mean in both scenarios, may be caused by a difference in placement of the Pico Flexx and the virtual camera, as the placement of the virtual camera was done by hand, as close to the Pico Flexx positioning as possible. As seen in Fig. 5, outliers are present, with the largest occurrence is in S1. The outliers appear at the pipe connection of the pipes which can be seen in Fig. 4, by the yellow and red colors. The remaining outliers in S1 are located at the end of the point cloud, likely caused by the lower point density. In S2, the outliers are most noticeable around the pipe connection point, but in this case they seem to be caused by the shadow cast by the rubber ring in S2. This can be seen in Fig. 4 as a lack of points. Considering the mean for all evaluations, in regards to the size of the pipes, the error rate is considered acceptable in order to classify e.g. displacements.

Fig. 5.
figure 5

Histogram of the absolute distances from points in the SPC, to the corresponding calculated least square plane in the PPC. Note the y-axis is log-scaled. Scenario 1 mean: 5.78 ± 8.92 mm. Scenario 2 mean: 7.58 ± 8.68 mm.

Full size image
Fig. 6.
figure 6

Error rate compared to the specified diameter of the real pipe from S1.

Full size image

Figure 6 shows the diameter size along the point clouds from the accuracy of point cloud comparison. The first 25 cm section of both the point clouds shows a diameter of 0 mm, and that is due to the distance between the sensor and the pipe surface. The next 35 cm section has increasing diameters size due to the slanted shape of the point cloud at the beginning as can be seen in Fig. 4, which results in a non-circular slice. From 60 until 150 cm, the mean diameter is 405 mm for the PPC and 376 mm for the SPC, which shows a small error in the PPC, believed to be caused by the lack of points in that area of the point cloud. However, the SPC shows almost a diameter size that matches the real pipe diameter for the majority of the pipe length. After 160 cm, major errors occur in both point clouds, which starts around the bend of the pipe, this is not of interest as the test assumes the pipe is straight. The point cloud comparison test indicates that the Pico Flexx yields more noise than expected, but to clarify this, further research is required. The accuracy of the point clouds test, indicates that the imperfections can be caused by the textures, imperfectness of the surfaces, dirt in the pipes or lighting that could reflect within the pipes’ internal surfaces. All these characteristics define the appearance of real surfaces, which can have an impact on other elements when implemented in computer graphics.

6 Conclusion

This paper has introduced a system which is capable of generating annotated SPCs based on the characteristics of the Pico Flexx in a synthetic sewer environment. Through experiments, it is concluded that the difference of the two point clouds is mainly caused by the lack of points in areas of the PPC. Furthermore, it is found that regions of the point clouds have points that approximately match the real data. Using a moving robot, a continuous good point cloud could be stitched together from multiple instances of the good section. However, if a larger portion of the point cloud is to be used, a more accurate simulation of the sensor might be required, for example an implementation of the MPI simulation as mentioned in related work Sect. 2. Considering the simplicity of the calculation behind the SPCs, the result seems promising. This especially applies in relation to the first part of the point clouds in S1. It should be noted that this is a controlled environment, and entering more realistic scenarios, that contain water, dirt etc. might challenge the Pico Flexx and therefore it yields noisier data. Here the solution might perform poorly due to it being too idealised, therefore reflection, refraction etc. need to be taken into considerations. Overall the solution is able to generate randomised sewer systems that contain defects using SDR. Moreover, SPCs can be generated which can potentially be used to accelerate the data acquisition process for machine learning algorithms.