Keywords

1 Introduction

The process of archaeological excavation is a careful effort to uncover and preserve historical artifacts and structures. Archaeologists start with detailed surveys, often using Geographic Information Systems (GIS) [24] to create maps and analyze the site. These surveys help determine where to dig. During the excavation, archaeologists remove soil and debris layer by layer, documenting each step with photos, sketches, and notes to keep track of where everything is found. Modern techniques like 3D reconstruction are now commonly used. Methods such as photogrammetry and structure from motion (SfM) allow for the creation of detailed 3D models of the site and artifacts.

One such site is Castellaraccio di Monteverdi, a deserted medieval village located on a hilltop 130 m above sea level, with remnants of structures hidden in the vegetation. Overlooking the Ombrone River and a ruined medieval bridge, its history dates back to the mid-12th century when it was part of the Monastery of Saint Salvatore of Giugnano and the Abbey of Saint Lorenzo all’Ardenghesca. In the 13th century, it changed hands between the Commune of Siena and various noble families, ultimately becoming abandoned by the late 13th century. Since 2017, as part of the IMPERO project [29], archaeological research has been excavating the site to map its extent, urban plan, and chronological use, uncovering early medieval phases not documented in historical records.

Photo-realistic 3D reconstruction of archaeological sites [6, 9, 10, 13, 20] has gained popularity following the development of sophisticated algorithms [5, 27, 28], affordable sensors [11, 14] and easy access to high-performance compute [26]. Such methods also provide insights into the spatial relationships, architectural details, and historical context of the structures.

A new way to enhance traditional photogrammetry and Structure from Motion (SfM) techniques involves the use of Gaussian splatting [19] and Neural Radiance Fields (NeRFs) [23] for 3D reconstruction. Gaussian splatting represents sparse point clouds as 3D Gaussians which are then optimized, leading to more efficient rendering and processing of 3D scenes. NeRFs employ neural networks to represent 3D scenes with exceptional detail and accuracy, optimizing both geometry and appearance from sparse input images.

While these 3D reconstruction methods offer high-fidelity, photo-realistic results, they require a substantial number of source images that fully cover a scene [31]. These input images can be captured using hand-held cameras [30], UAVs [32], or other autonomous platforms [7]. Typically, images are taken in a pre-configured pattern over the target scene with set overlap, resulting in uniform coverage but often leading to low resolution and artifacts.

To address this, we propose an innovative approach to optimize image capture using View Planning (VP) planning [8] and Gaussing splatting. The proposed method is currently under investigation with initial experiments showing a significant increase in quality of 3D models generated using our method. We intend to deploy the proposed approach at Castellaraccio di Monteverdi in the following year. The findings of this paper along with datasets and code will be open-sourced to the community following publication.

Fig. 1.
figure 1

Illustrates the Photogrammetry pipeline discussed in Sect. 2.1

Full size image

2 Related Work

2.1 Photogrammetry

In recent years, the popularity of 3D reconstruction has surged due to advancements in technology and its applications in fields such as virtual reality [6], augmented reality [33], and cultural heritage preservation [12]. One prominent method is photogrammetry, which involves using structure from motion (SfM) algorithms to create a detailed 3D model.

Photogrammetry creates 3D models by capturing multiple overlapping images of an object or scene from different angles, then using algorithms to triangulate the points where the images overlap to generate a detailed 3D representation. This process (shown in Fig. 1 involves aligning the images, extracting key features, and reconstructing the surface geometry to produce an accurate and photo-realistic 3D model. The surface reconstruction is typically performed using Poisson surface reconstruction [17] which attempts to create water-tight surfaces leading to over-smoothed outputs. This is a common source of artifacts when the under-lying point cloud is sparse or scene has high variance in geometry.

Fig. 2.
figure 2

Figure from [22], illustrates Neural Radiance Fields (NERFs)

Full size image

2.2 Neural Radiance Fields (NERFs)

Neural Radiance Fields (NeRFs) [4, 22] (Fig. 2) have gained significant attention due to their ability to produce highly detailed and realistic 3D reconstructions from 2D images. They work by training a neural network to model the way light interacts with objects in a scene. This process involves feeding the network a series of images taken from different angles, along with the corresponding camera positions which can be estimated using SfM methods. The network learns how light is absorbed, reflected, and transmitted by different surfaces in the scene.

The core idea is the use of a volumetric representation, where each point in the 3D space is associated with a color and opacity value. The network attempts at predicting these values based on the input images, effectively creating a continuous function that describes the scene. By sampling many points along rays that pass through the 2D images, the network can render new views of the scene with high accuracy and detail. NeRFs have become a powerful tool in applications ranging from virtual reality to digital content creation, offering an unprecedented level of realism. This comes at the cost of training time as NERFs often have long training times. A scene containing 100 800\(\,\times \,\)800 images can take 1–2 days to train [22].

Fig. 3.
figure 3

Figure from [3], illustrates the process of Gaussian Splatting

Full size image

2.3 Gaussian Splatting

Gaussian splatting [19] (Fig. 3), a newer technique in 3D reconstruction, offers competitive performance to NERFs while being extremely fast in training and inference. It uses simple Gaussian shapes (“blobs”) to represent 3D objects which is particularly useful for real-time applications because it reduces the amount of computing power needed. Gaussian splatting creates 3D models by approximating complex surfaces using a collection of Gaussian functions (Gaussians). These Gaussians serve as building blocks, each representing a small, localized area of the surface with parameters that define its position, size, orientation, and intensity.

The process begins by capturing multiple images of the object or scene from different angles, similar to traditional photogrammetry. Key features and points are then extracted from these images and used to define the initial positions and parameters of the Gaussians. Each Gaussian function acts like a smooth, flexible “blob” that can be adjusted to fit the shape and texture of the object’s surface accurately.

The parameters of the Gaussian functions are iteratively optimized to minimize the difference between the projected 3D model and the input images. This involves fine-tuning the position, size, and orientation of each Gaussian to ensure that the model accurately represents the underlying geometry and appearance of the scene. Its efficiency makes it ideal for situations where quick rendering is crucial, such as in virtual and extended reality applications.

2.4 View-Planning

For large scale 3D reconstruction of sites, especially with UAVs, a set of waypoints are chosen that maximize the amount of information received by the onboard sensor (monocular cameras, LiDARs). These waypoints tend to be grid-based or circular patterns. However, the regularity of the path may cause some areas to be occluded. These occlusions create gaps or reconstruction artifacts which would need additional data acquisition steps which is not economical. Minimizing the amount of views required to cover an object is called as a View Planning Problem and choosing a view on-line is called as Next-Best View [25]. Traditionally, VP algorithms utilize an octree approach whereby 3D locations are voxelized and queried as a tree. Each voxel is tagged as unmapped, free or occupied. At every iteration the planner (PRM [16], RRT [21], RRT* [15]) creates a path that maximally reduces the number of unmapped voxels by following only free voxels. This path considers the system state as well as the robot constraints and dynamics (Fig. 4).

3 Method

In order to accurately reconstruct Castellaraccio di Monteverdi, we collected images using a UAV (DJI Mavic 3 Enterprise RTK) and a hand-held camera (Sony A7R III + Sony 24–70 F4). The use of UAVs, equipped with Real-Time Kinematic (RTK) positioning, ensured high-precision geo-location data for each image. We employed a grid-surveying technique with the UAV, which involves flying the drone in a systematic grid pattern to capture overlapping top-down images of the site. This method ensures comprehensive coverage and consistency in image capture, which is essential for photogrammetry and structure from motion (SfM) processes.

Fig. 4.
figure 4

Illustrates the Castellaraccio di Monteverdi site and camera angles for the dataset that was captured

Full size image

Additionally, images were taken from varying angles with the hand-held camera to capture details that might be missed from aerial views. This combination of aerial and ground-level photography provides a rich dataset that enhances the accuracy and detail of the 3D reconstruction.

For photogrammetry we used popular software such as [1, 2] which offer an extensive suite of tools to build, customize and manipulate 3D models. The software outputs a dense point cloud and a textured mesh which is shown in Fig. 6. [28] was also used for reconstruction of the Castellaraccio site.

Photogrammetry and 3D reconstruction based view generation pipelines are feed-backward methods that generate pixels on an image plane one pixel at a time. To generate accurate lighting and shadows, they use an expensive ray-tracing operation. These methods also use interpolation to produce continuous meshes that often leads to over-construction that is commonly observed as smoothing.

Feed-forward methods, like Gaussian splatting, on the other hand, iterate over objects in the scene and produce color and alpha values for pixels in a process called splatting. In Gaussian splatting specifically, clusters of points on homogeneous surfaces can be represented by parametric Gaussians which leads to a reduction in the number of operations required to generated views. Further the illumination from light sources is precomputed before the view synthesis and need only be computed once rather than for every pixel in ray tracing. In effect, splatting leads to photo realistic volumetric rendering with lower point densities provided the gaussians are optimized well to represent both homogeneous and non-homogeneous structures.

The output of these modules are a set of SFM and Splat models. Due to the path, these are noisy models and may contain smoothed or missing chunks due to possible non-overlapping viewpoints. An improved plan is needed to efficiently retrieve these missing viewpoints. A limitation of traditional View Planning algorithms is that the space captured by the sensor is categorized as either occupied, free or unmapped. The algorithm would then optimize the path to minimize the unmapped region. Rather than binary categorization (free or occupied), we propose the use of 3D Gaussians from the splat output instead. Each 3D Gaussian has an associated Mean and Variance corresponding to the uncertainity in the distribution of the points.

A splat with low variance is preferred on surfaces with higher variance in position, surface normals, color, contrast and opacity whereas in large homogeneous areas, fewer gaussians with higher variance are sufficient to capture the lower detail. These are then used to determine “voxels” with maximum uncertainity. Based on these factors, the planning algorithm will aim to plan a path that captures the best views to optimize the gaussian parameters. Thus, re-planning to achieve minimum artifacting and over-smoothing. This pipeline is shown in Fig. 5.

In the next section, we demonstrate the photorealistic output of this new view synthesis method and detail how and why they fail under certain circumstances. We indicate our future research direction to use domain specific cues to improve this mode of visualization to aid archaeo-geophysical sensing of cultural heritage sites.

Fig. 5.
figure 5

Proposed Method for View-Planning

Full size image

4 Results and Discussion

4.1 Photogrammetry

Figure 6 (Left) show results using Photogrammetry to reconstruct Castellaraccio. While these models are of respectable perceptual quality, they contain over-smoothed regions and some artifacts. Artifacts are created due to the mesh generation process which is based on Poisson Surface Reconstruction which often smooths out surfaces when normal estimation is lacking or the underlying point cloud is not dense enough [18]. This can obscure intricate details and sharp edges leading to “bumps” and distortions.

4.2 Gaussian Splatting

Figure 6(Right), show 3D models (Splats) built using Gaussian Splatting. Splats are capable of synthesizing novel views which are visually superior to renderings of models built using photogrammetry. These confirm our hypothesis that Gaussian Splatting performs better than photogrammetry on the same dataset while offering comparable processing times. Although these splats are capable of representing the scene in high quality, they perform poorly when input images do not cover certain view-points. This is apparent in Fig. 6 where artifacts and noise elements are seen due to lack of coverage (Fig. 7).

Fig. 6.
figure 6

Figure shows 3D models of Castellaraccio generated using Photogrammetry (left) and Gaussian Splatting (right)

Full size image

4.3 Structure-Guided View Planing for Improved Gaussian Densification

Gaussian splatting uses fewer Gaussians to represent large homogeneous areas and collections of much denser Gaussians for other areas to reduce computational complexity and is one of ways through which this methods performs volumetric renderings in tens of minutes rather than days required by NeRF. An optimization process, termed densification, splits, merges and removes gaussians according to perceived complexity of the scene.

Fig. 7.
figure 7

Example of using View-Planning (VP) planning to improve Gaussian Splatting. Left to Right, model built using Photogrammetry, Gaussian Splatting and VP + Gaussian Splatting. Model built using images captured at the University at Buffalo.

Full size image

However, there are two key opportunities to make improvements to the domain-specific problem of generating photorealistic volumetric renderings for archeo geophysical sensing: (i) the densification in the original gaussian splatting is generalized and does not take any semantic cues from the scene, (ii) the densification process uses view-space positional gradients from a loss that is dominated by structural variations in the image space, as seen in Eq. 1, and (iii) a majority of the feedback from the loss function is biased by the image samples used to compute the loss. Therefore, splatting is clearly a downstream process that assumes the completeness of view samples and is largely influenced by the quality of the samples.

$$\begin{aligned} \mathcal {L} = (1 - \lambda ) \mathcal {L}_1 + \lambda \mathcal {L}_{D-SSIM} \end{aligned}$$
(1)

Since the structure of interest is bounded and can be known apriori, we are currently developing a method to plan the best views around the structure which can then be autonomously executed by a drone to produce an optimal set of views to: (i) better represent non-homogeneous surfaces often found during excavation, (ii) decrease geometric artifacts over structures largely caused by suboptimal densification, and (iii) improve the volumetric rendering in a 360\(^{\circ }\) viewing space around structures of interest.

5 Conclusion

In this paper we introduce a novel method for optimizing image capture by using View-planning. The images captured using this approach can be used to improve Gaussian splats by using the uncertainty information and by modeling the task as an information maximization problem. We describe our approach and demonstrate on a sample dataset. The method is set to be evaluated at Castellaraccio di Monteverdi in the following year following which we intend to publish our findings. The code, datasets and implementation details will be open sourced to the community following publication.