This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points).