JP7538862B2 - Creating an Arbitrary View - Google Patents

Description

CROSS-REFERENCE TO OTHER APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. No. 16/171,221, entitled "ARBITRAY VIEW GENERATION," filed Oct. 25, 2018, which is a continuation-in-part of U.S. patent application Ser. No. 15/721,421, entitled "ARBITRAY VIEW GENERATION," filed Sep. 29, 2017 (now U.S. Patent No. 10,163,249), which is a continuation-in-part of U.S. patent application Ser. No. 15/081,553, entitled "ARBITRAY VIEW GENERATION," filed Mar. 25, 2016 (now U.S. Patent No. 9,996,914), all of which are incorporated herein by reference for all purposes. U.S. Patent Application No. 15/721,421 (now U.S. Patent No. 10,163,249) further claims priority to U.S. Provisional Patent Application No. 62/541,607, filed August 4, 2017, entitled "FAST RENDERING OF ASSEMBLED SCENES," the latter of which is incorporated herein by reference for all purposes.

This application claims priority to U.S. Provisional Patent Application No. 62/933,254, filed November 8, 2019, entitled "QUANTIZED PERSPECTIVE CAMERA VIEWS," which is incorporated by reference herein for all purposes.

Existing rendering techniques face a trade-off between the conflicting goals of quality and speed. High-quality rendering requires significant processing resources and time. However, slow rendering techniques are unacceptable for many applications, such as interactive real-time applications. Typically, lower-quality but faster rendering techniques are preferred for such applications. For example, rasterization is commonly utilized by real-time graphics applications, sacrificing quality for relatively fast rendering. Thus, improved techniques that do not significantly compromise either quality or speed are needed.

Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.

1 is a high-level block diagram illustrating one embodiment of a system for generating an arbitrary view of a scene.

FIG. 13 is a diagram showing an example of a database asset.

1 is a flow diagram illustrating one embodiment of a process for generating an arbitrary perspective.

FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object. FIG. 1 illustrates an example embodiment of an application in which independent objects are combined to generate an ensemble or composite object.

1 is a flow chart illustrating one embodiment of a process for generating an arbitrary ensemble view.

The present invention may be implemented in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and/or a processor configured to execute instructions stored in and/or provided by a memory coupled to the processor. These embodiments or any other form the present invention may take may be referred to herein as techniques. In general, the order of steps of a disclosed process may be altered within the scope of the present invention. Unless otherwise noted, components such as a processor or memory described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

The following provides a detailed description of one or more embodiments of the present invention with reference to drawings illustrating the principles of the invention. The present invention has been described in connection with such embodiments, but is not limited to any of them. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

Techniques are disclosed for generating arbitrary views of a scene. The examples described herein provide high definition output with very low processing or computational overhead, effectively eliminating the difficult trade-off between rendering speed and quality. The techniques described are particularly useful for generating high quality output very quickly for interactive real-time graphics applications. Such applications depend on the substantially immediate presentation of a preferred high quality output in response to and according to user manipulation of a presented interactive view or scene.

1 is a high-level block diagram illustrating one embodiment of a system 100 for generating an arbitrary view of a scene. As shown, an arbitrary view generator 102 receives a request for an arbitrary view as an input 104, generates the requested view based on existing database assets 106, and provides the generated view as an output 108 in response to the input request. In various embodiments, the arbitrary view generator 102 may comprise a processor, such as a central processing unit (CPU) or a graphics processing unit (GPU). The configuration of the system 100 shown in FIG. 1 is presented for illustrative purposes. In general, the system 100 may comprise any other suitable number and/or configuration of interconnected components that provide the described functionality. For example, in another embodiment, the arbitrary view generator 102 may comprise different configurations of internal components 110-116, the arbitrary view generator 102 may comprise multiple parallel physical and/or virtual processors, the database 106 may comprise multiple network databases or clouds of assets, etc.

The arbitrary view request 104 includes a request for an arbitrary perspective of a scene. In some embodiments, the requested perspective of the scene, including other perspectives or viewpoints of the scene, is not already present in the asset database 106. In various embodiments, the arbitrary view request 104 may be received from a process or a user. For example, the input 104 may be received from a user interface in response to user manipulation of the presented scene or a portion thereof (e.g., user manipulation of a camera viewpoint of the presented scene). In another example, the arbitrary view request 104 may be received in response to specifying a path of movement or travel in the virtual environment, such as a fly-through of the scene. In some embodiments, the possible arbitrary views of the scene that can be requested are at least partially constrained. For example, a user may not be able to manipulate the camera viewpoint of a presented interactive scene to any random position, but is constrained to a particular position or perspective of the scene.

The database 106 stores multiple views of each stored asset. In the given context, an asset is an individual scene whose specifications are stored in the database 106 as multiple views. In various embodiments, a scene may include a single object, multiple objects, or a rich virtual environment. Specifically, the database 106 stores multiple images corresponding to different perspectives or viewpoints of each asset. The images stored in the database 106 include high-quality photographs or photorealistic renderings. Such high-definition or high-resolution images entered into the database 106 may be captured or rendered during offline processing or obtained from an external source. In some embodiments, corresponding camera characteristics are stored with each image stored in the database 106. That is, camera attributes such as relative position or location, orientation, rotation, depth information, focal length, aperture, zoom level, etc. are stored with each image. Additionally, camera optical information such as shutter speed and exposure may be stored with each image stored in the database 106.

In various embodiments, any number of different perspectives of an asset may be stored in the database 106. FIG. 2 shows an example of a database asset. In the given example, 73 views corresponding to different angles around a chair object are captured or rendered and stored in the database 106. The views may be captured, for example, by rotating the camera around the chair or rotating the chair in front of the camera. Relative object and camera position and orientation information is stored with each generated image. FIG. 2 specifically shows a view of a scene including one object. The database 106 may also store specifications for scenes including multiple objects or rich virtual environments. In such cases, multiple views corresponding to different positions or locations in a scene or three-dimensional space are captured or rendered and stored in the database 106 along with the corresponding camera information. In general, the images stored in the database 106 may include two-dimensional or three-dimensional images and may include stills or frames of an animation or video sequence.

In response to a request 104 for an arbitrary view of a scene that does not already exist in the database 106, the arbitrary view generator 102 generates the requested arbitrary view from multiple other existing views of the scene stored in the database 106. In the example configuration of FIG. 1, an asset management engine 110 of the arbitrary view generator 102 manages the database 106. For example, the asset management engine 110 may facilitate the storage and retrieval of data in the database 106. In response to a request for an arbitrary view of the scene 104, the asset management engine 110 identifies and retrieves multiple other existing views of the scene from the database 106. In some embodiments, the asset management engine 110 retrieves all existing views of the scene from the database 106. Alternatively, the asset management engine 110 may select and retrieve a portion of the existing views (e.g., views that are closest to the requested arbitrary view). In such a case, the asset management engine 110 is configured to intelligently select a portion of the existing views from which pixels can be collected to generate the requested arbitrary view. In various embodiments, multiple existing views may be retrieved together by the asset management engine 110 or on an as-needed basis by other components of the arbitrary view generator 102.

The perspective of each existing view retrieved by the asset management engine 110 is transformed by the perspective transformation engine 112 of the arbitrary view generator 102 into the perspective of the requested arbitrary view. As mentioned above, the exact camera information is known and stored with each image stored in the database 106. Thus, the perspective change from the existing view to the requested arbitrary view includes a simple geometric mapping or transformation. In various embodiments, the perspective transformation engine 112 may use any one or more suitable mathematical techniques to transform the perspective of the existing view into the perspective of the arbitrary view. If the requested view includes an arbitrary view that is not identical to any existing view, the transformation of the existing view into the perspective of the arbitrary view will include at least some unmapped or missing pixels, i.e., pixels at angles or positions introduced into the arbitrary view that are not present in the existing view.

Pixel information from a single perspective-transformed existing view cannot fill all the pixels of another view. However, in many cases, most, if not all, pixels of the requested arbitrary view may be collected from multiple perspective-transformed existing views. The merge engine 114 of the arbitrary view generator 102 combines pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. Ideally, all pixels that make up the arbitrary view are collected from existing views. This may be possible, for example, when a sufficiently diverse set of existing views or perspectives is available for the asset under consideration and/or when the requested perspective is not too different from the existing perspectives.

Any suitable technique may be used to combine or merge pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. In one embodiment, a first existing view that is closest to the requested arbitrary view is selected and retrieved from the database 106 and transformed into the perspective of the requested arbitrary view. Pixels are then collected from this perspective-transformed first existing view and used to fill the corresponding pixels in the requested arbitrary view. To fill the pixels of the requested arbitrary view that could not be obtained from the first existing view, a second existing view that includes at least a portion of these remaining pixels is selected and retrieved from the database 106 and transformed into the perspective of the requested arbitrary view. The pixels that could not be obtained from the first existing view are then collected from this perspective-transformed second existing view and used to fill the corresponding pixels in the requested arbitrary view. This process may be repeated for any number of additional existing views until all pixels of the requested arbitrary view are filled and/or until all existing views are exhausted or a predetermined threshold number of existing views are utilized.

In some embodiments, the requested arbitrary view may include some pixels that could not be obtained from any of the existing views. In such cases, the interpolation engine 116 is configured to fill in all remaining pixels of the requested arbitrary view. In various embodiments, any one or more suitable interpolation techniques may be used by the interpolation engine 116 to generate these unfilled pixels in the requested arbitrary view. Examples of available interpolation techniques include, for example, linear interpolation, nearest neighbor interpolation, etc. Interpolation of pixels introduces averaging or smoothing. While the overall image quality is not significantly affected by a certain degree of interpolation, excessive interpolation may introduce unacceptable blurring. Therefore, it may be desirable to use interpolation sparingly. As mentioned above, if all pixels of the requested arbitrary view can be obtained from existing views, then interpolation is avoided entirely. However, if the requested arbitrary view includes some pixels that cannot be obtained from any of the views, then interpolation is introduced. In general, the amount of interpolation required depends on the number of existing views available, the diversity of perspectives of the existing views, and/or how different the perspective of the arbitrary view is with respect to the perspective of the existing views.

For the example shown in FIG. 2, 73 views around the chair object are stored as existing views of the chair. Any view around the chair object that is different or unique from any of the stored views may be generated using a plurality of these existing views, preferably with minimal, if any, interpolation. However, generating and storing such a comprehensive set of existing views may not be efficient or desirable. In some cases, a sufficiently small number of existing views may instead be generated and stored to cover a sufficiently diverse set of perspectives. For example, the 73 views of the chair object may be reduced to a small set of fewer views around the chair object.

As mentioned above, in some embodiments, the possible arbitrary views that can be requested may be at least partially constrained. For example, a user may be restricted from moving a virtual camera associated with an interactive scene to a particular position. With respect to the example given in FIG. 2, the possible arbitrary views that can be requested are restricted to any position around the chair object and may not include any position below the chair object, for example, because insufficient pixel data exists for the bottom of the chair object. Such constraints on the allowed arbitrary views ensure that the requested arbitrary views can be generated from existing data by the arbitrary view generator 102.

The arbitrary view generator 102 generates and outputs a requested arbitrary view 108 in response to an input arbitrary view request 104. The resolution or quality of the generated arbitrary view 108 is the same or comparable to the quality of the existing view used to generate it, since pixels from the existing view are used to generate the arbitrary view. Thus, in most cases, using a high definition existing view will result in a high definition output. In some embodiments, the generated arbitrary view 108 may be stored in the database 106 along with other existing views of the associated scene and later used to generate other arbitrary views of that scene in response to future requests for arbitrary views. If the input 104 includes a request for an existing view in the database 106, the requested view does not need to be generated from the other views as described above; instead, the requested view is retrieved using a simple database lookup and directly presented as the output 108.

The arbitrary view generator 102 may be further configured to generate an arbitrary ensemble view using the described techniques. That is, the input 104 may include a request to combine multiple objects into a single custom view. In such a case, the above-described techniques are performed for each of the multiple objects and combined to generate a single integrated or ensemble view that includes the multiple objects. Specifically, existing views of each of the multiple objects are selected and retrieved from the database 106 by the asset management engine 110, the existing views are transformed to the perspective of the requested view by the perspective transformation engine 112, pixels from the perspective-transformed existing views are used to fill corresponding pixels of the requested ensemble view by the merge engine 114, and any remaining unfilled pixels in the ensemble view are interpolated by the interpolation engine 116. In some embodiments, the requested ensemble view may include perspectives that already exist for one or more objects that make up the ensemble. In such cases, an existing view of the object asset that corresponds to the requested perspective is used to directly fill the pixels that correspond to the object in the ensemble view, instead of first generating the requested perspective from other existing views of the object.

As an example of an arbitrary ensemble view that includes multiple objects, consider the chair object and the separately photographed or rendered table object of FIG. 2. The chair object and the table object may be combined using the disclosed techniques to generate a single ensemble view of both objects. Thus, using the disclosed techniques, separately captured or rendered images or views of each of the multiple objects may be consistently combined to generate a scene that includes the multiple objects and has a desired perspective. As described above, the depth information of each existing view is known. The perspective transformation of each existing view includes a depth transformation, allowing the multiple objects to be properly positioned relative to each other in the ensemble view.

The generation of an arbitrary ensemble view is not limited to combining multiple single objects into a custom view. Rather, multiple scenes with multiple objects or multiple rich virtual environments may be similarly combined into a custom ensemble view. For example, multiple separate and independently generated virtual environments (possibly originating from different content sources and possibly having different pre-existing individual perspectives) may be combined into an ensemble view with a desired perspective. Thus, in general, the arbitrary view generator 102 may be configured to consistently combine or blend multiple independent assets, possibly including different pre-existing views, into an ensemble view with a desired, possibly arbitrary perspective. Since all combined assets are normalized to the same perspective, a perfectly matched resulting ensemble view is generated. The possible arbitrary perspectives of the ensemble view may be constrained based on the pre-existing views of the individual assets available for generating the ensemble view.

FIG. 3 is a flow diagram illustrating one embodiment of a process for generating an arbitrary perspective. Process 300 may be used, for example, by arbitrary view generator 102 of FIG. 1. In various embodiments, process 300 may be used to generate an arbitrary view or an arbitrary ensemble view of a given asset.

Process 300 begins at step 302, where a request for an arbitrary perspective is received. In some embodiments, the request received at step 302 may include a request for an arbitrary perspective of a given scene that is different from any existing available perspective of the scene. In such a case, for example, the arbitrary perspective request may be received in response to a request to change the perspective of a presented view of the scene. Such a change in perspective may be prompted by a change or manipulation of a virtual camera associated with the scene, such as panning the camera, changing the focal length, changing the zoom level, etc. Alternatively, in some embodiments, the request received at step 302 may include a request for an arbitrary ensemble view. As an example, such an arbitrary ensemble view request may be received in relation to an application that allows selection of multiple independent objects to provide a unified perspective-modified ensemble view of the selected objects.

At step 304, a number of pre-existing images from which at least a portion of the requested arbitrary perspective is generated are retrieved from one or more relevant asset databases. The retrieved images may be associated with a given asset if the request received at step 302 includes a request for an arbitrary perspective of the given asset, or may be associated with multiple assets if the request received at step 302 includes a request for an arbitrary ensemble view.

In step 306, each of the multiple existing images retrieved in step 304 having different perspectives is transformed to the arbitrary perspective requested in step 302. Each of the existing images retrieved in step 304 includes associated perspective information. The perspective of each image is defined by the camera characteristics associated with the generation of that image, such as relative position, orientation, rotation, angle, depth, focal length, aperture, zoom level, lighting information, etc. Since the complete camera information is known for each image, the perspective transformation of step 306 involves simple mathematical operations. In some embodiments, step 306 optionally further includes an optical transformation such that all images are consistently normalized to the same desired lighting conditions.

In step 308, at least a portion of the image having the requested arbitrary perspective in step 302 is filled with pixels collected from the existing perspective-transformed image. That is, pixels from multiple existing perspective-corrected images are used to generate the image having the requested arbitrary perspective.

In step 310, it is determined whether the generated image having the requested arbitrary perspective is complete. If it is determined in step 310 that the generated image having the requested arbitrary perspective is not complete, it is determined in step 312 whether additional existing images are available from which to obtain any remaining unfilled pixels of the generated image. If it is determined in step 312 that additional existing images are available, one or more additional existing images are retrieved in step 314 and process 300 proceeds to step 306.

If it is determined in step 310 that the generated image having the requested arbitrary perspective is not complete and it is determined in step 312 that no existing image is available any more, then all remaining unfilled pixels of the generated image are interpolated in step 316. Any suitable interpolation technique or techniques may be used in step 316.

If the generated image having the requested arbitrary perspective is determined to be complete in step 310, or after interpolating all remaining unfilled pixels in step 316, the generated image having the requested arbitrary perspective is output in step 318. Thereafter, process 300 ends.

As mentioned above, the disclosed techniques may be used to generate arbitrary perspectives based on other existing perspectives. Because camera information is stored with each existing perspective, it is possible to normalize the different existing perspectives to a common desired perspective. A resulting image having the desired perspective can be constructed by taking pixels from the perspective-transformed existing images. The processing associated with generating arbitrary perspectives using the disclosed techniques is not only fast and nearly instantaneous, but also produces high-quality output, making the disclosed techniques particularly powerful for interactive real-time graphics applications.

The disclosed techniques further describe the generation of an arbitrary ensemble view that includes multiple objects using available images or views of each of the multiple objects. As described above, perspective transformation and/or normalization allows pixels that comprise separately captured or rendered images or views of multiple objects to be consistently combined into a desired arbitrary ensemble view.

In some embodiments, it may be desirable to first construct or assemble a scene or ensemble view by selecting and arranging content that is desired to be included in the scene or ensemble view. In some such cases, multiple objects may be stacked or combined like building blocks to create a composite object that comprises the scene or ensemble view. As an example, consider an interactive application in which multiple independent objects are selected and appropriately positioned, e.g., on a canvas, to create a scene or ensemble view. Interactive applications may include, for example, visualization or modeling applications. In such applications, due to perspective distortions caused by the associated focal lengths, arbitrary views of objects cannot be used to construct the scene or ensemble view. Rather, predetermined object views that are substantially free of perspective distortion are utilized as described below.

Orthographic views of objects are used in some embodiments to model or define scenes or ensemble views that include multiple independent objects. An orthographic view includes a parallel projection approximated by a (virtual) camera with a relatively long focal length, positioned at a large distance from the object relative to its size, such that the light or projection rays are substantially parallel. An orthographic view has no depth or a fixed depth, and therefore has little or no perspective distortion. Thus, orthographic views of objects may be used similarly to building blocks when specifying ensemble scenes or composite objects. After an ensemble scene containing any combination of objects has been specified or defined using such orthographic views, the scene or its objects may be transformed to any desired camera perspective using the arbitrary view generation techniques described above with respect to the description of Figures 1-3.

In some embodiments, the multiple views of an asset stored in database 106 of system 100 of FIG. 1 include one or more orthographic views of the asset. Such orthographic views may be captured (e.g., photographed or scanned) or rendered from a three-dimensional polygon mesh model. Alternatively, the orthographic views may be generated from other views of the asset available in database 106 according to any of the view generation techniques described above with respect to the description of FIGS. 1-3.

FIGS. 4A-4N show example embodiments of applications in which separate objects are combined to generate an ensemble or composite object or scene. Specifically, FIGS. 4A-4N show an example of a furniture assembly application in which various separate sofa components are combined to generate different modular sofa configurations.

Figure 4A is an example of a perspective view showing three separate sofa components (i.e., a one-seater with left arms, a loveseat without arms, and a chaise longue with right arms). Each of the perspective views in the example of Figure 4A has a focal length of 25 mm. As can be seen, the resulting perspective distortion prevents stacking of the components adjacent to one another (i.e., placing the components side-by-side), which may be desired when assembling a modular sofa configuration that includes the components.

Figure 4B shows an example of an orthographic view of the same three components as Figure 4A. As can be seen, the orthographic view of the objects is modular or block-like, suitable for stacking or placing next to each other. However, depth information is essentially lost in the orthographic view. As can be seen, in Figure 4A there are depth differences, particularly with respect to the chaise longue, but in the orthographic view, all three appear to have the same depth.

Figure 4C shows an example of combining the orthographic views of the three components of Figure 4B to define a composite object. That is, Figure 4C illustrates the creation of an orthographic view of a modular sofa by placing the orthographic views of the three components of Figure 4B side-by-side. As shown in Figure 4C, the bounding boxes of the orthographic views of the three sofa components fit closely next to each other to create the orthographic view of the modular sofa. That is, the orthographic views of the components facilitate user-friendly manipulation and precise placement of the components in the scene.

Each of Figures 4D and 4E shows an example of converting the orthographic view of the composite object of Figure 4C to an arbitrary camera perspective using the arbitrary view generation technique described above with respect to the description of Figures 1-3. That is, in each of the examples of Figures 4D and 4E, the orthographic view of the composite object has been converted to a normal camera perspective that accurately depicts depth. As can be seen, the relative depth of the chaise longue to the one-seater and loveseater, which was lost in the orthographic view, is now visible in the perspective views of Figures 4D and 4E.

4F, 4G, and 4H show examples of multiple orthographic views of a single seat with left arms, a loveseat without arms, and a chaise longue with right arms, respectively. As mentioned above, any number of different views or perspectives of an asset may be stored in the database 106 of the system 100 of FIG. 1. The set of FIGS. 4F-4H includes 25 orthographic views corresponding to different angles around each asset separately captured or rendered and stored in the database 106, from which any arbitrary view of any combination of objects may be generated. In a furniture assembly application, for example, a top view may be useful for floor placement, while a front view may be useful for wall placement. In some embodiments, to maintain a more compact reference data set, only a predetermined number of orthographic views are stored for an asset in the database 106, from which any arbitrary view of the asset may be generated.

Figures 4I-4N show various examples of generating an arbitrary view or perspective of an arbitrary combination of objects. Specifically, each of Figures 4I-4N shows the generation of an arbitrary perspective or view of a unit sofa that includes multiple separate sofa objects or components. Each arbitrary view may be generated, for example, by converting one or more orthographic (or other) views of the objects that make up the ensemble view or composite object into the arbitrary view using the arbitrary view generation techniques described above with respect to the description of Figures 1-3, incorporating pixels to fill the arbitrary view, and possibly interpolating any remaining missing pixels.

As mentioned above, each image or view of an asset in database 106 may be stored with corresponding metadata (such as relative object and camera position and orientation information, as well as lighting information). The metadata may be generated when rendering the view from a three-dimensional polygon mesh model of the asset, when imaging or scanning the asset (in which case depth and/or surface normal data may be estimated), or a combination of both.

A given view or image of an asset includes pixel intensity values (e.g., RGB values) for each pixel comprising the image, as well as various metadata parameters associated with each pixel. In some embodiments, one or more of the red, green, and blue (RGB) channels or values of a pixel may be used to encode pixel metadata. The pixel metadata may include, for example, information about the relative location or position (e.g., x, y, and z coordinate values) of a point in three-dimensional space that is projected onto that pixel. In addition, the pixel metadata may include information about the surface normal vector at that location (e.g., the angle with the x, y, and z axes). The pixel metadata may also include texture mapping coordinates (e.g., u and v coordinate values). In such cases, the actual pixel value at the point is determined by reading the RGB values of the corresponding coordinates in the texture image.

The surface normal vectors facilitate modification or alteration of the lighting of any generated view or scene. More specifically, altering the lighting of a scene includes scaling pixel values based on how well the pixel's surface normal vector matches the direction of a newly added, removed, or otherwise altered light source (e.g., which may be quantified, at least in part, by the dot product of the light source direction and the pixel's normal vector). Defining pixel values using texture mapping coordinates facilitates modification or alteration of the texture of any generated view or scene or part thereof. More specifically, the texture can be altered by simply exchanging or replacing a referenced texture image with another texture image having the same dimensions.

The disclosed arbitrary view generation techniques are effectively based on relatively low computational cost perspective transformations and/or lookup operations. An arbitrary (ensemble) view may be generated by simply selecting the correct pixels and appropriately filling the generated arbitrary view with those pixels. In some cases, pixel values may be optionally scaled, for example, if lighting has been adjusted. The low storage and processing overhead of the disclosed techniques facilitates fast, real-time, or on-demand generation of arbitrary views of complex scenes with quality comparable to the high-definition reference views from which they are generated.

As described above, assembling an ensemble or composite object or scene in some embodiments involves specifying the multiple object assets that make up the ensemble using an orthographic view. The orthographic view facilitates precise placement and alignment of the multiple objects or assets in the ensemble scene. The orthographic view of the ensemble scene may then be transformed to any arbitrary camera perspective, for example, to generate any desired or required perspective. Transforming the ensemble view to a given camera perspective may include individually transforming each of the multiple objects or assets that make up the ensemble scene to the given perspective using the techniques described above. While the techniques described above for generating an arbitrary ensemble view are relatively efficient, being even more efficient may be desirable in certain applications (e.g., applications that provide users with an interactive, real-time experience) where it is advantageous to generate output almost instantly or at least very quickly, with a latency penalty that is barely detectable to the end user.

In some embodiments, further improvements in efficiency may be facilitated, at least in part, by eliminating the processing associated with transforming a large portion of the objects or assets (e.g., their orthographic or other views) that make up the ensemble scene to the predetermined arbitrary perspective. Instead, the available existing views of the objects or assets that are closest or most similar to the predetermined arbitrary perspective for a given position and orientation of the objects or assets in the ensemble scene are used for the objects or assets when generating the output ensemble view or image that represents the predetermined arbitrary perspective. In most cases, the resulting output ensemble view is not completely perspective accurate, but provides a suitable approximation that is acceptable for many applications and is generated with significantly less latency than generating a completely perspective accurate output. The generation of such an approximation of an arbitrary ensemble view for an arbitrary camera pose using a maximally quantized subset of already existing reference views of one or more objects or assets that make up the ensemble is described in more detail below.

FIG. 5 is a high-level flow chart illustrating one embodiment of a process for generating an arbitrary ensemble view. In some embodiments, the process 500 is used to efficiently generate an output image of an ensemble scene based at least in part on appropriately combining or compositing a single best-match existing view of at least one or more objects or assets (if not most or all) that make up the ensemble scene.

The process 500 begins at step 502, where a request for a predefined perspective of an ensemble scene is received. The requested predefined perspective of the ensemble scene includes a selected or otherwise specified camera perspective for the ensemble scene, and may generally include any arbitrary view. An arbitrary view in a given context includes any desired view or perspective of a scene where the specification or camera pose is not known in advance prior to the request. The ensemble scene includes a composite view of multiple independent objects or assets. In general, the specification of an independent object or asset includes a set of existing reference images or views of the individual objects or assets with different camera perspectives and corresponding metadata, one or more of which may be used to generate or specify a portion of the ensemble scene associated with the object or asset. In some embodiments, the request of step 502 is received from an interactive mobile or web-based application that facilitates the manipulation of camera angles or camera poses in the ensemble scene space and/or the positioning of multiple objects or assets to create a composite or ensemble scene. For example, the request may be received from a visualization or modeling application or an augmented reality (AR) application. In some embodiments, the request of operation 502 is received with respect to an orthographic view of the ensemble scene, since an orthographic view facilitates easier manipulation, placement, and alignment of multiple objects or assets that make up the ensemble scene.

In step 504, the closest or most similar matching existing reference image or view is selected for each of at least a portion of one or more objects or assets that make up the ensemble scene. Step 504 may be performed sequentially and/or in parallel for individual or independent objects or assets that make up the ensemble scene. In some embodiments, only one or a single existing reference image or view is selected for an object or asset that best matches a predetermined perspective required for a given pose of the object or asset in the ensemble scene space. The ensemble scene space comprises an ensemble scene coordinate system having a predetermined origin (such as the center (e.g., center of gravity) of the ensemble scene) defined in an appropriate manner. In step 504, to select the closest matching existing reference image or view for an object or asset that makes up the ensemble scene, the position and orientation or pose of the object or asset with respect to the ensemble scene coordinate system is determined and then translated, transformed or otherwise correlated to an equivalent pose in the individual coordinate system associated with the existing reference image or view of the object or asset. Therefore, a simple camera metric calculation with relatively low computational complexity is performed based on the requested perspective and the relative object or asset poses in the ensemble scene so that the closest matching existing reference image or view can be selected in step 504.

One or more criteria and/or thresholds may be defined to determine or identify the closest matching existing reference image or view for the object or asset. In some cases, an existing reference image or view is selected in step 504 only if one or more such thresholds are met. In the ideal case, a perfect match is found and selected in step 504. However, in some cases, one or more selection criteria and/or thresholds cannot be met if the available existing reference image data set is too incomplete (e.g., if the available existing reference image or view of the object or asset is significantly different from the requested perspective) or if no reference image or view is available for the object or asset. In some such cases, a closest matching placeholder or ghost image or view of the object or asset is selected instead in step 504. Such a placeholder image or view represents the shape of the object or asset but lacks other attributes (e.g., texture and optical properties). In some embodiments, a set of placeholder images spanning a sufficiently dense set of possible views around the object or asset (e.g., including angles covering 360° around the object or asset) are generated and stored for each unique object shape using relatively low computational complexity rendering techniques. Placeholders are utilized when a fully rendered version of the object or asset is unavailable or exhibits unacceptable deviations from the requested perspective.

In step 506, an output image of the ensemble scene is generated for the requested predefined perspective, at least in part, by appropriately combining or compositing the closest matching existing reference images or views of the objects or assets that make up the ensemble scene selected in step 504. Step 506 may include appropriately scaling or resizing the closest matching existing reference images or views selected for the objects or assets, and/or determining a location or position within the ensemble view into which to paste or composite the closest matching existing reference images or views selected for the objects or assets. In most cases, the generated output image of the ensemble scene closely approximates the requested predefined perspective. Since most objects or assets that make up the ensemble scene are represented in the output image with their nearest or most similar existing pose available, the perspective is not completely accurate because these objects or assets are not exactly rendered or generated. That is, in most cases, these objects or assets will not have the requested predefined perspective in the output image unless a perfect match is found among the available existing images or views. While not all of the vanishing points of such objects or assets point to the same point in the output image, the objects or assets are in most cases offset or tilted by a small enough amount (e.g., a few degrees) to trick the human visual system into perceiving the output image as having the correct perspective for the most part.

Consistency in the output image of the ensemble scene is further promoted by generating at least some parts of the ensemble scene in a globally consistent or similar manner, which further facilitates a human interpretation of the output image as substantially visually accurate. For example, one or more objects or assets that make up the ensemble scene and/or surfaces (flat or otherwise), structural elements, global features, etc. that make up the ensemble scene may be rendered or generated to be perspective accurate, i.e., to have the requested predefined perspective, rather than an approximation of the requested perspective. For example, if the ensemble scene includes a space such as a room, the walls, ceiling, floor, rugs, wall hangings, etc. may be generated using the camera pose of the requested perspective, and therefore may be accurately represented in the output image of the ensemble scene generated in step 506. Furthermore, the output image of the ensemble scene may comprise a global lighting position that affects all parts of the scene in a similarly consistent manner, for example, when modifying the lighting using available metadata (such as surface normal vectors). Thus, by generating some parts of the ensemble view in a global or perspective-correcting manner and representing most of the independent objects that make up the ensemble view as a best approximation, an output is generated that is often nearly indistinguishable from a completely perspective accurate version. In some cases, some skewness may be observed, but this may still be acceptable for certain applications that do not require a completely accurate view, such as mood board applications or space/room planning applications where the designer or user benefits from seeing the ensemble of objects or assets regardless of the perspective accuracy. However, as the repository or database of available existing images or views grows over time, the disclosed technique continues to generate output that more and more accurately represents the desired predefined perspective. In the optimal case, an exact match is found for all objects or assets and is used to generate an output image that actually has the desired predefined perspective, not an approximation.

Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not limiting.
[Application Example 1] A method, comprising:
receiving a request for a given perspective of an ensemble scene including multiple assets;
generating an output image of the ensemble scene that approximates the requested predetermined perspective based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets;
A method comprising:
[Application Example 2] The method according to Application Example 1, wherein the request is received with respect to an orthogonal view of the ensemble scene.
[Application Example 3] The method according to Application Example 2, wherein the orthogonal projection view of the ensemble scene includes a combined orthogonal projection view of the multiple assets.
[Application Example 4] The method according to Application Example 1, further comprising selecting the single existing image of each of the at least a portion of the plurality of assets.
[Application Example 5] The method according to Application Example 4, wherein the selecting includes selecting an exact match with the requested predetermined perspective.
[Application Example 6] The method according to Application Example 4, wherein the selecting includes selecting the closest or most similar available match to the requested given perspective.
[Application Example 7] The method according to Application Example 4, wherein the selecting includes selecting based on a posture of a related asset within the ensemble scene.
[Application Example 8] The method according to Application Example 4, wherein the selecting includes selecting a rotated existing image of the related asset.
[Application Example 9] The method described in Application Example 4, wherein the selecting includes selecting the closest or most similar available match to the requested given perspective based on the pose of related assets within the ensemble scene.
[Application Example 10] The method described in Application Example 1, wherein generating the output image of the ensemble scene includes scaling the single existing image of one or more assets within the portion of assets.
[Application Example 11] The method described in Application Example 1, wherein generating the output image of the ensemble scene includes resizing the single existing image of one or more assets within the portion of assets.
[Application Example 12] A method as described in Application Example 1, wherein generating the output image of the ensemble scene includes determining a position to include the single existing image of each of at least the portion of the assets in the ensemble scene.
[Application Example 13] The method according to Application Example 1, wherein the combining includes synthesizing.
[Application Example 14] A method as described in Application Example 1, wherein generating the output image of the ensemble scene includes generating a view of at least one asset of the multiple assets having the requested predetermined perspective.
[Application Example 15] The method according to Application Example 14, wherein the view is generated using a plurality of existing images of the at least one asset.
[Application Example 16] A method as described in Application Example 1, wherein generating the output image of the ensemble scene includes generating at least one portion of the ensemble scene to have the requested predetermined perspective.
[Application Example 17] The method according to Application Example 16, wherein the at least one portion includes a surface of the ensemble scene.
[Application Example 18] The method according to Application Example 16, wherein the at least one portion includes a structural element of the ensemble scene.
[Application Example 19] The method according to Application Example 16, wherein the at least one portion includes a global characteristic of the ensemble scene.
[Application Example 20] The method according to Application Example 1, further comprising globally modifying the illumination of the generated output image of the ensemble scene.
[Application Example 21] The method according to Application Example 1, wherein the output image comprises a frame of a video sequence.
[Application Example 22] A system,
1. A processor comprising:
receiving a request for a given perspective of an ensemble scene including multiple assets;
a processor configured to generate an output image of the ensemble scene that approximates the requested predetermined perspective based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets;
a memory coupled to the processor and configured to provide instructions to the processor;
A system comprising:
[Application Example 23] A computer program product embodied in a non-transitory computer-readable storage medium,
computer instructions for receiving a request for a given perspective of an ensemble scene including a plurality of assets;
computer instructions for generating an output image of the ensemble scene that approximates the requested predetermined perspective based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets;
A computer program product comprising:

Claims (22)

1. A method comprising:
receiving a request for a given perspective of an ensemble scene including multiple assets;
generating an output image of the ensemble scene that approximates the requested predetermined perspective, but is not entirely accurate, based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets that are offset or tilted by several degrees;
wherein a single pre-existing image of each of at least a portion of the plurality of assets does not include the requested predetermined perspective . The method of claim 1, wherein the request is received for an orthogonal view of the ensemble scene. The method of claim 2, wherein the orthogonal view of the ensemble scene includes a combined orthogonal view of the multiple assets. The method of claim 1, further comprising selecting the single existing image of each of the at least some of the plurality of assets. The method of claim 4, wherein the selecting includes selecting an available match that is closest or most similar to the requested given perspective. The method of claim 4, wherein the selecting includes selecting based on a pose of a related asset in the ensemble scene. The method of claim 4, wherein the selecting includes selecting a rotated existing image of the related asset. The method of claim 4, wherein the selecting includes selecting the closest or most similar available match to the requested given perspective based on poses of related assets in the ensemble scene. The method of claim 1, wherein generating the output image of the ensemble scene includes scaling the single existing image of one or more assets within the portion of assets. The method of claim 1, wherein generating the output image of the ensemble scene includes resizing the single existing image of one or more assets within the portion of assets. The method of claim 1, wherein generating the output image of the ensemble scene includes determining a location to include the single existing image of each of at least the portion of assets in the ensemble scene. The method of claim 1, wherein the combining includes compounding. The method of claim 1, wherein generating the output image of the ensemble scene includes generating a view of at least one asset of the plurality of assets having the requested predetermined perspective. The method of claim 13 , wherein the view is generated using a plurality of existing images of the at least one asset. The method of claim 1, wherein generating the output image of the ensemble scene includes generating at least one portion of the ensemble scene to have the requested predetermined perspective. The method of claim 15 , wherein the at least one portion comprises a surface of the ensemble scene. The method of claim 15 , wherein the at least one portion includes a structural element of the ensemble scene. The method of claim 15 , wherein the at least one portion comprises a global characteristic of the ensemble scene. The method of claim 1, further comprising globally varying the illumination of the generated output image of the ensemble scene. The method of claim 1, wherein the output image comprises a frame of a video sequence. 1. A system comprising:
1. A processor comprising:
receiving a request for a given perspective of an ensemble scene including multiple assets;
a processor configured to generate an output image of the ensemble scene based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets offset or tilted by several degrees , the output image approximating the requested predetermined perspective, but not being completely accurate in perspective;
a memory coupled to the processor and configured to provide instructions to the processor;
wherein each single existing image of at least a portion of the plurality of assets does not include the requested predetermined perspective . A computer program product embodied in a non-transitory computer-readable storage medium, comprising:
computer instructions for receiving a request for a given perspective of an ensemble scene including a plurality of assets;
computer instructions for generating an output image of the ensemble scene based at least in part on combining a single pre-existing image of each of at least a portion of the plurality of assets offset or tilted by several degrees , the output image approximating the desired predetermined perspective, where the perspective is not entirely accurate;
wherein each single existing image of at least a portion of the plurality of assets does not include the requested predetermined perspective .

Images