tensorflow_graphics/notebooks/intrinsics_optimization.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NvTtGJ0buKqy"
      },
      "source": [
        "##### Copyright 2019 Google LLC.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "2ApyPUEKuQ68"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Bp1v8vOd0TSl"
      },
      "source": [
        "# Camera calibration\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/graphics/blob/master/tensorflow_graphics/notebooks/intrinsics_optimization.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/graphics/blob/master/tensorflow_graphics/notebooks/intrinsics_optimization.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tIPehekthWBr"
      },
      "source": [
        "Cameras are complex pieces of hardware that are able to capture 2D images of 3D objects. Indeed, to model real cameras precisely, one need to estimate many parameters including lens distortion, ISO, focal length, and exposure time. In the following, we restrict our focus to the projective camera model.\n",
        "\n",
        "![](https://storage.googleapis.com/tensorflow-graphics/notebooks/intrinsics/camera_model.jpg)\n",
        "\n",
        " This model is composed of two parameters that are often referred to as intrinsic parameters:\n",
        "- the principal point, which is the projection of the optical center on the image. Ideally, the principal point is close to the center of the image.\n",
        "- the focal length, which is the distance between the optical center and the image plane. This parameters allows to control the level of zoom.\n",
        "\n",
        "This notebook illustrates how to use [Tensorflow Graphics](https://github.com/tensorflow/graphics) to estimate the intrinsic parameters of a projective camera. Recovering these parameters is particularly important to perform several tasks, including 3D reconstruction.\n",
        "\n",
        "In this Colab, the goal is to recover the intrinsic parameters of a camera given an observation and correspondences between the observation and the render of the current solution.  Things are kept simple by only inserting a rectangle in the 3D scene, and using it as the source of correspondences during the optimization. The minimization is performed using the Levenberg-Marquardt algorithm."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7U2aF6a5z7_T"
      },
      "source": [
        "## Setup \u0026 Imports\n",
        "If Tensorflow Graphics is not installed on your system, the following cell can install the Tensorflow Graphics package for you."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "51bvxlEr0AQ3"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow_graphics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JEv08gaU0ECu"
      },
      "source": [
        "Now that Tensorflow Graphics is installed, let's import everything needed to run the demo contained in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "CXnl2yDzCcvP"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "#################################\n",
        "# Imports the necessary modules #\n",
        "#################################\n",
        "\n",
        "from tensorflow_graphics.math.optimizer import levenberg_marquardt\n",
        "from tensorflow_graphics.rendering.camera import perspective  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3pMk9JsWNTPb"
      },
      "source": [
        "## Understanding the perspective camera model\n",
        "To illustrate how this model works, we will assume that there is nothing in the scene in front of the camera, but a rectangle.\n",
        "Let's first define a function that will render this rectangle as observed by our camera."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "C5dSYND5E0h0"
      },
      "outputs": [],
      "source": [
        "def render_rectangle(rectangle_vertices, focal, principal_point, image_dimensions):\n",
        "  \"\"\"Renders a rectangle on the image plane.\n",
        "\n",
        "  Args:\n",
        "    rectangle_vertices: the four 3D corners of a rectangle.\n",
        "    focal: the focal lengths of a projective camera.\n",
        "    principal_point: the position of the principal point on the image plane.\n",
        "    image_dimensions: the dimensions (in pixels) of the image.\n",
        "\n",
        "  Returns:\n",
        "    A 2d image of the 3D rectangle.\n",
        "  \"\"\"\n",
        "  image = np.zeros((int(image_dimensions[0]), int(image_dimensions[1]), 3))\n",
        "  vertices_2d = perspective.project(rectangle_vertices, focal, principal_point)\n",
        "  vertices_2d_np = vertices_2d.numpy()\n",
        "  top_left_corner = np.maximum(vertices_2d_np[0, :], (0, 0)).astype(int)\n",
        "  bottom_right_corner = np.minimum(\n",
        "      vertices_2d_np[1, :],\n",
        "      (image_dimensions[1] - 1, image_dimensions[0] - 1)).astype(int)\n",
        "  for x in range(top_left_corner[0], bottom_right_corner[0] + 1):\n",
        "    for y in range(top_left_corner[1], bottom_right_corner[1] + 1):\n",
        "      c1 = float(bottom_right_corner[0] + 1 -\n",
        "                 x) / float(bottom_right_corner[0] + 1 - top_left_corner[0])\n",
        "      c2 = float(bottom_right_corner[1] + 1 -\n",
        "                 y) / float(bottom_right_corner[1] + 1 - top_left_corner[1])\n",
        "      image[y, x] = (c1, c2, 1)\n",
        "  return image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3juVb3ulSQft"
      },
      "source": [
        "The following cell defines default intrinsic parameters and renders the rectangle using these parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "SFML6hkRxaeZ"
      },
      "outputs": [],
      "source": [
        "# Sets up the vertices of the rectangle.\n",
        "rectangle_depth = 1000.0\n",
        "rectangle_vertices = np.array(\n",
        "    ((-150.0, -75.0, rectangle_depth), (150.0, 75.0, rectangle_depth)))\n",
        "\n",
        "# Sets up the size of the image plane.\n",
        "image_width = 400\n",
        "image_height = 300\n",
        "image_dimensions = np.array((image_height, image_width), dtype=np.float64)\n",
        "\n",
        "# Sets the horizontal and vertical focal length to be the same. The focal length\n",
        "# picked yields a field of view around 50degrees.\n",
        "focal_lengths = np.array((image_height, image_height), dtype=np.float64)\n",
        "# Sets the principal point at the image center.\n",
        "ideal_principal_point = np.array(\n",
        "    (image_width, image_height), dtype=np.float64) / 2.0\n",
        "\n",
        "# Let's see what our scene looks like using the intrinsic parameters defined above.\n",
        "render = render_rectangle(rectangle_vertices, focal_lengths, ideal_principal_point,\n",
        "                          image_dimensions)\n",
        "_ = plt.imshow(render)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-NaBsBom1_Wi"
      },
      "source": [
        "The focal length and the position of the optical center have very different\n",
        "effects on the final images. Use different configuration of the sliders below to\n",
        "convince you of the effect of each of these parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "Lz4OI0svv6nb"
      },
      "outputs": [],
      "source": [
        "###############\n",
        "# UI controls #\n",
        "###############\n",
        "#@title model parameters { vertical-output: false, run: \"auto\" }\n",
        "focal_length_x = 300  #@param { type: \"slider\", min: 100.0, max: 500.0, step: 1.0 }\n",
        "focal_length_y = 300  #@param { type: \"slider\", min: 100.0, max: 500.0, step: 1.0 }\n",
        "optical_center_x = 200  #@param { type: \"slider\", min: 0.0, max: 400.0, step: 1.0 }\n",
        "optical_center_y = 150.0  #@param { type: \"slider\", min: 0.0, max: 300.0, step: 1.0 }\n",
        "\n",
        "render = render_rectangle(\n",
        "    rectangle_vertices,\n",
        "    np.array((focal_length_x, focal_length_y), dtype=np.float64),\n",
        "    np.array((optical_center_x, optical_center_y), dtype=np.float64),\n",
        "    image_dimensions)\n",
        "_ = plt.imshow(render)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Oz9g_IiWPvd0"
      },
      "source": [
        "## Optimizing intrinsic parameters\n",
        "Every camera (e.g. smartphone camera) comes with its own set of intrinsics parameters. Among other applications, precise intrinsic parameters are used in 3D scene reconstruction, robotics, and navigation systems.\n",
        "\n",
        "A common way to estimate intrinsic parameters is to use a known 3D object. Using our current estimate of the intrinsic parameters, we can predict how the known 3D object should 'look', and compare that to the actual observation. \n",
        "\n",
        "Let's start by defining some helper functions which will help plotting results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "3Gg6BjRRRfEN"
      },
      "outputs": [],
      "source": [
        "def plot_optimization_step(observation, prediction):\n",
        "  plt.figure(figsize=(20, 10))\n",
        "  ax = plt.subplot(1, 3, 1)\n",
        "  ax.set_title(\"Observation\")\n",
        "  _ = ax.imshow(observation)\n",
        "  ax = plt.subplot(1, 3, 2)\n",
        "  ax.set_title(\"Prediction using estimated intrinsics\")\n",
        "  _ = ax.imshow(prediction)\n",
        "  ax = plt.subplot(1, 3, 3)\n",
        "  ax.set_title(\"Difference image\")\n",
        "  _ = ax.imshow(np.abs(observation - prediction))\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "def print_errors(focal_error, center_error):\n",
        "  print(\"Error focal length %f\" % (focal_error,))\n",
        "  print(\"Err principal point %f\" % (center_error,))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "O660v6cWVRQf"
      },
      "source": [
        "Let's now define the values of the intrinsic parameters we are looking for, and an initial guess of the values of these parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "qKKCZzWAZ8Jh"
      },
      "outputs": [],
      "source": [
        "def build_parameters():\n",
        "  # Constructs the intrinsic parameters we wish to recover.\n",
        "  real_focal_lengths = focal_lengths * np.random.uniform(0.8, 1.2, size=(2,))\n",
        "  real_principal_point = ideal_principal_point + (np.random.random(2) -\n",
        "                                                  0.5) * image_width / 5.0\n",
        "\n",
        "  # Initializes the first estimate of the intrinsic parameters.\n",
        "  estimate_focal_lengths = tf.Variable(real_focal_lengths +\n",
        "                                       (np.random.random(2) - 0.5) *\n",
        "                                       image_width)\n",
        "  estimate_principal_point = tf.Variable(real_principal_point +\n",
        "                                         (np.random.random(2) - 0.5) *\n",
        "                                         image_width / 4)\n",
        "  return real_focal_lengths, real_principal_point, estimate_focal_lengths, estimate_principal_point"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QTdXuY6BapnT"
      },
      "source": [
        "As described earlier, one can compare how the 3D object would look using the current estimate of the intrinsic parameters, can compare that to the actual observation. The following function captures a distance between these two images which we will seek to minimize."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "LKTAD6AUarl6"
      },
      "outputs": [],
      "source": [
        "def residuals(estimate_focal_lengths, estimate_principal_point):\n",
        "  vertices_2d_gt = perspective.project(rectangle_vertices, real_focal_lengths,\n",
        "                                       real_principal_point)\n",
        "  vertices_2d_observed = perspective.project(rectangle_vertices,\n",
        "                                             estimate_focal_lengths,\n",
        "                                             estimate_principal_point)\n",
        "  return vertices_2d_gt - vertices_2d_observed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TZeO_ESPdUOv"
      },
      "source": [
        "All the pieces are now in place to solve the problem; let's give it a go!\n",
        "\n",
        "Note: the residuals are minimized using Levenberg-Marquardt, which is particularly indicated for this problem. First order optimizers (e.g. Adam or gradient descent) could also be used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0f0m57YmDZ45"
      },
      "outputs": [],
      "source": [
        "# Samples intrinsic parameters to recover and an initial solution.\n",
        "real_focal_lengths, real_principal_point, estimate_focal_lengths, estimate_principal_point = build_parameters(\n",
        ")\n",
        "\n",
        "# Constructs the observed image.\n",
        "observation = render_rectangle(rectangle_vertices, real_focal_lengths,\n",
        "                               real_principal_point, image_dimensions)\n",
        "\n",
        "# Displays the initial solution.\n",
        "print(\"Initial configuration:\")\n",
        "print_errors(\n",
        "    np.linalg.norm(estimate_focal_lengths - real_focal_lengths),\n",
        "    np.linalg.norm(estimate_principal_point - real_principal_point))\n",
        "image = render_rectangle(rectangle_vertices, estimate_focal_lengths,\n",
        "                         estimate_principal_point, image_dimensions)\n",
        "plot_optimization_step(observation, image)\n",
        "\n",
        "# Optimization.\n",
        "_, (estimate_focal_lengths,\n",
        "    estimate_principal_point) = levenberg_marquardt.minimize(\n",
        "        residuals, (estimate_focal_lengths, estimate_principal_point), 1)\n",
        "\n",
        "print(\"Predicted configuration:\")\n",
        "print_errors(\n",
        "    np.linalg.norm(estimate_focal_lengths - real_focal_lengths),\n",
        "    np.linalg.norm(estimate_principal_point - real_principal_point))\n",
        "image = render_rectangle(rectangle_vertices, estimate_focal_lengths,\n",
        "                         estimate_principal_point, image_dimensions)\n",
        "plot_optimization_step(observation, image)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "intrinsics_optimization.ipynb",
      "provenance": [],
      "toc_visible": true,
      "private_outputs": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}