CISC 472 - 2D/3D Registration

2D/3D registration is done when we want to register a patient to a 3D model of that patient, but we can only capture 2D images of the patient.

The 3D model of the patient could be from a CT or MR scan.

The 2D images could be captured from:

fluoroscopy (i.e. intraoperative xray on a video screen), in which case the 2D image is a projection of the patient onto the image plane;
ultrasound, in which case the 2D image is a slice of the patient in the plane of the ultrasound probe; or
CT or MR, in which case the 2D image is a slice of the patient in the plane of the scanner.

The problem is: Given a set of 2D images in which the pose of the imaging device (fluoro, ultrasound, CT, MR) is known in a "world coordinate system", and given a 3D computer model of the patient in another "model coordinate system", find the transformation that maps the model into the world.

We saw how to do this with Procrustes if we know the positions of $n$ points in the world coordinate systems and the positions of $n$ corresponding points in the model coordinate system.

We could even do this if $n$ points are known in the world and their corresponding points are known on the 2D image plane. Various PnP ("points-N-perspective") algorithms could find this transformation; we have not discussed such algorithms.

In any case, we often don't have such point correspondences, and must rely upon 2D/3D image-based registration.

We'll discuss one example of 2D/3D image-based registration: that of registering a patient in the operating room to a pre-operative 3D CT scan, using a 2D fluoro image (or several 2D fluoro images) taken in the operating room.

The fluoro image provides a 2D x-ray projection of the patient onto the fluoro machine's image plane. The fluoro machine acts like a pinhole camera (but with x-rays instead of light), so its imaging parameters are completely defined by an extrinsic transformation (consisting of rotation and translation) and an intrinsic transformation (consisting of focal lengths, skew, and image centre).

We will assume that the intrinsic transformation is the identity, corresponding to a focal length of 1.0, no skew, and (0,0) at the centre of the image. (Given a fluro image with a non-identity intrinsic camera matrix, we can always generate a "normalized image" by applying the inverse of that matrix. So this assumption doesn't cause any problems.) We'll also assume that the flouro's coordinate system is aligned with the "world coordinate system".

The problem is now: Given a 2D fluoro image of the patient with the assumptions above, and given a 3D CT model, what is the transformation, $T$, that positions the 3D CT model such that its projection is the same as the fluoro image?

In the diagram below:

The MCS is the "model coordinate system" in which the 3D CT model of the patient is defined.
The WCS is the "world coordinate system" with which the fluoro is aligned.
The image plane is the plane onto which the fluoro projects its images. This is where the image of the patient, which is captured by the fluoro in the operating room, appears.
The transformation T maps from the model coordinate system to the world coordinate system.

Problem Definition

There are two images here:

$I_\textrm{fluoro}$ is the image captured by the fluoro.
$I_\textrm{proj}$ is the simulated projection of the 3D CT model onto the image plane, after transformation $T$ has been applied to the model.

So the goal of the registration algorithm is to find the $T$ that minimizes the difference between $I_\text{fluoro}$ and $I_\textrm{proj}$.

Problem Solution

Given a particular $T$, the algorithm can compute the projection image, $I_\textrm{proj}$.

Then the algorithm can apply a similarity measure (such as NCC or MI from the previous lectures) to determine how well this $I_\textrm{proj}$ matches with the $I_\textrm{fluoro}$ captured by the fluoro.

An optimization method (such as Powell's Method or Downhill Simplex from the previous lecture) can be applied to find the optimal $T$. The optimization method must search in a six-dimensional space consisting of three dimensions of translation and three dimensions of rotation. So each solution, $x$, referred to in the previous lecture would actually be a six-component vector.

If we happen to have multiple fluoro images taken from different fluoro positions and orientations, the similarity measure would be taken over all pairs of fluoro and projected images. The measure could be the average of the individual measures, or perhaps the minimum measure, which corresponds to the worst-matching pair of images.

The only thing missing here is a method to compute the projection image, $I_\textrm{proj}$. That's the subject of the next lecture.

2D/3D Registration

3D-CT-Model to 2D-Fluoro-Image Registration

Problem Definition

Problem Solution