Surgical tools can be tracked using a set of attached reflective spheres. The 3D positions of these spheres are detected by a tracker with a stereo infrared camera.
Each tool has a pre-defined arrangement of spheres and associated coordinate system.
After finding a set of spheres in the world, the tracker can find a matching between world spheres and the pre-defined spheres of the tool (based on different distances between pairs of spheres), then apply the Procrustes Method to find a transform from the tool coordinate system to the world coordinate system.
With infrared optical trackers, spheres will show up as very bright spots on the image.
Sphere Location from Two Images
If we can find the coordinates of a particular sphere as $x_1$ in camera 1's image, and as $x_2$ in camera 2's image, we have two lines going toward the sphere: One line goes from the centre of camera 1 through $x_1$; the other line goes from the centre of camera 2 through $x_2$.
We can then apply the "Closest approach of two lines" method (from these notes) to find the sphere position in 3D.
But the real problem is to establish a correspondence between spheres seen in two different camera images.
Finding Correspondences
Given a sphere location in one camera image, we need to find the location of the same sphere in the other camera image.
From the last lecture, we saw that we can determine the fundamental matrix, $F$, from two calibrated cameras.
Then all points, $x$, in camera 2's image that correspond to a point, $x_1$, in camera 1's image satisfy the equation $$x^T\ F\ x_1 = 0$$
(We have replaced $x_2$ with $x$ to make things cleaner below.)
Recall that $x$ is a point on the 2D image plane. But $x$ is a homogeneous point, so is represented by a 3D vector.
Then $$\begin{array}{rcll} x^T\ F\ x_1 &=& 0 \\ x^T\ f &=& 0 & \textrm{for } f = F\ x_1\\ x \cdot f &=& 0 \\ \end{array}$$
Since these are 3D vectors, we can write this with the components $(x,y,z)$: $$x_x\ f_x + x_y\ f_y + x_z\ f_z = 0$$
To check all points where the sphere might appear in camera 2's image, iterate through the values of $x_x$ from left to right and, for each value, compute the corresponding $x_y$. (We can set $x_z = 1$ as this is a homogeneous point.)
Pick the brightest spot among all the $(x_x,x_y)$ position and we're done.