When a ray passes from one medium to another, Snell's Law describes how it changes direction:

$\ni\ \sin\ti = \no\ \sin\to$.

$\ni$ and $\no$ are the indices of refraction of the two media and $\ti$ and $\to$ are the angles of incidence and of refraction, respectively.

The same applies in 3D, but in a plane that embeds the ray and the normal:

The plane needs a coordinate system. One of the coordinate axes can be $N$. If $\Ri$ is the direction of the incoming vector, then $\Ri \times N$ is perpendicular to the plane and

$M = {\large N \times (\Ri \times N) \over \large | N \times (\Ri \times N) |}$

is another coordinate axis. Note that this does not work if $\Ri$ is parallel (or almost parallel) to $N$; in this case the ray does not change direction across the interface.

Now $\ti$ can be computed using the dot product of $N$ and $\Ri$ (as long as $N$ and $\Ri$ have a positive dot product; reverse one of them if not). Or $\ti$ can be computed from the two-argument arctan (atan2() in C++) by calculating the projection of $\Ri$ onto $N$ and $M$.

Then $\to$ can be computed from $\ti$, $\ni$, and $\no$.

Then the reflection direction is

$\Ro = \cos\to\ \widehat{N} + \sin\to\ \widehat{M}$

where $\widehat{N} = \pm N$ and $\widehat{M} = \pm M$, with the signs depending upon the direction of $\Ri$ with respect to $N$ and $M$ (left as an exercise).

Total internal reflection occurs when a ray does not pass through the interface but, instead, gets totally reflected back into the medium. This occurs when the ray goes from a dense medium to a less-dense medium and

${\Large \ni \over \Large \no} \sin\ti > 1$.

Above, ${\ni \over \no} > 1$ so, when $\ti$ gets sufficiently close to 1, total internal reflection occurs. For the air/glass interface below (with $\ni = 1.510$ and $\no = 1.008$, instead), this occurs when

$\ti > \arcsin {\no \over \ni} = \arcsin 0.668 = 42^o$