$\begin{bmatrix} {1 \over \sqrt{2}} & 0 & {-1 \over \sqrt{2}} \\ 0 & 1 & 0 \\ {1 \over \sqrt{2}} & 0 & {1 \over \sqrt{2}} \end{bmatrix}$have on the vector $(1,2,3)$? solution
Rotates the vector by $-45^o$ about the $(0,1,0)$ axis.
In 2D, the rotation by $45^o$ is done with the matrix $$\displaystyle \left[\begin{array}{cc} \cos 45 & -\sin 45 \\ \sin 45 & \cos 45 \end{array}\right] = \left[\begin{array}{cc} {1 \over \sqrt{2}} & -{1 \over \sqrt{2}} \\ {1 \over \sqrt{2}} & {1 \over \sqrt{2}} \end{array}\right]$$
Implicitly, this rotation is being done in a 2D $x y$ plane that sits in a 3D space, with the $z$ axis pointing perpendicularly out of the $x y$ plane.
So this is a rotation by $45^o$ around the $z$ axis. If we were using 3D coordinates, it would be written as $$\left[\begin{array}{ccc} {1 \over \sqrt{2}} & -{1 \over \sqrt{2}} & 0\\ {1 \over \sqrt{2}} & {1 \over \sqrt{2}} & 0 \\ 0 & 0 & 1 \end{array}\right]$$
since that matrix does not change the $z$ coordinate.
To do the same rotation in the $x z$ plane, instead, we need to rotate around the $y$ axis. Swapping the roles of $y$ and $z$ in the matrix above yields $$\left[\begin{array}{ccc} {1 \over \sqrt{2}} & 0 & -{1 \over \sqrt{2}} \\ 0 & 1 & 0 \\ {1 \over \sqrt{2}} & 0 & {1 \over \sqrt{2}} \end{array}\right]$$
All vectors are mutually perpendicular:
$(0,1,0) \cdot ({1 \over \sqrt{2}},0,{-1 \over \sqrt{2}}) = 0$
$(0,1,0) \cdot ({1 \over \sqrt{2}},0,{1 \over \sqrt{2}}) = 0$
$({1 \over \sqrt{2}},0,{-1 \over \sqrt{2}}) \cdot ({1 \over \sqrt{2}},0,{1 \over \sqrt{2}}) = 0$
All vectors are unit length:
$(0,1,0) \cdot (0,1,0) = 1$
$({1 \over \sqrt{2}},0,{1 \over \sqrt{2}}) \cdot ({1 \over \sqrt{2}},0,{1 \over \sqrt{2}}) = 1$
$({1 \over \sqrt{2}},0,{-1 \over \sqrt{2}}) \cdot ({1 \over \sqrt{2}},0,{-1 \over \sqrt{2}}) = 1$
First, find the parallel vector that is the projection of $(1,0,2)$ to $(0,2,1)$:
$u = \left( { (1,0,2) \; \cdot \; (0,2,1) \over (0,2,1) \; \cdot \; (0,2,1) } \right) \; (0,2,1) = ( 0, { 4 \over 5}, {2 \over 5} )$.
Second, get the perpendicular vector by subtracting that parallel vector from $(1,0,2)$:
$v = (1,0,2) - ( 0, { 4 \over 5}, {2 \over 5} ) = (1, { -4 \over 5}, {8 \over 5})$.
Then $(1,0,2) = u + v = ( 0, { 4 \over 5}, {2 \over 5} ) + (1, { -4 \over 5}, {8 \over 5})$.
Note that u and v are perpendicular: $u \cdot v = ( 0, { 4 \over 5}, {2 \over 5} ) \; \cdot \; (1, { -4 \over 5}, {8 \over 5}) = 0$.
First, find vectors $u$ and $v$ along the triangle edges:
$u = (1,1,1) - (0,1,1) = (1,0,0)$
$v = (-2,2,-1) - (0,1,1) = (-2,1,-2)$
Second, take their cross product:
$u \times v = (1,0,0) \times (-2,1,-2) = \left[ \begin{array}{c} (0 {\small \times} -2) - (0 {\small \times} 1) \\ (0 {\small \times} -2) - (1 {\small \times} -2) \\ (1 {\small \times} 1) - (0 {\small \times} -2) \\ \end{array} \right] = \begin{bmatrix} 0 \\ 2 \\ 1 \end{bmatrix}$
Third, normalize the cross product:
$n = {u \times v \over |u \times v|} = { (0,2,1) \over |(0,2,1)| } = (0,{2 \over \sqrt{5}}, {1 \over \sqrt{5}})$
There are two such normal vectors: $n$ and $-n$.