errightarrow{AD}=-\hat{i}+2\hat{j}+2\hat{k} $$

First find the angle between $\overrightarrow{AB}$ and $\overrightarrow{AD}$.

Using dot product,

$$ \overrightarrow{AB}\cdot \overrightarrow{AD} =2(-1)+10(2)+11(2) $$

$$ =-2+20+22=40 $$

Now,

$$ |\overrightarrow{AB}|=\sqrt{2^2+10^2+11^2} =\sqrt{4+100+121} =\sqrt{225}=15 $$

and

$$ |\overrightarrow{AD}|=\sqrt{(-1)^2+2^2+2^2} =\sqrt{1+4+4} =\sqrt{9}=3 $$

If the angle between $\overrightarrow{AB}$ and $\overrightarrow{AD}$ is $\theta$, then

$$ \cos\theta=\frac{\overrightarrow{AB}\cdot \overrightarrow{AD}}{|\overrightarrow{AB}||\overrightarrow{AD}|} $$

$$ \cos\theta=\frac{40}{15\times 3} =\frac{40}{45} =\frac{8}{9} $$

So the original angle between $AB$ and $AD$ is $\theta$.

After rotation, $AD'$ becomes perpendicular to $AB$, so the angle between $AB$ and $AD'$ is $90^\circ$.

Since $AD$ is rotated by an acute angle $\alpha$ in the same plane,

$$ \alpha=90^\circ-\theta $$

Hence,

$$ \cos\alpha=\cos(90^\circ-\theta)=\sin\theta $$

Now,

$$ \sin\theta=\sqrt{1-\cos^2\theta} $$

$$ \sin\theta=\sqrt{1-\left(\frac{8}{9}\right)^2} $$

$$ =\sqrt{1-\frac{64}{81}} =\sqrt{\frac{17}{81}} =\frac{\sqrt{17}}{9} $$

So,

$$ \cos\alpha=\frac{\sqrt{17}}{9} $$

Correct option: Option B.

Vector Algebra

Continue this paper