The correct option is
A 3^j−^k√10Let unit vector be ^u=a^i+b^j+c^k|→u|.
Let →v=5^i+2^j+6^k , →x=^i−^j+^k and ^y=2^i+^j+^k.
Now as ^u and →v are orthogonal ∴^u.→v=0
(a^i+b^j+c^k)|→u|.(5^i+2^j+6^k)=0
5a+2b+6c=0→(1)
^u is coplanar with →x and →y means there scalar triple product will be zero.
→^u.(→x×→y)=0
=(a^i+b^j+c^k)→u.(→x×→y)=0
=(a^i+b^j+c^k).(→x×→y)=0
∣∣
∣∣abc1−11211∣∣
∣∣=0
=a(−1×1−1×1)−b(1×1−2×1)+c(1×1−2×(−1))=0
−2a+b+3c=0→(2)
Subtract equation (1)−2×(2)
7a=0
Put a=0 in (2), we get b+3c=0→b=−3c
∴^u=0^i−3c^j+c^k√02+(−3c)2+c2=−3^j+^k√10
Now, there is no option like this, so may be the the vector has opposite direction then what we have find out. So we reversed direction by just multiplying with −1. Because both are correct just directions are opposite.
^u=3^j−^k√10
Hence, (A)