The correct options are
B 3√10^j−1√10^k
D −3√10^j+1√10^k
Let a unit vector in the plane of 2^i+^j+^k and ^i−^j+^k be ^a=α(2×^i+^j+^k)+β(^i−^j+^k)
=(2×α+β)^i+(α−β)^j+(α+β)^k
As ^a is unit vector,we have
=(2×α+β)2+(α−β)2+(α+β)2=1
⇒6×α2+4×αβ+3×β2=1 on simplification
As ^a is orthogonal to 5^i+2^j+6^k,
we get 5×(2×α+β)+2×(α−β)+6×(α+β)=0
⇒18×α+9×β=0
⇒β=−2×α
From (i), we get 6×α2−8×α2+12×α2=1
⇒α=±1√10
⇒β=±2√10
Thus,^a=±(3√10^j−1√10^k)