The correct option is C →r=→a2−3→b4+54(→a×→b)
Any vector →r in space can be written as
→r=λ→a+μ→b+k(→a×→b) ⋯(1)
Taking dot product with (→a×→b), we get
→r⋅(→a×→b)=λ→a⋅(→a×→b)+μ→b⋅(→a×→b)+k(→a×→b)⋅(→a×→b)
⇒54=0+0+k(1)
⇒k=54
Taking dot product of (1) with →a, we get
→r⋅→a=λ
Similarly, →r⋅→b=μ
Given, →r⋅(3→a+2→b)=0
⇒3λ+2μ=0
Now, −4μ3∫−2λx+1x2+1dx=π2
⇒2λ∫−2λx+1x2+1dx=π2
⇒2λ∫01x2+1dx=π4
⇒tan−1(2λ)=π4
⇒λ=12 and μ=−34
∴→r=→a2−→3b4+5(→a×→b)4
and |→r|=√14+916+2516=√198
⇒|→r|2=198