The correct option is D 5π6
|→a|=1, |→b|=4, →a⋅→b=2
→c=(2→a×→b)−3→b
⇒→c+3→b=2→a×→b
∵→a⋅→b=2
|→a||→b|cosθ=2
⇒cosθ=2|→a||→b|=24=12
⇒θ=π3
|→c+3→b|2=|2→a×→b|2
⇒|→c|2+9|→b|2+2→c⋅3→b=4|→a|2|→b|2sin2θ
⇒|→c|2+9|→b|2+6→b⋅→c=48
⇒|→c|2+96+6(→b⋅→c)=0 …[1]
⇒→c=2→a×→b−3→b
⇒→b⋅→c=0−3×16=−48
Putting value of →b⋅→c in [1], we have
|c|2+96−6×48=0⇒|→c|=8√3
Putting this value in [1], we get
192+96+6|→b||→c|cosα=0
⇒cosα=−√32
or, α=5π6