The correct option is A π/3
Let cos−1x=y. Then x=cosy, so that 1/2≤x≤1, or 0≤y≤x/3, and
x2+12√3−3x2=12cosy+√32siny
=cosπ3cosy+sinπ3siny=cos(π3−y)
⇒cos−1(x2+12√3−3x2)=π3−y
because cos−1(cosx)=x if 0≤x≤π. Therefore, the given expression is equal to y+π/3−y=π/3