Consider the given function:
∫cosxcos3xdx
=∫cosx4cos3x−3cosxdx
=∫14cos2x−3dx
=∫14cos2x−3(sin2x+cos2x)dx
=∫13.cos2−sin2xdx
=∫1sin2x(3−cot2x)dx
=∫1sin2x3−cot2xdx
=∫cosec2x3−cot2xdx
putcotx=t
−cosec2xdx=dt
=∫dt3−t2
=12.1√3logx√3+t√3−t+c
=12√3log√3+cotx√3−cotx+c
Hence this is the answer.