The correct options are
A sin(A−B)=−12√2sin2B
C sin(A−B)=13
D sin(A−B)=−13
√2sinA=sinB−sin3B ...(1)
√2cosA=cosB+cos3B ...(2)
Now,
eqn(1)×cosB−eqn(2)×sinB, gives
√2(sinAcosB−cosAsinB)=−sin3BcosB−cos3BsinB
⇒√2sin(A−B)=−cosBsinB
⇒sin(A−B)=−12√2sin2B ...(3)
Now,
2(cos2A+sin2A) =(cosB+cos3B)2+(sinB−sin3B)2
⇒2=1+2(cos4B−sin4B)+(cos6B+sin6B)
⇒1=2(cos2B−sin2B)+(cos4B+sin4B−cos2Bsin2B)
⇒1=2cos2B+(cos2B+sin2B)2−3cos2Bsin2B
⇒1=2cos2B+1−34sin22B
⇒0=8cos2B−3sin22B
⇒3cos22B+8cos2B−3=0
⇒(3cos2B−1)(cos2B+3)=0
⇒cos2B=13, cos2B≠−3
sin2B=±2√23
From eqn(3),
sin(A−B)=±13