The correct option is A ifa=b or θ=nπ, n is an integer
Given
A=[cosθ−sinθsinθcosθ]
and B=[a00b]
⇒AB=[cosθ−sinθsinθcosθ][a00b]
=[acosθ−bsinθasinθbcosθ]
BA=[a00b][cosθ−sinθsinθcosθ]
=[acosθ−asinθbsinθbcosθ]
Now AB=BA
⇒[acosθ−bsinθasinθbcosθ]=[acosθ−asinθbsinθbcosθ]
On comparison
asinθ=bsinθ
(a-b) sinθ=0
So either a=b or θ=nπ
(i.e. A and B commute when a=b (or) θ=nπ is an integer .