The correct option is B AB and BA are square matrices.
If both AB and BA exist, then the number of columns of A is equal to the number of rows of B. Therefore,
n+5=m ....... (1)
And the number of columns of B is equal to the number of rows of A. Therefore,
11−n=m ...... (2)
Solving (1) and (2), we get n=3 and m=8. Hence, A has order 8×8 and B has order 8×8. Hence, both AB are BA are square matrices.