The two ways of relating what a person has said are 1)direct and 2)indirect (reported).
In the direct speech, the exact words of the original speaker are repeated.
In the indirect speech (reported speech), the exact meaning of what the speaker told is expressed but not necessarily in the speaker’s exact words.
Option B is correct as the appropriate introductory verb 'asked' is used with the past continuous tense.
Options A, C, and D are wrong as 'that what' cannot be used, 'I' cannot come after the helping verb, and 'here' cannot be used int the reported speech respectively.
The correct answer is BMy brother asked me what I was doing there.