Dear Student
NaBr give bromide ion which is weak base so displace of OH- is not possible. When H2SO4 is added it protonate the OH group so not displacement is the water molecule it can be displaced by bromide as water is less basic then bromide.
Regards