Dear stident,
We need to solve the problem using BODMAS rule.
In the brackets, the order of opening the brackets will be,
Simple bracket, square bracket and then curly bracket or the double bracket.
So the answer will be,
a-[b-{a-(b-1)+3a}]
a-[b-{a-b+1+3a}]
a-[b-{4a-b+1}]
a-[b-4a+b-1]
a-[2b-4a-1]
a-2b+4a+1
5a-2b+1