First we have to remove the small brackets, or parentheses, ( ), then the curly brackets, { }, and then the square brackets, [ ].
Therefore, we have
20 - [5xy + 3{x2 - (xy - y) - (x - y)}]
= 20 - [5xy + 3{x2 - xy + y - x + y}]
= 20 - [5xy + 3{x2 - xy + 2y - x}]
= 20 - [5xy + 3x2 - 3xy + 6y - 3x]
= 20 - [2xy + 3x2 + 6y - 3x]
= 20 - 2xy - 3x2 - 6y + 3x
= - 3x2 - 2xy - 6y + 3x + 20