The correct option is A a(2a+b)(5a−8b)
In the given expression, common term has to be separated:
Thus, 10a3−11a2b−8ab2 becomes a(10a2−11ab−8b2)
By middle term splitting we can write the above expression as:
a(10a2−16ab+5ab−8b2)
which further becomes a(2a(5a−8b)+b(5a−8b))
On Further simplification it becomes; a(2a+b)(5a−8b)
So, the correct answer is option (a).