Tr=8!a! b! c!(1)a(x)b(x2)c, where a+b+c=8
For coefficient of x5, b+2c=5
(b,c)=(1,2),(3,1),(5,0)
∴a5=8!5!⋅1!⋅2!+8!4!⋅3!⋅1!+8!3!⋅5!⋅0!
⇒a5=168+280+56=504
Alternate Solution :
[(1+x)+x2]8=8C0(1+x)8+8C1(x2)1(1+x)7+8C2(x2)2(1+x)6+8C3(x2)3(1+x)5+⋯
First three terms consist x5.
So, a5=8C0⋅8C5+8C1⋅7C3+8C2⋅6C1
⇒a5=56+280+168=504