By the Binomial Theorem, we see that
1,n−1,(n−3)(n−2)⌊2,(n−5)(n−4)(n−3)⌊3,...
are the coefficients of xn,xn−2,xn−4,xx−6,.... in the expansions of (1−x)−1,(1−x)−2,(1−x)−3,(1−x)−4,.... respectively. Hence the sum required is equal to the coefficient of xn in the expansion of the series
11−bx−ax2(1−bx)2+a2x4(1−bx)3−a3x6(1−bx)4+...,
and although the given expression consists only of a finite number of terms, this series may be considered to extend to infinity.
But the sum of the series =11−bx÷(1+ax21−bx)=11−bx+ax2
=11−(a+1)x+ax2, since b=a+1.
Hence the given series =coefficient of xn in 1(1−x)(1−ax)
=coefficient of xn in 1a−1(a1−ax−11−x)
=an+1−1a−1.