The correct option is D ⎡⎢⎣855030558⎤⎥⎦
The characteristic equation of the matrix A is
⎡⎢⎣(2−λ)110(1−λ)011(2−λ)⎤⎥⎦=0
λ3−5λ2+7λ−3=0
According to cayley -Hamilton theorem, we have
A3−5A2+7A−3I=0
A5(A3−5A2+7A−3I)+A(A3−5A2+7A−3I)+A2+A+I
=A2+A+I
A2=⎡⎢⎣211010112⎤⎥⎦⎡⎢⎣211010112⎤⎥⎦=⎡⎢⎣544010445⎤⎥⎦
A2+A+I=⎡⎢⎣544010445⎤⎥⎦+⎡⎢⎣211010112⎤⎥⎦+⎡⎢⎣100010001⎤⎥⎦=⎡⎢⎣855030558⎤⎥⎦