y(3x−y)x(4x+y)=512⇒12y(3x−y)=5x(4x+y)⇒36xy−12y2=20x2+5xy⇒20x2−31xy+12y2=0⇒20x2−15xy−16xy+12y2=0⇒5x(4x−3y)−4y(4x−3y)=0⇒(5x−4y)(4x−3y)=0
5x−4y=0 ⇒x=4y5 ---(1)
or 4x−3y=0 ⇒x=3y4 ---(2)
Using(1)
x2+y2(x+y)2=(4y5)2+y2(4y5+y)2=16y225+y2(9y5)2=41y22581y225=4181