Let a circle passing through origin be
x2+y2+2gx+2fy=0...........(1)For two circles to be orthogonal, 2gg′+2ff′=c+c′
In x2+y2−6x+8=0,g′=−3,f′=0,c′=8(Comparing with x2+y2+2gx+2fy+c=0)
In x2+y2−2x−2y−7=0,g′=−1,f′=−1,c′=−7
So if circle (1) is orthogonal with x2+y2−6x+8=0, then
2.g(−3)=8
or g=−43
If circle (1) is orthogonal with x2+y2−2x−2y−7=0, then
2.g(−1)+2f(−1)=−7
or,−2(−43)+(−2f)=−7
or,−2f=−7−83
or,−2f=−293
or, f=296
So, the circle is x2+y2−83x+293y=0
or, 3x2+3y2−8x+29y=0