The correct option is
A x2−y2−5x−9y−26=0Let
C1:x2+y2−6x−8y−7=0∴(x2−6x)+(y2−8y)−7=0
∴(x2−6x+9)−9+(y2−8y+16)−16−7=0
∴(x−3)2+(y−4)2=32
∴(x−3)2+(y−4)2=(√32)2
Comparing this equation with the standard equation of circle i.e. (x−h)2+(y−k)2=r2, we get,
h=3 or k=4 and r1=√32
Thus, coordinates of center of circle C1 are A(3,4) and radius is √32
Let C2:x2+y2−4x−10y−3=0
∴(x2−4x)+(y2−10y)−3=0
∴(x2−4x+4)−4+(y2−10y+25)−25−3=0
∴(x−2)2+(y−5)2=32
∴(x−2)2+(y−5)2=(√32)2
Comparing this equation with standard form of equation of circle i.e. (x−h)2+(y−k)2=r2, we get,
h=2, k=5 and r2=√32
Thus, coordinates of center of C2 are B(2,5) and radius is √32
Now, this line AB is diameter of another circle C3. Mid-point of this line will be center of circle C3
Let coordinates of center of circle C3 are C(h,k)
Thus, by mid-point formula, we can write,
h=3+22
∴h=52
Similarly, k=4+52
∴k=92
Thus, coordinates of center are C(52,92)
Now, as shown in figure, CA and CB are radii of circle C3
Thus, by distance formula,
CA=√(3−52)2+(4−92)2
∴CA=√(12)2+(−12)2
∴CA=√(14)+(14)
∴CA=√12
∴CA=1√2
Thus, equation of circle C3 is given by,
(x−52)2+(y−92)2=(1√2)2
∴x2−5x+254+y2−9y+814=12
∴x2+y2−5x−9y+1064=12
∴x2+y2−5x−9y=12−1064
∴x2+y2−5x−9y=−1044
∴x2+y2−5x−9y=−26
∴x2+y2−5x−9y+26=0
Thus, answer is option (B)