A circular current loop of radius R is placed inside square loop of...

Using the reciprocity theorem, assume a current $I$ flows through the larger square loop and compute the flux through the smaller circular loop. The mutual inductance is given by $M = \dfrac{\Phi}{I}$.

Magnetic field at the centre of the square loop:
For a square of side $L$, the perpendicular distance from the centre to any side is $d = \dfrac{L}{2}$, and each side subtends angles $\theta_1 = \theta_2 = 45^\circ$ at the centre.

The field due to one finite straight segment is:
$B_{\text{segment}} = \dfrac{\mu_0 I}{4\pi d}(\sin\theta_1 + \sin\theta_2)$

$B_{\text{segment}} = \dfrac{\mu_0 I}{4\pi \cdot \frac{L}{2}}\left(\dfrac{1}{\sqrt{2}} + \dfrac{1}{\sqrt{2}}\right) = \dfrac{\mu_0 I}{2\pi L} \cdot \sqrt{2} = \dfrac{\sqrt{2}\,\mu_0 I}{2\pi L}$

All four sides contribute equally in the same direction at the centre, so:
$B_{\text{total}} = 4 \cdot \dfrac{\sqrt{2}\,\mu_0 I}{2\pi L} = \dfrac{2\sqrt{2}\,\mu_0 I}{\pi L}$

Flux through the circular loop:
Since $L \gg R$, the field $B_{\text{total}}$ can be treated as uniform over the area $A = \pi R^2$ of the circular loop.

$\Phi = B_{\text{total}} \cdot A = \dfrac{2\sqrt{2}\,\mu_0 I}{\pi L} \cdot \pi R^2 = \dfrac{2\sqrt{2}\,\mu_0 R^2 I}{L}$

Mutual inductance:
$M = \dfrac{\Phi}{I} = \dfrac{2\sqrt{2}\,\mu_0 R^2}{L}$

Hence, the correct option is $(4)\ \dfrac{2\sqrt{2}\,\mu_0 R^2}{L}$.