The mass density of a planet of radius R varies with the distance r...

$dm = ρ \times 4{\pi x}^{2}dx = {ρ}_{0}\left(1 - \dfrac{{x}^{2}}{{r}^{2}}\right) \times 4{\pi x}^{2}dx$

$m = 4\pi ρ{∫}_{0}^{r}\left({x}^{2} - \dfrac{{x}^{4}}{{R}^{2}}\right)dxs$

$m = 4{\pi ρ}_{0}\left|\dfrac{{r}^{3}}{3} - \dfrac{{r}^{5}}{5{R}^{2}}\right|$

$E = \dfrac{Gm}{{r}^{2}} = \dfrac{G}{{r}^{2}} \times 4{\pi ρ}_{0}\left(\dfrac{{r}^{3}}{3} - \dfrac{{r}^{5}}{5{R}^{2}}\right)$

$E = 4{\pi Gρ}_{0}\left(\dfrac{r}{3} - \dfrac{{r}^{3}}{5{R}^{2}}\right)$

$E$ is maximum when $\dfrac{dE}{dt} = 0 ⇒ \dfrac{dE}{dr} = 4{\pi Gρ}_{0}\left(\dfrac{1}{3} - \dfrac{3{r}^{2}}{5{R}^{2}}\right) = 0$

$⇒ r = \dfrac{\sqrt{5}}{3}R$

{E}_\max = 4{\pi Gρ}_{0} \times \dfrac{\sqrt{5R}}{3}\left[\dfrac{1}{3} - \dfrac{1}{5} \times \dfrac{5}{9}\right]

{E}_\max = \dfrac{8\sqrt{5}}{27}{\pi Gρ}_{0}R