In Rutherford's scattering experiment, which specific observation...

In Rutherford's $\alpha$-particle scattering experiment, the observation that a very small fraction of the $\alpha$-particles (about $1$ in $20,000$) bounced back or were deflected by nearly $180^{\circ}$ indicated that they experienced a massive repulsive force.

This strong repulsion could only occur if the entire positive charge and almost all the mass of the atom were concentrated in a very small, dense region at the center, which Rutherford called the nucleus.

The observation that most particles passed straight through indicated that most of the atom is empty space, while small deflections indicated the presence of a positive center. However, the nearly $180^{\circ}$ deflection specifically proved the extreme density and small size of this central mass and charge.

Answer: A very small fraction of the $\alpha$-particles bounced back, experiencing a deflection of nearly $180^{\circ}$.