> They essentially learn probabilities of different word combinations.

These isn't dichotomous with having a set of rules. The rules operate at a deeper (less interpretable level -- some may say "subsymbolic") compared to GOFAI. The whole setup of model+gradient descent correspond to having some update rules (based on partial differentations and such). In practice they aren't fully continuous either (though in theory they are) because of floating point approximations and underlying digitization.