That‘s true for most of Europe for most of the middle ages, yes. Something I‘ve not seen mentioned though, is that in the absence of oxen, farmers would also use cows as beasts of burden. Especially here in Switzerland where cows were rather common in the high- and late middle ages. This was mostly due to the the nature of the terrain; up on the mountain slopes you can‘t feasibly farm grain, but you can have pastures for cattle, simply called „alps“ in German. The main settlements would generally remain in the down valleys where farming was possible, while the wealthier farmers could join the „Alpgenossenschaften“ for a fee, so they could send their cattle to the communally owned mountain pastures. This way cheese became the dominant export good for many alpine regions, something Switzerland is still known for today. It‘s worth mentioning also that, certainly in Switzerland, there tended to be more „free“ farmers toward the end of the middle ages (we even had a couple civil wars where farmers fought for their rights in the early modern period). Additionally, in Switzerland farmers in service to a lord tended to have more freedoms later on, having a fairly complex societal structure in which the lord didn‘t just have complete authority over the villages. Rather the village’s community appointed representatives, which then had to be either accepted or denied by the lord‘s representative. other than that, the lord‘s influence was mostly limited to managing legal matters and taxes, though often a sort of veto would be retained by the lord‘s representatives if they didn‘t like what the village council came up with, but it seems like this was rarely used. In many regards they enjoyed a surprising degree of autonomy. So, it was a lot more cooperative than you‘d think in a feudal society. I don‘t know how common this was in the rest of Europe though. Swiss farmers were rather infamous for rising up against the „natural order of things“ (the estates), especially after the Habsburg Wars and especially by Swabian lords before, during and after the Swabian War (called Swiss War in Germany). But that‘s another matter entirely.

As a quick aside: Hugary is also interesting to mention, as Hungarian cows and oxen were exported en masse into central Europe during the middle ages.

But yes, horses were really rather rare and expensive during much of the European middle ages. not just to buy but also to maintain, as they ate more than cows of the period (there is a pletitude of sources for this here in Switzerland, eg. Alpgenossenschaftsbücher). I read that English kings would often dwell in the houses of wealthy citizens when they were travelling, which put a large logistical burden on the host, since he had to feed all those horses - a king never travels alone. On a smaller scale, I‘ve seen the same sort of principle in a source about a prestigious abbess from western Germany, when she was visiting her domain. Apparently it put quite the strain on the hosts, not because they had to feed the abbess and her entourage, but because they suddenly had to feed dozens of horses. That leads me to believe that, even if they could afford one, peasants would likely not buy a horse since they cost a lot of upkeep. Oxen were the obvious choice.