There is a easy way to test for this. Set a magnet on your desk. Now, rotate it 180 degrees, and set it down again.

Does the magnet suddenly snap back to its original position, or otherwise slide over your desk in search of magnetic north? You are probably (correctly) thinking this is a ridiculous experiment. It's obvious that earth's magnetic field is not strong enough to do stuff like that.

So, if the earth's magnetic field is too weak to even move a unsecured magnet that probably only weighs a few grams, you're expecting it to affect a billion tons of rock in the form of continental plates?

You can even see a complete absence of effect ib the exact same rocks you are referencing. These rocks provide evidence of past pole reversals, because different layers have different alignments.

If the polarized rock moved into alignment when the fields shifted, new layers would always be deposited with the same alignment as the old ones. The only reason these rocks are even referenced in the same breath as pole realignments is because this doesn't happen, and cannot happen.

Finally, to see how magnetic the average rock is, obtain a compass, a magnet, and a piece of gravel. Gravel is made exclusively from the least valuable (e.g. Most common) form of rock readily available, so it is a good stand in for the average composition of the earth's crust.

Find the furthest distance you can that leads the compass to point towards the magnet, rather than magnetic north. If the compass is pointing at the magnet, this means it's feild is dominating over the earth's field at that distance. Replace the compass with gravel. Does the gravel slide towards the magnet at this distance? If not, this shows that the magnetic minerals in the rock are insufficient to move it, despite being in the presence of a local magnetic field strong enough to dominate over the earth's natural field (as shown by the compass)

We routinely experience magnetic fields thousands of times stronger than the earth's own, and nothing tends to happen. The same is true of most rocks.