There would need to be a critical mass of highly magnetic material in the crust, and it would have to be highly polarized and the flip would have to happen extremely fast to have any impact on the crust at scale.

Most highly magnetic material is in the core (iron) while the crust is mostly silicates which don't care about the field all that much.

The stuff that is magnetic wouldn't be polarized enough to prefer one pole to the other. There would be equal "north" and "south " pull.

Lastly the flip is thought to take around 2-12k years, with the magnetic field flickering and changing over that time