For newer developments there's usually a plat map that gives dimensions along the edges of the lot. I think they store those at city hall somewhere. So one stake being out of place or lost wouldn't be a huge problem. And for larger plots all the states not on the east coast follow a fairly standard system. So if your land boundary is on the range line or town line or some quarter section line then that's a known thing and stake position isn't going to matter as much.

With that said, what really matters is whether it's the kind of thing that's worth going to court over or bringing in a surveyor. But even if it isn't right now it could still be a problem in the future. Stuff does get errantly built outside the bounds of peoples' property and that's a legal mess when somebody discovers it.