That's true if there is 100 fish - but not if there is 1 000 000 fish.

If there's a huge number of fish, the chance that a given gene is removed randomly is very low. Unless it provides a disadvantage, it's entirely random. There have been experiments, both in digital environments and in sealed live environments, tracking genetic drift (the change in gene representation in a population over time); and pretty consistently there are cases of genetic variations that end up spreading by chance that don't do anything.

The classic example of real-life variations are two different genes that code for the same amino acid chain using different base pairs - there's no advantage one way or another. In such a case, the most common result over time is that both versions of the gene persist; even if you start with one variation is less common than the other.

The exception is if you simulate bottlenecks - like the "100 fish" scenario you posted. With such small numbers getting through, it becomes a lot more likely for some genes to be lost forever by random chance - including useful genes that happen to get unlucky.