The 20% figure is very relevant though. Per Wikipedia, an estimated 20% of distinctly Neanderthal genes are still extant.

The process here is not that different from mtDNA. When a couple produces a child, each gene from either parent has a roughly 50% chance to be passed to that child. (Ignoring mutations, where the gene might eventually become unrecognizable. Also its chance of being passed on further depends on whether the gene is adaptive or not, but let’s assume it’s neutral.) MtDNA is different, in that it is always passed from the mother, but the child has a roughly 50% chance to be female and thus to be capable of passing on those genes. Chromosomal genes can swap between chromosomes, but this mostly doesn’t affect the 50% probability of each gene being passed on. So the statistics for mtDNA (as a whole since it doesn’t recombine) and for other genes should be roughly similar, perhaps with different rate constants (eg due to men having a wider variation in how many children they have).

In any case, if only 1/5 of distinctly Neanderthal genes have survived this process, it’s not too surprising that their mtDNA didn’t make it (as far as we know).