A few things; 1, there's seems to be two inter breeding "events" 140k years ago, and 40k years ago, and the mitochondria DNA of Neanderthal prior to the 140k time period appears very similar to Denisovans, while after looked more like Sapiens, suggesting a replacement of Neanderthal mothers with Sapien ones, likely do to our more rounded shaped skull and adapted birth canal (also we have a few fossilized examples of Neanderthal mothers with near full term Sapien "hybrids." 2, there's a growing (slowly) position that Sapiens, Neanderthal, Denisovans, etc should actually be considered subspecies of Homo Erectus, as we appear to be able to fully, and successfully breed with one another, meaning a speciation event never happened (although I would argue by now that has likely occurred.)

Also there's the misconceptions of the genetic bottle neck, that supposedly cause only 50 to 5000 pairs (depending of the source) of humans to be alive at one point, and that's completely false. If a father has only daughters (or only his daughters have offsprings) then his Y legacy ends, and in modern genetic testing he never existed. Same with a mother, if she only has sons (or only her sons successfully procreate) then her mitochondria DNA ends, and we will have no evidence of her genetic existence.

Just remember, we, as a collective, have at least 6 other hominids DNA within our genome (2 of which we only know from genetic testing).