Reaction mass is what some people have said, if you don't weigh much, when you connect, you're more likely to push yourself back off them than if you were heavier.

Another factor is the speed of your muscles. All things being equal, if you were lighter (and your fist/arm was too), you could make up for that with a faster punch.
Less mass in your arm, but more speed to make up for it. This is one of those Newtonian things involving collisions, if everything worked out without losses, if you wacked a 2kg ball into a 1kg ball and it transferred all the energy, the 1kg ball would shoot off at double the speed.
However, there's going to be a max speed based on your muscles, you wouldn't just be able to halve the weight of your arm and then expect it to move twice as fast. In other words, you can get faster with your punches, but there's a sort of preferred speed from your muscles which it'll tend to hang around.

eg- your muscles will move your arm in a particular speed range no matter what. Double the weight of your arm won't halve the speed, half-weight arms won't double the speed. You can add extra mass though, so if you gripped a lump of steel in your fist, you'd have a stronger punch, as there would be spare capacity in your muscles to account for the extra mass without taking a big hit in the speed to level it back out.

This part however is just one part of the whole punch, it's more like the mechanics of throwing a limp arm at someone, and doesn't account for the followthrough which would give you a stronger punch from having a heavy overall physique.