As others have mentioned your walls and plaster and wood lath. The wood lath is nailed to the wood studs and then the plaster base coat pushed onto the wood lath. The plaster that is pushed in between the lath strips are called keys. This is what holds the plaster onto the wall. The finish coats of plaster are then applied on top of that base coat. When you've been drilling through the walls you're basically breaking away the plaster keys. This is the crumbing sound you hear.

Your Stud finder is going to pickup all of the lath so you won't have much success with that tool.

If what you're hanging is not to heavy then I would recommend using toggle bolts or snap toggles. These don't expand into the plaster which would cause cracking. The toggles sandwich the wall and pull from the backside of the wall which is much more stronger.

When drilling through the plaster it would be best to use a masonry drill bit, this will give you a clean hole through the plaster. The masonry bit may not drill through the wood so you may need to switch to a standard wood drill bit.

When you buy your toggle bolts the box should tell you what size drill bit to use.

In the event you do find a stud then you can just drill into it with a regular wood screw. Make sure you account for the plaster thickness when you buy the screws.

If you're hanging a curtain rod near a window opening you may run into brick or other masonry behind the plaster. If this is the case you could use a plastic wall anchor made for masonry. Use a masonry drill bit to make the hole, then insert the anchor and then the screw. You could chose to add some construction adhesive to strengthen it.

Hope this helps.