Unless he's trying to "forbid" you, then you can just keep doing what you're doing. If he does try, have a sit down and ask if he would feel more comfortable if you just avoided doing it when he's around.