Honestly it's the opposite for me. Nobody else does my job so when I'm out of office work piles up, emails come in, stressful crap happens and I end up having to deal with it anyway or when I get back and I'm buried.

However, if I take my vacation days during the holiday season (like right now when globally my industry more or less shuts down) then I'm more likely to get uninterrupted restful time off and not come back to the office walking into a dumpster fire.