This is the answer. I had a wood stove at my old house and if the stove piping and flue weren’t already warmed up, smoke would come back into the house rather than up the chimney. This usually happened this time of year when the stove wasn’t cranking 24/7.

Open the closest window or 2 to get additional air flow up the chimney will solve the issue.