Install a two channel relay wherever the feed to the light/fan splits to both devices. Look into the UL listed Shelly brand relays that can be wired in device junction boxes. You would wire the physical switch to the switch input on the channel you want to turn on/off (eg fan). The other switch input on the relay can be controlled via the shelly app and uses Wi-Fi (no hub required). Might confuse the next home owner but does what you need.

If you need help understanding the wiring, I can send you a little diagram.