In regards to Silicon Valley/San Francisco specifically it's due to the long commute times. 90 min commutes mean wasting 3 hours per day in traffic. This makes an 8 hour workday into a 11'er, 12 with a lunch break. Most people won't tolerate that unless the money is really good, which it increasingly isn't. This is caused by a supply problem, there's not enough transit and physical mobility in the region, and this problem will persist through the end of the decade.

We can talk about Office-or-not all day but the plain fact is, if the Office is completely divorced from your community and not involved in your personal life in any way then you have no commitment to it. Workers are isolated from their physical workspaces and even the products they create, to the point where they'd rather quit than subject themselves to such a dehumanizing, alien experience. Which follows as it's only human nature to want some amount of control over our daily routines.

To have a serious discussion about Office-ing, the Office and areas around it have to be desirable. If they aren't, it's over. The same problem vexes industrial workers who are just told to eat it, and largely do because they didn't finish college. These are also the people most likely to replace former Office jobs, presuming companies can't find a machine to do it better.