I did the same exact thing to the top half of a shower wall. The adhesive and the drywall came in very, very similar plastic buckets. "I don't remember the tile adhesive being this lightly colored..........oh for F's sake."
Take them down, rinse them off and do it right.