Besides the obvious class differences, the largest difference is train access. Rye Brook does not have a station, and the nearest one, Port Chester, doesn't have express trains.
Fewer people in Rye Brook commute to the city daily. In Rye, you are unlikely to be able to afford to live there in the 1st place if you don't.
I would be very comfortable choosing Rye Brook if my husband didn't commute long hours in the city. Houses are far cheaper than in Rye, and you don't get your town invaded by outsiders every June-September.