If money is no object, I'd take the Metro-North Hudson Line to Grand Central, and then take the (6) down to 23rd Street. Google Transit says that it should take about 45 minutes door-to-door, which I think is pretty reasonable.
It also suggests taking the (1) to Times Square for the (N) or (R), and that takes about 50 minutes, but seeing that the (1) makes all local stops, I don't see how it only adds 5 minutes to the trip. It probably adds around 15 minutes, which would bring the travel time to about an hour.
The thing is that Metro-North is faster, but it's less frequent, so if you don't want to be tied down to the schedule, take the subway (or if it's a weekend, I guess you could drive, but parking is still going to be pretty hard to find, unless you use a garage, but that could be expensive. A lot of them charge $20 an hour for parking)