East 128 between 5th and Park is a pretty attractive street of brownstones, most of which are kept nicely. East of Park Ave, 128 is dominated by mid-century institutional-type buildings, including a NYCHA project, Jackie Robinson Houses.
5th and Madison are both mostly residential around those blocks. The only businesses are occasional small bodegas, laundromats, and the like. So you have the trade-off of the area being quiet on one hand... and desolate on the other. That would be the safety issue, that there's little street life at night in that area. You'd be walking down to 125th or over to Lenox to do your errands.
The area around 125th and Lexington is often crowded, loud, dirty, and full of sketchy drug addicts hanging out. It's a big transfer point between buses and trains. On the other hand, there's a steady stream of commuters transferring between the 4/5/6 and the Metro-North, and the area's busy enough at all hours to feel safe to me.