HA You'd think the city of Albany was the South Bronx circa 1978 from some of these posts. Trust me the two ghettos in Albany (South End and Arbor Hill don't hold as candle to the worst ghettos in parts of NYC and especially North Philly, but I wouldn't go there at night.

I've been through there during the day plenty and not been accosted yet. Downtown Albany though is a world apart and you don't feel in danger there, and going toward the more west side of town there are many nice neighborhoods.
I live on the outside (suburbs) so I can't tell you about living in the city proper, but I can list off some nice things about the city of Albany....
Riverfront Park: nice green space and amphitheater along the Hudson connected to Downtown with a new pedestrian Bridge over I-787. Summer concert series held here.
Capital Park: lovely park with great Spring Flowers below the Capitol Building
Washington Park and the Lakehouse: park with landscaping from the era of Central Park in NYC ..think its older. Visit in the Spring for the Tulip Festival and Pinksterfest (old Dutch tradition born out of Pentecost Sunday originally).
Holiday lights tour through the park in December.
NY State Capitol and Empire Plaza. Huge public space in the plaza with several Towers...good views from the top of the Corning Tower. The Cultural Education Center at the southern end houses the state museum. The Egg is a performing arts center. The underground concourse holds many events such as auto shows, ski expos etc. The architecture of the complex is is very post modern ca. 1970 (think old World Trade Center), but its a nice facility overall.
Palace Theater: classic early 20th Century theater now gets many events and touring bands etc.
Times Union Center: down the hill from the Capitol a bit...a typical arena with the AHL Albany River Rats hockey club and larger concert tours etc. Indoor walkway connects the TU Center with the plaza above. The neighboring area along the northern part of South Pearl Street has a decent nightlife scene now. Stay above Madison Ave. on South Pearl because below that is the infamous South End.
Remaining vestiges of the old Dutch/Colonial Heritage....
Cherry Hill Mansion Historic Site
Ten Broek Mansion Historic Site
Schuyler Mansion
Urban Cultural Center exhibit at Quackenbush Square
Albany has some great old architecture such as the old NY Central Headquarters at the base of State Street...now SUNY Headquarters. It was designed by the same architect that designed the parliament buildings in Ottawa. The City Hall and don't miss the occasional carillon bell concerts there. The Capitol building itself and neighboring (now restored) Alfred E. Smith Building. Lots of stately homes going west on Madison then Western Aves.
Albany Musem of History and Art... A good collection of paintings from the Hudson River school masters, plus a lot more about history and art...
So anyway ...its not all bad about Albany ...and you are a 2.5 hour amtrak ride into Manhattan from the very nice new train station just across the river.