Manorhaven is more blue-collar than the rest of Port so that is why some people may have a negative impression of it.
My observation is that some streets have a high number of absentee rentals and thus are a little run down (but certainly not ghetto), while others are well kept and beautiful.
The area is really diverse, there are lots of older Italian folks and blue collar white families mixed with central American immigrants (both legal and illegal) and a fair number of Asians too. It also seems to be popular with mixed race couples (especially where one spouse is white and the other Hispanic). There are also higher income people that live in condos along the water. For a suburban neighborhood it really is kind of unique. They even have a community of houseboats!
The area has a nice park (Manorhaven Park) with a very nice pool and a great playground. People from all over Port use it. There is also an awesome pizzaria called La Piccola Angelina
http://www.lapiccolaangelina.com/ . Definately the best pizza in Port!
There was a problem a few years back with burgleries and vandalism but since the NCPD started focusing more attention on the area it has improved and I haven't heard of any problems lately.
If you want to live in other areas of Port there are houses that occasionally pop up in areas outside of Manorhaven for under 500K. There is a very small house in my neighborhood (The Terrace) that is listed for only 349K. There are others right around or just over 500K that are on decent streets.
Good Luck and welcome to Port! Don't let people who know nothing about the area scare you away from it.