I live in central Maine (Newport), but Portland ME is one of my all-time favorite cities.
I really wish I could afford to live there.
When I originally moved to Maine two years ago, I started looking at houses in and around Portland. House prices (to me) were way out of my price range for what I needed, so I started looking in Bangor. Bangor's prices were much closer to what I was ready to pay, but found a place really cheap in Newport, about 30 minutes away from Bangor. I work from home, so commute time wasn't ever an issue.
In any case, I wouldn't worry about crime in Portland either. The 11:00 news spends a lot more time talking about "soft" news then they do about violence, drugs, and other crimes.
The problem with living in Portland, or any other place in Maine, will be the cost of heating oil, electricity, car registration. Property tax in southern Maine will be worse than in other places in Maine (like Bangor) too.
Last year, we paid $352/mo (every month) for heating oil for our 5 br house. This year, we'll pay $402/mo. You overpay in the summer to compensate for underpaying in the winter.
Electricity here is unregulated, so you pay roughly 8 cents/kwh for the electricity, and another 6 cents/kwh for the "delivery", meaning electricity is really 14 cents/kwh total.
In Idaho (our previous state) we paid natural gas each month, and NEVER paid more than $90 for heat. During the summer, we paid $14 (for hot water only--I miss those days of complaining about a $90 heating bill.) Electricity in Idaho was 7 cents/kwh too, so our electricity bill effectively doubled. We pay roughly $160 - $190 every month for electricity in Maine.
Car registration costs are ridiculous. Registring our new 2006 Toyota Sienna was $650. Our old 1997 Toyota Camry was $150. You can estimate your car registration costs here:
http://www.ci.bar-harbor.me.us/vehicle/extaxcal.html (you'll need your vehicle's MSRP for when it was new)
All that being said, if you can handle the cost of heat, electricity, and car registration, living in Maine is wonderful. There is no traffic, virtually no crime, the scenery is always beautiful, and the people are really friendly.