The high crime areas in Annapolis are: Robinwood (off of Forest Drive), Harbor House (in Eastport), Clay Street (off of West Street).
Eastport (aside from the public housing) is a beautiful and generally safe area - It really doesn't fall into the category of "affordable" by most standards. Depends on whether you plan to rent or buy.
There are still some affordable single family homes on the outskirts of Eastport, but they are affordable because they could use some fixing up, are closer to the higher crime areas, or are on a busy road.
If you are looking for affordable single family homes in Annapolis, you may want to consider the neighborhoods: Arundel On The Bay, Oyster Harbor, Highland Beach. Eastport and the historic district are easily accessible and they are safe. Also, Cape Saint Claire is an affordable option - but it may as well be another city. It's pretty far out from downtown and other Annapolis amenities.
If you are looking for condo/apartment living, there are oodles of options - and plenty of new construction.
They are building the
Annapolis Towne Centre at Parole, which will have loads of new housing options next year.
1901 West was recently built. The condos on Bestgate are within walking distance to the Annapolis Mall.
Hope this helps!