Well, basically anywhere you go you're going to be inundated with frat houses, party hubs, rap hoods (mostly pseudo rap hubs, really) etc. etc. It's a college town, so that's what you get.
I think the neighborhood your're talking about is the Oceanaire area (I could be wrong though) and that is a HUUUUGE party neighborhood. Also good if you're an avid pot smoker and beer pong afficianado.
There really aren't too many sketchy neighborhoods in SLO, all in all it's a fairly safe place. The houses are small (landlords know they can make bank by charging per person, so if you have a house that has 3 bedrooms you can usually get 6 kids in there, charge them the going rate of $600 or so, especially with mommy and daddy paying, and you've just made a handsome profit!) and many are kind of run down (ex college houses that weren't taken care of) but you can usually find a good one if you look hard enough.
I've had the chance to live in SLO many times, but I won't do it unless it's out of the city area. I'm too damn old for college parties
I'm a light sleeper as well, so I don't like people coming and going at all hours of the night.
Where are you planning on working?