My family had a summer cottage in the Wareham area when I was a kid. I've got a ton of happy memories
OTOH I remember my mother having to drive to the A&P at the bridge for groceries or having to drive to Hyannis to do any other kind of shopping because back then Wareham/Buzzards Bay was a very sleepy place with a few restaurants scattered on Cranberry Highway.
Its main problem IMO is location. Other than Rte. 25 there is no direct access to any of the major highways. I have a coworker who lives down that way...she has to negotiate back roads to avoid the bridge to get to Rte. 3 for work, and it can take her up to an hour or more depending on the weather and/or time of year.
At one point it was on our short list when we were thinking of moving. We discarded it because of the the location.
BTW, the downtown area as well as Onset has always had that "seedy" feel. You didn't want to stay too long in either place even back when I was a girl and that was a long time ago.