I don't live In Grafton but in Northborough, and I love it. However, as you have noted, it is expensive; I would say that property taxes for an average home are $4-5K. Yet the expense is justified because the area is safe (you can leave your doors unlocked by mistake without incident) and the public schools are really good. Grafton High School is relatively small [for a high school]: 700 students. The building is only 4 years old and is state-of-the-art. Westborough Elementary is supposedly the best in the area, and Westborough was named [in 2014] one of the 10 best towns in America for families by Family Circle. The Fay School is private and is located in the most affluent town in the area, Southborough.
There are lots of amenities in the area--great grocery stores (Wegman's Stop&Shop, Roche Bros.), fitness options, restaurants, etc. There is a gold course in Northborough (Juniper Hill), Natick Mall is 25 minutes away and Hopkinton State Park is 20 minutes away. The area is very neighborly (most people sort of expect you to be friendly and wonder if you don't say 'hello') and family oriented (most people are married and have children), so there isn't a huge dating scene. If you are interested in going to a club, you would most likely have to go into Boston (I don't recommend Worcester).
Grafton is less affluent b/c it is a bit father away from Boston and is still more rural. However, because of its proximity to the more affluent boroughs of Worcester and its lower home prices, it is very up-and-coming. And you are still only 45 minutes from Boston.
The only concern I would have is this recent development in North Grafton:
http://www.railwayage.com/index.php/...-terminal.html