Coaticook is charming and closest to the border, but if you're in that area you might as well go two towns over to the city of Sherbrooke, which is like an actual small city with about 150,000 people. Magog's nice to, really pretty waterfront.
Things to do in Sherbrooke