Sometimes I think Milford tries too hard to live up to the cute "small town" ideal.
No idea about drug use, but I will say that Milford has/attracts a lot of drinkers, aside from the bars in the area of the oval, each of the (mostly Chinese/fusion) restaurants along 101A has a pool of regulars propping up the lounge side of the business. NTTAWWT.
Not sure about the rental house market in S.NH, but it seems pretty limited. Overall, housing is a bit more expensive than towns further into NH, in part because Milford is within commuting distance of not only Manchester and Nashua, but also some parts of the 495 tech corridor in Massachusetts.