You have aptly described one of Charlottesville's drawbacks with your Abercrombie and Fitch parallel. Besides this annoying phenomenon, it is simply a hard town to break into if you don't have a ready-made community (ex: sorority). Even as a native, when I moved back in my 20s to go to UVa, I found it quite hard to create a community for myself. Don't fear, there is diversity in the town and plenty to do! I would suggest looking for events and things to do on and around the downtown mall area. Pick up a
C-ville and the
Hook to see what's around that interests you... It is a wonderful place. Good luck to you!