Charlottesville's not my cup of tea at all, but I do have friends/acquaintances who live(d) there and love(d) it. It has an old college town feel to it, which makes sense because it is an old college town (University of Virginia is there, founded by Thomas Jefferson). But just because it's a college town doesn't mean that you'll be surrounded by undergrads the minute you walk out your front door.
Richmond, the state capital, is in the process of transforming its downtown, mainly due to Virginia Commonwealth University's aggressive expansion. There are still large parts of the city (mainly south of the James River) where you don't want to venture after dark. But north of that, there are sections that are really nice and historic, including the Fan District. If you want to live in the suburbs, there's Henrico County surrounding the city and Chesterfield County to the south and west if you're looking for more upscale areas.
It all depends what you're looking for, though, because Richmond and Charlottesville have little in common. They are only an hour and a half apart, though, so it wouldn't be a strain on your gas budget to go visit your people in C'ville if you lived in Richmond.