Wherever you end up (can make a strong case for either downtown), add these to your list:
So & So Bookstore in Raleigh - a small independent bookstore that supports local authors and poets. You can read about the owners here:
New bookstore gets personal with patrons - Technician: News
The Person St/Peace College/Seaboard area (near Oakwood and Mordecai neighborhoods) is really coming together. It has several great places like Yellow Dog Bakery, PieBird, Peace Street Wine Bar, The Station. Scott Crawford, the executive chef from Heron's, is apparently leaving to open a new restaurant/grocery/urban farm there as well (sounds like he is going to take over Market's space).
Of course, there is also Quail Ridge Book Store in Raleigh (a couple miles from downtown) that has poetry readings the 1st Sunday of each month and all sorts of other interesting literary events.
I'm thinking that if you come to visit, you'll feel an affinity for one or the other...just remember, as noted above by MyGhost, there is plenty of stuff in Raleigh proper, it's just more spread out than in Durham (downtown is more compact).