I'd guess touring Pakistan is not very safe, even for Pakistanis. Ive known a few who have been kidnapped in their own country, later to be released but out in the middle of nowhere. Surely many have gone but I'm sure some haven't been treated so kindly.
Check out
Terrorism | South Asia Terrorism Portal and see when and where the most recent happenings have been. I too would love to visit as it seems like a place full of culture and interests that you won't find in much of the world but I'd use caution as a primary factor in determining when and where I visit.