The summer season usually starts around mid to late May. If you're wanting to do outdoorsy things, especially anything involving getting wet, I wouldn't go in April. It can be chilly sometimes in April and rainy.
You should be "safe" from tourists if you go in early to mid May. A lot of people don't go to the beach until their kids are out of school. We used to go in May until our kids started school. Now we have to wait 'til June. I really liked going in May. The temperatures are usually pretty nice (70s) and the humidity isn't high yet.
September would also be a good month to go. Just keep an eye on the hurricane forecast.
Have fun!