Oak Island Drive is 35 mph from the Barbee Bridge to about East 45th Street. Middleton is 35 mph from Oak Island Drive to Beach Drive.
I can see your dilemma; if you were on say, W. Yacht Drive at 10th Street, your only path to the beach would be via 10th to W. Oak Island Drive (which is 45 mph) and then Middleton to the beach.
One solution would be to stick to the eastern end of the island where Oak Island Drive is 35 mph.
I'd call the Oak Island Police Department and ask them about it. The Town ordinance on golf carts can be found here:
https://library.municode.com/nc/oak_...RTIIGOCAMOASDE
It says golf carts are not permitted on Oak Island Drive, Beach Drive or Ocean Drive, but I have seen them being operated on those streets.