Yes, I would stay well away, and avoid crossing when they are actually casting. Just because you are out of range of the general movement of the rod/lure doesn't mean you are safe. Guy could screw up, snap bail shut, snap the line, and send lure flying behind him backward, straight up, or whatever.
It goes the other way too. I've done some surf fishing myself. I've had pedestrians walk right behind me and a friend as we motioned that we were preparing to cast. You do not want a lure with treble hooks in the face-trust me!(a lot of surf casting lures are also very heavy) Same goes for folk in the water who felt that our particular section of beach was the perfect place to exit. It also goes without saying that it is general impolite to snoopily walk up behind a fishermen with his rod at ready for a cast and peak in the bucket and ask if they have caught anything.
Maybe I would be if you moved!