Many years ago, I was driving on a busy road when I suddenly saw four kittens by the roadside - some cruel person had dumped them there.
There was no shoulder, no break in traffic, and nowhere to turn around. So I had no choice but to keep going. Even today I wonder how those babies ended up.
You're doing right by those strays, OP. Don't worry about being away from them for a few days now and then. They'll know to stick around.