We had the same problem in our rental last year! Awful. We were always afraid something was going to happen to the dogs when we left them alone in the house.
At first, we assumed the wasps were coming in through the exhaust fan in the bathroom, which is not unusual, apparently. But we discovered they were coming in through gaps in the old window hardware in one of our bathrooms (in between the top and bottom pane, also up the tracks in the bottom pane that allow you to open the window). We finally narrowed it down by "pre-treating" suspect areas with Raid and then watching for results, so maybe that's something you could try.
Just spray wherever you suspect they're getting in - like any crevices in your window, for example - and if you find dead yellow jackets near your window, then you know you've found the culprit. Of course, use your judgment with kids and pets when treating with Raid.
Once you've found the entry point, hopefully it's something you can easily address. If you can't find the entry point, an experienced professional should be able to help by doing an inspection.
Best of luck! The paranoia is killer, I know.