I've used this to get latex paint off of glass and it worked really well......it won't hurt the brick, so you might want to give it a try:
Windex - just spray it on liberally, wait just a few seconds for it to soak in, then scrape (maybe with a razor or a butter knife) off the latex paint.
It kind of makes the paint rubbery for a few minutes and makes it pliable enough to get it off.
Worth a shot
.
*the only thing I would not do is anything harsh or abrasive - I would not use a pressure washer, a sandblaster, etc.