The TOWN of Newburgh is TOTALLY fine. Yes, avoid the CITY of Newburgh. Look to stay WEST of Route 300 in the TOWN of Newburgh. Previous poster "underwater" probably scared you away though. Still think you can get a reasonable and safe place in the TOWN for the time being, though.
I don't think your $1200 / month goes all too far in Rockland County. It is also more commercialized on the whole as compared to Orange (you specified a desire for country living in your OP) + higher taxes.
Also, where will your husband be working in New York (street)? This will help me out vastly in recommendations. If in Midtown, he should be along the Hudson Line. If in Lower Manhattan, he should be along the Port Jervis Line / West-of-Hudson.
Either way, your money will go the farthest in Orange and Dutchess Counties. You have specified that the initial move will be temporary. A move to Orange or Dutchess likely gives you the most long-term financial flexibility. Once you are ready, you can move to a move upscale spot down the Hudson in Westchester or Rockland County.
Again, the following locations are desirable and close to the Hudson River for optimal Hudson Line commuting: Beacon, Wappingers, Fishkill, New Hamburg, TOWN of Newburgh, New Windsor, Montgomery, and Maybrook.