SwedenBob, you might want to take a look at
Parker, AZ, located in the northwest part (on the border of Arizona/California) on the Colorado River, 35 miles north of the I-10 freeway and 40 miles south of Lake Havasu. (Cross the river and you're in California.) The weather might be about what you're looking for (right now, at 2:45 PM it is 19°C), and from there it is not difficult to get to either LA or Vegas.
It's really kind of an appealing area in many ways for the type of lifestyle you seem to be looking for, with the added attraction of 16 miles of Colorado River called the Parker Strip. It's not a big town, with a population of only a few thousand, but near enough to bigger Lake Havasu City for emergencies. Take a look and see if it might suit you. A local realtor can give you info on buying land and whatever regulations there are on building, etc.
Parker, AZ: http://parkerareatourism.com/
From Wikipedia:
http://en.wikipedia.org/wiki/Parker,_Arizona