I think I understand what the OP is trying to say....
Before he purchased the house the previous owners installed pavers and a paver retaining wall around the property.
When the previous owners installed the pavers, there was probably a nice big house behind their lot that was sold to developers who tore it down and built attached homes.
Then the builder puts up a fence for the attached houses...
and the fence the builder installs is a foot or so into the OP's property line leaving the paver retaining wall on the side of the fence that borders the attached houses.
and the OP is concerned because he doesn't have access to the retaining wall, and/or those neighbors will remove it.
I think..
I could be wrong...
or maybe the gypsies did it.