For walls and buildings I think this mechanic is just a very cheesy and cheap thing. It is just pretty ridiculous how this mechanic works somehow, being able to block any army in seconds and the army just can watch the vill completing. I think make the building foundation have a very weak armor like (-5/-5) for example for both buildings and walls.