You don't buff an unit by giving it 36 secret bonuses and techs, you buff it by changing its core stats
Look at the knight-line, they have zero secret bonuses against any units, they're just good,: good speed, good attack, good melee and pierce armor. Longswordsmen and up should take example of that.
FE devs are beating around the bush for years, increasing bonus attack against eagles, increasing bonus attack against buildings, free Tracking, introducing Arson tech, Supliiiiies tech, all this kind of nonsense when all they needed was probably a tiny speed boost.
In fact, I'd get even crazier and say let them build any building a villager could build. In those times, soldiers would build entire encampments complete with palisade walls and housing. It would fit the theme. Plus, it allows for more interesting civ bonuses and balance considerations.
I mean, for god's sake, they're countered by everything. If you can't stop them from building a forward base then that's on you.