If people feel as though walls + drush fc is too strong, another way to deal with it is just to nerf knights and crossbows, and/or increase the cost of TCs. That would lessen the power spike of reaching Castle Age, and make it less likely that the benefits of a slightly faster Castle Age will outweigh the costs in economy and map control.
Also it would probably make Castle Age more balanced overall if those two units weren't so strong.
Also it would probably make Castle Age more balanced overall if those two units weren't so strong.