No you don't, unless you deleted vills since you were pop-limit.Then you have to include the opportunity cost of having used that pop for wood chopping which isn't negligible.
Anyway, this argument that a lot of games end before the game is over is pretty bad. If someone adds additional villagers just before the enemy resigns, that investment didn't pay off, but that doesn't mean that it was a wrong decision to add those additional villagers. The same goes for adding trade carts, although the time before it pays off is a bit longer than with villagers, but the point is still the same: If the enemy had resigned a few minutes later, the investment would've been worth it.