If you have a map with really good fish for villagers you could probably get an even faster time than that. If you didn't get loom I think you could probably get enough gold to go castle by selling your stone. It might be an interesting math problem to figure out what the fastest theoretical time would be in that scenario. (Even though it would be completely useless in an actual game.)