Twice now I've seen it happen where downloading a map as you load in for the first time is extremely slowly (10+ minutes) for everyone on the server who hasn't previously downloaded the map. I'm not sure if this is tied to specific maps or just happens sometimes for another reason. Quitting out of the download and grabbing the same map off the workshop fixed the issue when I tried it on dinkledome. I've seen it with the following maps:
Dinkledome
ttt_summermansion_b3
Dinkledome
ttt_summermansion_b3