I'm sorry it used to happen to ours years ago but I cant remember how it was fixed. It sounds like the server is doing something before the map starts. Have you tried deleting the server logs ? so they are not very big to load / read and having no skins or anything running, just basic server with silent on it. Are you running punkbuster still ?