A while back I had a very strange problem where for a very few users site load times would blow up to 30-60 seconds of delay (something happening prior to the firebug NET timing) It was only a very few users and in the end the solution was to remove the Expires header from the web response (which makes no sense I know) Here is my blog post about it
Now it's back, but it must be something else because the Expires is already gone.
Just two users out of several hundred have the problem only on this site on this server (a different site with the same code and design on a different server works just fine for them http://www.ccga12.org/ ) again firebug net timings show the delay occurs before it even gets to firebug. They experience the same delay in all browsers but the two users are on completely different hardware and connection types.
I fear it is another oddball problem somewhere deep in the bowels of the Server->ISP->User path but it makes no sense - When I last asked this question nobody trying from other places around the world had any issues - so I dunno if hitting either of these sites will help narrow it down - but I'm open to any ideas at this point.
Since it seems to be particular users that have the problem - and only on one site (not on an identical site with a different server), it sounds like a cache/cookie problem, since these are unique to a single user, and are stored by website. If you can get the users experiencing the problem to clear cache and delete cookies, you can rule out (or fix) this possibility.
If it's not either of those, I'd lay my money on it being a server configuration issue. (I don't know what you mean by "The person can access other web sites from the same server".) This doesn't seem like enough to preclude the possibility of server configuration problem. But then, I've only done enough web server configuration to be impressed by the number of ways things can be totally messed up. :-)