In a previous post I mentioned that you should probably just give up the idea of using Polling Duplex Communication endpoints in your Silverlight application if you’re going to host your WCF services from SharePoint 2007.
The main reason I cited was that hosting the service(s) in SharePoint 2007 meant you probably wanted access to the SPContext which meant you must force your WCF services to run in ASP.NET Compatibility mode. This alone doesn’t cause the problem and I did mention the issue stems from contention trying to access session state data. I dumped the whole idea of using Polling Duplex for my needs since access to the session state was necessary. It occurs to me now that not everyone needs it.
Check out this article which provided me a good deal of information when I was debugging my issue. It provides a different approach for resolving this slowness. Simply put your Polling Duplex stuff in its own service with session state disabled. That should keep things quick and happy and prevent your session-needing service calls from getting blocked.
Hope this helps,