Update: Scott Seely explained that this is by design.  So I'm changing the title in my post. Also if you want to do this, use async methods as he pointed out.

Last night my friend Michele pointed me an interesting thing on the WCF client side runtime. Basically, she had a service method like as follows.

public void SendMessage(string message)

{

    Console.WriteLine("SendMessage: {0}", message);

    MessageBox.Show(String.Format("Received message: '{0}'", message));

}

(We know, we know, one should not display message boxes from a service method but here the idea was basically blocking the thread that executes the service method interactively.)

Further more the service was configured as PerCall/Multiple and was running on netTcp.

Then we tried to call this service with an instance of svcutil generated proxy in several threads simultaneously (see below). 

 private void button1_Click(object sender, EventArgs e)

{

    MethodInvoker m = new MethodInvoker(CallService);

    m.BeginInvoke(null, null);

 }

 private void CallService()

 {

     proxy.SendMessage(string.Format("Message {0}", ++counter));

 }

When we ran client and the service, we expected to see multiple messages boxes appearing in the service as we click on the button1 in the client. In theory this should work because, although we use a single tcp session, our service is configured for PerCall and Multiple. So the message pump in the service side ChannelHandler can use a new thread to pump the next message from the same tcp session and dispatch it to a new service instance object while the previous message is being processed.

However, things did not workout the way we want.  When we clicked the button, first message box appeared in the service. But subsequent clicks did not do so until the first message box was closed (i.e. thread was released). However, when all pending messages are displayed, everything started to work as expected for the subsequent requests.

Let the fun begin!  :)

Out of curiosity I got a snapshot of all threads while the client is blocked after the few very first calls. Call stack of one blocking service call revealed quite a lot about what's going on. The stack was like this (irrelevant frames and parameters are removed for simplicity sake).

[In a sleep, wait, or join]        

mscorlib.dll!System.Threading.WaitHandle.WaitOne()

mscorlib.dll!System.Threading.WaitHandle.WaitOne()

System.ServiceModel.dll!System.ServiceModel.TimeoutHelper.WaitOne()        

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOnceManager.SyncWaiter.Wait()

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce()     

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.EnsureOpened()

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call()

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call()        

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService()   

System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke()

mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke()

Aha! Interesting! On the client side WCF internally uses ServiceChannel.Call to send/receive messages from the service (regardless of whether you use svcutil generated proxy or manual ChannelFactory and IClientChannel). So it seems like the call is not going out of ServiceChannel.Call at all. It's blocking at a call to EnsureOpened method. Then I opened up Reflector hoping to find what's going on in this method (Lutz, I thank you every time I use it :)).

Refelctor demystified some of the interesting stuff WCF do in the client side. When we call service methods, we simply call them with just a single line (we hardly call Open explicitly).  Therefore WCF has to make sure that the underlying channel is Open before sending the request. This opening is just one time action. So internally WCF uses a thing called CallOnceManager to make sure that  actions like this are performed only once.

The ServiceChannel.EnsureOpened method calls the CallOnceManager which is responsible for calling Channel.Open once to open the underlying channel (look at the following code I extracted from Reflector). 

internal void CallOnce(TimeSpan timeout, ServiceChannel.CallOnceManager cascade)
{
    SyncWaiter item = null;
    bool flag = false;
    if (this.queue != null)
    {
        lock (this.ThisLock)
        {
            if (this.queue != null)
            {
                if (this.isFirst)
                {
                    flag = true;
                    this.isFirst = false;
                }
                else
                {
                    item = new SyncWaiter(this);
                    this.queue.Enqueue(item);
                }
            }
        }
    }
    SignalNextIfNonNull(cascade);
    if (flag)
    {
        bool flag2 = true;
        try
        {
            this.callOnce.Call(this.channel, timeout);
            flag2 = false;
        }
        finally
        {
            if (flag2)
            {
                this.SignalNext();
            }
        }
    }
    else if (item != null)
    {
        item.Wait(timeout);
    }
}

CallOnceManager services the simultaneous requests trying to use it for the first time in a FIFO fashion using a queue. This queue is initialized in the ctor. When the CallOnce is invoked it checks whether this queue is available and if "yes", it checks whether this call is the first one. If it's NOT, a waitable object is created and placed in the queue and then CallOnce method blocks on this newly created waitable object.

So out of several simultaneous *first* calls to CallOnce the one that wins open the channel and returns back to the ServiceChannel.Call frame. After doing some more work it finally uses this newly opened channel to send the message. When the reply is returned, it calls CallOnceManager.SingnalIfNotNull method which in turn calls the CallOnceManager.SignalNext method to signal the next waitable object in the queue. When it's signaled the relevant CallOnce call that was waiting on it gets released and the next request is sent to the service.

So in our case first request did not return until we closed the message box. So the subsequent service method calls were hanging at the CallOnceManager.CallOnce method. Because first call has to return and call CallOnceManager.SingnalIfNotNull to get one of the waiting calls released.

IMO, this is too bad. This should essentially check whether the Open has called and if it has, it should just flow without blocking.

On the second part of the question. I was wondering why it was working after end all pending calls. I opened up the SignalNext method and the answer was there.

internal void SignalNext()
{
    if (this.queue != null)
    {
        IWaiter state = null;
        lock (this.ThisLock)
        {
            if (this.queue != null)
            {
                if (this.queue.Count > 0)
                {
                    state = this.queue.Dequeue();
                }
                else
                {
                    this.queue = null;
                }
            }
        }
        if (state != null)
        {
            IOThreadScheduler.ScheduleCallback(signalWaiter, state);
        }
    }
}

When the SignalNext is invoked it dequeues the next waitable object from the queue and signals that. If the queue is empty (i.e. no more items striving to be first), it sets the queue to null. Therefore the next call to CallOnceManager.CallOnce just exists without doing anything because the queue is null.

IMO, this is a little bug we have there ;). Dear team, please correct me if I'm missing anything here or if this is by design please explain us why.

Meanwhile if someone is trying to get over the problem I've also found a nice solution. The CallOnceManager is used for automatic opening of channels. But if we call Open explicitly in our code we can get rid of it (Look at how ServiceChannel.EnsureOpened is called in ServiceChannel.Call method).

if (!this.explicitlyOpened)
{
    this.EnsureDisplayUI();
    this.EnsureOpened(rpc.TimeoutHelper.RemainingTime());
}

So we can turn on this explicitlyOpened flag by calling Open *once* in our proxy or the channel before invoking the method.   

After all I started to wonder why I did not spend little bit of more time to reflector the client side runtime. There is a lot of fun out there as well!!! :)