XmlValidationStep:  It is the equivelent to using SelectSingleNode and must resolve to a node set

XmlValidationStepEx:  It utilizes XPathNavigator.  XmlValidationStepEx should be used when evaluating against functions such as Count()

This particular client appear to programmatically connect to the BizTalk server via Telnet as a way to keep the socket open.  In return, BizTalk would treat the connection as having a missing body (not to be confused with an empty body). To solve the problem, although it’s not ideal, is to put a try/catch around the GetOriginalDataStream() and return null if it encounters an error.   Unfortunately, the side affect of this is an active blank message instance in the BizTalk Administration Console.  But at least there are no more error event logs and suspended message instances.  

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
    IBaseMessagePart msgBody = pInMsg.BodyPart;

if (msgBody != null)
{

string oldStr = string.Empty;

try
{
    using (StreamReader sr = new StreamReader(msgBody.GetOriginalDataStream()))
    {
         oldStr = sr.ReadToEnd();
    }
}
catch
{
    return null;
}

<< INSERT STUFF YOU NEED TO DO HERE>>

}

}

For awhile I thought the issue has to do with the stability of their data connection. But the client have since moved to a full T1 on a different carrier but the problem persists.

While I am not a 100% sure I believe I have narrowed the problem down to the socket connection disconnecting and reconnecting. Let me explain…

Each of the MLLP receive port in question has persistence set to false and a 15 minute timeout period. To check on my theory, I changed the timeout period to 1 minute and used netstat to monitor remote socket switching (i.e. a disconnect and reconnect to a different port due to socket TIME_WAIT). Guess what? Right after the socket switch, there was a suspended message with the error stated earlier.

So if my theory is correct, what are those empty messages? And if a timeout should only occur if the socket is idle beyond the timeout period, what are those “forcibly closed by the host” error messages?

Please leave me a comment if you have any suggestions!

Update: The issue does appear to have to do with the stability of their data connection. Earlier today, their bandwidth was completely peaked out and the said problem popped up right on queue. Now, the only question is, anything can be done to prevent the errors? And at time of error, are we losing messages?

Here are the steps I took with which you SHOULD NOT follow:

1. Shut down all services gracefully and domain. Restart as required.
2. Recreate all appropriate users, groups and rights in the new domain
3. Add new user credentials to folder structures and database security
4. Change the appropriate services (i.e. host instances, BAM services, SQL, etc) to run as the new users.
5. Change all host instances to run using the appropriate new user
6. Start SQL
7. Start BizTalk services

Step 7 was where the fun began. SSO and BAM started ok for me. But as soon as I tried to start the host instances, SSO freaked out with errors like “… cannot perform
encryption or decryption because the secret is not available from the master
secret server…”

At this point, it would have been helpful if I had backed up the SSO master secret because that file was the key (pun intended) to restoring SSO ability to work under a different user account, and subsequently save me hours of fun reconfiguration BizTalk from scratch.

The client thought it has to do with network latency and started going down that road. However, I knew that latency can’t be the issue because we were getting the inbound messages just fine and that the ACK delivery time degrades through the day. So I started looking into the configuration for that hospital and here’s what I found:

• This high-volume hospital shares the same receive and orchestration process host as another high-volume hospital. So deploying dedicated process hosts for the hospital provided minor relieve.
• The orchestration for this hospital executes a send pipeline inside an Expression Shape and slowed things down quite a bit. A simple re-architecture did the trick.

Due to my lack of knowledge with the HL7 Accelerator, I had assumed ACKs are returned to the originating party soon after the message gone through the receive pipeline. Instead, it this scenario, it appears ACKs are not generated/returned until after the Orchestration notifies the Receive Port that the message has processed.

Live and learn!

PID.3_PatientIdInternalId identifies the unique patient
PID.18_PatientAccountNumber identifies the unique visit