6

Resolved

Trigger message sent twice at the same time

description

Hello,

I'm using the scheduled task adapter to trigger orchestrations, using the XmlStringStreamProvider, and to create a singleton pattern in the orchestration.
The receive location is configured to trigger a message every minute and the orchestration is designed with a loop to ignore other trigger messages in case the duration exceed 1 min.
It's not perfect and I get Zombie messages.

The issue is that 2 messages are sent to my orchestration every minute ..

I did a test, triggering an orchestration without the singleton pattern loop, and two instances started at the same time, after 1 min.
And when I look at my zombie messages in the suspended instance, they are always by pair.

In the example attached:
  • The orchestration started at 3:55:00 (triggered by 1 message from the adapter, and ignore the duplicate)
  • At the end, there are 4 zombie messages, you can group them by 2: same message ID, same creation date.
  • 2 messages created at 3:55:58 and 2 messages created at 3:55:59
So for a schedule set up every minute, something is wrong ...

Any help would be welcome :)

Thank you.

file attachments

comments

rrossi wrote Jul 29, 2016 at 9:39 AM

Hello,

We are experiencing similar issues in three different environments. The adapter occasionally generates two messages at the same time. While this happens, it looks like the ScheduledTaskAdapter.NextScheduleTime property value is same as the "last activated" value. This causes the adapter to fire a new message instanty.


Any help would be welcome also :)

Thanks.

wrote Aug 11, 2016 at 12:39 AM

tpo wrote Nov 2, 2016 at 2:12 PM

That happens to me too. I'm using the XMlString task component.
Sometimes works nice, sometimes generate two messages.
Sandro, can you fix it pls?

wrote Nov 2, 2016 at 2:12 PM

tpo wrote Nov 2, 2016 at 2:46 PM

The problem is in GetNextActivationTime(DateTime now) method in TimeSpanSchedule class.
Code was changed from the previous version,
Do you know why this code was changed? I can fix it, but I'll have to know why this code was changed.

Tks
Tiago

wrote Dec 18, 2016 at 5:11 PM

BizTalkers wrote Jan 27 at 1:26 PM

I am facing the same problem. This bug is serious because:
  • it lulls into a false sense of security. All seems good, then it will create multiple messages out of the blue!
  • because it's unexpected, it's unlikely solutions will have the necessary defensive logic in place, such as the singleton orchestration and checking for duplicate triggers
  • the impact can be significant, especially if downstream LoB interfaces were not developed as idempotent
I really appreciate the effort that has gone into its development, but the scheduled task adaptor is so widely used, I think the functionality should be provided with BizTalk - covered by warranty! If you agree, please vote here: https://biztalk.uservoice.com/forums/383724-biztalk-server/suggestions/17771293-make-biztalk-scheduled-task-based-adapter-out-of-b

wrote Feb 1 at 6:26 PM

wrote Feb 1 at 6:52 PM

nubpotato wrote Feb 1 at 6:52 PM

Came to upvote this issue, my biztalk team has been experiencing the same issue with this adapter using the XMLStringProvider.

We recently migrated one of our solutions using this adapter from Biztalk 2010 to Biztalk 2013 R2. After the migration, we started noticing occasional double triggers from the Scheduled Task adapter in the newer BT2013 environment. It happens only once every couple hours, as shown in our database log screenshot, but its still an annoyance because it causes us to get a lot of false errors/alerts on a daily basis (we use the trigger to kick off a webservice call, which does not support simultaneous processes).

We've already tried different ways of rectifying it, changing time intervals (tried 5 mins to 30 mins), ensuring only one host instance is running, using only 32bit host, using only 64bit host, all to no avail.

Not sure if this might be related, but we do have a number of receive locations running the scheduler adapter across different biztalk applications, all on different time intervals.

Hoping this problem will get looked at.

sandro_asp wrote Mar 1 at 4:02 PM

I am making some modifications and test to try to solve this issue... that is not easy to test it.
But be aware that the adapter cannot only be running in one host instance, if you have a multi-server installation, the host need to be clustered

sandro_asp wrote Mar 4 at 11:42 AM

Again difficult to debug this problem, but I am trying to emulate with 30m, 10m, 2H and even with 1s and everything is running fine with the new release I am working on - only one message is being triggered.

And again, if you have several BizTalk boxes, you can only run the host instance in one of the servers, otherwise, you need to cluster the host.

wrote Mar 4 at 11:43 AM

wrote Mar 20 at 5:47 PM

nubpotato wrote Mar 20 at 5:47 PM

Hi Sandro,

We have always only had one host instance running the scheduler, so that is for sure not the issue here. In fact, I was able to reliably replicate the issue on an isolated Biztalk 2013 R2 Development VM running the scheduler over the weekend (single instance Biztalk + SQL on same server).

To mimic what we have in our production environment, I have setup 3 parallel receive ports/locations with scheduler adapter (XML String Provider) with parameters as follows:
RP_TEST_5MIN - Runs every 5 minutes from 0:00 of the hour
RP_TEST_10MIN_1 - Runs every 10 minutes from 0:05 of the hour
RP_TEST_10MIN_2 - Runs every 10 minutes from 0:00 of the hour

Each receive location has a simple logging pipeline to log the datetime and message into table on database, which is where these logs were pulled.

I have attached the resulting logs from this past weekend in 'SchedulerAdapterTest-Logs.xlsx'. As you will see, the duplicate calls occur a few times every 24hr period for each RP. Hopefully this is enough detail for us to reopen this issue.