TREP resilient issue: 2 type of error, 1 successful, 1 failed to failover to next channel.

fuweng.ng

We are using Ema-3.3.1.0.jar in our TREP application to get live/delay feed. We setup 4 channels in our EmaConfig.xml. However, we notice that the resiliency failed on some scenario.

Scenario 1:

Error Message: A21: Access Denied. Waiting to connect to DACS server. Try request again.

Result: Resiliency works, the application able to failover to next channels and re-establish the connection

Scenario 2:

Error Message: A21: Access Denied. Timed out waiting for response from DACS server. Try request again.

Result: Resiliency failed, the application failed to failover to next channel and the application lost connection to TREP

May I know why the resiliency fail for second Scenario?

umer.nalla

Hi @fuweng.ng

As explained in my earlier post, the ADS is the one that interacts with the DACS server - not your application. Your application only talks to the ADS.

Therefore, you need to speak to your Market Data team and/or DACS administrators to resolve this issue. If they are uncertain how to resolve they should raise a My.Refinitiv ticket for technical support for the ADS / DACS issue.

fuweng.ng

This is the log after 1st Scenario:

2020-11-28 21:12:30.005 WARN [pool-2-thread-1] OmmConsumerImpl - loggerMsg

ClientName: ChannelCallbackClient

Severity: Warning

Text: Received ChannelDownReconnecting event on channel Channel_3

RsslReactor @59dbf726

RsslChannel @571c126d

Error Id -1

Internal sysError 0

Error Location WlLoginHandler.readStatusMsg

Error text Received login response with Closed/Recover stream state. Disconnecting.

loggerMsgEnd

2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:

2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0

UpdateSubMgr statiistics end. <<

2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end

2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<

2020-11-28 21:12:34.023 INFO [pool-2-thread-1] OmmConsumerImpl - loggerMsg

ClientName: ChannelCallbackClient

Severity: Info

Text: Received ChannelUp event on channel Channel_4

Instance Name Consumer_1_1

Component Version ads3.3.1.L1.linux.tis.rrg 64-bit

loggerMsgEnd

This is the log after 2nd scenario:

2020-11-28 18:12:27.640 ERROR [pool-2-thread-1] RTROmmConsumer - [registerClient] Item: app_eqtrade / Service: <not set> [State: Closed / Suspect / Timeout / 'A21: Access Denied. Timed out waiting for response from DACS server. Try request again.']

2020-11-28 18:12:27.640 WARN [pool-2-thread-1] RTROmmConsumer - No. connection failed detected :2

2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:

2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0

UpdateSubMgr statiistics end. <<

2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end

2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<

2020-11-28 18:12:58.202 INFO [Thread-7] SL_SubMgr - proUnsub starting check

2020-11-28 18:12:58.202 INFO [Thread-7] SL_SubMgr - proUnsub check end. Count=0

2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:

2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0

UpdateSubMgr statiistics end. <<

2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end

2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<

umer.nalla

Hi @fuweng.ng

Looking at the output you have provided, it would appear that there are problems with your DACS server - which is not a data provider - it is Data Access Control system.

In your consumer configuration, you have setup 4 channels - these alternate channels should be for different ADS instance. When your application connects to an ADS, the ADS must check your login credentials with the DACS server to ensure you are a valid user and then as you request data, it will again check with the DACS to ensure you are entitled to each of the data items you are requesting.

The problem is that If the DACS server(s) is slow to start or crashing - even if all your ADS are working fine they cannot proceed as they are unable to validate your credentials.

In the above examples you have posted, it is unlikely the failover is even being activated - you are most likely connecting to the first ADS. In scenario 1, the ADS then tries to validate your credentials with a DACS server, it waits a while and then it would appear the DACS server does respond before the timeout period, so the ADS can continue and process your login + data request.

In scenario 2, the ADS waits for the DACS server to respond which fails to do so in the timeout period so the ADS has no choice but to close the connection. There would be little point in failing over to a different ADS because the 1st ADS itself is running just fine.

I would speak to your Market Data team and identify why the DACS server is failing to respond to the ADS request to validate your credential. They need to resolve the DACS issues with a high priority. It is quite rare for a DACS server to fail to respond or crash - so there must some serious issue at play here. They can raise a support ticket at MyRefinitiv if they need assistance.

fuweng.ng

Thanks alot for your prompt reply. In this case, may i know where can i change the timeout for DACS server? is the below setting in EmaConfig.xml the timeout for DACS?

<ConnectionPingTimeout value="30000"/>

I am thinking if we can set the timeout longer, we can avoid the 2nd Scenario. Thanks.