reactor is shutdown message

joe.hong

Our EMA consumer periodically stops working due to an error regarding the reactor shutting down.

loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Error
    Text:    Received ChannelDown event on channel Channel
	Instance Name EmaConsumer_1
	RsslReactor @4f363abd
	RsslChannel @21a2fda5
	Error Id -1
	Internal sysError 0
	Error Location WlItemHandler.readUpdateMsg
	Error text ReactorCallbackReturnCodes.FAILURE was returned from defaultMsgCallback(). This caused the Reactor to shutdown.
loggerMsgEnd

Once this occurs, all subsequent requests will fail with additional errors...

loggerMsg
    ClientName: SingleItem
    Severity: Error
    Text:    Internal error: rsslChannel.submit() failed in SingleItem.submit(RequestMsg)RsslChannel 0
	Error Id -1
	Internal sysError 0
	Error Location ReactorChannel.submit
	Error Text Reactor is shutdown, submit aborted.
loggerMsgEnd

com.thomsonreuters.ema.access.OmmInvalidUsageExceptionImpl: Failed to open or modify item request. Reason: ReactorReturnCodes.FAILURE. Error text: Reactor is shutdown, submit aborted.

Then our EMA client has to be restarted to get it to work again.

Can anyone explain why this occurs and how we can fix it? The errors do not provide any additional information as to where to start investigating.

Nipat Kunvutipongsak

@joe.hong

There is a clue in the first EMA log output given:

Received ChannelDown event on channel Channel

In my test, when I tried to connect to a server, then I killed the server process or cut the connection, I got "Received ChannelDownReconnecting event on channel Channel_1" instead.

So, I searched for the meaning of the difference between them which are as follows:

CHANNEL_DOWN
ReactorChannel has failed (e.g. the connection was lost or a ping timeout expired) and can no longer send or receive data. There is no connection recovery for this event.

CHANNEL_DOWN_RECONNECTING
ReactorChannel has failed and connection recovery has started. This only occurs on client connections since there is no connection recovery for server connections.

Referring to the meaning of CHANNEL_DOWN definition, the problem should be from connectivity issue between EMA and the server.

Regarding the second EMA log output, it seems like the application tried to send a message to the server, but the reactor had been shut down already. I believe that it happened because of the connectivity problem earlier.

Therefore, I have several questions below:

1. What's the server?
2. Did the consumer application perform any posting
3. Did the consumer application send item requests at the beginning only or it sent requests on the fly?

If the application didn't perform activities in question 2 and 3, I suggest you test the application with the testing server such as IProvider example, to verify whether this problem still persists when connects to the local server or not.

joe.hong

Thanks, Nipat.

For your questions...

1. What's the server?

   Java application on Linux

2. Did the consumer application perform any posting

   Yes, the second part of the log output occurs when the consumer is posting

3. Did the consumer application send item requests at the beginning only or it sent requests on the fly?

   Not understanding the question.

If the Channel_Down error occurs, I thought the system automatically tries to reconnect, but we have no log entries to support that it is. Is there something we need to do to enable the retries? To get 'CHANNEL_DOWN_RECONNECTING' instead of just 'CHANNEL_DOWN' and stop working.

Nipat Kunvutipongsak

@joe.hong, to elaborate my previous questions, I observed that the application logged the error message while it was trying to send outgoing messages to the server, which could be the scenarios that...

• The application performed posting messages
• The application sent requests. In the normal situation, the consumer application typically sends requests at the beginning rather than in the middle of running

These scenarios are my previous question 2 and 3 respectively.

We're trying to check which example can be the test server and run on the same machine as the consumer application. I'll be back soon.

Nipat Kunvutipongsak

joe.hong, With reference to the definition of 'CHANNEL_DOWN' from Eta ref manual (ETA is an underlying layer of EMA), it indicates that the problem is from either connection lost or ping timeout expired.

To perform an experiment whether the original problem relates to connectivity issue or not, I suggest you connect the application to the local test server, which is an ETA's Interactive Provider example. This example locates in <Elektron SDK Package>/Eta/Applications/Examples (package: com\thomsonreuters\upa\examples\provider). It can support both On-stream and Off-stream post.

Nipat Kunvutipongsak

To add on top of my previous answer, you can modify a couple of lines in Provider.java to make it is easy for tracking output result as follows:

1. Extend the default runtime ( default runtime is just 1200 seconds or 2 minutes )

2. Prevent the provider generates updates and sends to a consumer automatically to remove unnecessary information except posting data only.

Nipat Kunvutipongsak

Moreover, this Interactive Provider example accepts the following fields only.

Fid: 2 Name = RDNDISPLAY
Fid: 4 Name = RDN_EXCHID
Fid: 38 Name = DIVPAYDATE
Fid: 6 Name = TRDPRC_1
Fid: 22 Name = BID
Fid: 25 Name = ASK
Fid: 32 Name = ACVOL_1
Fid: 11 Name = NETCHNG_1
Fid: 267 Name = ASK_TIME

You may need to consider changing your application to send out known fields such as BID or ASK.

Nipat Kunvutipongsak

Have you tried connecting the local provider? If so, does the application still encounter the connection down problem from Reactor?