Auto listener occupies 70-100% of CPU as we are waiting for the update(Consumer.dispatch)

Unknown

We are trying to code as below.

1.Auto Listener waits for updates for a single dummy rate.

2.When there is an update the listener will pick rates for the rest of the 600 CCY rates from ATS.

However, Auto listener occupies 70-100% of CPU as we are waiting for the update(Consumer.dispatch) and also when we try to retrieve 600 CCY rates from Reuters. Sometime it hangs the server. As it does registerclient, it starts many threads which slows down the server.

Queries,

How do we overcome this issue? Is there a way to retrieve rates for 600 CCY rates in a single thread?

OmmConsumer consumer = null;
AppClientLive appClient = new AppClientLive();
try {
	OmmConsumerConfig config = EmaFactory.createOmmConsumerConfig();
	consumer = EmaFactory.createOmmConsumer(config.host(props.getProperty(MMRatesConstants.HOST_NAME_PROP)).username(props.getProperty(MMRatesConstants.USER_NAME_PROP)));
	consumer.registerClient(EmaFactory.createReqMsg().serviceName(props.getProperty(MMRatesConstants.SERVICE_NAME_PROP)).name("MMRTUDT=MMR"), appClient);
	while (true) {
		if (MMRatesConstants.IS_MARKET_UPDATED){
		break;
	}
	consumer.dispatch(1000);
    }
}

Olivier DAVANT

Hi @Pulaputra.Sudheer,

I see very good suggestions in the replies from Umer, Nick
and Zoya. I will not
add anything new here but try to summarize what has been said so far to help you tackling down your issue.

Here is your code snippet (reformatted):

OmmConsumer consumer = null;
AppClientLive appClient = new AppClientLive();

try 
{
    OmmConsumerConfig config = EmaFactory.createOmmConsumerConfig();

    consumer = EmaFactory.createOmmConsumer(config
          .host(props.getProperty(MMRatesConstants.HOST_NAME_PROP))
          .username(props.getProperty(MMRatesConstants.USER_NAME_PROP)));

    consumer.registerClient(EmaFactory.createReqMsg()
           .serviceName(props.getProperty(MMRatesConstants.SERVICE_NAME_PROP))
           .name("MMRTUDT=MMR"), 
        appClient);

    while (true) 
    {
        if (MMRatesConstants.IS_MARKET_UPDATED)
        {
            break;
        }

        consumer.dispatch(1000);
    }
}

The first problem of this code is that you’re mixing the
two EMA operational models for dispatching messages. EMA allows you to dispatch
messages either from an EMA internal thread (API_DISPATCH model) or from your
own thread(s) (USER_DISPATCH model). These two modes must not be mixed.

By default, the API_DISPATCH model is used. With this model,
your application must not implement a dispatch loop but just wait for the
EMA thread to dispatch messages either by calling Thread.Sleep(…) or by doing
another activity in your main thread. As Nick suggested you will find more about this in the Requesting MarketPrice Data tutorial

If you want to dispatch messages from your own thread(s),
then you must tell EMA you’re using the USER_DISPATCH model and you must
implement a dispatch loop as the one you’ve done:

consumer = EmaFactory.createOmmConsumer(config
                .host(props.getProperty(MMRatesConstants.HOST_NAME_PROP))
                .username(props.getProperty(MMRatesConstants.USER_NAME_PROP))
                .operationModel(OperationModel.USER_DISPATCH));
...
while (true) 
{
    ....
    consumer.dispatch(1000);
}

So you have to choose.

Either you use API_DISPATCH model and then you replace your dispatch loop by a Thread.Sleep() or another activity.

Or you tell EMA you use the USER_DISPATCH model as shown
below and you keep your dispatch loop.

The second (potential) problem is that your code snippet creates
an OmmConsumer and a dispatch loop for a single instrument. As Umer said, if
you register to 600 instruments you should create one consumer and one loop for
each of them. It may cause performance issues.

I hope this clarification will help

Lars-at-Addicticks

Would it be possible for you to reformat the question (use the features provided), so that it gets easier to read?

zoya faberov

@Pulaputra.Sudheer

Believe dispatching is done by EMA for you, is not necessary, and also eats a bit of CPU.

Where you have

"consumer.dispatch(1000) "

try

"while (true)

Thread.sleep(1000);"

instead. Do you see the decrease?

A double-check regarding "if (MMRatesConstants.IS_MARKET_UPDATED){"

You are not checking a constant, that does not change, you are probably checking if your fid of interest was updated, right?

umer.nalla

Hi @Pulaputra.Sudheer

Are you creating a separate OmmConsumer and/or separate appClient for each of your 600 rates?

I ask the above because your question could be read to suggest that you are doing the above.

If you are, then be aware that you should not (need to) do so. A single instance of OMMConsumer can be used to subscribe all 600 instruments (by using Batch or repeatedly calling registerClient for all your instruments) and a single instance of the AppClient can be used to process all 600 instruments (by specifying the same client in the registerClient call).

nick.zincone

@Pulaputra.Sudheer,

For a basic guideline in building an EMA Consumer application, I would follow the tutorials within the Developer Portal. As mentioned in the other answers, EMA can automatically perform dispatching for you. However, this will occur on it's own thread. If you prefer to manage everything within your own thread, you can override this default behavior. Specifically, refer to the Requesting MarketPrice Data tutorial within the portal that discusses the different Operation Models within the "Message Processing - dispatch" section. By default, the Api Operation Model is defined. Alternatively, you can override this by specifying a User Operation Model.

To setup a User Operation Model, you modify your consumer config as:

config.host(props.getProperty(MMRatesConstants.HOST_NAME_PROP))
      .username(props.getProperty(MMRatesConstants.USER_NAME_PROP)
      .operationModel(OmmConsumerConfig::UserDispatchEnum)
));

Once defined, you can perform the dispatching yourself.

nick.zincone

@Pulaputra.Sudheer,

Regarding CPU usage, I would encourage you to update your EMA version to the latest (1.0.8 at the time of this post). For example, in the release notes for EMA Java 1.0.8 (release 3.0.4.L1 RRG), it states:

	[EMA-828] - Occasional 100% CPU usage on Consumer applications.

I've encountered unnecessary CPU usage when running very simple EMA Java consumer applications using version 1.0.7 for example.