IProvider - Managing temporary unavailability of data

Paolo Parlapiano

Hi,

please help me with this case.

Scenario: Interactive Provider providing data to downstream consumers.

For reasons external to the application, the Interactive Provider may not have yet the items requested by the consumers (e.g. in startup scenarios).

If in this timeframe requests are received, they are replied with a "Closed" status.

Note: the provider is aware of the structure of an item (i.e. the fields composing the RIC) only when this item is eventually known to it.

In this specific scenario, the subscribers are not able to recover from the "Closed" status, e.g. by re-trying the subscription at a later time.

As per my understanding, the user cannot leverage the "notInCacheData" feature of the ADS to mitigate the issue, as it applies to NonInteractive Providers only.

Is there any alternative / recommendation (including possibly changing the application logic) that could help in this scenario?

Thanks

Regards,

Paolo

Gurpreet

Hi @Paolo Parlapiano,

I think you already know the issue here - since its an interactive provider, the infrastructure (ADS) really doesn't play much role in terms of state management. A subscriber might directly connect to the provider, and it will respond with "closed" status since the item doesn't exist in its cache yet.

One way to mitigate this issue is to not start listening for connections (keep iprovider down) until the upstream is ready to provide data and the items universe is known.

Paolo Parlapiano

Thanks,
@Gurpreet,

but unfortunately in this case there's no event or condition signaling that the upstream data has been completely received, I'll need to find another way to handle the issue.

Regards,

Paolo

wasin.w

Hello @Paolo Parlapiano

Is it possible to send the Open/Suspect status message like "Open / Suspect / None / 'Service not available" to the consumer while the upstream data is not ready?

Paolo Parlapiano

Hi @wasin.w,

are you suggesting replying with a status message to the request?

Is it fine?

I may be wrong, but I remember to have read somewhere that the recommendation is always to reply with a refreshMessage to a request?

In this case, I would not know the set of fields of the item, so either I implement something like the "notInCacheData" ADS feature, or I provide an empty fieldSet? Does it make sense?

Thanks

Regards,

Paolo

wasin.w

Hello @Paolo Parlapiano

About replying with the status message while the upstream data is not ready, I got the idea from the ADS behavior that returns the "Open / Suspect / None / 'Service not available" status message to the consumer if the service is not available (down or not exist). Once the service is up, the API on the consumer side automatic recovery subscription for the consumer application.

Example with IProvider example 100 --> ADS --> EMA Consumer scenario:

< -- IProvide is not start -->
StatusMsg
    streamId="5"
    domain="MarketPrice Domain"
    state="Open / Suspect / None / 'Service not available'"
    name="USDSWFSR="
    serviceName="DIRECT_FEED"
StatusMsgEnd
< -- IProvide start -->
RefreshMsg
    streamId="5"
    domain="MarketPrice Domain"
    solicited
    RefreshComplete
    state="Open / Ok / None / '*Refresh Completed'"
    itemGroup="00 01"
    name="USDSWFSR="
    serviceId="10004"
    serviceName="DIRECT_FEED"
    Payload dataType="FieldList"
        FieldList FieldListNum="0" DictionaryId="0"
            FieldEntry fid="22" name="BID" dataType="Real" value="39.90"
            ....

I am not sure if it matches your scenario.

However, according to the ETA Java Development guide document, the Provider should send a StatusMsg to indicate the reason and close the stream if it cannot satisfy the request.

At last, I think it is based on your IProvider and Consumer requirements/agreements about how to handle each message scenario.

Paolo Parlapiano

Hello @wasin.w,

thanks for the reply, but If I understood correctly, I'm afraid that this approach would not work on my side: in my scenario, some of the RICs may be available, and some others may not be, so replying with a status like "Service not available" will probably be incorrect/misleading here, especially if the requests come from the same consumer. I mean, If you imply changing the status of my service (so to leverage the auto-recovery mechanism later), I think I should not do it in this case..

In the meantime, I'm exploring the option of sending blank images, in a way similar to the Source Distributor, so by sending replies like "DSPLY_NAME, Not In Cache", since the consumers of my scenario work fine with this functionality.

A first PoC on this appears promising, and working as usual after the target item becomes eventually available..Anyway I have a question about this option...once the data is available, the provider sends a solicited refresh message to all the active clients, with the actual data: should I set the "clearCache" flag of the refresh message to true so that the consumers can remove any cached data of the blank image ("DSPLY_NAME, Not In Cache")?

Thanks

Best Regards,

Paolo

wasin.w

Hello @Paolo Parlapiano

Thank you for the clarification.

According to the EMA API RDM Guide document, the application should clear the cache when performing a refresh as follows:

However, I think this unsolicited refresh message is suitable for this "missing data is now available" scenario because it is a refresh message that updates data of the previous solicited refresh message (contains "not in cache").

wasin.w

Hi @Paolo Parlapiano

The first picture above comes from the EMA J RDM Usage Guide document.

The second picture comes from the Transport API J Developers Guide document.

If you are using RTSDK C/C++, the documents are available here.

Paolo Parlapiano

Hi @wasin.w,

many thanks for the details.

I was thinking to send the refreshMessage with actual data with clearCache=true because, if the actual data does not carry any value for DSPLY_NAME, the final image on the consumer could still contain DSPLY_NAME="Not In Cache" after receiving this message, does it make sense as a reasoning?

wasin.w

Hello @Paolo Parlapiano

Do you mean this scenario?

1. The provider sends the first refreshMsg with DSPLY_NAME="Not In Cache" to downstream
2. The provider gets DSPLY_NAME data from its backend
3. The provider then sends a second refreshMsg with DSPLY_NAME="Value" and "clearCache" flag to downstream

If my assumption is correct, I think the 2nd refreshMsg (step 3 above) should be

• Refresh Message
• Indications: "clearCache" flag = True
• Solicited = false (unsolicited)
• Payload: DSPLY_NAME="Value" ,...

Paolo Parlapiano

Hi
@wasin.w,

yes, exactly.

In my scenario, I was also supposing that the step 2 is not present at all (i.e. the case of blackend providing a set of fields not containing at all the DSPLY_NAME one).

Anyway, apart from this, I'm implementing precisely the logic that you reported in the step 3 (unsolicited refresh message, with clearCache set to True).

Thanks

Best Regards,

Paolo