Performance Issue with Delay in Batch Request Processing In EMA API
log.txtI am currently conducting a performance test using the EMA API and have encountered an issue with the timing of batch request processing. My test involves sending 40 batch requests simultaneously, each containing 22 symbols.
I received all 40 status messages indicating "Close/OK/None/Stream closed for batch" at 16:30:06.xxx.However, I observed that the last outgoing reactor message(the request message) was sent to the EMA API at 16:30:10, which is 4 seconds later than the status messages. and the last incoming message(the refresh message) was received at 16:30:11. This delay seems to indicate that not all request messages are being sent simultaneously, resulting in a 5-second lag in receiving the final refresh message. My goal is to optimize this process so that all request messages are sent without this lag, allowing for earlier reception of refresh messages.
Are there specific settings within the EMA configuration that I can modify to enhance the performance and reduce the delay in message processing.
I have additional observation from the log files. At the start, the application sends about 10-20 requests to the EMA API. After receiving the first response(the refresh message), the application shifts to a pattern of sending a request and then processing a response before sending the next.
I'm using EMA JAVA 3.7.2.
Best Answer
-
Thank you for reaching out to us.
I assume that each batch contains the same items and you are using snapshot requests.
Typically, EMA supports watch list and fanout so it sends one request message for duplicated subscriptions. Then, it can fanout the retrieves messages to all subscriptions.
However, according to the log, some subscriptions may be overlapped. For example, the application may register the same item after the request message was sent. Therefore, EMA will sent another request after receiving the refresh message, as shown below.
<!-- rwfMajorVer="14" rwfMinorVer="1" -->
<REFRESH domainType="MARKET_PRICE" streamId="8" containerType="FIELD_LIST" flags="0x1FA (HAS_PERM_DATA|HAS_MSG_KEY|HAS_SEQ_NUM|SOLICITED|REFRESH_COMPLETE|HAS_QOS|CLEAR_CACHE)" groupId="0" seqNum="29168" permData="0301 0184 C0" Qos: Realtime/TickByTick/Static - timeInfo: 0 - rateInfo: 0 State: Non-streaming/Ok/None - text: "*All is well" dataSize="141">
<key flags="0x07 (HAS_SERVICE_ID|HAS_NAME|HAS_NAME_TYPE)" serviceId="257" name="AFN.TO" nameType="1"/>
<dataBody>
<fieldList flags="0x09 (HAS_FIELD_LIST_INFO|HAS_STANDARD_DATA)" fieldListNum="0" dictionaryId="1">
<fieldEntry fieldId="6" data="094E 9530"/>
...
</fieldList>
</dataBody>
</REFRESH>
<!-- Outgoing Reactor message -->
<!-- java.nio.channels.SocketChannel[connected local=/10.57.137.133:64548 remote=/ 10.215.149.140:14002] -->
<!-- Wed Nov 29 18:57:32 EST 2023 -->
<!-- rwfMajorVer="14" rwfMinorVer="1" -->
<REQUEST domainType="MARKET_PRICE" streamId="8" containerType="ELEMENT_LIST" flags="0x442 (HAS_PRIORITY|HAS_QOS|HAS_VIEW)" Qos: Realtime/TickByTick/Static - timeInfo: 0 - rateInfo: 0 priorityClass="1" priorityCount="1" dataSize="80">
<key flags="0x03 (HAS_SERVICE_ID|HAS_NAME)" serviceId="257" name="AFN.TO"/>
<dataBody>
<elementList flags="0x08 (HAS_STANDARD_DATA)">
<elementEntry name=":ViewType" dataType="UINT" data="1"/>
...I checked the EMA configurations and was unable to find any configurations that can control the number of request messages sent by the API.
If you would like to determine the performance of the API, you can refer to the EMA Performance Tools (\Java\Ema\PerfTools) in the package.
0
Answers
-
log.txt
attached is a sample of the log file0 -
Thanks for quick response. Based on the guidance about EMA's handling of duplicated subscriptions and fanout, I conducted an additional test to further investigate the issue.
I send 100 snapshot batch request, each containing only one symbol. 50 request were for BMO.TO and another 50 for TD.TO.
Observation in Logs:
The last status message indicating "Stream closed for batch" was logged at 08:02:20.809
The first refresh message was received at 08:02.20:822
The last refresh message was received at 08:02:25
The last outgoing reactor message (request message) was sent at 08:02:25
I observed a 5 second delay between the last "Stream closed for batch" status message and the sending of the last outgoing reactor request message. I expected the outgoing request messages to be send immediately after receiving the "Stream closed for batch" message, not 5 seconds later.
What could be causing this delay in sending out the request messages?
However, Sent one batch request containing 200 items. Notably, all outgoing request messages for this large batch were sent at the beginning, and the entire process was completed within 500 milliseconds
log1.txt log2.txt
0 -
upload log file again
0 -
Thank you for the information.
This forum is aimed at answering "how to" types of questions about using Refinitiv APIs. To verify the behavior in the API regarding performance, please contact the API support team direclty via Contact Premium Support.
However, you need to be a RDC named user in order to access Contact Premium Support. You can contact your Refinitiv account team or sales team direclty for more information regarding a RDC subscription (Refinitiv Developer Connect).
Otherewise, you can raise this issue to the development team directly via GitHub.
0
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛