OMM Provider Application Crashes Periodically When Sending Status Message
Hi Developers,
We have received several reports from different customers over the past few months of our OMM provider application crashing seemingly at random. Upon examining the memory dumps provided by the customers, the common point of failure is in the rfa::sessionLayer::RSSL_Prov_ChannelSession::removeToken(rfa::sessionLayer::RSSLRequestToken*) method. This is called when posting an OMM status message that closes the stream.
Partial Stack Trace from LLDB:
frame #0: 0x00007f6193959033
libRFA_RSSL_Prov_Adapter.so`rfa::sessionLayer::RSSL_Prov_ChannelSession::removeToken(rfa::sessionLayer::RSSLRequestToken*)
+ 483
frame #1: 0x00007f6193959414 libRFA_RSSL_Prov_Adapter.so`rfa::sessionLayer::RSSL_Prov_ChannelSession::processStatusMsg(RsslMsg*, rfa::sessionLayer::RSSLRequestToken*, unsigned char, unsigned char, rfa::common::RFA_String&) + 260
frame #2: 0x00007f61932f03a8 libRFA_SessionLayer_OMM.so`rfa::sessionLayer::OMMProviderImpl::submitCmd(long, rfa::sessionLayer::OMMSolicitedItemCmd const&, void*) + 664
frame #3: 0x00007f61932f05c1 libRFA_SessionLayer_OMM.so`rfa::sessionLayer::OMMProviderImpl::submit(rfa::sessionLayer::OMMCmd const*, void*) + 337
frame #4: Our code...
From looking through the disassembly of removeToken it appears that the crash occurred outside of a critical section (pthread_mutex_unlock appears to have been called recently), so I don't know if it's some kind of race condition, but if you could let me know what the root cause might be and/or if there is a fix available I'd be very grateful.
The weirdest part is that this can happen twice in a week, then go silent for months. We've had 3 reports of this in the last year from different customers. It is noted that the changelog for 7.5.1.L1 that there was a fix for "OMM RFA provider crashes randomly in removeToken function", could this be a regression from that release or is there something else at fault (either on the RFA side or our application code)?
Other info:
- We're currently using RFA 7.6.2.E2 on Linux (RHEL6 x64 GCC44). Unfortunately we cannot upgrade to RFA8 since we need to continue supporting a limited number of RHEL5 customers as well as RHEL6.
- The crashing application is a multi-threaded OMM provider application.
- The full stack trace indicates that our application attempted to send a Status message that closed the stream (in this case, the requested item was not found on the server.
- We are not able to reliably replicate the issue in our lab tests.
I am more than happy to provide more information as required.
Many thanks.
Best Answer
-
A RequestToken is considered closed and deleted when the provider finishes processing the close request from the consumer or the provider application closes the stream (i.e. send StreamState Closed message).
It is the provider application responsibility to avoid using the deleted RequestToken.
RFA does attempt to catch and fire an error message when the application attempts to use the invalid RequestToken but it may cover every situation.
From the Dev guide section 11.3.3.2 Request Tokens
If an application attempts to send data on a request token that is closed, it may receive an OMMCmdErrorEvent in response. The application will receive the error event one time per provider on the first submit after the token has been closed. All data sent on a closed request token will be dropped.
Providers should not attempt to submit data in the following cases:
• After the associated client session has become inactive
• After the application has unregistered the client session associated with the token
• After a complete refresh has been submitted for a non-streaming request
• After the provider has received and processed a close request for an item
0
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛