Filling EMA messages is not multi-thread-safe.
Hello
I have an interactive provider, that uses a dedicated thread per connected consumer to build up EMA messages and send them to the corresponding consumer.
Of course I use individual message objects per thread, of type RefreshMsg / UpdateMsg with a FieldList therein.
However, if the code that fills the message objects is not lock-protected, the data sent to to the consumers get corrupted (the fields in the field list get random values). If there is a lock protection arround the code, which fills the (thread-specific!) message objects, everything works fine. Obviously it is not safe, to build (different!) messages at the same time in multiple threads. Why ?!?
This is annoying, because the lock protection slows down the program.
Best Answer
-
Hello Michael @miczuc,
Off-topic, a sense of humor is not as common among developers, as one would think, and is appreciated .
On the topic:
The multi-threading in EMA gives you a lot, on consumer side of the equation, and less on the provider side of it.
Many improvements to our APIs originate with developer clients, such as yourself, and are prioritized, if several clients are interested, then the suggestion gains in priority. The other consideration is your Refinitiv account team may aid you in prioritization, for example, if your development is part of a strategic initiative, etc.
If you find something like this, we encourage you to let us know, so we can improve.
On how you'd prefer to proceed:
I can add this (in my opinion) useful suggestion, to Elektron SDK tracked issue list, as I understand it, on your behalf? If you would like your enhancement to be tracked as customer, please comment with your organization info, as you have registered on the portal with a private email address.
Or, alternatively, you can make the suggestion directly, via Elektron SDK on GitHub, including details, optionally a link to this forum, optionally adding me on the tag (zfarberov), to enable me to keep the community members interested, updated of any suggestions or progress?
0
Answers
-
Hello @miczuc,
Sorry to hear you are finding this slow.
In my understanding, the thread-safe behavior per EMA design, is centered around safe dispatch by multiple threads. The dispatch mechanisms, the callback processing, both for consumers and providers, that implemented, are described in EMA Developer Guide.
I do not believe that the thread-safeness as documented was designed to cover your requirement by the library. I would venture a guess, that creating and populating messages for the purposes of publication is not considered as strictly time-sensitive.
Therefore, what you describe would be an enhancement request rather then an expected behavior.
If on your side you consider this very important, rather then nice to have, please submit the description of requirement directly to Elektron SDK Development on GitHub so the enhancement can be considered and if accepted, prioritized.
Would it make sense for your requirements, to create and populate messages from a single thread, perhaps to partially pre-populating the contents known ahead of time?
0 -
I did't say, "I am finding this slow". I said, the need to protect the building of messages by a lock is slowing down the program. That means, the program could run faster (with higher troughput), if the building of messages was multi-thread-safe. One should expect, that building messages is multi-thread-safe ...
So maybe I have to register on the GitHub site and report my problems there ....
0 -
Hello @miczuc,
I hear you and if you would like to report this to ESDK development team, you are entitled to do so.
However, I am also thinking as to what kind of numbers or ballpark rates on the provider side you are looking to achieve, and of the best way to achieving those numbers. If you are targeting very high numbers, configuration tuning, input and output buffers tuning is usually required. It may also be considered if EMA is the right API tier for you or if ETA is a better choice.
0 -
Hello @zoya.farberov ,
you don't want to program yourself to death using the ETA C library directly.
You need a simplifying object-oriented layer on top of that like EMA.
There are many good things in the EMA library, I don't damn it.
Unforntunately EMA is not well designed concerning multi-threading.
E.g. the submit()-methods in class OmmIProviderImpl all lock the same mutex
in their interior, around all code therein. That means, sending data in multiple theads at
the same time is dramatically hindered, even if each thread sends data on a different connection
("channel"), because effectively only one thread can execute
the method at a time while the others are blocked.
Of course you need some locking to protect internal data structures,
but sending data on different connections should not interfere with each other.
So that could be done with less locking (smaller periods of time holding locks),
but that is more difficult to implement.
In my option this is another big problem, that should be reported to the development.
Regards, Michael Zuck0
Categories
- All Categories
- 6 AHS
- 39 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 60 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛