Serialize/Deserialize RespMsgs in RFA.NET
Is there a way how to (de)serialize entire RespMsg in RFA.NET so that those can be efficiently (means without generating GC garbage and without unnecessary processing) stored and then decoded later?
The use case behind this question is logging. We need to log each incoming message (along with our receiving and processing time stamps). Logs are not expected to be human readable. But they are expected to contain all needed information for investigations - means we need to be able to retrieve same info that the application did.
Manual transforming of the message to string (similar to example app) is a very inefficient way how to achieve this.
One solution to this seems to be by using
\Connections\<connectionName>\traceMsgToFile = true
This however seems to log messages in very verbose format, plus we are unable to add our application timestamps
Best Answer
-
For incoming data you can grab the encoded buffer backing the RespMsg with getEncodedBuffer() then insert that into a binary log, e.g. GZIP-enabled Protobufs works well at 100k msg/s+ rates.
This buffer then would be independent of transforms and dictionary issues and can be re-decoded post-facto to replay any issues.
RFA/C++ example with SSL/Marketfeed into a Protobuf coded stream.
0
Answers
-
Great answer!
Before marking this as answer - do you have any suggestion how to get content of the ThomsonReuters.RFA.Common.Buffer without incurring memory allocation?
I'm missing some some 'CopyTo(byte[] preallocatedArray)' method.
The only workarounds I could think off involves accessing the private _Buffer member with reflection or pointers casting and then accessing it in order to manually copy to preallocated buffer.
0 -
I'm posting as separate answer since comment legth restriction won't let me post the code sample. The main credit still goes to Steven McCoy.
Here is the hack code that I stitched together to copy the RFA Buffer without memory allocations. It relies on knowledge of internals of the Buffer implementation, however until there is some 'CopyTo' method, I guess that that's the only option.
public static class RfaBufferUtils
{
public static unsafe void CopyRfaBufferToBuffer(ThomsonReuters.RFA.Common.Buffer rfaBuffer,
byte[] destination, out int length)
{
length = 0;
if(destination.Length < rfaBuffer.Size)
throw new ArgumentException("Insufficient buffer");
FieldInfo fi = typeof (ThomsonReuters.RFA.Common.Buffer).GetField("_Buffer",
BindingFlags.NonPublic | BindingFlags.Instance);
var bufferPtr = fi.GetValue(rfaBuffer);
//rfa.common.Buffer* bufferPtr
long pointrValue = *(long*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 8);
if (pointrValue == 0L)
{
destination[0] = 0;
return;
}
length = *(int*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 20);
Marshal.Copy(new IntPtr((void*) pointrValue), destination, 0, length);
}
}This is how it can be called:
public void ProcessEvent(Event evt)
{
RespMsg respMsg = (RespMsg)((OMMItemEvent)evt).Msg;
long before = GC.GetTotalMemory(false);
long length;
RfaBufferUtils.CopyRfaBufferToBuffer(respMsg.EncodedBuffer, _preallocatedByteArray, out length);
long after = GC.GetTotalMemory(false);
Debug.Assert(after - before == 0);
// Store the _preallocatedByteArray content as needed
//...
}0
Categories
- All Categories
- 6 AHS
- 39 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛