Caching of OMMStreamCache objects using a lot of memory
We have a use case where we allow users to scan market data for instruments that meet certain criteria. These scans can request historical data snapshots across 1000-2000 instruments or more. Memory profiling shows that after performing a scan of almost 2000 instruments the OMMConnection object was using 158MB of memory (see attached screenshot from dotMemory).
Is there a way to not cache OMMStreamCache objects if it is a snapshot request, or could the amount of data held in memory be reduced? Here is our code when making a historical snapshot request...
public IObservable<WorkspaceSnapShotStreamingResponse> GetTimeSeriesDataSnapShot(
ISession workspaceSession,
WorkspaceSnapShotStreamingRequest request,
WorkspaceMetadataResponse metadataResponse)
{
return Observable.Create<WorkspaceSnapShotStreamingResponse>(observer =>
{
SerialDisposable activeSubscriptionToken = new SerialDisposable();
if (workspaceSession.OpenState == Session.State.Opened)
{
// Build the request
var summariesDefinition = BuildRequest(request);
// Make the request
activeSubscriptionToken.Disposable = summariesDefinition
.GetDataAsync(workspaceSession)
.ToObservable()
.Subscribe(OnSubscribeAction, observer.OnError);
// Deal with the response
void OnSubscribeAction(IDataSetResponse workspaceResponse)
{
if (workspaceResponse.Data != null)
{
if (workspaceResponse.Data.Table != null)
{
observer.OnNext(ConvertDataTableToWorkspacePriceRecords(
workspaceResponse.Data.Table,
request.Interval,
request.View,
TimeSeriesUpdateType.Update));
observer.OnCompleted();
}
else
{
Logger.Warn(workspaceResponse.Data.Errors);
observer.OnNext(new WorkspaceSnapShotStreamingResponse());
}
}
}
}
return activeSubscriptionToken;
});
}
private ISummariesDefinition BuildRequest(WorkspaceSnapShotStreamingRequest request)
{
var summariesDefinition = Summaries
.Definition(request.Symbol)
.Interval(ResponseHelper.ConvertInterval(request.Interval, Logger))
.Fields(request.View.MapTo.ToArray());
if (request.Count > 0)
{
summariesDefinition.Count(request.Count);
}
else if (request.RequestRange.StartDateTime.HasValue)
{
summariesDefinition.Start((DateTime)request.RequestRange.StartDateTime);
}
else if (request.RequestRange.EndDateTime.HasValue)
{
summariesDefinition.End((DateTime)request.RequestRange.EndDateTime);
}
return summariesDefinition;
}
Best Answer
-
If by snapshot you mean a historical pricing snapshot using .GetData(), then my suspicions is that there may be other requests going out within your application the uses .GetStream(), via historical or Pricing. For example, if I were to write a small, standalone example that only uses .GetData() for the historical summaries interface, there would be no streaming connection nor any requests for streaming data in this scenario. If you are seeing the opposite, can you put together a small test example our team can try to replicate? Or, maybe you can try one of the github examples to confirm.
thanks.
0
Answers
-
The OMMStreamCache is a container managing open stream IDs and references as opposed to holding a cache of the data objects. If you are asking for snapshots, the OMMStreamCache will clean up the references once a refresh or status has been sent. In theory, the OMMStreamCache should be clean after fulfilling a request. There may be something else dangling within this container.
I can bring it to the attention of the development team to observe.
thanks.
0 -
If you want to perform a streaming snapshot, then you should make a call something like this:
var snap = Pricing.Definition("CAD=").Fields("BID", "ASK", "DSPLY_NAME")
.GetStream().Streaming(false);When you specify .Streaming(false) this will automatically clean up the cache with the details I mentioned above. The default is: .Streaming(true) and this will keep the streams open for every item requested.
Now, if you're mixing this with historical streaming bars, by definition, the historical streaming bars keep the stream open. I don't fully understand what you are trying to retrieve given that a historical snapshot using .getData() already gives you the latest values within the bars. But if your requirement is to retrieve latest realtime values, then you can perform the snapshot based on the code segment I provided above.
0 -
Thanks Nick, we'll give that a try.0
-
For our use case we need to retrieve historical data (~1250 records) for each one of the ~2000 instruments. It's the historical data request using the Summaries API that is holding open the OMMStreamCache object so setting Streaming to false won't help us much. Is there anything else that might help?0
-
Hi @cory.schmidt.1
How are you retrieving historical data? Are you using .GetStream() and opening? Or are you simply using .GetData()? If you're using .GetData(), there is no attempt to open any stream.
0 -
We're calling .GetDataAsync() to retrieve history. That's what prompted this question... it seemed odd to have an IOMMItemStream object created and held in memory for each instrument when we are requesting a snapshot not a stream.
0 -
I'm still working on getting a sample put together to illustrate the problem. Stay tuned...0
-
@nick.zincone You were correct about having requests that were creating subscriptions. We had a section of code checking QoS for each requested instrument creating a stream. Once we added Streaming(false) to get a snapshot instead then the memory consumption was greatly reduced. Thanks for the feedback.1
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛