EMA memory usage increase

leonardo.pupacic
leonardo.pupacic
in EMA

The issue:

Increase in memory usage over time, signifying a possible memory leak.


Platform and software:

Refinitiv Real-Time-SDK 2.2.1, EMA C++ on Linux


The EMA usage profile:

  1. ~3 million RICs subscribed for MMT_MARKET_PRICE, spread over 10 Edge host, with two connections to each, i.e. 20 client channels
  2. Subset of that RIC universe is also subscribed to MMT_MARKET_BY_PRICE and MMT_MARKET_MAKER
  3. RefreshMsg, StatusMsg and UpdateMsg copied using their copy-constructor, in the client callback, for processing on separate threads.

The behavior:

  • Base case
    • Items 1 and 2 remained as is.
    • For item 3 above, not making copies, simply receiving callbacks and doing nothing, results in small non-increasing memory footprint.
  • Fast increase in memory usage
    • All three items above remained as is.
    • Occurs when exceptions are thrown by EMA with high frequency, e.g.:
      • Exception Type='OmmInvalidUsageException', Text='The enum value 2315 for the field Id 374 does not exist in the enumerated type dictionary', ErrorCode='-22'
      • Exception Type='OmmInvalidUsageException', Text='The enum value 1115 for the field Id 374 does not exist in the enumerated type dictionary', ErrorCode='-22'
      • Exception Type='OmmInvalidUsageException', Text='Failed to convert to UTF8 in RmtesBufferImpl::toString(). Reason: RSSL_RET_FAILURE', ErrorCode='-1'
  • Slow increase in memory usage
    • All three items above remained as is.
    • Occurs when no exceptions are thrown
    • There is nothing in the logs to signify an issue

Investigation is ongoing on our end to profile the application to try to identify the source of increase in memory usage.

Tagged:

Answers

  • Gurpreet
    Gurpreet

    Hello @leonardo.pupacic,

    You have correctly identified that the memory leak is happening somewhere in the event loop, where somehow copied objects or some other data are not being released into the heap. Only suggestions I can offer are:

    1. Try an EMA example with the same set of instruments - this will rule out the memory leak in the SDK

    2. Use profilers to identify which objects are not being reclaimed.

Categories