How to recover from encodeSummaryDataComplete failure?
If when encoding a map or a vector you call encodeSummaryDataComplete(encodeIterator,true) and you receive a BUFFER_TOO_SMALL the encodeIterator becomes 'broken' and you cannot complete the encoding.
,
Our application expects to be able to encode a message in a single pass. In order to avoid unnecessary garbage collection we allocate a fixed number of buffers at application start up. If a message cannot fit within a buffer we expect to get a CodecReturnCode.BUFFER_TOO_SMALL, when we get that return code we start calling encodeComplete(encodeIterator,true) on all the nested containers, we remember where we are in the published data so we can start a continuation message re encode the nested containers and continue processing the data. Also If we get BUFFER_TOO_SMALL on creating a container we can roll it back with encodeComplete(encodeIterator,false).
BUT we can also get BUFFER_TOO_SMALL when we call encodeSummaryDataComplete(encodeIterator,true). After we receive this result code the encodeIterator seems to be 'broken', that is we cant callEncodeComplete to produce a message, The summaryData container has been encodeCompleted(true) so cant be undone either.
If you are encoding a Map/Vector with Map/Vector entries it gets more serious because failing to complete a summarydata on one of the Entry payloads causes the whole message to fail, including all the previous entriess, which in our case we dont have any more.
Best Answer
-
In UPA C, when rsslEncodeMapSummaryDataComplete(encIter, RSSL_TRUE) returns RSSL_RET_BUFFER_TOO_SMALL, the application should calls rsslEncodeMapComplete(encIter, RSSL_FALSE) in order to roll back to the last previously successful encoded point in the contents.
if ((retVal = rsslEncodeMapSummaryDataComplete(encIter, RSSL_TRUE)) < RSSL_RET_SUCCESS)
{
printf("Error %s (%d) encountered with rsslEncodeMapSummaryDataComplete(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
if (retVal == RSSL_RET_BUFFER_TOO_SMALL)
{
retVal = rsslEncodeMapComplete(encIter, RSSL_FALSE);
…
…Then, you can use rsslRealignEncodeIteratorBuffer to assign a new larger buffer.
//newBuffer contains a new larger buffer
retVal = rsslRealignEncodeIteratorBuffer(encIter, &newBuffer);After that, you need to call rsslEncodeMapInit() to start encoding map and summary data again.
if ((retVal = rsslEncodeMapInit(encIter, &rsslMap, 0, 0 )) < RSSL_RET_SUCCESS)
{
/* error condition - switch our success value to false so we can roll back */
success = RSSL_FALSE;
/* print out message with return value string, value, and text */
printf("Error %s (%d) encountered with rsslEncodeMapInit(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
}1
Answers
-
OK make it public, Just to note that we get the same problem with both the C and Java UPA encoders
0 -
The question has been escalated to TRDC.
0 -
In java we allocate all the buffers up front to avoid garbage collection so it is not really possible to meaningfully allocate a larger buffer.
Also our contract with the publishing application is that we only have one pass through the data.
After we have successfully encoded something we always expect to be able to perform an encodeComplete(true) on all the enclosing containers. When EncodeInit or Encode fails we
call encodeComplete(false) for that object, but then call encodeComplete(true) on all enclosing containers. We can then publish the buffer and then encode the containers and continue for the next buffer. EncodeComplete(true) should not really make the message longer because it is just populating the length of the encoded object in the message. Since we have to set the hasSummaryData flag in the Map/Vector before calling encodeInit the encoder has enough information to reserve the space up front.
0 -
@kim.valentine
From my test, Map.encodeSummaryDataComplete(true) requires few bytes to complete. For example, before calling Map.encodeSummaryDataComplete(true), buffer position in EncodeIterator is 143. After calling Map.encodeSummaryDataComplete(true), buffer position is moved to 146.
For this reason, Map.encodeSummaryDataComplete(true) can return BUFFER_TOO_SMALL.
0 -
I would like to add the answer for UPA Java for this question.
When you receive the return
code -21 (Buffer Too Small) after executing encodeSummaryDataComplete(iterator, true), you should- Roll back the
iterator to the last complete encoded position by calling encodeComplete(iterator,
false) - Realign the new
larger buffer to the iterator by calling realignBuffer function. - It is mandatory to
reinitialize the iterator again. With that, the iterator should be now ready
and can perform encoding from the last completed point.
Please
find the snippet below./* complete encoding of summary data.
If any field list encoding failed, success is false */\
if ((retVal = vector.encodeSummaryDataComplete(encIter, success)) < CodecReturnCodes.SUCCESS)
{
/* error condition - switch our success value to false so we can roll back */
success = false;
System.out.printf("Error %s (%d)
encountered with EncodeVectorSummaryDataComplete. Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal));
System.out.printf("Rolling back the summary data.....\n");
if ((retVal = vector. encodeComplete(encIter, success)) < CodecReturnCodes.SUCCESS) // Rollback with the success = false
{
System.out.printf("Error %s (%d) encountered with EncodeVectorSummaryDataComplete. Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal));
} else
{
// Realign buffer
// Creating a new larger buffer
Buffer newBuff = CodecFactory.createBuffer();
newBuff.data(ByteBuffer.allocate(1000));
if ((retVal = encIter.realignBuffer(newBuff)) < CodecReturnCodes.SUCCESS)
{
System.out.printf("Error %s (%d) encountered with realignBuffer. Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal));
} else
{
System.out.printf("SUCCESS: realignBuffer");
// From here the iterator is now larger with the previous data being copied as well.
// You can continue your operations from here.
}
}With this approach, you will need to re-encode the entire data over again since it rolls back to the very beginning of the vector you have initialized. That means the raw data should be reusable at least until the entire vector data will be encoded completely.
0 - Roll back the
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛