How to identify "truncated" responses?
Hello everybody,
I am trying to obtain some data via EIKON Data API and Python. Taking a closer look at my obtained result, I observed the following: from time to time after a different amount of "complete responses", I only obtain results for some of my requested fields. I attached a Screenshot to show you what I mean. Does this mean that the very specifc request has been "truncated" at that point (as described in the API documentation)?
In the documentation it says:
"When the limits for datapoints per request are reached, responses are simply truncated and only the available cells/headlines/results are returned ..."
kind regards
Best Answer
-
@s2782245 So the Eikon Data API get_data api has a limit of 10K datapoints per call - if your request is larger than this it attempts to deliver the maximum possible and will NA the rest. You need to break up your request into smaller chunks and iterate those calls to build larger dataframes. Alternatively - and I would suggest you do - use our Refinitiv Data Libraries to retrieve data - these can do the iteration for you depending on the function for example get_history. You can play around with the API in the Codebook app (type CODEBK into Eikon Search bar). Go to the examples folder and look at the Access layer samples for both get_data and get_history. I hope this can help.
0
Answers
-
Hi,
thank you very much for your responses.
@wasin.w - yes I can replicate the issue on demand.
In the code below I reproduce the issue for the Stock with the ISIN "CNE100004116". What the code essentially does is taking the Stock, a list of TR.Functions, and a list of Parameters (here an indicator for years e.g. 0CY, -1CY, -2CY etc.) and puts them into the get.data function. Whil the Stock and the TR.Functions are held constant, a loop works through the years definied in the parameter list. Please not that I use an R wrapper for the Python library (please let me know if you want to see it in plain python; however the logic is the same).
Example Code for "CNE100004116":
#### Setting up the connection ####
eikonapir::set_proxy_port(9000L)
eikonapir::set_app_id('XXXXXX')
#### Creating Year Parameter ####
year_df <- as.data.frame(0:-25)
year_df <- year_df %>% rename("yearindikator"="0:-25")
year_df <- year_df %>%
mutate("yearindikator"=paste(year_df$yearindikator,"CY",sep ="" ))
#### Reuters Functions to be pulled ####
investors_reuters_function <- list(
"TR.InstrumentType",
"TR.CompanyName",
"TR.FreeFloatPct",
"TR.InvestorFullName.investorpermid",
"TR.InvestorFullName",
"TR.HoldingsDate",
"TR.EarliestHoldingsDate",
"TR.SharesHeld",
"TR.PctOfSharesOutHeld",
"TR.SharesHeldValue",
"TR.InvestorType",
"TR.InvParentType",
"TR.InvInvmtOrientation",
"TR.FilingType",
"TR.ConsHoldFilingDate",
"TR.NbrOfInstrHeldByInv",
"TR.InvAddrCountry",
"TR.NbrOfInstrBoughtByInv",
"TR.NbrOfInstrSoldByInv")
#### Example-ISIN for truncated response ####
example_truncated_stock <-"CNE100004116"
#### Main loop to get the needed data ####
for (q in 1:nrow(year_df)){
# This loop now takes one "Parameter Item" from the year_df created earlier
# and attaches it to the "get_data" function
### To avoid overload
tmsleep<- sample(1:6,1)
Sys.sleep(tmsleep)
print(q)
next_df <-
get_data(example_truncated_stock,
investors_reuters_function,
parameters = list("SDate" = year_df[q,1]))
### As sometimes, empty responses are received for no obvious reason,
### a while loop is implemented to try again
counter <- 0
while(dim(next_df)[1]==0 & counter <5){
counter <- sum(counter,1)
tmsleep<- sample(1:5,1)
Sys.sleep(tmsleep)
print(counter)
next_df <-
get_data(example_truncated_stock,
investors_reuters_function,
parameters = list("SDate" = year_df[q,1]))
}
### Saving the answers
savingsname <- paste("CNE100004116",".csv",sep="")
savingspfad <- paste("P:/17 ...",savingsname,sep="")
write_delim(next_df,
savingspfad,
delim = ";",
append = TRUE,
col_names = !file.exists(savingspfad)
)
}@jason.ramchandani01 Thank you very much for your answer. Is there any possibility to know beforehand whether the answer will exceed the 10.000 datapoint limit? Having checked the obtained files, i noticed that the "truncated repsonses" start way before even being close to the 10.000 datapoint limit....
I also have attached an image of the output file for the stock mentioned. As I marked in yellow, the answers start to be "truncated" already after 2536 datapoints...
Kind regards
0 -
Hello @s2782245
I am not sure why you get "truncated" response. Did you try to break your request into smaller chunks like @jason.ramchandani01 suggested yet?
I noticed that you are using the eikonapir library. Please be informed that the eikonapir is not Refinitiv product, you may need to contact the library developer directly via the GitHub issue page.
- Note: The official Eikon API product is Eikon Data API (Python) and Refinitiv Data Libraries (Python, .NET, TypeScript)
0
Categories
- All Categories
- 6 AHS
- 39 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 60 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛