Refinitiv API errors
I tried running this code (attached below). The code is inconsistent in giving results. It worked in the very first attempt for 12 companies but then it throws an error when I ran it for more than 2 companies and sometimes it runs in 3rd/4th attempts. In this inconsistency problem, we can not just say that this code will run for x companies in y attempts (y<=5). Is this a server overload problem or inherent issues with this API?
Code
rd.open_session()
# Define a function to chunk a list into smaller lists of a specific chunk size
def chunk_list(first, chunk_size):
for i in range(0, len(first), chunk_size):
yield first[i:i + chunk_size]
# Set the chunk size
chunk_size = 1000
# Define a list of RICs
RIC_list = ['GE', 'ROK','TT', 'EMR', 'DHR'] #, 'SCHN.PA', 'ABBN.S', 'SIEGn.DE', 'DHR','HON.O','JCI'] #, 'ZOMT.NS', 'INAR.NS', 'TATE.NS', 'COMU.NS', 'CENA.NS',
fields1 = [
'TR.FundClassId','TR.FundPortfolioName','TR.FundParentNameHistory', 'TR.FundPortfolioName.investorid',
'TR.FundOwnershipTurnover(Sdate=2024-06-30)',
'TR.FundAdjShrsHeld(Scale=6, Sdate=2024-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2024-03-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-09-30)',
'TR.FundAdjShrsHeld(Scale=6, Sdate=2023-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-03-31)', 'TR.FundAdjShrsHeld(Scale=6, Sdate=2022-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2022-09-30)',
'TR.FundAdjShrsHeld(Scale=6, Sdate=2022-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2022-03-31)', 'TR.FundAdjShrsHeld(Scale=6, Sdate=2021-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2021-09-30)',
'TR.FundHoldingsDate(Sdate=2024-06-30)','TR.FundHoldingsDate(Sdate=2024-03-31)','TR.FundHoldingsDate(Sdate=2023-12-31)','TR.FundHoldingsDate(Sdate=2023-09-30)',
'TR.FundHoldingsDate(Sdate=2023-06-30)','TR.FundHoldingsDate(Sdate=2023-03-31)','TR.FundHoldingsDate(Sdate=2022-12-31)','TR.FundHoldingsDate(Sdate=2022-09-30)',
'TR.FundHoldingsDate(Sdate=2022-06-30)','TR.FundHoldingsDate(Sdate=2022-03-31)','TR.FundHoldingsDate(Sdate=2022-12-31)','TR.FundHoldingsDate(Sdate=2022-09-30)',
'TR.FundHoldingsDate(Sdate=2022-06-31)', 'TR.FundTotalEquityAssets','TR.FundInvtStyleCode','TR.FundInvestorType','TR.FundAddrCountry','TR.FundParentType','TR.FdShrhldrActivistType', 'TR.FundInvtOrientation','TR.FundEarliestHoldDate','TR.FundAggPE12MonthsFwd','TR.FundContactFirstName','TR.FundContactMiddleInit','TR.FundContactLastName','TR.FundestorContactTitle',
'TR.FdConPrimaryFunction','TR.FundContactEmailAddr','TR.FundContactTelCntry','TR.FundContactTelAreaCode','TR.FundContactTelExt','TR.FundContactTelNumber','TR.FundAggRevenue','TR.FundAggReturnOnEquity','TR.FundAggReturnOnAssets'
]
parameters = {}
df_combined = None
retry_max = 5
retry_count = 1
# Define a function to get data with retry logic
def get_data_with_retry(chunk, fields, parameters):
global retry_count
while retry_count <= retry_max:
try:
df = rd.get_data(chunk, fields=fields, parameters=parameters)
return df
except RDError as e:
print(f"RDError code: {e.code}")
print(f"RDError message: {e.message}")
print(f"Attempt {retry_count} failed. Retrying...")
time.sleep(2**retry_count) # Exponential backoff
retry_count += 1
raise Exception("Maximum retry attempts reached.")
# Try to retrieve data in chunks and handle errors
try:
for idx, chunk in enumerate(chunk_list(RIC_list, chunk_size)):
print('i:', idx , 'c:', chunk)
df = get_data_with_retry(chunk, fields1, parameters)
if df_combined is None:
df_combined = df
else:
df_combined = df_combined.append(df)
print(f'{len(chunk)} chunks completed')
retry_count = 1 # Reset retry count for the next chunk
except Exception as e:
print(f"An error occurred: {e}")
Error:
Any help to understand this error and make code consistently give results would be much appreciated.
Best Answer
-
Thank you for reaching out to us.
There are two types of timed out.
1. The client timed out
The client timed out can be configured via the configuration file (refinitiv-data.config.json) or the code.
{
"http": {
"request-timeout": 60
},
...
}The code is:
config = rd.get_config()
config.set_param("http.request-timeout", 60)
rd.open_session()2. The server timed out
The server timed out can't be modified. Therefore, the number of items, fields, or date range should be reduced.
According to the error, it should be the client timed out. Please try to increase the request-timeout value in the RD configurations.
1
Answers
-
Hi @Jirapongse
Thank you for your response.
I have tested the code with your above-mentioned suggestion. This helped resolve most of the errors we were facing earlier. However, we still encounter some errors infrequently. While the solution works well for smaller lists of companies, it does not scale to the level we were aiming for.
We appreciate your assistance.
0 -
Yes, it wil help the client timed out. However, it will not solve the server timed out if the application is still requesting a lot of data. Please try to reduce the number of items, fields, or requested date range in each request.
As far as I know, the API is not designed for applications to request a lot of data in one request. You may contact your LSEG account team or sales directly to discuss alternative methods.
1 -
Sure, Thank you for the clarification.0
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛