screening with a loop and throwing fatal errors
Hello! I am querying Refinitiv Workspace for equities based on a screen. I used to run this screen, or ones just like it, with no problem, even while using the same Python 3.11.4 that I'm using now. But now it's throwing fatal errors and not always at the same place in the loop. When I take the specific screen text that threw the error in the loop and run it on its own, it returns results with multiple companies. So the underlying screener query wouldn't seem to be a problem. So what is wrong with my below code?
import refinitiv.data as rd
rd.open_session()
import pandas as pd
indcode = [50,51,52,53,54,55,56,57,59,60,62,63]
WEurope = ["AT,BE,DK,FO,FI,FR,DE,GI,GL,GG,IS,IE,IM,IT,JE,LI,LU,MC,NL,NO,PT,SM,ES,SE,CH,GB"]
NAmerica = ["US,CA"]
CNJP = ["CN,JP"]
region = WEurope + NAmerica + CNJP
CATfields = ['TR.CommonName','TR.HeadquartersCountry','TR.HQCountryCode']
syntax = 'SCREEN(U(IN(Equity(primary,public))),\
IN(TR.TRBCEconSectorCode,{j}), IN(TR.HQCountryCode,{i}),\
TR.TRESGScore(Period=FY0)>0)'
dataCAT2 = []
df_CAT_list = []
for j in range(len(indcode)):
for i in range(len(region)):
dataCAT1 = rd.get_data(syntax.format(i=region[i],j=indcode[j]),CATfields)
dataCAT2.append(dataCAT1)
df1 = pd.concat(dataCAT2)
print(df1)
Very oddly, I can get the screen to run successfully in my global environment rather than virual (i.e., the fatal errors are only thrown when I'm in the virtual environment). This makes me wonder whether I have some package inconsistency error -- but having just installed pandas and refinitiv-data today, that wouldn't seem to be the problem.
What do you think is wrong with my code?
Best Answer
-
Hi, @LRE42 ,
For the use case with region = "GB" server returns empty data with code 200.
{"responses":[{"columnHeadersCount":0,"data":[],"rowHeadersCount":0,"totalColumnsCount":0,"totalRowsCount":0}]}
There is difference between RD versions you run in global and local env.
In RD 1.0 we have the logic to log errors but not raised the exceptions at all.
In RD 1.4.0 we have approach we decided to let users know about server-side issues.
- raise no exception in case when any of requests under the hood of get_history() at least have partial data -> build Dataframe.
- raise an exception if all requests failed;
- raise an exc if it was only one request (for example to Datagrid endpoint) and it failed.
In current case we raised an exception because datagrid platform returned response without header, no possible to build Dataframe with data.
In the next release we will make a fix to build empty dataframe that avoid exceptions in case of headers and data absence.
I will let you know about the new version.
Thank you.
0
Answers
-
Hi @LRE42 ,
May I ask: (i) what IDE are you using to run all this (e.g.: CodeBook, VSCode, ...), (ii) what session are you using (info on this here) (guessing from the code above, in your question, I think you're using the desktop session and that's causing the issue), (iii) how does your virtual environment (e.g.: with VSCode) limit proxy usage (I believe that's where the issue is and your virtual env. is limiting access to the desktop application proxies needed for authentication to LSEG API services in desktop session mode), and (iv) what error do you get exactly?
1 -
Hi @jonathan.legrand -- thanks for responding. Here I will answer your questions:
i. VSCode is the IDE
ii I am using desktop session
iii I am using a virtual environment with VSCode. I don't know what limits it has. Would this be an issue with my local network? Something I noticed after asking my initial question is that while my virtual environment has RD 1.4, my global environment has RD 1.0, which may be relevant.
iv Here is the full text of the error message:
RDError Traceback (most recent call last)
Cell In[1], line 20
18 for j in range(len(indcode)):
19 for i in range(len(region)):
---> 20 dataCAT1 = rd.get_data(syntax.format(i=region[i],j=indcode[j]),CATfields)
21 dataCAT2.append(dataCAT1)
22 df1 = pd.concat(dataCAT2)
File [local address redacted]\.venv\Lib\site-packages\refinitiv\data\_access_layer\get_data_func.py:98, in get_data(universe, fields, parameters, use_field_names_in_headers)
96 if exceptions and all(exceptions):
97 except_msg = "\n\n".join(exceptions)
---> 98 raise RDError(-1, except_msg)
100 hp_and_cust_inst_data = HPAndCustInstDataContainer(stream_columns, stream_data, stream_df)
101 adc_data = ADCDataContainer(adc_raw, adc_df, fields)
RDError: Error code -1 | Backend error. 400 Bad Request Requested universes: ['SCREEN(U(IN(Equity(primary,public))), IN(TR.TRBCEconSectorCode,54), IN(TR.HQCountryCode,CN,JP), TR.TRESGScore(Period=FY0)>0)']. Requested fields: ['TR.COMMONNAME', 'TR.HEADQUARTERSCOUNTRY', 'TR.HQCOUNTRYCODE']0 -
Hi @jonathan.legrand , I accidentally posted as an answer instead of a comment - I hope you'll still see my response.
0 -
Hi @LRE42, Are you able to use any rd.get_data call at all in your virtual environment at all? I have the impression that your VSCode environment blocks incoming data in packages larger than a specific amount. This could be a setting in VSCode maybe?
0 -
Hi @jonathan.legrand, thanks again. To answer your question, yes, I am able to download data for several loop iterations. It's not always the same loop iteration that throws the fatal error.
Below I show some code. First I ran it with the long list of countries, and then I ran it with just GB. Both of these failed. I would think that the combination of one country and industry code wouldn't produce "too" much data, but that one also threw the same error.
import refinitiv.data as rd
rd.open_session()
import pandas as pd
CATfields = ['TR.CommonName','TR.HeadquartersCountry','TR.HQCountryCode']
syntax = 'SCREEN(U(IN(Equity(primary,public))),\
IN(TR.TRBCEconSectorCode,{j}), IN(TR.HQCountryCode,{i}),\
TR.TRESGScore(Period=FY0)>0)'
region = "AT,BE,DK,FO,FI,FR,DE,GI,GL,GG,IS,IE,IM,IT,JE,LI,LU,MC,NL,NO,PT,SM,ES,SE,CH,GB" # ran this one first
region = "GB" # ran this one second
indcode = "62"
df2 = rd.get_data(syntax.format(i=region,j=indcode),CATfields)
print(df2)(I wasn't able to post this as a comment so trying now as an answer)
0 -
Thank you. I have installed v1.0 in my virtual environment and was able to run screening queries without any fatal errors. So that has solved my problem for now. It will be good to have the updated version when this is fixed in the next RD release. Thank you0
Categories
- All Categories
- 6 AHS
- 37 Alpha
- 161 App Studio
- 4 Block Chain
- 4 Bot Platform
- 16 Connected Risk APIs
- 47 Data Fusion
- 30 Data Model Discovery
- 608 Datastream
- 1.3K DSS
- 577 Eikon COM
- 4.9K Eikon Data APIs
- 7 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- Trading API
- 2.7K Elektron
- 1.3K EMA
- 236 ETA
- 519 WebSocket API
- 33 FX Venues
- 10 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 20 Messenger Bot
- 2 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 59 Open Calais
- 264 Open PermID
- 39 Entity Search
- 2 Org ID
- PAM
- PAM - Logging
- 8.4K Private Comments
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 20 RDMS
- 1.4K Refinitiv Data Platform
- 367 Refinitiv Data Platform Libraries
- 3 Refinitiv Due Diligence
- LSEG Due Diligence Portal API
- 3 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.1K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 10 World-Check Customer Risk Screener
- 990 World-Check One
- 44 World-Check One Zero Footprint
- 45 Side by Side Integration API
- Test Space
- 3 Thomson One Smart
- 1.2K TR Internal
- Global Hackathon 2015
- 2 Specialists Who Code
- 10 TR Knowledge Graph
- 150 Transactions
- 142 REDI API
- 1.7K TREP APIs
- 4 CAT
- 21 DACS Station
- 117 Open DACS
- 1.1K RFA
- 103 UPA
- 172 TREP Infrastructure
- 224 TRKD
- 886 TRTH
- 5 Velocity Analytics
- 5 Wealth Management Web Services
- 59 Workspace SDK
- 9 Element Framework
- 5 Grid
- 13 World-Check Data File
- Yield Book Analytics
- 46 中文论坛