rd.get_data takes too long to load

julius.mendez

I'm trying to load certain data using get_data function from refinitiv data library but it takes too long to load. A quicker approach is using the same function name under eikon built within refinitiv.data but very unreliable as I often encounter 400 bad request error. Can you provide any better solution using the first approach?

Below is the sample code using first approach.

import refinitiv.data as rd

rd.get_data(

universe='DEA1867P=BE',

fields=['BID', 'CTBTR_1','CF_BID', 'TR.BIDPRICE', 'CF_LOW', 'BID_LOW_1']

)

raksina.samasiri

Hi @julius.mendez ,

I'm really sorry for the issue you've been facing. Ticket number 12061741 has been raised on your behalf and the technical support team is going to contact you directly to assist and investigate this issue.

Please let me know in case you have any further questions.

raksina.samasiri

Hi @julius.mendez ,

I'm sorry to hear that you're facing this issue. However, for further investigation, could you please clarify more on the takes too long to load and how long it take in this case? You could use the code below to measure it

my execution on the Codebook app took 2.71 seconds for rd.get_data() and 1.67 seconds for ek.get_data()

import time

# get the start time
st = time.time()

df = rd.get_data(universe='DEA1867P=BE',
                 fields=['BID', 'CTBTR_1','CF_BID', 'TR.BIDPRICE', 'CF_LOW', 'BID_LOW_1']
                )

# get the end time
et = time.time()

# get the execution time
elapsed_time = et - st
print('Execution time:', elapsed_time, 'seconds')
display(df)

julius.mendez

raksina.samasiri, thanks for reverting back to my query! I'm using jupyter notebook within the terminal to run python.

I included below a simple code that enabled me to run functions within set time limit in seconds. Here, I set ek_get_data which uses eikon within refinitiv.data to 30 seconds time limit and set rd_get_data which uses the get_data function directly from refinitiv.data to 300 seconds.

After running the output, I noticed that the ek_get_data() performed within expectation but rd_get_data() threw an TimeoutError denoting that the function was stopped as it continued to run for more than 300 seconds without providing the return output.

I could go on and extend the time limit for as long as I want but the fact that you were able to run rd.get_data for less than 10 seconds means that there something wrong with the rd.get_data when I run it on our end.

Is there any reason that you can think of as to why this is happening?

Current refinitiv-data library version: 1.0.0b24

CODE

import refinitiv.data as rd
import refinitiv.data.eikon as ek
import threading
import time

ek.set_app_key('APP_KEY')
rd.open_session()

# Create wrapper function that would halt the code function run if it exceeds defined threshold in seconds

class TimeoutError(Exception):
    pass

class InterruptableThread(threading.Thread):
    def __init__(self, func, *args, **kwargs):
        threading.Thread.__init__(self)
        self._func = func
        self._args = args
        self._kwargs = kwargs
        self._result = None

    def run(self):
        self._result = self._func(*self._args, **self._kwargs)

@property
    def result(self):
        return self._result


class timeout(object):
    def __init__(self, sec):
        self._sec = sec

    def __call__(self, f):
        def wrapped_f(*args, **kwargs):
            it = InterruptableThread(f, *args, **kwargs)
            it.start()
            it.join(self._sec)
            if not it.is_alive():
                return it.result
            raise TimeoutError('execution expired')
        return wrapped_f

# Test results
@timeout(30)
def ek_get_data():
    df_data, err = ek.get_data(instruments = 'DEA1867P=BE', 
                               fields = ['BID', 'CTBTR_1', 'CF_BID', 'TR.BIDPRICE', 'CF_LOW', 'BID_LOW_1']
                              )
    return df_data

@timeout(300)
def rd_get_data():
    df_data = rd.get_data(universe='DEA1867P=BE',
                     fields=['BID', 'CTBTR_1','CF_BID', 'TR.BIDPRICE', 'CF_LOW', 'BID_LOW_1']
                )
    return df_data

print(ek_get_data())
print(rd_get_data())

rd.close_session()

RESULT

    Instrument    BID CTBTR_1  CF_BID  Bid Price  CF_LOW  BID_LOW_1
0  DEA1867P=BE  89.62  Berlin   89.62     89.505   89.62       <NA>
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
Input In [4], in <cell line: 2>()
      1 print(ek_get_data())
----> 2 print(rd_get_data())

Input In [2], in timeout.__call__.<locals>.wrapped_f(*args, **kwargs)
     31 if not it.is_alive():
     32     return it.result
---> 33 raise TimeoutError('execution expired')

TimeoutError: execution expired

fabian.echterling

In my case it takes 13 (!) seconds. Why does it take so long?

I face the same problem when I pull `TR.UltimateParentID` for appr. 3250 ISINs. It takes forever! Never observed a finish (even after one hour).