Restarting session after 401 error in Python

r.fernandez

I'm using a basic python setup, following the session example on GitHub.

  import refinitiv.data as rdfrom refinitiv.data.content import fundamental_and_reference rd.open_session() # run routines ->

After a number of routines, some 1000 correct calls with http 200 I sometimes receive http code 401. Token de-activated. The only way to solve for this is to restart the script.

My question: How can I programmatically restart the session.

I would expect something as:

  response = fundamental_and_reference.Definition(universe=['example']).get_data() if response._http_status['http_status_code'] == 401:        # solution like this would be nice:    rd.restart_session()

rd.restart_session obviously does not exist. rd.close_session() and rd.open_session() does not work as well. The code freezes and does not continue.

In the previous answer I got referred to the GitHub, but there is no code explaining there what to do.

What can we do?

jonathan.legrand

Hi @r.fernandez,

We advise coding defensively in these instances. I manufactured an error in the code below to exemplify what I mean:

import refinitiv.data as rd
from refinitiv.data.content import fundamental_and_reference

rd.open_session()

request = fundamental_and_reference.Definition(
    universe=['LSEG.L'],
    fields=['ASK'])

try:
    response = request.get_data()
except Exception as e:
    print(e.__dict__)

try:
    response = request.get_data()
except Exception as e:
    if e.__dict__['code'] == -1:
        print("The Error Is -1")

As you can see, you can use a try and if loop to come to a similar functionality to what you asked. Please let me know if this makes sense.

r.fernandez

Hi
@jonathan.legrand

This does not answer my question. Obviously in the production code we have a try/except around it.

My very very specific question: how do I restart the session after receiving a 401 error?

Can you please think about an answer and email this to me? Thanks

jonathan.legrand

Hi @r.fernandez, If you would like to keep your code lines' framework, you could try something like this:

if response._http_status['http_status_code'] == 401:

    rd.close_session()
    rd.open_session()
    response = fundamental_and_reference.Definition(universe=['example']).get_data()

If you often encounter this issue, I would advise making a function to capture all calls:

import time

def get_fundamental_and_reference_data(
        universe,
        fields,
        max_attempts=4,
        parameters=None,
        row_headers=None,
        use_field_names_in_headers=None,
        extended_params=None
        ):

    for i in range(max_attempts):
        response = rd.content.fundamental_and_reference.Definition(
            universe=universe,
            fields=fields,
            parameters=parameters,
            row_headers=row_headers,
            use_field_names_in_headers=use_field_names_in_headers,
            extended_params=extended_params).get_data()
        if response._http_status['http_status_code'] == 401:
            time.sleep(0.5)
            rd.close_session()
            time.sleep(2.5)
            rd.open_session()
            time.sleep(1)
        elif response._http_status['http_status_code'] != 401:
            return response

r.fernandez

Hi @jonathan.legrand

As I wrote in my question: d.close_session() and rd.open_session() does not work as well. The code freezes and does not continue.

I will show you the code I have (similar to your code I've created a function to capture all calls). The error handling specifically for 401 can be written out with time.sleep in between:

This results in the following routine. As you see in the command line, the code runs perfectly, and after about 10 minutes (approximately), the returned error is 401. As you can see, I've setup the routine, and as you see the code freezes on "rd.open_session()"

I've waited for over 5 minutes, no result. The code freezes.

Can you come up with a method which does not result in this behavior?

jonathan.legrand

Hi @r.fernandez,

Yes, at my end, code like the below does not result in this behavior:

import refinitiv.data as rd
import time

rd.open_session(
    "desktop.workspace",
    config_name="C:\\Example.DataLibrary.Python-main\\Configuration\\refinitiv-data.config.json")

def get_fundamental_and_reference_data(
        universe,
        fields,
        max_attempts=4,
        parameters=None,
        row_headers=None,
        use_field_names_in_headers=None,
        extended_params=None
        ):

    for i in range(max_attempts):
        response = rd.content.fundamental_and_reference.Definition(
            universe=universe,
            fields=fields,
            parameters=parameters,
            row_headers=row_headers,
            use_field_names_in_headers=use_field_names_in_headers,
            extended_params=extended_params).get_data()
        if response._http_status['http_status_code'] == 401:
            time.sleep(0.5)
            rd.close_session()
            time.sleep(2.5)
            rd.open_session()
            time.sleep(1)
        elif response._http_status['http_status_code'] != 401:
            return response

data=get_fundamental_and_reference_data(universe="VOD.L", fields="TR.ClosePrice")

data.data.df

jonathan.legrand

Hi @r.fernandez,

May I ask

(i) what message you get with the 401 error? is it something like `"message":"Invalid authentication credentials"`?

(ii) You mentioned following the session example on GitHub, which session out of the ones shown there are you using?

r.fernandez

hi @jonathan.legrand ,

The error I receive is: "Error code 401 | token expired"

I've replicated the full code below, based on your example. I've ran the code in 3 command windows in parallel to replicate the production environment. After 1975 calls, 4 minute 20 seconds, I receive this Exception and the code freezes.

Note: Your code you check 401 in the http response, but in practice 401 raises an error and thus must be caught using try/catch.

See attached for the .py file (added as .txt file)

See: _test.txt

r.fernandez

@jonathan.legrand

Did you receive the message yesterday on the error code 401 and the script? Have you been able to replicate the error code after running it for some ten minutes?

r.fernandez

@jonathan.legrand

I've answered your thread last week, but have not received a reaction. Could you please take a look at it? My note:

The error I receive is: "Error code 401 | token expired"

I've replicated the full code below, based on your example. I've ran the code in 3 command windows in parallel to replicate the production environment. After 1975 calls, 4 minute 20 seconds, I receive this Exception and the code freezes.

Note: Your code you check 401 in the http response, but in practice 401 raises an error and thus must be caught using try/catch.

See code:

import refinitiv.data as rd
import time
from datetime import datetime 
import random

rd.open_session()
start = datetime.now()

def get_fundamental_and_reference_data(
        universe,
        fields,
        max_attempts=4,
        parameters=None,
        row_headers=None,
        use_field_names_in_headers=None,
        extended_params=None,
        ncalls = 0
        ):

    for i in range(max_attempts):

        try:
            response = rd.content.fundamental_and_reference.Definition(
            universe=universe,
            fields=fields,
            parameters=parameters,
            row_headers=row_headers,
            use_field_names_in_headers=use_field_names_in_headers,
            extended_params=extended_params).get_data()
            return response._http_status['http_status_code']
        except Exception as e:
            error = str(e).split('Requested')[0]

            # check if 401 error, if so restart and fully exit script
            if 'code 401' in error or 'token expired ' in error:
                print(f'\n\n-> Error 401: {error}')
                print(e)
                print('Script started at: ', start)
                print('Current time: ', datetime.now())
                print('Number of calls done: ', ncalls)

                time.sleep(1)
                print(f'{datetime.now()} -- closing session')
                rd.close_session()
                print(f'{datetime.now()} -- session closed')
                time.sleep(2.5)
                print(f'{datetime.now()} -- opening new session')
                rd.open_session()
                print(f'{datetime.now()} -- session opened')
                time.sleep(1)
                print(f'{datetime.now()} -- continuing')

            return e

ncalls = 0
while True:
    ncalls += 1
    http=get_fundamental_and_reference_data(universe="VOD.L", fields="TR.ClosePrice", ncalls=ncalls)
    print(f'    - call {ncalls} -- http: {http}', end='\r')

    #  to ensure no 429 data limiting calls
    time.sleep(random.uniform(0, 0.5))

FYI, reran the code on Monday 3 June. Performed the code in 2 command windows, and first window received 401 error after just 64 calls. See screenshot:

Running on Mac OSX, latest version of osx, python 3.9 and pip version.

jonathan.legrand

Hi @r.fernandez,

Sorry for the tardive answer, but I ran the _test.txt code three times on two different days and got the same:

The discrepancy between your experience and mine is making me think that this may be a permission issue, whereby you may be using credentials aimed solely at Desktop Sessions and not Platform Sessions.
The code you kindly shared with us includes the line `rd.open_session()`; may I know what session you have as default in your configuration file? Is it Desktop Sessions? Or maybe you do not use a configuration file? Because in this case, what may be happening is that the proxy on Workspace may be shutting down after a long while of inactivity on the Desktop Application or after a session limit, leading to the issues you are experiencing. In this case, a solution would be to shut down and restart the Workspace Desktop Application every time your code runs into this issue. If your LSEG License allows for Platform Sessions, I would advise using this in your instance; you may be requesting a lot of data, which reflects the use of Platform Sessions for which these were made.

r.fernandez

Hi @jonathan.legrand

The discrepency is I think in the fact that I run multiple commandlines, perhaps the token invalidates that case. As you can also not be logged in on the website and the refinitiv app at the same time.

Can you run the command in 2 command windows, and see if you get the same issue?

jonathan.legrand

Hi @r.fernandez

You are correct. I was testing several things and I took a screenshot of the test with a single instrument. I tested multiple instruments just now to test and returned with the same:

jonathan.legrand

The discrepancy between your experience and mine is making me think that this may be a permission issue, whereby you may be using credentials aimed solely at Desktop Sessions and not Platform Sessions.
The code you kindly shared with us includes the line `rd.open_session()`; may I know what session you have as default in your configuration file? Is it Desktop Sessions? Or maybe you do not use a configuration file? Because in this case, what may be happening is that the proxy on Workspace may be shutting down after a long while of inactivity on the Desktop Application or after a session limit, leading to the issues you are experiencing. In this case, a solution would be to shut down and restart the Workspace Desktop Application every time your code runs into this issue. If your LSEG License allows for Platform Sessions, I would advise using this in your instance; you may be requesting a lot of data, which reflects the use of Platform Sessions for which these were made.

jonathan.legrand

The discrepancy between your experience and mine is making me think that this may be a permission issue, whereby you may be using credentials aimed solely at Desktop Sessions and not Platform Sessions.
The code you kindly shared with us includes the line `rd.open_session()`; may I know what session you have as default in your configuration file? Is it Desktop Sessions? Or maybe you do not use a configuration file? Because in this case, what may be happening is that the proxy on Workspace may be shutting down after a long while of inactivity on the Desktop Application or after a session limit, leading to the issues you are experiencing. In this case, a solution would be to shut down and restart the Workspace Desktop Application every time your code runs into this issue. If your LSEG License allows for Platform Sessions, I would advise using this in your instance; you may be requesting a lot of data, which reflects the use of Platform Sessions for which these were made.