Exception from get bundle data in datastream

Blue.suit

Hi,

Im using the Datastream get_bundle_data to returns several items for a universe of stocks. We have set up a loop to cycle our universe. It works fine,but then randomly we get an error and it says it has did connected. Can you please advise me on the following. I should not we are within the max requests and on retry it works.

1. How can we handle this so it avoids this disconnection. We currently have the program use tenacity retry, a wrapper from python but this is less than ideal. We want to avoid these errors all together.

2. Can we use multithreading to make this faster, we don't have this in place currently but would like to do it. Or any other optimisations you can recommend based on screenshots.

here is the exception:

on 40: HTTPSConnectionPool(host='product.datastream.com', port=443): Read timed out.

on 40: _get_json_Response: Exception Occured:

on 40: HTTPSConnectionPool(host='product.datastream.com', port=443):

Read timed out.

on 40: get_bundle_data: Exception Occured

on 40: (<class 'TypeError>, TypeError("argument of type 'None Type' is not

Iterable"), <traceback object at 0x00000227252BCC40>)

on 40: None

on 40: Traceback (most recent call last):

File "C:\Users\sayersqvm\miniconda3/envs/qvmLive\lib\site-

packages\DatastreamDSWS\DS_Response.py", line 209, in get_bundle_data

if 'DataResponses' in json_Response:

TypeError: argument of type 'None Type' is not iterable

on 62: ("Connection aborted.", RemoteDisconnected('Remote end closed connection without response"))

on 62: _get_json_Response: Exception Occured:

on 62: ('Connection aborted.", RemoteDisconnected('Remote end closed

connection without response'))

on 62: get_bundle_data: Exception Occured

on 62: (<class 'TypeError>, TypeError("argument of type 'None Type' is not

Iterable"), <traceback object at 0x0000022724802040>)

on 62: None

Reply

31

Search

Jirapongse

@Blue.suit

I found that the default HTTP timeout of the DatastreamDSWS Python library is 180 seconds (3 minutes). Therefore, if you got an error after 3 minutes, it could relate to this setting.

You can change this timeout by using the configuration file (config.ini).

[app]
timeout=300
[url]
path=https://product.datastream.com

Then, set this configuration file in the config parameter.

ds = DSWS.Datastream(username = username, password = password, config='config.ini')

However, if it doesn't solve the issue, it could be a connection or server issue.

Regarding using multiple threads with DatastreamDSWS, please refer to this article: Load Data using Multiprocessing in Python.

raksina.samasiri

Hi @Blue.suit ,

Regarding an error you got, have you had a chance to check this thread?

The suggestion from the thread is you can use the requests Python library to check network settings. Please try the following code in the same environment and let us know the result

import requests
 
url = 'https://product.datastream.com/DSWSClient/V1/DSService.svc/rest/GetToken';
myobj = {"Password": "<username>", 
         "Properties": [{"Key": "__AppId", "Value": "PythonLib-1.1.0"}], 
         "UserName": "<password>"}
 
x = requests.post(url, json = myobj)
print(x.status_code)
print(x.text)

You may need to enable HTTP logging in Python.

import logging
import http.client
 
http.client.HTTPConnection.debuglevel = 1
 
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Hope this helps and please let me know in case you have any further questions

Blue.suit

Sorry but this is not helpful. This example is not getting access at all. For our issue, we are able to get data, but it will intermittently freeze the call and result in an error. When it is retried, it works fine.