Python websocket connection to Refinitiv Realtime fails with SSL error

_Tom_

I am trying to connect to the Refinitiv Realtime Optimized platform in AWS using websockets in python.

I am using the market_price_rdpgw_service_discovery.py example from https://github.com/Refinitiv/websocket-api/tree/master/Applications/Examples/RDP/python#windowslinuxmacos

The HTTP requests for service discovery work and I can setup a token but the websocket connection fails:

Sending Refinitiv Data Platform service discovery request to https://api.refinitiv.com/streaming/pricing/v1/

Refinitiv Data Platform Service discovery succeeded. RECEIVED:
....

Connecting to WebSocket wss://eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net:443/WebSocket for session1...
Traceback (most recent call last):
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_app.py", line 312, in run_forever
    self.sock.connect(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_core.py", line 248, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 121, in connect
    sock = _ssl_socket(sock, options.sslopt, hostname)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 256, in _ssl_socket
    sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 231, in _wrap_sni_socket
    return context.wrap_socket(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1131)
EOF occurred in violation of protocol (_ssl.c:1131) for session1
WebSocket Closed for session1

I've tried disabling all SSL verification and fixing the version to TSL 1.2 but still get the same error on handshake with the socket.

 def connect(self):
        # Start websocket handshake
        ws_address = "wss://{}/WebSocket".format(self.host)
        print("Connecting to WebSocket " + ws_address + " for " + self.session_name + "...")
        self.web_socket_app = websocket.WebSocketApp(ws_address,
                                                     on_message=self._on_message,
                                                     on_error=self._on_error,
                                                     on_close=self._on_close,
                                                     on_open=self._on_open,
                                                     subprotocols=['tr_json2'])  
        sslopt = {
            'check_hostname': False,
            "verify_mode": ssl.CERT_NONE,
            "cert_reqs": ssl.CERT_NONE,
            "ssl_version": ssl.PROTOCOL_TLSv1_2,
        }
        wst = threading.Thread(target=self.web_socket_app.run_forever, kwargs={'sslopt': sslopt})
        wst.start()

Here are some of the relevant library version:

pip list |grep 'socket\|request\|http\|cert\|url'
certifi              2021.10.8
ndg-httpsclient      0.5.1
requests             2.22.0
requests-toolbelt    0.9.1
urllib3              1.25.11
websocket-client     1.1.0

Has anyone else seen this and been able to resolve?

This is running behind a corporate proxy which I have no control over, but the HTTP requests for service discovery are working which makes me thing the SSL setup is OK in python requests generally, but there is an issue with websockets.

Jirapongse

@_Tom_

I tested with the mitmdump (Man In The Middle Dump) proxy.

The code can run properly with the http_proxy environment variable.

umer.nalla

Hi @_Tom_

We cannot provide much Websocket API guidance in terms of dealing with corporate proxies as there can be many factors at play and variations depending on your local security implementations.

However, a few things to try

• set proxy parameters for the WebSocketApp instance in the code (websocket/_app.py — websocket-client 1.2.3 documentation )
• set the https_proxy and http_proxy parameters as environmental variables before running the script

Certainly, I am aware of a couple of users where the 2nd option allowed to connect.

You should speak to your internal security team for further guidance. You can also raise a ticket with the RTO team - who may be able to provide some assistance to your security team.

_Tom_

I've tried setting http_proxy and https_proxy, it works for the initial http requests but not for the websocket. If I set them incorrectly then I can break the normal requests, setting them correctly goes back to trying the websocket connection and failing:

Connecting to WebSocket wss://eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net:443/WebSocket for session1...
Traceback (most recent call last):
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_app.py", line 312, in run_forever
    self.sock.connect(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_core.py", line 248, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 117, in connect
    sock = _tunnel(sock, hostname, port, auth)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 287, in _tunnel
    raise WebSocketProxyException(
websocket._exceptions.WebSocketProxyException: failed CONNECT via proxy status: 504
failed CONNECT via proxy status: 504 for session1
WebSocket Closed for session1

Basic network connectivity works on port 443 based on telnet:

telnet eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net 443
Trying 159.43.201.118...
Connected to eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.biz.
Escape character is '^]'.

_Tom_

For anyone else reading this thread with similar problems it turned out to be network configuration issues. The hosts returned from the service discovery call resolve to internal IPs that we didn't have routing for (e.g. refinitiv.net were CNAMEs for refinitiv.biz internal addresses).

We did have network connectivity setup for us-east region and just set that group of hosts. The EOF error from the websocket ssl handshake is really because the resolved IP is not a refinitiv RTO server due to the network setup.

_Tom_

Thanks for checking the proxy setup, I've commented below on the real issue which was company specific network setup. But this will be useful for others.