How do I fix this error?

bill39

How do I fix this error?

Error:

2020-10-20 05:40:08,286 -- pyeikon -- INFO -- Set App Key: None 

2020-10-20 05:40:12,522 -- pyeikon -- INFO -- Response : 500 - {"code":500,"message":"Cannot find module \".\"","statusMessage":"Internal Server Error"} 

2020-10-20 05:40:12,523 -- pyeikon -- INFO -- Port 9000 was retrieved from .portInUse file 

2020-10-20 05:40:12,524 -- pyeikon -- INFO -- Try to handshake on url http://localhost:9000/api/handshake... 

2020-10-20 05:40:12,612 -- pyeikon -- INFO -- Response : 400 - Handshake payload is invalid. 

2020-10-20 05:40:12,613 -- pyeikon -- INFO -- Set Proxy port number to 9000 

2020-10-20 05:40:12,613 -- pyeikon -- INFO -- Port 9000 on local proxy was detected 

2020-10-20 05:40:12,614 -- pyeikon -- DEBUG -- Request:{'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'instruments': ["lists('Inv Trust List')"], 'fields': [{'name': 'TR.RIC'}]}]}}} 

2020-10-20 05:40:31,164 -- pyeikon -- DEBUG -- HTTP Response code: 200 

2020-10-20 05:40:31,165 -- pyeikon -- DEBUG -- HTTP Response: {"ErrorCode":400,"ErrorMessage":"Backend error. 400 Bad Request"} 

2020-10-20 05:40:31,166 -- pyeikon -- ERROR -- Backend error. 400 Bad Request

My code:

It downloads data via the Eikon API and exports them as a CSV each morning. I use Windows Task Scheduler to initiate the Batch file which runs the python file.

#WARNING: the Eikon terminal must be running in order to connect to the API

#import packages
import eikon as ek # the Eikon Python wrapper package
import pandas as pd
import numpy as np
import datetime
from datetime import timedelta, date, datetime
from pandas.tseries.offsets import BDay
import logging.config

#logs info for bug fixing
ek.set_log_level(logging.DEBUG)

#connects to Bill's Eikon terminal
ek.set_app_key('XXXX')

#retreive the RICs from Eikon
df_rics,e = ek.get_data("lists('Inv Trust List')","TR.RIC")

#convert that into a list and set as an object
ric_list = df_rics['Instrument'].tolist()

#Slice, loop and concatenate the retrieval request - must be done in order to avoid a backend request timeout which
 #happens when we use too many RICs. n can be toggled below. 
n = 10
 df = pd.DataFrame()
for ric_chunk in [ric_list[i:i + n] 
                  for i in range(0, len(ric_list), n)]:
    tmp_df, e = ek.get_data(ric_chunk,
                    ['TR.RNSFilerName',
                     'TR.RNSAnnouncedDate',
                     'TR.RNSTransactionType',
                     'TR.RNSARNumShrsTransacted',
                     'TR.RNSARPctOSTransacted',
                     'TR.RNSARTransactionPrice',
                     'TR.RNSARMktValTransaction',
                     'TR.RNSARTotShrsPostTrans',
                     'TR.RNSARPctOSPostTrans'])
    df = tmp_df.append(df)


#set boundary dates of the download
end_date = date.today()
 start_date = end_date - BDay(2)

#set those dates in the necessary string format
end_date_str = datetime.strftime(end_date, "%Y-%m-%d")
 start_date_str = datetime.strftime(start_date, "%Y-%m-%d")

#denote dates in datetime format
df['RNS Announced Date'] = pd.to_datetime(df['RNS Announced Date'])

#set the date constraints as an object
mask = (df['RNS Announced Date'] >= start_date_str) & (df['RNS Announced Date'] < end_date_str)

#set the dataframe as only those with values inside the boundary dates
df = df.loc[mask]

#rename some of columns headings, some for compatability purposes - some programs don't like the '£' symbol
df.rename(columns={'RNS Announced Date': 'Transaction Date','RNS AR Price (at Transaction) - £': 'RNS AR Price (at Transaction) GBP',
                    'RNS AR Market Value of Transaction - £': 'RNS AR Market Value of Transaction - GBP'},
           inplace=True)

#create file name and export as CSV
todays_date = date.today()
 todays_date_str = datetime.strftime(todays_date, "%Y%m%d")
 df.to_csv('Daily API Download_' + todays_date_str + '.csv')

zoya faberov

Hello all,

For additional details, pertaining to this question, please refer to this previous discussion thread

bill39

Hi Zoya, No one responded to my final question in that thread. I note also since then I have been collecting error reports as you suggest. The above error is the core issue I believe. Please will you help me? Ideally if you would call me it would be much appreciated. Thanks

zoya faberov

Hi @bill39,

Sorry about that delay.

The reason it happened is, because you did not tag (@), it was very easy for the developer who was in discussion with you to miss your last comment, and it is a very busy time of the year for us. I have mentioned, and you should hear from us soon.

Please zip the log, and attach it in the original question.

I would also update to the latest version of Eikon library 1.1.7, as there has been error logging improvements introduced, so this may allow us to learn more of your issue.

The issue continues to be intermittent, about once a week, correct? Does it mean you run the code every day and it fails roughly 1/7 times, or do I misunderstand?

bill39

Hi @zoya.farberov, It says I am neither able to attach a text document nor an excel file because "This attachment is not permitted because the file type is invalid." What file types can I attach? There are three scenarios: 1) download works (happens c.25% of the time), 2) download fails because Eikon Terminal is not open at the time of running (happens c.25% of the time) -this is a Windows Task Scheduling error and NOT one I need help fixing, 3) download fails and the error code originally posted at the beginning of this new thread is generated (happens 50% of the time) - I don't know what this problem is and is where I need your help please. Thanks

zoya faberov

Hello @bill39

Thanks for this info.

Could you please zip the log, and attach the zipped file? Then it will be of a valid type to attach.

bill39

Error log.zipHi @zoya.farberov, find that log here

Alex Putkov.1

@bill39

While we're looking into the root cause of the error in the response from the backend, as an immediate workaround you can simply retry the request when you encounter this error. E.g. the following code snippet retries the request up to two times when "Backend error. 400 Bad Request " is encountered.

for ric_chunk in [ric_list[i : i + n] for i in range(0, len(ric_list), n)]:
    for j in range(3):
        try:
            tmp_df, e = ek.get_data(ric_chunk, 
                                    ['TR.RNSFilerName', 
                                     'TR.RNSAnnouncedDate',
                                     'TR.RNSTransactionType',
                                     'TR.RNSARNumShrsTransacted',
                                     'TR.RNSARPctOSTransacted',
                                     'TR.RNSARTransactionPrice',
                                     'TR.RNSARMktValTransaction',
                                     'TR.RNSARTotShrsPostTrans',
                                     'TR.RNSARPctOSPostTrans'])
            df = tmp_df.append(df)
            break
        except ek.EikonError as err:
            if err.code == 400:
                pass

zoya faberov

Hello @bill39,

Did you have a chance to try the suggestion from @Alex Putkov.?

I agree that this may be the best approach to take, as just like an umbrella guarding from many types of rain, this defensive coding will allow your script to proceed successfully in an event of many kind potential outage, network connectivity, server overloaded, etc.

bill39

Hi @zoya.farberov and @Alex Putkov, I have implemented the error catch workaround you have suggested. It worked yday but today produced the attached error reportpyeikon.20201028.05-40-08.zip Do you have any other suggestions? Thanks

Alex Putkov.1

@bill39

The defensive code that I suggested was to protect specifically against occasional time outs from the backend. There's no code that can protect you from every eventuality. The log you shared indicates that Eikon API Proxy was unavailable and did not respond to the handshake request. One reason this could happen is if Eikon application was not running or was not signed in. Unfortunately from this log alone we cannot say definitively why Eikon API Proxy was unavailable. To establish the root cause would require creating several different logs all with higher logging level than the standard logs created by default.

bill39

Hi @Alex Putkov,
Sure. How do I create those logs? The Eikon Terminal was open at the time of this request. Thanks

Alex Putkov.1

To increase the logging level in Eikon Data APIs library call ek.set_log_level(5). Chapter 3 "Verify that Eikon Desktop is running properly and APIPROXY service is enabled" in this article contains instructions for enabling logging in Eikon application. The specific log files we're interested in are APIProxy.YYYYMMDD.HHMMSS000.pXXXX.txt and SxS.YYYYMMDD.HHMMSS000.pXXXX.txt. I think it's only practical to enable logging if the issue is reproducible. You wouldn't want to keep logging enabled for days or weeks on end if the issue only happens once in a blue moon.

bill39

Hi @alex putkov, @zoya.farberov,

I added ek.set_log_level(5) to my code but I don't know where the logs it creates are stored. In any case, it appears the logs you require were already being stored on my computer. I have attached the two requested on the last day the API failed, being 3 Nov 2020. What do these logs tell you about my problem? Thanks APIProxy.20201103.083952000.p6628.zipSxS.20201103.083952000.p6740.zip

Alex Putkov.1

@bill39

The logs you just sent indicate that Eikon API Proxy was successfully started at 8:40 am on 03-Nov-2020. Are you saying that on Nov 3 after starting Eikon around 8:40am you experienced "Error code 503 | Server Error: API Proxy is not available" when running your code? If yes, do you have pyeikon log from Nov 3 similar to the one you posted on Oct 28? ek.set_log_level(5) increases the logging level in pyeikon log, so hopefully pyeikon log from Nov 3 was created after you added ek.set_log_level(5) to your code.

bill39

@alex putkov, thanks for the explanation. Eikon Terminal automatically opens at 5:40am and the code runs at c.6am. I'm sorry I'm not sure what happened on 3 Nov but I suspect the terminal wasn't running which doesn't sound to be a coding issue necessarily. So far the error-catch re-run amendment you suggested has been successful (apart from on 3 Nov) so it might have been a one-off issue on that day 3 Nov. For now, I'll continue tracking the errors and report back with the logs if it fails again. Thanks for all of your help. Bill

bill39

Hi @alex putkov, the API failed today. I ran the API at 09:48 and attached are the associated logs.SxS.20201106.094358000.p12848.zipAPIProxy.20201106.094358000.p12856.zip

Alex Putkov.1

@bill39

The logs you provided tell that Eikon API Proxy was successfully started at 9:44am. The logs didn't capture any requests sent from your Python script. Based on the symptoms you describe, I'm guessing that EDAPI library was unable to establish the connection with Eikon API Proxy. This is merely a guess, as the inly info I have to go by is the statement that "API failed at 9:48". For further diagnostic we need pyeikon log, which I'm afraid you didn't provide. If you have pyeikon log file from Nov 6th, could you post it here?

bill39

Hi @Alex Putkov. attached is the file you requested.pyeikon.20201106.05-40-07.zip Does that help?

Alex Putkov.1

@bill39
This pyeikon log says that your Python script attempted to connect to Eikon on 2020-11-06 at 05:40. The connection attempt failed. The most likely reason is that Eikon desktop application was not running on this machine at the time. From the previous logs we learned that on 2020-11-06 Eikon application was successfully launched at 9:44. We don't have Eikon logs for 2020-11-06 05:40, hence we cannot definitively know in what state Eikon application was at that time. However, from the logs we do have so far, everything points to Eikon desktop not running at the time, which explains why Eikon Data APIs library for Python couldn't establish connection with Eikon API Proxy on 2020-11-06 at 05:40.

bill39

Hi @Alex Putkov., please can we set up a remote session so that you can help me to add any additional logging to my code that is required in order to get to the bottom of this? I have on several occasions in a manual log noted instances where the connection fails yet where my Eikon terminal has been open. Thanks

bill39

@Alex Putkov., @zoya.farberov, @chavalit.jintamalit, @wasin.waeosri, @pierre.faurel Please will someone help me via remote session? I pay £750 per month for Eikon and I've been struggling with the problem for months now without proper help. I'm very dissatisfied and frustrated.

Alex Putkov.1

@bill39

I don't think configuring additional logging is what's required here. The logs that you've been sharing capture everything these logs are supposed to capture. The problem as I see it is that the logs you've shared do not correspond to the time when the event you'd like investigated occurred. In any case I've just sent you an email and we can take this conversation offline.