Searched for Futures and Options unsuccessfully with status code 502 in between of multiple long run

trisha.gehlot

DSS Rest API

Domain - https://selectapi.datascope.refinitiv.com/RestApi/v1/

URL - https://selectapi.datascope.refinitiv.com/RestApi/v1/Search/FuturesAndOptionsSearch

Total instruments for 1 day data - 418788

Details of issue -
As more then 400 instruments were giving server busy error, tried querying this above URL in chunks with instruments = 400 and 'respond-async;odata.maxpagesize=1000'.

After running for 5 loops(instrument chunks), it failed and raised error -

Processing instruments bucket No.5 for searchFuturesOptions with len:400 Instrument Range BLSR.TA<-->RKH.L

ERROR Searched for Futures and Options unsuccessfully with status code 502

msg: 'Content of request: b\'{"error":{"message":"An error occurred on the server. If this problem persists, contact Support (referenci' (str)

response: '<Response [502]>' (Response)

Could you please guide what is the good way to query large amount of instruments, as if its fails in between, time and processing both gets wasted.

Jirapongse

@trisha.gehlot

Thank you for reaching out to us.

According to the DSS Status Codes page, the 502 is Bad Gateway.

502 Bad Gateway indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request. The application can retry.

What do you mean by "Total instruments for 1 day data - 418788"?

Please share the request message that you are using. I will try it on my machine.

trisha.gehlot

@Jirapongse Thank you for your response!!

As it was a Bad Gateway, it might not arise again, but could please suggest best way to deal with large amount of instruments.

418788 this is the total number of instruments to be searched.

Request body-

searchRequest = {"IdentifierType": identifierType, "PreferredIdentifierType": preferredIdentifierType}

if underlyingRIC:

searchRequest["UnderlyingRic"] = identifiers

requestBody = json.dumps({"SearchRequest": searchRequest})

return requestBody

Request Body -

{"SearchRequest": {"IdentifierType": "Ric", "PreferredIdentifierType": "Ric", "UnderlyingRic": "AAL.O,ABT.N,AMD.O,APD.N,AEP.O,AXP.N,AMGN.O,AAPL.O,ADM.N,ADSK.O,ADP.O,AVY.N,BLL.N,BSET.O,BAX.N,BDX.N,HRB.N,BA.N,BGGSQ.PK,BMY.N,BFb.N,BC.N,CNA.N,CSX.O,CPB.N,CAT.N,CLX.N,KO.N,CL.N,CAG.N,CTB.N,GLW.N,CR.N,CMI.N,DE.N,DLX.N,DDS.N,DIS.N,RRD.N,DOV.N,ETN.N,ECL.N,EMR.N,ETR.N,FMC.N,NEE.N,FMCC.PK,FNMA.PK,FLTWQold.PK,TGNA.N,GPS.N,GD.N,GE.N,GIS.N,GCO.N,GPC.N,GT.O,GRA.N,GWW.N,HAL.N,LHX.N,HAS.O,HP.N,HSY.N,HPQ.N,HD.N,HUM.N,ITW.N,INTC.O,IBM.N,IFF.N,IP.N,JNJ.N,K.N,KMB.N,KR.N,LLY.N,LB.N,LNC.N,LPX.N,LOW.N,LUB.N,MMC.N,MAS.N,MAT.O,MDRIQ.PK,MCD.N,SPGI.N,MDT.N,CVS.N,MRK.N,MDP.N,MSI.N,NL.N,NC.N,THC.N,NAV.N,NYT.N,NEM.N,NKE.N,JWN.N,NSC.N,NOC.N,NUE.N,OXY.N,OKE.N,ORCL.N,PHM.N,PNC.N,PPG.N,PCAR.O,PCG.N,PLLdum.N,PH.N,CPPRQ.PK,PEP.O,PFE.N,PBI.N,PG.N,PEG.N,RAD.N,ROK.N,R.N,EIX.N,SPXC.N,SLB.N,SCI.N,SHW.N,SKY.N,SNA.N,SO.N,SWK.N,STI.N,SYY.N,TJX.N,TXN.O,TXT.N,TKR.N,GL.N,UALAQold.OB,UN.N,UNP.N,RTX.N,VFC.N,WMT.N,WBA.O,WLBAQ.PK,WY.N,WHR.N,WMB.N,XRXold.N,ADBE.O,AIN.N,MATX.N,APC.N,AONOLD.N,APAold.O,AMAT.O,AVT.O,BKH.N,CMS.N,CBT.N,CDNS.O,CAH.N,CRS.N,CHKAQ.PK,CRUS.O,CTAS.O,CMA.N,DBD.N,EBF.N,EFX.N,FSS.N,FISV.O,BEN.N,GATX.N,AJRD.N,HOG.N,HSC.N,HE.N,IDA.N,IPG.N,KELYA.O,LANC.O,LEG.N,MKC.N,MU.O,MLHR.O,MUR.N,MYL.O,NI.N,UVV.N,CSCO.O,DG.N,FITB.O,.SPX,EAT.N,KEY.N,NFG.N,NPK.N,NDSN.O,ES.N,NTRS.O,ODPold.O,OGE.N,OLN.N,OMC.N,PAYX.O,PNR.N,PNW.N,PGR.N,SCHW.N,SEE.N,SON.N,LUV.N,STT.N,TFX.N,TDS.N,TER.O,TMO.N,TIF.N,TRN.N,TSN.N,UFI.N,VLO.N,GHC.N,WEC.N,XLNX.O,AGCO.N,AES.N,RAMP.N,AEM.N,Y.N,AMSC.O,AP.N,APH.N,ARCB.O,ARW.N,ASTE.O,ATO.N,AZO.N,BBBY.O,BBY.N,BIO.N,BSX.N,CACI.N,ELY.N,CHH.N,FUN.N,CAKE.O,CINF.O,CTG.O,CRVL.O,CACC.O,XRAY.O,DGII.O,DXYN.O,DY.N,EA.O,VALPQ.PK,ENZN.PK,FAST.O,GILD.O,HUBB.N,JBHT.O,IDXX.O,AEGN.O,IONS.O,KLAC.O,KEM.N,LSCC.O,MTX.N,PRCP.O,SKYW.O,SWN.N,SHYF.O,SPLS.O,PSA.N,SNPS.O,SNV.N,TCO.N,TECD.O,TPLold.N,TR.N,TSS.N,UGI.N,UHS.N,VSH.N,VNO.N,WRI.N,WMK.N,WDC.O,WSM.N,WWW.N,ZBRA.O,YPF.N,L.N,EQC.N,HRL.N,KIM.N,KSS.N,LEE.N,JEF.N,MBI.N,MDC.N,MTG.N,MGM.N,MAN.N,NJR.N,ORI.N,OMI.N,RHI.N,HST.O,MSFT.O,PRGO.N,EMN.N,SBUX.O,TV.N,RYN.N,MGA.N,QCOM.O,SPY.P,ALB.N,INTU.O,TEF.N,LRCX.O,SPG.N,KOF.N,TOT.N,DRE.N,IMO.A,.RUI,.RUT,ALL.N,MLM.N,HBAN.O,MXIM.O,VLY.O,ENIA.N,JJSF.O,MCHP.O,NWN.N,REGN.O,MEOH.O,TEVA.N,PZZA.O,ADTN.O,GBX.N,COF.N,STRT.O,HIW.N,LMT.N,DRI.N,AFG.N,INCY.O,BWA.N,RHHBY.PK,FCX.N,AKS.N,TGS.N,UL.N,CPSS.O,AEG.N,SWM.N,IT.N,ITT.N,HIG.N,TUP.N,DOFSQ.PK,FMS.N,ATGE.N,CBRI.PK,AGLJ.J,SASY.PA,TDC.CO,ALSO.PA,CNPP.PA,PUBP.PA,AGFB.BR,SAN.MC,WHLJ.J,ADEN.S,ITUJ.J,ABBN.S,NELES.HE,IDR.MC,HSBA.L,INCH.L,NTGY.MC,HUDN.AS,LKOH.MM,MSNG.MM,RTKM_p.MM,ALB.MC,BMPS.MI,OML.L,TEF.MC,RTKM.MM,ABB.ST,SAF.PA"}}'

response = requests.post(

DataScopeURL.SEARCH_FUTURES_OPTIONS,

requestBody,

headers=populateHttpRequestHeaders(self.loginDetails, self.proxyServers, prefer='respond-async;odata.maxpagesize=1000', passToken=True),

proxies=self.proxyServers,

)

Jirapongse

@trisha.gehlot

I found the answer in this discussion.

trisha.gehlot

@Jirapongse Thank you for this paging logic, will try to increase page size!!

But the issue above is, when I send 418788 instruments all together, the server give exception.

And when I sent instrument Chunks for eg. 2000 instruments is taking 12 hours in querying this Rest API

I need to check, what is the best practice to deal with this, while sending the query to FuturesAndOptionsSearch

*Also if you can confirm, how many concurrent queries they support for our account?

Jirapongse

@trisha.gehlot

I think 418788 instruments are too much for search endpoint.

Please contact your LSEG account team or sales team directly to verify this use case.

We may be able to suggest other products for this use case.