Time and sales too huge and unable to download in compressed format

lim.yongyik
lim.yongyik
in DSS

Hello. I'm using Stored & Scheduled Workflow to download Tick History Time And Sales from DSS.

When i download the file using Python, i noticed that the format is raw and too huge.

Below is a sample i tried.

https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractedFiles('VjF8MHgwOGQzYTE2MjA1Nzk5OTYyfA')/$value

The output is below in which i need to write into a file.

1708439541083.png

what can i do to download directly from the link below (GUI) ? in the Download link, the file size is already in gz format and relatively smaller. How can i get this using Python?

1708439816073.png

Tagged:

Best Answer

  • Jirapongse
    Jirapongse
    Answer ✓

    @lim.yongyik

    Thank you for reaching out to us.

    We have an article and Python sample regarding this workflow.

    The article is DSS REST API: Downloading Completed Extraction Files by a Schedule Name. The Python example is on GitHub.

    The download code is in the DSSClient.py.

        def download_file(self, extracted_file, aws):
             decode = True
             headers={}
             headers["Prefer"] = "respond-async"
             headers["Authorization"] = "Token "+self.__token__
             if aws == True:
                 headers["X-Direct-Download"] = "True"
             if extracted_file["ExtractedFileName"].endswith("gz") == True:
                 decode = False


             requestUrl = self.__dss_uri__+self.__download_file_endpoint__.format(extracted_file["ExtractedFileId"])
             r = requests.get(requestUrl,headers=headers,stream=True)
             chunk_size = 1024 
             with open(extracted_file["ExtractedFileName"], 'wb') as fd:
                for data in r.raw.stream(chunk_size,decode_content=decode):
                    fd.write(data)

    The code checks the file's surname before opening the stream. Please check if this code works.

Answers

  • lim.yongyik
    lim.yongyik

    Thanks @Jirapongse. The key point here is decode = False to ensure the file is downloaded in .gz. However i noticed, although i use AWS to download, the speed is still slow. Any recommendation for this?

Categories