TickHistoryRawExtractionRequest REST API throws an error using Curl

rahul.deshmukh

I have followed below steps using curl:

1) get the authentication token which is working and generated token:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -H "Content-Type: application/json" -H "Prefer: respond-async" -X POST -v -d "{\"Credentials\": {\"Username\": \"501\",\"Password\": \"xxxxxx\"}}" https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken

2) Use this authentication token in the below curl request:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X POST \
    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw \
    -H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
    -H 'Content-Type: application/json' \
    -d '{
        "ExtractionRequest": {
        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
        "ContentFieldNames": [  
       
        ],
        "IdentifierList": {
          "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
          "InstrumentIdentifiers": [
            {"Identifier":"IBM.N","IdentifierType":"Ric"},{"Identifier":"CARR.PA","IdentifierType":"Ric"},{"Identifier":"BEGRs.TQE","IdentifierType":"Ric"},{"Identifier":"US1156371007.GTX","IdentifierType":"Ric"},{"Identifier":"JP3249600002.GTX","IdentifierType":"Ric"}
          ]
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "2021-08-20T00:00:00.000Z",
            "QueryEndDate": "2021-08-23T00:00:00.000Z",
            "Fids": "3,5,12,13,15,16,19,22,23,26,27,4196,5632,5633,2130",
            "DisplaySourceRIC": true
      }
      }
 }'

The output response for the above request is as below:

@{"@odata.context":"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae207dfead0ec8","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 7701\nExtraction ID: 2000000296050738\nSchedule: 0x07ae207dfead0ec8 (ID = 0x0000000000000000)\nInput List (1 items):  (ID = 0x07ae207dfead0ec8) Created: 23.08.2021 15:58:07 Last Modified: 23.08.2021 15:58:07\nReport Template: _OnD_0x07ae207dfead0ec8 (ID = 0x07ae207dfecd0ec8) Created: 23.08.2021 15:56:55 Last Modified: 23.08.2021 15:56:55\nSchedule dispatched via message queue (0x07ae207dfead0ec8), Data source identifier (8A3F4C373BA140F98836FA4F5B502FC8)\nSchedule Time: 23.08.2021 15:56:56\nProcessing started at 23.08.2021 15:56:56\nProcessing completed successfully at 23.08.2021 15:58:07\nExtraction finished at 23.08.2021 13:58:07 UTC, with servers: tm02n02, TRTH (41.83 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nTotal instruments after instrument expansion = 1\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}

3) This step i am not clear yet. In this step i tried to get the JobID and also previous token i have used from step 1 and use it another curl request to get the response values for the fids:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json_values.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X GET \
    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \
    -H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
    -H 'Content-Type: application/gzip' \
    -H 'X-Direct-Download: True' 

And this step is throwing an error as below:

ODI-1226: Step Copy of Refinitiv_RESTAPI_REQUEST fails after 1 attempt(s).
ODI-1241: Oracle Data Integrator tool execution fails.
Caused By: com.sunopsis.dwg.function.SnpsFunctionBaseException: ODI-30038: OS command returned 1. Error details are [sh: -c: line 1: syntax error near unexpected token `('
sh: -c: line 1: `    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \'
].
    at com.sunopsis.dwg.tools.OSCommand.actionExecute(OSCommand.java:535)
    at com.sunopsis.dwg.function.SnpsFunctionBaseRepositoryConnected.execute(SnpsFunctionBaseRepositoryConnected.java:235)
    at oracle.odi.runtime.agent.execution.SessionTask.execIntegratedFunction(SessionTask.java:949)
    at oracle.odi.runtime.agent.execution.SessionTask.executeOdiCommand(SessionTask.java:607)
    at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:32)
    at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:18)
    at oracle.odi.runtime.agent.execution.TaskExecutionHandler.handleTask(TaskExecutionHandler.java:52)
    at oracle.odi.runtime.agent.execution.SessionTask.processTask(SessionTask.java:216)
    at oracle.odi.runtime.agent.execution.SessionTask.doExecuteTask(SessionTask.java:128)
    at oracle.odi.runtime.agent.execution.AbstractSessionTask.execute(AbstractSessionTask.java:886)
    at oracle.odi.runtime.agent.execution.SessionExecutor$SerialTrain.runTasks(SessionExecutor.java:2225)
    at oracle.odi.runtime.agent.execution.SessionExecutor.executeSession(SessionExecutor.java:610)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:718)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:611)
    at oracle.odi.core.persistence.dwgobject.DwgObjectTemplate.execute(DwgObjectTemplate.java:203)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor.doProcessStartAgentTask(TaskExecutorAgentRequestProcessor.java:800)
    at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor.access$1400(StartSessRequestProcessor.java:74)
    at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor$StartSessTask.doExecute(StartSessRequestProcessor.java:702)
    at oracle.odi.runtime.agent.processor.task.AgentTask.execute(AgentTask.java:180)
    at oracle.odi.runtime.agent.support.DefaultAgentTaskExecutor$2.run(DefaultAgentTaskExecutor.java:108)
    at java.lang.Thread.run(Thread.java:748)

zoya faberov

Hello @rahul.deshmukh ,

Working with RTH REST via curl is stable. However, curl is a test tool, rather then a programming language. Requests via curl returns a lot of verbose output that you have indicated you would prefer not to have.

Have you had a chance to review Programming Without SDK tutorials? They go over the recommended steps and link to RTH Downloads that includes RTH Python Code Samples.

I would suggest using On-Demand examples( please see tutorials for the complete information on On-Demand and Scheduled/GUI) and replace the request in the example (IntradayBars request) with your required RTH request (TickHistoryRaw).

I have just used this approach, to make a python example for TickHistoryRaw on demand:

RTH_OnDemand_Raw.20210827.zip

Please find it attached, replace the credentials with yours, and I hope you find this example useful

rahul.deshmukh

@zoya.farberov @Gurpreet can you please help me here...thanks

zoya faberov

Hello @rahul.deshmukh ,

There was a mandatory migration of the endpoints, for DSS and RTH,

Please find the requirement details on DSS API front page and more in-depth from RTH API front page

The tutorial on how to migrate that should be helpful can be found at Refinitiv Datascope Select API Migration Tutorial, this information is relevant for both RTH and DSS migration requirements.

rahul.deshmukh

@zoya.farberov Thanks now i modified as per the migration, there is no error now but i am not getting the correct response for fields:

Below is the response:

@{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae62f53ead0f40","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9028601\nExtraction ID: 2000000296068231\nSchedule: 0x07ae62f53ead0f40 (ID = 0x0000000000000000)\nInput List (2 items):  (ID = 0x07ae62f53ead0f40) Created: 23.08.2021 17:08:43 Last Modified: 23.08.2021 17:08:43\nReport Template: _OnD_0x07ae62f53ead0f40 (ID = 0x07ae62f53ecd0f40) Created: 23.08.2021 17:07:40 Last Modified: 23.08.2021 17:07:40\nSchedule dispatched via message queue (0x07ae62f53ead0f40), Data source identifier (DED82B93BB234FF882B35BEE6E2510B3)\nSchedule Time: 23.08.2021 17:07:40\nProcessing started at 23.08.2021 17:07:40\nProcessing completed successfully at 23.08.2021 17:08:43\nExtraction finished at 23.08.2021 15:08:43 UTC, with servers: tm03n03, TRTH (49.021 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nInstrument <RIC,CARR.PA> expanded to 1 RIC: CARR.PA.\nTotal instruments after instrument expansion = 2\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: CARR.PA,Market Price,,,Inactive,0\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}

I am just doing some tests initailly with some Rics and fids to check if i am getting response from this RAW request.

rahul.deshmukh

@zoya.farberov Thanks now i modified as per the migration, there is no error now but i am not getting the correct response for fields:

Below is the response:

@{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae62f53ead0f40","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9001\nExtraction ID: 2000000296068231\nSchedule: 0x07ae62f53ead0f40 (ID = 0x0000000000000000)\nInput List (2 items):  (ID = 0x07ae62f53ead0f40) Created: 23.08.2021 17:08:43 Last Modified: 23.08.2021 17:08:43\nReport Template: _OnD_0x07ae62f53ead0f40 (ID = 0x07ae62f53ecd0f40) Created: 23.08.2021 17:07:40 Last Modified: 23.08.2021 17:07:40\nSchedule dispatched via message queue (0x07ae62f53ead0f40), Data source identifier (DED82B93BB234FF882B35BEE6E2510B3)\nSchedule Time: 23.08.2021 17:07:40\nProcessing started at 23.08.2021 17:07:40\nProcessing completed successfully at 23.08.2021 17:08:43\nExtraction finished at 23.08.2021 15:08:43 UTC, with servers: tm03n03, TRTH (49.021 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nInstrument <RIC,CARR.PA> expanded to 1 RIC: CARR.PA.\nTotal instruments after instrument expansion = 2\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: CARR.PA,Market Price,,,Inactive,0\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}

rahul.deshmukh

I am just doing some tests initailly with some Rics and fids to check if i am getting response from this RAW request. Is there anything wrong with the fids or Rics or do i need to modify condition or any parameters ? I just need to see response with the values correctly...Why its showing inactive in response...i have edited my question and updated the extraction request which i am using..

zoya faberov

Hello @rahul.deshmukh ,

Let me just confirm, you are doing all steps via curl as per

REST API Tutorial 8: On Demand raw data extraction

1. Authenticating and obtaining a valid token

2. Issuing request with valid token => getting back a submitted jobId/ExtractionID

3. Checking request status on the job ID => confirming it is completed = 200 ( not accepted = 202). And if status=202, checking until status is completed = 200.

4. Once the status becomes 200, extracting result via {{protocol}}{{host}}{{api}}Extractions/RawExtractionResults('{{jobId}}')/$value

This sequence of steps, as described in detail in the tutorial for Postman, will need to be done via Postman, or via curl, or implemented in any language supporting HTTP REST.

?

rahul.deshmukh

@zoya.farberov i did step 1 and2 but not have done step 3 and directly proceed for step 4 where i provide token and my extraction request...all this steps i have done using curl...i already used curl with the same steps for FundAllocationExtractionRequest rest api and it works fine...i am doing the tick history raw request first time so not sure if i miss anything during extraction request...

zoya faberov

Hello @rahul.deshmukh ,

The step 3 is required, to be sure that the job has completed, prior to extracting the result.

The processing is asynchronous, and the time it will take to process may differ based on the job size, and service load at the the time.

If you do not check status, you can not be confident that your result is ready when you extract it.

Please try to check the status, and only extract when the result is ready?

rahul.deshmukh

@zoya.farberov i dont understand how to do implement step 3...during implementation its going to be difficult...i think without step 3 also we can run the extraction request and get the reponse as mentioned in the tutorial this step is optional...the REST API Reference Tree is not yet updated and migrated and its still old...

zoya faberov

Hello @rahul.deshmukh ,

I have just tested the following step 4 curl, on a completed TickHistoryRaw result with status = 200

"curl-7.70.0-win64-mingw\bin\curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults(';MYJOBID')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MyProxy:8080 -v -N -o "tst.csv.gz"

I got the result downloaded, as expected.

Hope this information helps

rahul.deshmukh

@zoya.farberov can you please tell how do i find 'MYJOBID' in my response ? And unlike other Rest API this looks like new step for Raw extraction Request for tick history ? This makes the implementaion very difficult when we try to automate our programming script...because we have to put token and now 'MYJOBID' in the extraction request ? Your GET request looks different than i used in my code: "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('MYJOBID')/$value"

Also does this JobID change everytime like token for each request?

zoya faberov

@rahul.deshmukh ,

You assigned jobId comes back in the response in step 2 above. Do you have it?

I think working through tutorial REST API Tutorial 8: On Demand raw data extraction may be helpful to you.

Postman tool is free, you will be able to download it from https://www.postman.com/downloads/, also by downloading our working request collection that comes with REST tutorials, you will be able to run this and other requests included with the collection.

rahul.deshmukh

@zoya.farberov i have jobID.. you can also see in the reponse which i am getting back...can you please help how to use Postman and the link which you share is not working...

In the first step i generated token

In second step i used this token in my extraction request which i have mentioned in the question

In third step again i have to use curl request with the JobID like you mentioned in your curl to get the response back with fids value ?

To be honest Tutorial is bit confusing for me and not understandable and not all the steps are mentioned with the actual complete request...So i am trying to figure out the soltuion with my example..Is it possible for you just to use my request in question and provide soultion ...thanks...so that i can follow the same step..

rahul.deshmukh

what wrong i am doing here ...in the last step again i am trying to use same token and provided jobid but getting error as :

Caused By: com.sunopsis.dwg.function.SnpsFunctionBaseException: ODI-30038: OS command returned 1. Error details are [sh: -c: line 1: syntax error near unexpected token `('
sh: -c: line 1: `    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \'

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json_values.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X GET \
    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \
    -H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
    -H 'Content-Type: application/gzip' \
    -H 'X-Direct-Download: True' \

zoya faberov

Hello @rahul.deshmukh ,

I have verified the link to tutorial again. It requires the user to be logged into dev portal, maybe that is the issue:

https://developers.refinitiv.com/en/api-catalog/refinitiv-tick-history/refinitiv-tick-history-rth-rest-api/tutorials#rest-api-tutorial-8-on-demand-raw-data-extraction

Please note:

My step 2 returns, in headers, assigned jobID = ExtractionID. It is part of Header response Location, for example:

https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId=';0x07ae630fc0dd0f40')

same as in tutorial

"Request raw data - HTTP response"

Step 3 checks the status of the request.

I will try to prepare all steps in curl as example, please allow some time for this - we have fully working example for Postman, available with tutorial.

rahul.deshmukh

@zoya.farberov i never faced this issue when i used previous refinitiv rest api...it will be really helpful if you can prepare stepwise steps with my given example of curl...please take your time as this is extremely important for us here to make it work....i have user login and when i used your link it again shows the same tutorial...i have updated my question now with the steps i followed using curl and where i actually got stuck to avoid confusion...i think you can use my example as well just need to generate new token and jobid i think...thanks

zoya faberov

@rahul.deshmukh

I suspect the job is not processed yet, when you request the result.

The right way to be sure the job has completed, is to check as in step 3.

Possibly, if(?) by now, a long time have passed from the time you have submitted step 2, you can retry only(!) step 4, to see if now you will be able to retrieve the result, as by now it might have completed.

This is a guess.

zoya faberov

Hello @rahul.deshmukh ,

Steps that we use to extract TickHistoryRaw content with curl:

1. Authenticate and obtain a valid token

...curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async"  https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @creds.txt

where creds.txt file is located in the same directory and contains

{
 "Credentials": { 
      "Username": "MYVALIDDSSUSERID", 
      "Password": "MYVALIDDSSPASSWORD"
 }
}

Resulting in

@{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Edm.String","value":"MYVALIDTOKEN"}HTTP/1.1 404 Not Found
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 615

2. Use the valid token to create TickHistoryRaw extraction request:

...\curl-7.70.0-win64-mingw\bin\curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @requestBody.txt
HTTP/1.1 200 Connection established

Contents of requestBody.txt file saved into the same directory

{ 
  "ExtractionRequest": { 
    "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest", 
    "IdentifierList": { 
      "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",  
      "InstrumentIdentifiers": [{
        "Identifier": "CARR.PA",
        "IdentifierType": "Ric"
      }]
    },
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2016-09-29T12:00:00.000Z",
      "QueryEndDate": "2016-09-29T12:10:00.000Z",
      "ExtractBy": "Ric",
      "SortBy": "SingleByRic",
      "DomainCode": "MarketPrice",
      "DisplaySourceRIC": true
    }
  }
}

obtaining Extraction ID and status:

HTTP/1.1 202 Accepted
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:30:25.3627291
BeginRequestDate: 2021-08-23
Status: InProgress
Progress:
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae64718bad0f41/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 26.47529
RequestsPerSec: 0
W3WP-PrivateBytes: 7120
Date: Mon, 23 Aug 2021 21:30:55 GMT
Content-Length: 0

3. Check status of the extraction via job ID

...curl" -i -X GET -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:8080

until it becomes status 200=completed

HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:57:10.4224282
BeginRequestDate: 2021-08-23
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae85c312ed0f67/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 17.59716
RequestsPerSec: 7.619745
W3WP-PrivateBytes: 1569704
Date: Mon, 23 Aug 2021 21:57:10 GMT
Content-Length: 1414

4. Pull down the extraction result and store it into a file

... curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults(';0x07ae64718b9d0f41')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -v -N -o "tst.csv.gz"

rahul.deshmukh

thanks
@zoya.farberov does this works for you ? Is it working example ? I will give try today and let you know about the result...

zoya faberov

Hello @rahul.deshmukh ,

This is a fully working example

rahul.deshmukh

@zoya.farberov i am able to generate token in step1. But when i use your second step i am getting error now:

test.txt

Below is my script:

curl -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw" -H "Authorization: Token MYTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:8080 -d { 
  "ExtractionRequest": { 
    "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest", 
    "IdentifierList": { 
      "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",  
      "InstrumentIdentifiers": [{
        "Identifier": "CARR.PA",
        "IdentifierType": "Ric"
      }]
    },
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2016-09-29T12:00:00.000Z",
      "QueryEndDate": "2016-09-29T12:10:00.000Z",
      "ExtractBy": "Ric",
      "SortBy": "SingleByRic",
      "DomainCode": "MarketPrice",
      "DisplaySourceRIC": true
    }
  }
}

What mistake i did ?

zoya faberov

Hello @rahul.deshmukh ,

Do you see this part:

""No permission for template \"TickHistoryRawReportTemplate"?

Looks like you are not permissioned for TickHistoryRaw retrieval.

Please contact your Refinitiv account manager, to verify your RTH permissions.

rahul.deshmukh

@zoya.farberov thanks i will contact my refinitiv account manager and try again this request and let you know in case any issues

rahul.deshmukh

@zoya.farberov now i tried with another refinitiv userid from which i have permission for tickhistoryraw retrival and got the below message. does it mean i can use use this extraction id now to check the status and pull the extraction result ? Or is still have issues ? I dont know why it has too many text values in the output..

test.txt

zoya faberov

Hello @rahul.deshmukh ,

Yes, you can use this Extraction ID for status check.

This is why the majority of our clients prefer Postman tool and our provided collection for this type of testing and learning, you have indicated that you prefer curl. Some prefer curl.

With Postman, one has a very clear GUI-interface view of HTTP requests and responses only, and is very easy to interpret.

rahul.deshmukh

@zoya.farberov in the step 4. what result did you got ? i am not getting result ..i have attached the file and why does this all files has raw and unwanted values and how do we remove it becuse at the end we want to import this data into oracle database and if we have unwanted values in file its very difficult to import it ....also in your script you mentioned tst.csv.gz what does this mean ? For step 4 can we also export the data in csv file which will then be easy for us to import into database table...Currently i dont know how to use Postman and as we have to automate our process daily for extraction i think this is not possible with the Postman tool ?final_extraction_result.txt

zoya faberov

Hello @rahul.deshmukh ,

I strongly recommend against using curl for production integration with RTH REST.

Curl is intended primarily for testing, debugging, and learning APIs capabilities.

I recommend integrating, using a programming language that supports HTTP REST, of your preference or RTH .NET SDK.

Postman can be used for automation, however, it is also targeted and intended as a testing tool. Allows you to test HTTP REST quickly and efficiently.

I would like to suggest to review our Programming without SDK Tutorial and .Net SDK Tutorials (with downloadable code examples) as working examples of programmatic integration, so you can make best design choices per your requirements, so hope you will find the tutorials very helpful.

zoya faberov

Hello @rahul.deshmukh

Please confirm, did step 3 return status =200 before you have run step 4?

Once this is confirmed, please confirm if when you run step 4, a the file named tst.csv.gz is created in the same folder, when you run step 4 - as I see no errors in your verbose curl output,

rahul.deshmukh

@zoya.farberov As we are using ODI as ETL tool where we are trying to use curl RTH REST for extracting the response. Is working with curl is not stable or have any performance issue with RTH REST ? I have seen in tutorial about the Python RTH code example here RTH Python and would like to know is it working sample example and can we try this for TickHistoryraw ? Is there any working example for TickHistoryRaw REST Api for Python ?

rahul.deshmukh

Yes step 3 returns status=200 but for step 4 there is no file names tst.csv.gz created...And i dont understand why we have unwanted result values in output files...cant we remove it ?