RDP news with no payload

OneTick
OneTick
in Refinitiv Data Platform Libraries

We have a python application using RDP API receiving news in json format. Most of the json objects received contain payload. For example

{
  "attributes": [
    {
      "domain": {
        "type": "string",
        "value": "story"
      }
    }
  ],
  "envelopeVersion": "1.0",
  "ecpMessageID": "urn:newsml:reuters.com:20230721:nBw70J7x7a:1",
  "sourceSeqNo": 1153992,
  "distributionSeqNo": 478052,
  "sourceTimestamp": "2023-07-21T10:50:00.131Z",
  "distributionTimestamp": "2023-07-21T10:50:03.243Z",
  "payloadVersion": "2.0",
  "subscriptionID": "535c1fcb-6756-439b-8fb9-f453ca89acc3",
  "payload": {
    "newsItem": {
      "_guid": "tag:reuters.com,2023-07-21:newsml_Bw70J7x7a",
      "itemMeta": {
...

But some json objects received do not contain payload. For example

{
  "attributes": [
    {
      "domain": {
        "type": "string",
        "value": "story"
      }
    }
  ],
  "envelopeVersion": "1.0",
  "ecpMessageID": "urn:newsml:reuters.com:20230721:nRSU8251Ga:1",
  "sourceSeqNo": 1146276,
  "distributionSeqNo": 477496,
  "sourceTimestamp": "2023-07-21T10:15:15.352Z",
  "distributionTimestamp": "2023-07-21T10:15:16.888Z",
  "payloadVersion": "2.0",
  "subscriptionID": "535c1fcb-6756-439b-8fb9-f453ca89acc3",
  "href": "https://api.refinitiv.com/file-store/v1/files/4bef-dcc1-89bf39f4-bf4f-79d00c6565d5/stream",
  "internalS3Url": "a204892-content-eds-alt-prod-use1/news-stories/024c/2023-07-21/acf01783-38e4-4ef2-b537-b339a94dcf93.json"
}


How to interpret such news items withput payload? Is there an API to get expanded news story?


Tagged:

Best Answer

  • raksina.samasiri
    raksina.samasiri admin
    Answer ✓

    Hi @OneTick ,

    Is /alerts/v1/news-stories-subscriptions the endpoint you're using?

    If so, in it's reference page in the API doc, there's a paragraph explaining this

    Downloading with Messages without "payload" field and it contains href link

    Actually, we have maximum size limitation to deliver messages to user SQS queue that it's 180 KB and if message size is larger than this limit, we'll remove "payload" field and put the link into "href" fields as shown below

    1. Here is example messages

       {
       "attributes": [{
               "attribute1": ["value1"]
           }, {
               "attribute2": ["value2"]
           }
       ],
       "envelopeVersion": "1.0",
       "ecpMessageID": "99d512f7-d4b3-4eb9-8ad3-01b0a0cded33",
       "sourceSeqNo": 123,
       "distributionSeqNo": 1,
       "distributionTimestamp": "2021-02-06T18:52:11.673Z",
       "payloadVersion": "1.0",
       "subscriptionID": "48e4f082-6993-445a-85fc-bbc2e7eaf553",
       "href": "https://api.ppe.refinitiv.com/file-store/beta1/files/42e5-0b2d-eb853c37-abd4-8c8ec6a60e56/stream",
       "internalS3Url": "a204892-content-eds-alt-dev-use1/news-stories/test/temp_large_msg_5xMB.json",
       "ecpID": "23644"
   }

    2) Once you receive this message, you have 2 method to download this file as following below

    2.1 Download this file from RDP API Playground via web-interface

    2.1.1) APIDOCS GET File Stream API

    2.1.2) From href, fileId will be located in the url. In this case, your fileId is 42e5-0b2d-eb853c37-abd4-8c8ec6a60e56

    2.1.3) From No.1, please replace {fileId} in this url https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream and you'll get actual payload from the response as example below

    {
    {
        "newsMessage": {
            "header": {
                "channel": [
                    {
                        "$": "TXT"
                    }
                ],
                "destination": [
                    {
                        "_qcode": "NP:BSW"
                    },
                    {
                        "_qcode": "NP:CNR"
                    }
                ],
                "priority": {
                    "$": 3
                },
                "sender": {
                    "$": "reuters.com"
                },
    .....................................................
    }
}

    2.1.4) There are another option, if you would like to download file later you can specify with this url https://api.refinitiv.com/file-store/v1/files/{fileId}/stream?doNotRedirect=true and you'll receive s3 url and the link will be expired within 6 hours. You have to call this fileStream API again once it expire for getting new s3 url and example response is shown as following below

        {
        "url": "https://a204892-content-eds-alt-prod-use1.s3.amazonaws.com/news-stories/0025/2021-01-21/5e6247d7-e53d-41fb-a59a-81a82990c8f0.json?x-request-Id=2e7d4b2c-0aba-4d08-8750-3dec9d2570b2&x-package-id=4b74-070f-1558af6d-b1ec-3ade7455b05b&x-client-app-id=6bc155331c3f4c78bae8b0389c262ec17f2a9191&x-file-name=filesets-4bdbe568-052e-4c45-a73d-40dcfb4473c5-1612980871&x-fileset-id=45ce-0e39-1e51d321-877c-217289179fdc&x-bucket-name=protocol-alarm&x-uuid=GE-203&x-file-Id=4411-c086-cba648a7-8c0d-6c06529e8588&x-fileset-name=bus&x-event-external-name=cfs-claimCheck-download&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELL%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQCA4Efh5JfLsAif5vTxhVW7fAzT%2BLKkf6f9Twxw0hutUwIgeQcYDvbwDk9O9jFErTTs30n2eHUGA5cZTLF40hs7We8q2AMIq%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw2MDgwMTQ1MTUyODciDAZrUt4WYi%2FX1RH9QCqsAxtw%2ByGIsYzVsLSrnh%2BL3zFVP8VBVNXYHUwmo7LLhk6Hpi8oZIL38S5rWsbnTO%2BHdG9M5V%2FMBIYqyaRpY8Lt9iThpEEQxTa6Yfsgyc0gER4l2Bhh%2B4MJH8Z%2FrigL7z3us49r5dxgd5aepEWgixbutqFL5pQPUcrHcpukbVRbM9mk%2F9d82VkHnXX62gBFoNJk6fQM5Io7ycvdcDw%2Fqz70DXYpzOMvlx8NF9nSgrFklBw9L%2F9tKMOrnwioKqmW5FUdlgaH3COkAQBabxFVg2rMiaEOpVaOH9Fn%2F2qdZ%2B4q4OQhzEkYo2UaRN0s1Y26TotdGhk2zAkVHSIghvUBfWjzdBnAu7rV9xptWNJ7AyJbDjw0G3yQxU1nznHRqjZpMA7jv8KBU5b6j89IlV7arfzH6mw0c990ziO5tWNPCxIHn6cY%2BPgBOtcqUK%2BisPDTDaWlAxLvIVWYc1%2Bi0IPd7OuT6h5cG5sodyQsB%2FA%2B3kYHuaHBGZ%2FwBKwiLoVQ7f6qGgF8wkBuWXCk5w21Ety1DCPRSWtLkK94gvP4hMgNIc9jXZg4GyauDNGQZ4Em2%2FOwMMewkIEGOuwB7ScuAYjKxmcztZU2heFRheWByvVm20XkycUKNM%2B1mC5GpVXnP75YiNjXcngDedzeo%2Bv6yfmNo3upsTzqsdvNu%2Fl8bAK68xUJC6oiq3gmhJ6d3CYdkAUGcUElQ2VX7cVCrHAMn41lTIjx5l0iCPNeR6firb5m%2Bl6%2FZYFs1VX5mfHrxMjv3qvHoAv1h6HIoN4YJQKipq%2Fyh4auwW5ESA135i1ZRPpZzmKfjTcDbj0ha2jCNw9x5Wmg35u5x7cPZaVI4%2F%2FngCoqLVG9H3q3CRJSS%2FsD4NhoiXtmUTRk3kVDTrJ%2BA4cBebhtXegWkC0%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210210T191613Z&X-Amz-SignedHeaders=host&X-Amz-Expires=21599&X-Amz-Credential=ASIAY3ED5HRL2GZ4Y354%2F20210210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1d535ea691fbff82121a56f7f7b4525d0cd247cc44f2841e8fd6b0199318268d"
    }

    2.1.5) You can download file from "url" field directly

    2.2 Download this file from RDP API Gateway

    2.2.1) Login with your username and password to retrieve request token from RDP API Gateway via https://api.refinitiv.com/auth/oauth2/v1/token and please make sure you specify "trapi.cfs.subscriber.read" scope in your request as example below.

        - Headers --> Key: Content-Type  Value: application/x-www-form-urlencoded
    - requestBody --> grant_type=password&username=<USERNAME>&password=<PASSWORD>&scope=trapi.cfs.claimcheck.read trapi.cfs.subscriber.read&client_id=<CLIENT_ID>&takeExclusiveSignOnControl=true
    Note: If you don't have CLIENT_ID please generate via [AppKeyGenerator](https://apps.cp.thomsonreuters.com/apps/AppkeyGenerator).

    2.2.2) Please make sure you found scope name "trapi.cfs.claimcheck.read" or "trapi.cfs.subscriber.read" as example response below

        {
       "access_token": "eyJ0eXAiOiJhdCtqd3QiLCJhbGciOiJSUzI1NiIsImtpZCI6IlNMVWxGQVlkZUhCb1dmMnplTG9lMTZKY2JrcVpXLU9hRVh2NTZsVkZ3MzAifQ.eyJkYXRhIjoie1wiY2lwaGVydGV4dFwiOlwiUlpvc1BKX0F2MnlWWUNKamtGTGhxR2liR1owZDRqVUNTU0VFamYxX2Y5cU1CcFh6NFphMWp3TFA3NzFraHdBOGZBdFh4WnZESVo5RXg5M0s5VTFsTmxfWjNMSi0tQTRCRjhwYm8tZjhHLWZBYklnS2NJR3JHSGdZbklEeWxWcGpJbnlIQm5jYnR1Y2pydWV3M1RBY0xPUlhaZmVxeWNSVkRPOHMzcHRIS01XbXRzdzk3WFBjd2hFbVF4YWR1UzZWcEV2ak1KMDNkT1dlV18zQWJDZkJqZmw0QkZQRjhpOHhnRGtKUTR3Wmo2WVVrdGNBWEY1SmZsbGtUZVQ3RVhUS09ZR0hKSlhILU9HVUdRYThDaEk0cy14ZUVsbDRtaHBpYm5xWjE1d3BubXNFQktqNS1ERENrUk1tZjVqMGJVUDZUYTJCdngwMjJiRjczQTh0VzhUYy0tZ0ZIdnlRbE0zMnk5ajJyQWFNS3p3b04xTVU5SmxPQnRVWWpKWmVrSUpWQlp1Tm5lYlc4RTNTUE5fMnlHVjFKUVljZmQ5aXJ5cXNWTmdzWDY5NDBIbjJnZ3dwZzRCTXA2N2ZEajNDUmdfT3hWZTc0c2xIYmkzWnNpS0djbDUzUUNtdm9oVm92c1U1bUdBXCIsXCJpdlwiOlwiWFB2NDAyeEswYWdKU1M3TFwiLFwicHJvdGVjdGVkXCI6XCJleUpoYkdjaU9pSkJWMU5mUlU1RFgxTkVTMTlCTWpVMklpd2laVzVqSWpvaVFUSTFOa2REVFNJc0lucHBjQ0k2SWtSRlJpSjlcIixcInJlY2lwaWVudHNcIjpbe1wiZW5jcnlwdGVkX2tleVwiOlwiQVFJQkFIamxWM2dlbGtxMGxibUhEckpsU3BQTGlLZTRqM2NaeDJLbXR3djBUdnFZTGdFbUxFamU3WGZ3a1pTd3N0Vk5nVndaQUFBQWZqQjhCZ2txaGtpRzl3MEJCd2FnYnpCdEFnRUFNR2dHQ1NxR1NJYjNEUUVIQVRBZUJnbGdoa2dCWlFNRUFTNHdFUVFNOEs3UzFBYUU2V00wWnQ4UUFnRVFnRHNHZjNVQy02SGpHazFFZ3J4dVhPbThVMm9VdzRVMTkwOFdibzZRUTdnQ3NRSUhTNWstbkpsajBXcG8yY0M4VFREQWZ5anhwNmgyQUxDVjhBXCIsXCJoZWFkZXJcIjp7XCJraWRcIjpcImFybjphd3M6a21zOmFwLXNvdXRoZWFzdC0xOjY1MzU1MTk3MDIxMDprZXkvNjJhMzY4MzUtYmNiNS00N2Y5LWJiMDUtNzk5MjA1N2MxNWM3XCJ9fSx7XCJlbmNyeXB0ZWRfa2V5XCI6XCJBUUlDQUhqOW9sZDI5RXB3VU1hZTZCSjRRVWVrOGZMQTlxQ2t3YlBKUFF1QV9tMEdwUUVMekthMWlZUXFvRy0zeGZ2aGJKTDJBQUFBZmpCOEJna3Foa2lHOXcwQkJ3YWdiekJ0QWdFQU1HZ0dDU3FHU0liM0RRRUhBVEFlQmdsZ2hrZ0JaUU1FQVM0d0VRUU1EM1dPT0NZdUxrcmZKcGxNQWdFUWdEc2gwWHVfb2kzNlptUFR6VFQ4X0loNTR2Y0dGS3J4UnVfTjNVS01zQ2w1a3Z0Yk1Vanl2c2ZxZTdzeTAyUlI0M2Q3ZWJlV1JfVmtEeDMwMXdcIixcImhlYWRlclwiOntcImtpZFwiOlwiYXJuOmF3czprbXM6dXMtZWFzdC0xOjY1MzU1MTk3MDIxMDprZXkvNjUwOGQzM2YtMjU4My00YjNmLTlkZjItYmQwZTQ0YzUxZTQzXCJ9fSx7XCJlbmNyeXB0ZWRfa2V5XCI6XCJBUUlDQUhoWEtRM2g5YU9NVThPX24tZy1ISV93SmYwTmRCRzc0RHpLSjAzVDk0T2d3UUcxZURVVmQ4TmhGS0c4aUdBM2RHbjRBQUFBZmpCOEJna3Foa2lHOXcwQkJ3YWdiekJ0QWdFQU1HZ0dDU3FHU0liM0RRRUhBVEFlQmdsZ2hrZ0JaUU1FQVM0d0VRUU1BT1NHV3RiUEEzOHNYQ2FDQWdFUWdEdEJBU1RSenpxd00wYUxQWnRTeEVoellkb3RZYWlxYWVDczlkSVRwV0hJTFdBbDBjOEZPNzhLVlVsZENzYmFqVmNvN0toWVVxeWZyZW1FbWdcIixcImhlYWRlclwiOntcImtpZFwiOlwiYXJuOmF3czprbXM6ZXUtd2VzdC0xOjY1MzU1MTk3MDIxMDprZXkvOGQwOTQzM2YtYWNmZS00MjQ0LWExYjktZTcwOTVjOGFkNTZlXCJ9fV0sXCJ0YWdcIjpcImFYeFA2d3RydlhQalFzcnNBZVdOWUFcIn0iLCJyczEiOiIwYmM3ZWE2NTUyM2UxZDY5MGIzODFjYWE3Y2Y2NTJlOWUxYjc5MjA3IiwiYXVkIjoiNmJjMTU1MzMxYzNmNGM3OGJhZThiMDM4OWMyNjJlYzE3ZjJhOTE5MSIsImlzcyI6Imh0dHBzOi8vZmluYW5jaWFsLnRob21zb25yZXV0ZXJzLmNvbS9hcGkvaWRlbnRpdHkvc3RzX3ByZXByb2QiLCJleHAiOjE2MTI5ODQ4NzAsImlhdCI6MTYxMjk4NDU3MH0.hJLO255l2Im7vwIzf2H2Bt8iToeJ8wIWpc7DavIkWNLTzQFsJEHNQOsTlnC5dUtvMFzKBd2DoXGTJO1Oll42pQnf5knD7HuQLK-9pCND1jtt1_t_0q0qnVjIce_050pm-wn8giLkGZsdE1ZNXBIEU2rPgQqpg2yjF7ADXKRIMcXfqDBkuHzMvvx_X_cGDaroTUcBvT3iC1hxFMmgBNK56rV_uaHTuPM9LbP1mpdn_27z7dHmCVB-tAaBsf_OBSYDvxfIIdikYVf69a_1t4QN6rlbvH9OstTXVrkGtUYrGvDb9kUAzzxQbIhjn5zlFkJGC29-dcWSMug5tUDAppTC8w",
       "refresh_token": "aa02622a-dd18-431d-937e-01f63ced29df",
       "expires_in": "300",
       "scope": "trapi.cfs.claimcheck.read trapi.cfs.subscriber.read ....",
       "token_type": "Bearer"
    }
     Note: Please keep refresh_token to renew expired token and access_token for downloading actual payload via file stream API

    2.2.3) When token is expired please retrieve token from RDP API Gateway as same as 2.2.1 but it's different request body as shown below

    - Headers --> Key: Content-Type  Value: application/x-www-form-urlencoded
- requestBody --> grant_type=refresh_token&username=<USERNAME>&refresh_token=<REFRESH_TOKEN>&client_id=<CLIENT_ID>
Note: it's more secure and more comfortable that you don't have to supply password and scope every time you retrieve token. However, if you could not use REFRESH_TOKEN to retrieve a new one please use ACCESS_TOKEN by follow 2.2.1 instead

    2.2.4) Result should be similar with 2.2.2

    2.2.5) Please check "href" field from messages that you receive and you can choose whether you want it redirects to download actual file or download it later by keeping s3 url from 2 url as following below

    - Redirect --> https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream
- Do not redirect --> https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream?doNotRedirect=true
For example results please check 2.1.3 and 2.1.4. Here's request information below 
- Headers --> Authorization: "Bearer <ACCESS_TOKEN>" &nbsp;

Answers

Categories