Inconsistent number of data points returned for historical data request

GoGoGroundhog

I would like to raise your attention on an unexpected behaviour regarding the API's time series data request.

I can reproduce the problem with the DataApiUsageExampleTimeseriesData that can be downloaded here with slight code changes.

In TimeSeriesRequestExample.Launch() I set up a request like this:

request = timeSeries.SetupDataRequest("CFI2Z7") 
.WithView("TRDPRC_1")
.WithAllFields()
.WithInterval(CommonInterval.Intraday1Minute)
.WithNumberOfPoints(20000)
.OnDataReceived(chunk => { Console.WriteLine("Count={0}",chunk.Records.Count()); })
.CreateAndSend();

When requesting
20000 data points now (27th Oct 2017, 14:30 CEST), the number of records provided will be
14945.

Note that the number returned is less then the number requested. However, when requesting
30000 data points, the result set is much larger,
18586 data points.

Also note that when requesting 40000 data points you still get only 18586 records delivered, so I assume this is all data available for that symbol.

Why don't we get back the full amount of data back when requesting 20000 records even though it fully fits the requested number?(Also note that in all case I will get two chunks sent, with the second having 0 records and IsLast == true).

Alex Putkov.1

Well, DTEGn.DE is not as liquid as you think. If you look close you'll see gaps in the minute bars returned. E.g. today Nov 7 there were no trades on this stock between 16:31:47 and 16:33:11 exchange time, which results in the 16:33 bar missing. But you're right that what I mentioned is not the full story. There are other factors affecting how many rows are returned. Those factors include predictable things such as trading sessions. E.g. right now if I request 1,000 rows, which is less than the number of rows I would get for two full trading sessions if this stock traded every minute, I get back 994 rows going to the start of trading session yesterday. But if I request 1,100 rows, which is more than the number of rows I would get for two full trading sessions if the stock traded every minute, I get back full 1,100, of which 106 rows belong to the session 3 business days ago. There are also factors that cannot be predicted such as what's ready available in the cache in the Timeseries Proxy, which is an element of distributed delivery infrastructure. The exact same request submitted multiple times may return different number of rows. The bottom line though is that you can rely on this parameter to limit the number of rows returned to below the value you specify, but you cannot rely on it to return the exact number of rows. Unless the instrument is so liquid that it's guaranteed to have a trade in every 1 minute interval. Try your experiment with IBM.N.

Alex Putkov.1

WithNumberOfPoints parameter can be used to limit the number of rows in the response to below the specified threshold, but you cannot rely on this parameter for the request to return exactly the number of rows you specify unless the instrument is very liquid and is guaranteed to have at least one trade within every minute of every trading session. For less liquid instruments such as this future there are minutes when no trades occur, which you can think of as "empty rows" that still count towards the threshold specified in WithNumberOfPoints parameter. The other limit you need to be aware of is the depth of price history stored on the backend, which is provided here. As you can see for 1 minute interval the depth is 90 days, and this is the limit you hit when you increased the value of WithNumberOfPoints parameter to 30K. As of 8pm New York time today Oct 31st 2017 price history in 1 minute aggregation goes back to Aug 1st 2017 and this entire timeseries is 18,652 rows.

GoGoGroundhog

Hi Alex,

thanks a lot for your response.

In my opinion though, this cannot explain how the API behaves in this regard.
For example, requesting a very liquid equity, DTEGn.DE 1 Minute data successively increasing WithNumberOfPoints by 1000 for each request yields the following:

DTEGn.DE Intraday1Minute 1Min Requested: 1000 Received: 1000
DTEGn.DE Intraday1Minute 1Min Requested: 2000 Received: 1986
DTEGn.DE Intraday1Minute 1Min Requested: 3000 Received: 3000
DTEGn.DE Intraday1Minute 1Min Requested: 4000 Received: 4000
DTEGn.DE Intraday1Minute 1Min Requested: 5000 Received: 5000
DTEGn.DE Intraday1Minute 1Min Requested: 6000 Received: 6000
DTEGn.DE Intraday1Minute 1Min Requested: 7000 Received: 6665
DTEGn.DE Intraday1Minute 1Min Requested: 8000 Received: 8000
DTEGn.DE Intraday1Minute 1Min Requested: 9000 Received: 8929
DTEGn.DE Intraday1Minute 1Min Requested: 10000 Received: 10000

Note that you do only get 8929 data points back when requesting 9000 points, but you do get the full 10000 when requesting 10000 points.

GoGoGroundhog

Thanks for the additional explanation. I guess if one's app needs to provide a certain amount of data points, it boils down to making multiple requests, stitching results together until you reach your required number of data points (or no additional data is delivered by the API).