Login Page - Create Account

DTC Protocol Discussion Forum


Date/Time: Tue, 13 Nov 2018 04:18:46 +0000


Historical data zip format?

Support Request:
[2016-10-24 17:44:35]
DabbaDo - Posts: 103
From Sierra Chart server, I receive historical price data without problems when it is NOT zipped.
When zipped (i.e. the header shows UsingZLibCompression == 1), the next bytes coming across the wire are in what format? The first two bytes seem NOT to be compressed. Are they a batch size? And the next two bytes seem not to be a DTCMessageType, unless perhaps they are compressed.
In other words, is there some documentation/guidance about the bytes sent from the server following a HistoricalPriceDataResponseHeader message where UsingZLibCompression == 1?

Cheers,
Dale

P.S. I figured this out. I must skip past the 2-byte ZLib header (ref. https://tools.ietf.org/html/rfc1950) and then the DeflateStream class decompresses the records with no problems -- ref. https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx [From version 4.5, .Net started using ZLib under the covers.]

[But it still might be worth a documentation one-liner, IMHO.]
Date Time Of Last Edit: 2016-10-25 09:16:09
[2016-10-25 04:26:03]
DTC Engineering - Posts: 270
We are not familiar with this because we use this library which handles all of this:
http://www.zlib.net/
[2018-01-06 21:18:37]
AndrewAMD1 - Posts: 26
I have a question about zlib with DTC. (Long term, I might implement this if I run into bandwidth problems.)

So with uncompressed messages, I can always read the type and size up front and size my buffers accordingly. (This is using C++17 & Boost Asio.) But I think it is not this simple when using zlib.

Okay, so I request historical data and get an uncompressed header. But then the rest of the messages are in compressed zlib? I need a feasible way to allocate buffers and decode without ruining the decompression process.

1) In general, how should I approach receiving zlib data from the historical server?

2) How am I supposed to know when I received all messages for a particular request if the "last message" flag is already encoded?
[2018-01-07 03:22:45]
DTC Engineering - Posts: 270
Okay, so I request historical data and get an uncompressed header. But then the rest of the messages are in compressed zlib?
Yes.

1. You should do this incrementally. When you get a batch of data on the socket, give it to zlib. It will respond with some uncompressed data and you put that into a buffer and then process as many complete messages out of it that you can. Continue with this process.

2. You will be looking at the decompressed messages and you know you are done, when there is the flag indicating the final record has been sent. s_HistoricalPriceDataRecordResponse::IsFinalRecord

To post a message in this thread, you need to login with your Sierra Chart account:

Login


Login Page - Create Account