Login Page - Create Account

DTC Protocol Discussion Forum


Date/Time: Tue, 13 Nov 2018 04:18:21 +0000


Does SC historical connection support Google Protocol Buffers?

Support Request:
[2017-05-05 22:02:05]
Queeq - Posts: 31
Hi,

I am testing DTC protocol with Sierra Chart (version 1549). SC server encoding is set to automatic. Intraday logon sequence goes fine, SC acknowledges the use of Protobuf, and the following requests are flowing successfully. Here's the Encoding Response:


b'\x07\x00\x00\x00\x04\x00\x00\x00DTC\x00'
^^^^^^^^


However, when connecting to Historical data port, Encoding Response indicates binary:


b'\x07\x00\x00\x00\x00\x00\x00\x00DTC\x00'
^^^^^^^^

Consequently, the login sequence fails. SC itself reports the following in the log:


HD Server | Unexpected DTC encoding request: 4 | 2017-05-05 21:41:07

The client program is written in Python and protobuf classes are generated from proto v2 file.

Have I missed something?
Date Time Of Last Edit: 2017-05-05 22:03:04
[2017-05-12 09:50:05]
Queeq - Posts: 31
Answering my own question: there is no support for anything else than binary (https://www.sierrachart.com/index.php?page=doc/DTCServer.php#HistoricalPriceDataServer ):
Only the DTC Binary Encoding is currently supported.

For other people looking for answers, there's much longer discussion on Sierra Chart forum: https://www.sierrachart.com/SupportBoard.php?ThreadID=11110

Dear SC team, I need to understand how to move forward. Any plans or better ETA for other encodings support?
Date Time Of Last Edit: 2017-05-12 09:50:29
[2017-05-14 04:18:21]
DTC Engineering - Posts: 270
We have to look into this. This may not be difficult for us to add support for the other encodings for the historical data server within Sierra Chart.
[2017-05-14 04:43:24]
DTC Engineering - Posts: 270
We will try to get all encodings supported with the DTC Protocol historical data server in Sierra Chart version 1558.

This should be out this coming week.

Only binary encoding and binary encoding with variable length strings will support compression. The others will not.
Date Time Of Last Edit: 2017-05-15 05:56:57
[2017-05-17 15:45:25]
DTC Engineering - Posts: 270
Version 1558 of Sierra Chart has been released and it supports all of the DTC Protocol encodings including Google protocol buffers for historical data. Please test and let us know how that works.
[2017-06-09 05:45:57]
Queeq - Posts: 31
Tested with historical data connection (the whole sequence from encoding request to data reception). So far so good. I will report if there are any issues. Thanks a lot!
[2017-06-09 20:44:58]
DTC Engineering - Posts: 270
This is good.
[2017-12-28 08:34:15]
EricH - Posts: 17
Queeq, Can you share with us the python with google protobuf code?
I am stuck even at the login message...

Thanks!
[2017-12-28 08:38:12]
DTC Engineering - Posts: 270
Make sure you read the information here about using Google protocol buffers:
https://www.sierrachart.com/index.php?page=doc/DTCProtocol.php#GoogleProtocolBuffers
[2017-12-28 08:43:37]
Queeq - Posts: 31
Hi EricH,

I was going to open-source the code I've written for myself, but it requires quite some cleanup (mostly getting rid of multithreading as it turned out to be slower than single thread), "librarization" and packaging, so it's easily pluggable to any project. This hasn't been a priority for me, though, but I'll do it sooner or later.
Date Time Of Last Edit: 2017-12-28 08:45:34
[2017-12-28 08:51:08]
EricH - Posts: 17
Dear both, the last URL of 'header/msg length' note does the tricks.
And your code did exactly that! I thought it was specifically for encoding request!

Still, look forward to your github :)
[2018-01-02 02:29:06]
EricH - Posts: 17
To continue the Python/google protobuf and DTC experiments.

Today's issue is on 'how do you specify the 'start/end date' for a historical data request.
From the spec, both start/end time should of unix epoch time in sfixed64.
So my question is what's the equivalent of sfixed64 in python?

I tried the following:

>>>
message = Dtc.HistoricalPriceDataRequest()
message.RequestID = 99
message.Symbol = "CRM"
message.RecordInterval = Dtc.INTERVAL_1_MINUTE
message.StartDateTime = 0
message.EndDateTime = long(1512000000)
message.MaxDaysToReturn =int(30)
message.UseZLibCompression = 0
>>>

and send the request to SC. The EndDateTime, 1512000000 is 11/30/2017.


SC log shows

HD Server | Received login. Requesting authorization. | Username: EricHW. | 2018-01-02 10:15:14
HD Server | Login has been authorized. | Username: EricHW. | 2018-01-02 10:15:14
HD Server | HD Request #99 | Socket: 6986 | Symbol: CRM | ServiceCode: | RecordInterval: 60 | MaximumDaysRequested: 30 | Start date-time: Since beginning | ClientRequestID: 99 | Username: EricHW. | 2018-01-02 10:15:14
HD Server | HD Request # 99 | Socket: 6986 | Symbol: CRM | ServiceCode: | RecordInterval: 60 | Records Served: 17651 | ClientRequestID: 99 | Username: EricHW. | 2018-01-02 10:15:14
HD Server | Timed out waiting for client to close socket. | Username: EricHW. | 2018-01-02 10:16:14


From python debugger I can see roughly 60 days of records, 1st and last's timestamp decoded as
time.struct_time(tm_year=2017, tm_mon=10, tm_mday=31, tm_hour=11, tm_min=27, tm_sec=5, tm_wday=1, tm_yday=304, tm_isdst=0)
time.struct_time(tm_year=2017, tm_mon=12, tm_mday=29, tm_hour=22, tm_min=29, tm_sec=45, tm_wday=4, tm_yday=363, tm_isdst=0)

So my questions are:

1. how do you specify the EndDate in this case? I've tried in python casting to int and long, or no casting, all with the same results.
2. From the log it seems SC does not properly decode the 'enddate' since there's no log message about it.
3. Why is 30-day request not respected?
Date Time Of Last Edit: 2018-01-02 15:07:44
[2018-01-03 09:30:06]
DTC Engineering - Posts: 270
1, 2, 3. These particular message fields are not currently supported but we will get these supported in the next release of Sierra Chart.
[2018-01-03 09:57:44]
Queeq - Posts: 31
EricH,

I'm going to release what I have this week. I've already extracted it from my project, but now need to test it as a separate library. It has several limitations (e.g. only historical data), but could still save you quite some time.
[2018-01-04 09:52:23]
Queeq - Posts: 31
https://github.com/Queeq/pydtc

Although working, it's at a very basic alpha stage. Feel free to post bugs and pull requests on Github or privately adjust for your own needs.
Date Time Of Last Edit: 2018-01-04 09:53:15
[2018-01-04 15:31:27]
EricH - Posts: 17
Queeq
Thanks a lot!!
[2018-01-04 21:01:29]
DTC Engineering - Posts: 270
In regards to:
HISTORICAL_PRICE_DATA_REQUEST::EndDateTime
HISTORICAL_PRICE_DATA_REQUEST::MaxDaysToReturn

Refer to the paragraphs after the steps in this section here:
https://www.sierrachart.com/index.php?page=doc/DTCServer.php#HistoricalPriceDataServer

And use Sierra Chart prerelease version 1682 for the latest support for these.
[2018-01-15 04:12:25]
EricH - Posts: 17
I can confirm 1682's endDateTime/MaxDaysToReturn as documented above is working fine.
More questions on historical quotes:

Can SC/DTC server accept multiple HISTORICAL_PRICE_DATA_REQUEST within the same 'login session'?

I.e., I tried login, then his_data_request, and can get the correct records back from SC, but when I issue the next his_data_request, and then wait (indefinitely) on getting the SC to respond to this request.
Also SC log did not show the 2nd HD Request...
[2018-01-15 05:00:44]
DTC Engineering - Posts: 270

Can SC/DTC server accept multiple HISTORICAL_PRICE_DATA_REQUEST within the same 'login session'?
No, this is not supported.
Date Time Of Last Edit: 2018-01-15 05:00:51

To post a message in this thread, you need to login with your Sierra Chart account:

Login


Login Page - Create Account