Login Page - Create Account

DTC Protocol Discussion Forum


Date/Time: Tue, 13 Nov 2018 04:18:48 +0000


Python interface for DTC protocol library

Support Request:
[2017-04-08 13:16:42]
Queeq - Posts: 31
Hi,

I would like to use DTC protocol from Python, but haven't found any available solutions. Thus, I decided to try SWIG to generate it. As my knowledge about C/C++ is very basic, I encountered some problems during the process. I ended up with the following (doing it on Linux Mint,
GCC 5.4.0, SWIG 3.0.8):

1. Put DTCProtocol.h and DTCProtocol.cpp in a target folder.

2. Create DTCProtocol.i with the following contents:

%module DTCProtocol

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%include "DTCProtocol.h"


3. Run the following (according to SWIG manual):
swig -c++ -python DTCProtocol.i
gcc -O2 -fPIC -std=c++11 -c DTCProtocol.cpp
gcc -O2 -fPIC -std=c++11 -c DTCProtocol_wrap.cxx -I/usr/include/python3.5m
gcc -shared DTCProtocol.o DTCProtocol_wrap.o -o _DTCProtocol.so

During the swig run there were many warnings like these:

DTCProtocol.h:15: Warning 315: Nothing known about 'std::int32_t'.
DTCProtocol.h:16: Warning 315: Nothing known about 'std::int64_t'.
DTCProtocol.h:17: Warning 315: Nothing known about 'std::uint16_t'.
DTCProtocol.h:18: Warning 315: Nothing known about 'std::uint32_t'.

Somewhere on Stackoverflow it was mentioned that it is not critical.

Further on the way, I had to add
#include <cfloat>
to DTCProtocol.h, otherwise there were errors like the following:


In file included from DTCProtocol_wrap.cxx:3204:0:
DTCProtocol.h: In member function ‘void DTC::s_MarketDataSnapshot::Clear()’:
DTCProtocol.h:834:29: error: ‘DBL_MAX’ was not declared in this scope
    SessionSettlementPrice = DBL_MAX;
                             ^
DTCProtocol.h: In member function ‘void DTC::s_MarketDataUpdateBidAsk::Clear()’:
DTCProtocol.h:1495:15: error: ‘DBL_MAX’ was not declared in this scope
    BidPrice = DBL_MAX; //This also signifies the BidQuantity is unset
               ^
DTCProtocol.h: In member function ‘void DTC::s_MarketDataUpdateBidAskCompact::Clear()’:
DTCProtocol.h:1574:15: error: ‘FLT_MAX’ was not declared in this scope
    BidPrice = FLT_MAX;
               ^
DTCProtocol.h: In member function ‘void DTC::s_OrderUpdate::Clear()’:
DTCProtocol.h:2493:13: error: ‘DBL_MAX’ was not declared in this scope
    Price1 = DBL_MAX;
             ^

When trying to import the resulting object, it gives the following errors:


➜ python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import DTCProtocol
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/queeq/Work/wat/wat/DTC/DTCProtocol.py", line 28, in <module>
_DTCProtocol = swig_import_helper()
File "/home/queeq/Work/wat/wat/DTC/DTCProtocol.py", line 24, in swig_import_helper
_mod = imp.load_module('_DTCProtocol', fp, pathname, description)
File "/usr/lib/python3.5/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /home/queeq/Work/wat/wat/DTC/_DTCProtocol.so: undefined symbol: _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE

I tried readelf on the .so file, and it seems that the above function (or whatever it is) is there:

➜ readelf -sW _DTCProtocol.so | grep _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE
29: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE
3407: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE

I don't really know how should it look like. Could someone help me with this? Would be really great to have DTC protocol available for use in Python, and with SWIG it seems to be pretty straightforward.

Thanks!

P.S. SWIG manual used is here: http://www.swig.org/Doc3.0/Python.html#Python
Date Time Of Last Edit: 2017-04-08 19:24:57
[2017-04-09 16:29:17]
Queeq - Posts: 31
I finally managed to make it importable. Had to add some ignores to DTCProtocol.i:

%module DTCProtocol

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%ignore s_MarketDepthFullUpdate20;
%ignore s_MarketDepthFullUpdate10;

%include "DTCProtocol.h"

And also add -lstdc++ when creating .so. Thus, the complete list of commands is as follows:

swig -c++ -python DTCProtocol.i
gcc -O2 -fPIC -std=c++11 -c DTCProtocol.cpp DTCProtocol_wrap.cxx -I/usr/include/python3.5
gcc -shared DTCProtocol.o DTCProtocol_wrap.o -o _DTCProtocol.so -lstdc++

Seems to work now:

>>> import DTCProtocol
>>> print(DTCProtocol.cvar.CURRENT_VERSION)
<Swig Object of type 'int32_t *' at 0x7fc50173fa20>

Not sure how to use it from a Python script yet, but I think I'll figure it out.
Date Time Of Last Edit: 2017-04-09 16:31:40
[2017-04-09 17:06:32]
Queeq - Posts: 31
Alright, had to include stdint.i to DTCProtocol.i, so the whole file now looks like this:

%module DTCProtocol

%include "stdint.i"

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%ignore s_MarketDepthFullUpdate20;
%ignore s_MarketDepthFullUpdate10;

%include "DTCProtocol.h"

Now:

>>> import DTCProtocol
>>> print(DTCProtocol.CURRENT_VERSION)
7

[2017-04-10 17:41:42]
DTC Engineering - Posts: 270
In regards to Python, we recommend using Google Protocol Buffers encoding instead.

Here is the Google documentation for that:

https://developers.google.com/protocol-buffers/docs/pythontutorial

The current protocol buffer file can be found here:
http://dtcprotocol.org/DTC_Files/DTCProtocol.proto
Date Time Of Last Edit: 2017-04-10 17:42:29
[2017-04-17 07:53:32]
Queeq - Posts: 31
Thanks for the hint. I tried to test it, but I can't make it work.

What I did is I compiled the proto3 file, taken from my SC installation, using latest google.protobuf. It compiled successfully and I am able to access messages from Python. I wrote the following script as a test:


""" Data retrieval from SierraChart services """

import socket
import struct

from DTC import DTCProtocol_pb2 as Dtc


def construct_message(m, message_type):
""" Prepends message size and type, appends ProtocolType and packs it to binary

Args:
m (DTCProtocol): DTC protobuf message
message_type (int): DTC message type

Returns:
bytes: Binary string
"""
try:
# Try to set ProtocolVersion as most messages contain this field
m.ProtocolVersion = Dtc.CURRENT_VERSION
except AttributeError:
pass
m.ProtocolType = "DTC" # ProtocolType is always the same
total_len = 4 + m.ByteSize() # 2 bytes Size + 2 bytes Type
header = struct.pack('<HH', total_len, message_type) # Prepare 4-byte little-endian header
binary_message = m.SerializeToString()
print(binary_message) # <-- This print outputs b'\x08\x07\x1a\x03DTC', which is wrong
return header + binary_message

SC_INTRADAY = ("127.0.0.1", 11100)
SC_HISTORICAL = ("127.0.0.1", 11101)
ENCODING = Dtc.BINARY_ENCODING

# Create encoding request
message = Dtc.EncodingRequest()
message.Encoding = ENCODING
data = construct_message(message, Dtc.ENCODING_REQUEST)

# Connect
s = socket.socket()
s.connect(SC_INTRADAY)

# Send encoding request
s.send(data)
# Receive encoding response
response = s.recv(1024)
# 2 bytes with offset of 2 must indicate this is encoding response message type
assert struct.unpack_from('<H', response, 2)[0] == Dtc.ENCODING_RESPONSE
encoding_response = Dtc.EncodingResponse()
print(encoding_response.ParseFromString(response[4:]).Encoding)

For some reason, EncodingRequest message doesn't get correctly serialized: it gives b'\x08\x07\x1a\x03DTC'. As a result, SC logs "Unexpected DTC encoding request: 4412484".

I'm pretty stuck here and would appreciate some help.

Thanks!
[2017-04-17 17:43:30]
Queeq - Posts: 31
I figured I did it wrong: would need to set/negotiate the protobuf encoding on/with the server.
[2017-04-17 21:46:16]
DTC Engineering - Posts: 270
Yes, the client makes a request to the server for the encoding.

Also we recommend using this DTC protocol buffer definition file instead of version 3 :

http://dtcprotocol.org/DTC_Files/DTCProtocol.proto
Date Time Of Last Edit: 2017-04-17 22:02:04
[2017-04-17 21:58:10]
Queeq - Posts: 31
Why is it recommended? I'm kind of used to the notion that "latest=best" :)
[2017-04-17 22:01:05]
DTC Engineering - Posts: 270
It has not been tested by us and may not have all of the latest updates.
Date Time Of Last Edit: 2017-04-17 22:01:34

To post a message in this thread, you need to login with your Sierra Chart account:

Login


Login Page - Create Account