-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
208 lines (177 loc) · 23.5 KB
/
atom.xml
File metadata and controls
208 lines (177 loc) · 23.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[Scout Alarm Engineering]]></title>
<link href="http://ScoutAlarm.github.io/atom.xml" rel="self"/>
<link href="http://ScoutAlarm.github.io/"/>
<updated>2013-12-29T23:10:15-06:00</updated>
<id>http://ScoutAlarm.github.io/</id>
<author>
<name><![CDATA[Scout Alarm]]></name>
</author>
<generator uri="http://octopress.org/">Octopress</generator>
<entry>
<title type="html"><![CDATA[REST'ing on Binary]]></title>
<link href="http://ScoutAlarm.github.io/blog/2013/12/29/resting-on-binary/"/>
<updated>2013-12-29T14:41:51-06:00</updated>
<id>http://ScoutAlarm.github.io/blog/2013/12/29/resting-on-binary</id>
<content type="html"><![CDATA[<p>@ Scout we live at the intersection of web and hardware technologies. At first you may make the same mistake I did and assume that the hardware revolution can/ will just assimilate itself to standards established in the web 2.0 era. WRONG.</p>
<p>Disclaimer, This is my first encounter with hardware devices that communicate with the internet (IoT, internet of things).</p>
<p>A short background story, currently we are developing the internal infrastructure for Scout, a few weeks ago I began writing a web-service that would be a gateway for our hardware devices to communicate with our ecosystem. Naively I created a RESTful api using my favorite web-service DSL (<a href="https://github.com/mattetti/Weasel-Diesel">https://github.com/mattetti/Weasel-Diesel</a>). I defined some resources and got as far as establishing all the important information we needed to capture.</p>
<p>Tangent, data transfer is not a big consideration when developing for the web. For mobile, developers get a little bit more conscious but as long as you are sending only the data you need and using JSON, further optimization isn’t worth the effort (imho). The reason behind this is that technology has advancement so much, servers have gotten so powerful and cheap that protocol overhead and payload management isn’t relevant like it used to be when the internet infrastructure was in its infancy.</p>
<p>Back to the story, With the following knowledge there was almost no reason for me to think about data transfer and how resources intensive protocols like HTTP are. Why would I, I have never seen a webserver or client choke when trying to process an http request/ response.</p>
<p>Another Tangent, enter Internet of Things (iot), billions of small devices with sensors that are going to take over our lives. Most of these sensors are simple in implementation and are battery powered which gives them the mobility to be anywhere and everywhere. Here comes the fun part, all these battery powered devices are not going to come loaded with the latest and greatest in client-server technology. Imagine a scenario where someone loads an RC car with a cinder block and expects it to run circles around their dog, not gonna happen. As such these devices have microcontrollers that are just powerful enough to do their tasks and at the same time enabling the device to operate for as long as possible in remote locations on battery.</p>
<p>Circling back, again, to the story, if we continued down the HTTP path our device would literally be maxing out their resources doing string manipulation, building http requests and parsing JSON responses. Imagine that, we are in the middle of a hardware revolution, and we cant even use strings… Scout also has the ability to operate over GSM so the more data that we send over the wire the more we dig into our margins. So the incentives aligned to find a lower level protocol to communicate over.</p>
<p>The solution, TCP/IP + binary. Lets let that soak in a little. I haven’t touched binary in years and if there was ever a case where I needed to read a file through a stream, the binary piece is so abstracted that it’s a black box for most developers.</p>
<p>The hunt for a binary server began. Most production tested frameworks that I like/ have worked with (sinatra, padrino, rails) don’t have great solution for handling TCP/IP. Sure ruby has a built in TCPSocketServer, but I don’t want to have to write a server framework from scratch…</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='ruby'><span class='line'><span class="n">s</span> <span class="o">=</span> <span class="no">TCPSocket</span><span class="o">.</span><span class="n">new</span> <span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">2000</span>
</span><span class='line'><span class="k">while</span> <span class="n">line</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">gets</span>
</span><span class='line'> <span class="nb">puts</span> <span class="n">line</span>
</span><span class='line'><span class="k">end</span>
</span></code></pre></td></tr></table></div></figure>
<p>After about three days of searching for a suitable ruby framework I decided to look at nodejs. I have never used it, but in my past life I dealt extensively in javascript client frameworks (backbone, angular, extjs, and dojo). Turns out, there is a great project called actionHero (<a href="https://github.com/evantahler/actionHero">https://github.com/evantahler/actionHero</a>). evantahler created an awesome solution that gives a developer the ability to define server types within a framework. And it just so happened that it included a json socket server out of the box… jackpot! (Finding this was a needle in a haystack situation). What was left was converting json input / output into binary opt codes + parameters. So I cloned the json scoket server, and redefined the parseRequest function (see below) and implemented a binaryActionRequestParser. The binaryActionRequestParser creates an object that acts in the same way the json socket server expects its data. This way I was able to throw the object at the server.processAction() and let it take care of the rest.</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='ruby'><span class='line'><span class="n">api</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">binaryActionRequestParse</span> <span class="o">=</span> <span class="n">function</span><span class="p">(){</span>
</span><span class='line'> <span class="n">function</span> <span class="no">Parser</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="n">dissolve</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">this</span><span class="p">);</span>
</span><span class='line'> <span class="n">this</span><span class="o">.</span><span class="n">loop</span><span class="p">(</span><span class="n">function</span><span class="p">(</span><span class="k">end</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="n">this</span><span class="o">.</span><span class="n">mcaction16</span><span class="p">(</span><span class="s2">"action"</span><span class="p">)</span><span class="o">.</span><span class="n">tap</span><span class="p">(</span><span class="n">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="n">switch</span> <span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">vars</span><span class="o">.</span><span class="n">action</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="k">case</span> <span class="s1">'signin'</span><span class="p">:</span> <span class="n">this</span><span class="o">.</span><span class="n">uint32be</span><span class="p">(</span><span class="s1">'uuid'</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span>
</span><span class='line'> <span class="k">case</span> <span class="s1">'status'</span><span class="p">:</span> <span class="k">break</span><span class="p">;</span>
</span><span class='line'> <span class="k">case</span> <span class="s1">'updateFwVersion'</span><span class="p">:</span> <span class="k">break</span><span class="p">;</span>
</span><span class='line'> <span class="o">.</span><span class="n">.</span><span class="o">.</span>
</span><span class='line'>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="p">})</span><span class="o">.</span><span class="n">tap</span><span class="p">(</span><span class="n">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="n">this</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">this</span><span class="o">.</span><span class="n">vars</span><span class="p">);</span>
</span><span class='line'> <span class="n">this</span><span class="o">.</span><span class="n">vars</span> <span class="o">=</span> <span class="p">{};</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'>
</span><span class='line'> <span class="n">util</span><span class="o">.</span><span class="n">inherits</span><span class="p">(</span><span class="no">Parser</span><span class="p">,</span> <span class="n">dissolve</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'> <span class="o">.</span><span class="n">.</span><span class="o">.</span>
</span><span class='line'>
</span><span class='line'> <span class="k">return</span> <span class="kp">new</span> <span class="no">Parser</span><span class="p">();</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='ruby'><span class='line'><span class="n">var</span> <span class="n">parseRequest</span> <span class="o">=</span> <span class="n">function</span><span class="p">(</span><span class="n">connection</span><span class="p">){</span>
</span><span class='line'> <span class="n">parser</span> <span class="o">=</span> <span class="kp">new</span> <span class="n">api</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">binaryActionRequestParse</span><span class="p">();</span>
</span><span class='line'> <span class="n">parser</span><span class="o">.</span><span class="n">on</span><span class="p">(</span><span class="s2">"readable"</span><span class="p">,</span> <span class="n">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="n">var</span> <span class="n">e</span><span class="p">;</span>
</span><span class='line'> <span class="k">while</span> <span class="p">(</span><span class="n">e</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> <span class="p">{</span>
</span><span class='line'> <span class="n">connection</span><span class="o">.</span><span class="n">params</span><span class="o">[</span><span class="s2">"action"</span><span class="o">]</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">action</span><span class="p">;</span>
</span><span class='line'> <span class="n">connection</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="n">e</span><span class="p">;</span>
</span><span class='line'> <span class="n">connection</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="n">null</span><span class="p">;</span>
</span><span class='line'> <span class="n">connection</span><span class="o">.</span><span class="n">response</span> <span class="o">=</span> <span class="p">{};</span>
</span><span class='line'>
</span><span class='line'> <span class="n">server</span><span class="o">.</span><span class="n">processAction</span><span class="p">(</span><span class="n">connection</span><span class="p">);</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'>
</span><span class='line'> <span class="n">parser</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">connection</span><span class="o">.</span><span class="n">rawConnection</span><span class="o">.</span><span class="n">socketDataString</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>ActionHero has proven to be an awesome tool for getting a binary server up and running, and it has been performing brilliantly. For parsing and producing binary data in node I used this repo (<a href="https://github.com/deoxxa/dissolve">https://github.com/deoxxa/dissolve</a>) and this repo (<a href="https://github.com/deoxxa/concentrate">https://github.com/deoxxa/concentrate</a>). Deoxxa, has a great solution for working with binary data. It also gives you the ability to define your own patterns for consuming data if the built in functions aren’t enough</p>
<p>Chaining calls:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="k">this</span><span class="p">.</span><span class="nx">uint32be</span><span class="p">(</span><span class="s2">"uuid"</span><span class="p">).</span><span class="nx">uint16</span><span class="p">(</span><span class="s2">"opt_code"</span><span class="p">).</span><span class="nx">uint8bool</span><span class="p">(</span><span class="s2">"on_battery_power"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<p>Custom types:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="kd">var</span> <span class="nx">mcstring16</span> <span class="o">=</span> <span class="kd">function</span> <span class="nx">string16</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="kd">var</span> <span class="nx">len</span> <span class="o">=</span> <span class="p">[</span><span class="nx">name</span><span class="p">,</span> <span class="s2">"len"</span><span class="p">].</span><span class="nx">join</span><span class="p">(</span><span class="s2">"_"</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">uint16be</span><span class="p">(</span><span class="nx">len</span><span class="p">).</span><span class="nx">tap</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">buffer</span><span class="p">(</span><span class="nx">name</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">len</span><span class="p">]</span> <span class="o">*</span> <span class="mi">2</span><span class="p">).</span><span class="nx">tap</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="k">delete</span> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">len</span><span class="p">];</span>
</span><span class='line'>
</span><span class='line'> <span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="nx">i</span><span class="o"><</span><span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">].</span><span class="nx">length</span><span class="o">/</span><span class="mi">2</span><span class="p">;</span><span class="o">++</span><span class="nx">i</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="kd">var</span> <span class="nx">t</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">][</span><span class="nx">i</span><span class="o">*</span><span class="mi">2</span><span class="p">];</span>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">][</span><span class="nx">i</span><span class="o">*</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">][</span><span class="nx">i</span><span class="o">*</span><span class="mi">2</span><span class="o">+</span><span class="mi">1</span><span class="p">];</span>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">][</span><span class="nx">i</span><span class="o">*</span><span class="mi">2</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="nx">t</span><span class="p">;</span>
</span><span class='line'> <span class="p">}</span>
</span><span class='line'>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">].</span><span class="nx">toString</span><span class="p">(</span><span class="s2">"ucs2"</span><span class="p">);</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="kd">var</span> <span class="nx">uint8bool</span> <span class="o">=</span> <span class="kd">function</span> <span class="nx">uint8bool</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">uint8be</span><span class="p">(</span><span class="nx">name</span><span class="p">).</span><span class="nx">tap</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">vars</span><span class="p">[</span><span class="nx">name</span><span class="p">]</span> <span class="o">==</span> <span class="mi">1</span> <span class="o">?</span> <span class="kc">true</span> <span class="o">:</span> <span class="kc">false</span><span class="p">)</span>
</span><span class='line'> <span class="p">});</span>
</span><span class='line'><span class="p">};</span>
</span></code></pre></td></tr></table></div></figure>
<p>Its incredible to think that with hardware taking such a big step forward in accessibility and implementation we (as developers) are forced to take steps back in communication protocols. I was a victim of trying to impose web standards in a field that is so vastly different. There is not this 1:1 correlation that I assumed/ wished there was. We had to go back to basics and build a solution that worked best for the problem. It’s too early to tell, but I think the difference solution is for the best. It’s like the saying goes; don’t use an anvil when a hammer will do.</p>
<p>It is an exciting time and it will be very interesting to see what new frameworks will be developed on languages such as ruby, node, python that will help usher in the new hardware era.</p>
]]></content>
</entry>
</feed>