Sense API post.

_posts/2016-04-10-sense-api.md

@@ -0,0 +1,233 @@
+---
+layout: post
+title: Reverse engineering the Sense API
+strapline: Who needs API docs when you can do a MITM attack?
+---
+
+### Meet Sense
+
+<div class="image right">
+ <img src="/res/images/sense/sense.jpg" alt="Sense">
+</div>
+
+<a href="https://hello.is/">Sense</a> is a little device that sits by your bedside and, in
+conjunction with a little 'pill' attached to your pillow, monitors your sleeping patterns and
+any environmental conditions that might hamper them. Android and iOS apps show you your sleep
+history, and offer suggestions for improvements.
+
+<!--more-->
+
+Sense was <a href="https://www.kickstarter.com/projects/hello/sense-know-more-sleep-better/description">Kickstarted</a>
+in August 2014, raising over 2.4 million US dollars, and shipped to backers in mid 2015. The
+campaign blurb included this snippet:
+
+> Building with Sense
+>
+> You'll always have access to your data via our API. Take it, play with it, graph it, do whatever
+> you want with it. It's yours. That's important to us.
+>
+> We enjoy tinkering with and building on-top of other products we like. Sense will let you have
+> that experience.
+>
+> We'd love to hear your thoughts on what you might want to build with Sense, and how you could
+> directly interact with the hardware, and the data it collects.
+
+Sounds great! But a year after shipping, there's no sign of an API, and some of us who enjoy
+tinkering are getting a bit restless...
+
+### Reverse engineering to the rescue
+
+Data from the Sense hardware is transmitted off onto the Internet somewhere, and then pulled
+down by the mobile apps. If we can snoop that traffic, we can probably figure out how to grab
+the data from the Sense.
+
+So the first step is to capture the raw network traffic while the Sense and the app are in use. I
+crafted an extremely over-the-top setup that involves my phone and the Sense being on a new,
+separate wireless network that gateways through a Linux server that can look in on all the
+traffic (because you never know when a dedicated capturing network will come in handy, right?...)
+
+Running `tshark` while the app and device are active showed they were making HTTPS requests an
+Amazon Elastic Cloud instance. That's not really much use, as we can't see the content of the
+requests. The next step is to move up the network stack and target the HTTPS traffic specifically.
+Step in, <a href="https://mitmproxy.org/">`mitmproxy`</a>. This automates man-in-the-middle attacks
+on HTTPS requests. When it receives such a request, it cooks up its own certificate and sends that
+to the client, and then sits in between while the client and the server communicate.
+
+The certificates generated by `mitmproxy` aren't issued by a trusted source, so (well behaved)
+browsers and apps will throw up lots of warnings and refuse to communicate with it for the most
+part. This is the mechanism that stops anyone impersonating your banking website when you access it
+over HTTPS, so is generally a very good thing. In this case, both the Sense itself and the Android
+app refused to talk with the blatant attacker.
+
+We control the Android phone, though, so can just tell it that we trust the dodgy certificate
+issuer. This involves grabbing the certificate generated by mitmproxy, and adding it to Android's
+key store. The process is <a href="http://docs.mitmproxy.org/en/stable/certinstall.html#installing-the-mitmproxy-ca-certificate-manually">documented nicely in the mitmproxy docs</a>
+and is pretty straightforward. With Android trusting our certificate authority, the Sense app
+starts talking through the proxy and we can look at the plain HTTP requests.
+
+### The API
+
+Logging in to the Android app and navigating through the various screens shows that the API is
+actually pretty nice. It's RESTful, and uses an OAuth bearer token for authorisation, as you can
+see in the `mitmproxy` output of the authentication and first few requests:
+
+{% highlight text %}
+POST https://api.hello.is/v1/oauth2/token
+    ← 200 application/json 151B 1.17s
+GET https://api.hello.is/v2/account/preferences
+    ← 200 application/json 124B 109ms
+GET https://api.hello.is/v1/account
+    ← 200 application/json 181B 388ms
+GET https://api.hello.is/v2/devices
+    ← 200 application/json 235B 126ms
+POST https://api.hello.is/v1/app/checkin
+    ← 200 application/json 106B 119ms
+GET https://api.hello.is/v2/insights
+    ← 200 application/json 1.4kB 157ms
+POST https://api.hello.is/v1/notifications/registration
+    ← 204 [no content] 364ms
+GET https://api.hello.is/v2/timeline/2016-04-09
+    ← 200 application/json 160B 550ms
+GET https://api.hello.is/v1/questions?date=2016-04-10
+    ← 200 application/json 284B 132ms
+PATCH https://api.hello.is/v1/app/stats
+    ← 202 [no content] 126ms
+GET https://api.hello.is/v1/app/stats/unread
+    ← 200 application/json 71B 171ms
+{% endhighlight %}
+
+#### Authorisation
+
+To get a bearer token, you send a POST request to `/v1/oauth2/token`, supplying the username and
+password, and a client ID and secret. Presumably if the API is ever opened up you'll be able to
+register clients and get your own ID and secret, but for now reusing the ones from the Android
+app works fine. [I've prettified and linewrapped the content to make it a bit easier to read.]
+
+{% highlight http %}
+POST /v1/oauth2/token HTTP/1.1
+Host: api.hello.is
+Accept: */*
+Content-Length: 123
+Content-Type: application/x-www-form-urlencoded
+
+grant_type=password
+ &client_id=8d3c1664-05ae-47e4-bcdb-477489590aa4
+ &client_secret=4f771f6f-5c10-4104-bbc6-3333f5b11bf9
+ &username=USERNAME
+ &password=PASSWORD
+{% endhighlight %}
+{% highlight http %}
+HTTP/1.1 200 OK
+Cache-Control: no-cache
+Content-Type: application/json
+Date: Sun, 10 Apr 2016 17:25:16 GMT
+transfer-encoding: chunked
+Connection: keep-alive
+
+{
+  "token_type":"Bearer",
+  "expires_in":31536000,
+  "account_id":"1234",
+  "access_token":"2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
+  "refresh_token":"2.yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
+}
+
+{% endhighlight %}
+
+#### Sleep timeline
+
+Getting the timeline for one night is just a straight up GET request. It gives a detailed log of
+events, including different 'depths' of sleep (I've cut out a whole bunch here for simplicity).
+It also shows metrics relating to the entire night, and their condition ('warning' or 'ideal').
+
+{% highlight http %}
+GET /v2/timeline/2016-04-08 HTTP/1.1
+Host: api.hello.is
+Accept: */*
+Authorization: Bearer 2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+{% endhighlight %}
+{% highlight http %}
+HTTP/1.1 200 OK
+Cache-Control: no-cache
+Content-Type: application/json
+Date: Sun, 10 Apr 2016 17:29:00 GMT
+transfer-encoding: chunked
+Connection: keep-alive
+
+{
+  "score":76,
+  "score_condition":"WARNING",
+  "message":"You were asleep for **8.2 hours**, and sleeping soundly for 3.2 hours.",
+  "date":"2016-04-08",
+  "events":[
+    {
+      "timestamp":1460168700000,
+      "timezone_offset":3600000,
+      "duration_millis":60000,
+      "message":"You went to bed.",
+      "sleep_depth":0,
+      "sleep_state":"AWAKE",
+      "event_type":"GOT_IN_BED",
+      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
+    },{
+      "timestamp":1460169600000,
+      "timezone_offset":3600000,
+      "duration_millis":60000,
+      "message":"You fell asleep.",
+      "sleep_depth":100,
+      "sleep_state":"SOUND",
+      "event_type":"FELL_ASLEEP",
+      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
+    },{
+      "timestamp":1460177040000,
+      "timezone_offset":3600000,
+      "duration_millis":60000,
+      "message":"You were moving around quite a bit.",
+      "sleep_depth":2,
+      "sleep_state":"AWAKE",
+      "event_type":"GENERIC_MOTION",
+      "valid_actions":["VERIFY","INCORRECT"]
+    },{
+      "timestamp":1460199000000,
+      "timezone_offset":3600000,
+      "duration_millis":60000,
+      "message":"Good morning.",
+      "sleep_depth":0,
+      "sleep_state":"AWAKE",
+      "event_type":"WOKE_UP",
+      "valid_actions":["ADJUST_TIME","VERIFY","INCORRECT"]
+    }
+  ],
+  "metrics":[
+    { "name":"total_sleep", "value":490, "unit":"MINUTES", "condition":"IDEAL" },
+    { "name":"sound_sleep", "value":190, "unit":"MINUTES", "condition":"IDEAL" },
+    { "name":"time_to_sleep", "value":15, "unit":"MINUTES", "condition":"IDEAL" },
+    { "name":"times_awake", "value":2, "unit":"QUANTITY", "condition":"IDEAL" },
+    { "name":"fell_asleep", "value":1460169600000, "unit":"TIMESTAMP", "condition":"IDEAL" },
+    { "name":"woke_up", "value":1460199000000, "unit":"TIMESTAMP", "condition":"IDEAL" },
+    { "name":"temperature", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
+    { "name":"humidity", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
+    { "name":"particulates", "value":null, "unit":"CONDITION", "condition":"WARNING" },
+    { "name":"light", "value":null, "unit":"CONDITION", "condition":"IDEAL" },
+    { "name":"sound", "value":null, "unit":"CONDITION", "condition":"IDEAL"}
+  ]
+}
+{% endhighlight %}
+
+#### Other interesting resources
+
+Pretty much everything displayed in the app is available as a REST resource. They clearly
+put some thought into API design, which makes it even more of a shame that they haven't made it
+public yet. Some of the other interesting resources are:
+
+* `/v1/alarms`: the alarms set on the Sense.
+* `/v1/room/current?temp_unit=c`: gives the latest environmental readouts from the Sense.
+* `/v1/room/all_sensors/hours?quantity=2&from_utc=<timestamp>`: sensor history.
+* `/v2/devices`: the paired devices, including battery status of pills.
+* `/v2/insights`: JSON representation of the cards displayed on the main screen (recommendations,
+  analysis, etc).
+* `/v2/trends/LAST_WEEK` (or `MONTH` or `3_MONTHS`): gives graph data for sleep scores, duration
+  and depths.
+
+
+Now the API is figured out, all that's left is to build something that uses it...