Thursday, January 22, 2015

R Package to Download Fitbit Data

Edit 4/8/2015: Updated to use newer functions…
Edit 3/26/2015: fitbitScraper 0.1.2 is now on CRAN with new heart rate data capabilities.
Edit 2/2/2015: fitbitScraper 0.1.1 is now on CRAN

Fitbit is a device that tracks your daily activity. It's basically a pedometer, but it does a little more. It has an altimeter, so it can count flight of stairs climbed. It can detect your sleeping activity and give you a read on how often you are tossing and turning. Any how, I received a Fitbit One for Christmas and have been unexpectedly into this thing. I absolutely did not expect to be motivated by monitoring my fitness activity. It's been fun.

The problem is that Fitbit doesn't allow you to export your data under their “free” data tracking tier. It costs $50 a month to upgrade to “premium” to allow for data export. It's my data, I should be able to get it. So, I made an R package to do just that.

fitbitScraper is my new R package, hosted on Github for now, to download Fitbit data. It uses the internal API that fitbit uses to populate your dashboard. Here, they give data as granular as every 15 minutes. Also, there's daily aggregate data available. Basically, the package logs into fibit.com using your email / password, and is returned a cookie. Using that cookie, the internal API is accessible. For the time being, I can only get that cookie if you use email / password to login. Facebook and Google login are available, but I haven't put any effort into figuring out how to get a cookie if one of those methods are used to login. One could copy and paste their cookie information into R and use this package that way…

Here's a quick example of downloading and graphing step data for a given day. On this particular day, I played basketball from about 6am until 9am.

install.packages("fitbitScraper")  
library("fitbitScraper")
# just reading from file to hide pw and to make .Rmd document to work...
mypassword <- readLines("pw.txt")
cookie <- login(email="corynissen@gmail.com", password=mypassword) 
df <- get_intraday_data(cookie, what="steps", date="2015-01-10")  
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=steps, fill=steps), stat="identity") + 
             xlab("") +ylab("steps") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none") 

plot of chunk unnamed-chunk-1

Here's an example of the daily data.

df <- get_daily_data(cookie, what="steps", start_date="2015-01-13", end_date="2015-01-20")  
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=steps), stat="identity") + 
             xlab("") +ylab("steps") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none") 

plot of chunk unnamed-chunk-2

I have to give a should out to this guy for inspiration. I don't know perl, so I couldn't really use anything here, but it gave me confidence that I could figure it out.

I hope this is of some use to some of you R hackers out there…

28 comments:

  1. This is great! I've been wanting this, but have been too busy (lazy?) to figure out the details.

    One thing: I get a POSIX time zone warning, as the tz isn't defined in the `get_15_min_data()` call to `as.POSIXct()`.

    > df <- get_15_min_data(cookie, what="steps", date="2015-01-10")
    Warning messages:
    1: In strptime(xx, f <- "%Y-%m-%d %H:%M:%OS", tz = tz) :
    unknown timezone '%Y-%m-%d %H:%M:%S'
    2: In as.POSIXct.POSIXlt(x) : unknown timezone '%Y-%m-%d %H:%M:%S'
    3: In strptime(x, f, tz = tz) : unknown timezone '%Y-%m-%d %H:%M:%S'
    4: In as.POSIXct.POSIXlt(as.POSIXlt(x, tz, ...), tz, ...) :
    unknown timezone '%Y-%m-%d %H:%M:%S'

    What timezone does the fitbit API return? Local? Perhaps the `get_*_data()` functions should have a call to specify timezone, with a default value of `Sys.timezone()`.

    Cheers,
    Clark

    ReplyDelete
    Replies
    1. Weird... I wasn't getting that warning on Ubuntu. Any how, I've added the tz parameter as you've suggested. Thanks for the feedback.

      Delete
    2. Cool -- my hack was to just load the lubridate package and force the time vector to `Sys.timezone` after extracting the data from the web.

      I wonder if you have a default timezone set in an R startup file? Something like this:

      http://www.r-bloggers.com/doing-away-with-%E2%80%9Cunknown-timezone%E2%80%9D-warnings/

      I definitely don't set a standard tz, as (almost) everything I do I need to have in UTC, so I make sure that it gets set explicitly in my code.

      Delete
    3. Also, I think timezones are mostly handled by the system, so there are some instances where different OSes produce different results. I'm on OSX, so I would have expected it to be similar, but who knows ...

      Delete
    4. Cool yeah. Sys.timezone() is NULL for me. But, Sys.time() has a timezone, oddly enough... so I added this to the package...

      tz <- Sys.timezone()
      if(is.null(tz)){tz <- format(Sys.time(),"%Z")}

      Delete
  2. Hey, where is the documentation/source code? I think I must have missed it on Github and I don't feel like patiently waiting for CRAN ;)

    ReplyDelete
    Replies
    1. https://github.com/corynissen/fitbitScraper

      Code to install this via the devtools package is in the blog post and in the github README.md file

      Delete
  3. Really cool! I've been using an IFTTT recipe to get my daily summaries for about 6 months now, but the more granular data will be much more fun.

    ReplyDelete
  4. This is great! I was able to use your package to whip up a script to download the entirety of my 15 minute data (about 1.5 years worth).

    ReplyDelete
  5. It's great, but an exception is raised,
    Error in fromJSON(content, handler, default.size, depth, allowComments, :
    invalid JSON input

    ReplyDelete
    Replies
    1. Looks like they made a slight change to the API. I fixed the package. Use the dev branch on github until I get a chance to submit the fix to CRAN... devtools::install_github("corynissen/fitbitScraper@dev")

      Delete
    2. I tried to use it. It seems like everything is working but get_intraday_data
      It returns:
      Error in fromJSON(content, handler, default.size, depth, allowComments, :
      invalid JSON input
      I have R 3.2.1 on Windows. I used devtools to install fitbitScraper.

      Delete
    3. I can't replicate the issue. Can you please try updating the "httr" package and see if that helps?

      Delete
  6. Sorry If this posts twice but I think it would be nice if you added a column next to the intraday HR data that specifies the type of activity being performed during that time window (e.g. "running" "sleeping" "weight lifting") this info is already available for the Surge on the online dashboard and would be a great addition to add context to the HR data.

    ReplyDelete
  7. FitBit recently took away the ability to see (or download) things like Blood Pressure, Glucose and Body Measurement data.
    According to a moderator on the forums page, the data is still in their database, but they've got no way to display or
    export the data :(

    Do you know how we might figure out the database table name for these things (other than hit-or-miss hacking and trying, which I may do to get the body measurements back)?

    I'll be trying this tool after work tonight - thank you!

    ReplyDelete
  8. I was unable to access the fitbitscraper package from CRAN for R version 3.2.3
    *************************
    Warning message:
    package ‘fitbitScraper’ is not available (for R version 3.2.3)
    > library("fitbitScraper")
    Error in library("fitbitScraper") :
    there is no package called ‘fitbitScraper’
    >

    ReplyDelete
    Replies
    1. I don't know man... can you install other packages? Are you connected to the internet? Did you try downloading the zip files and installing from that? Do you have the following packages installed: httr, stringr, RJSONIO, methods, utils? Could be lots of things... I don't have enough information to help you.

      Delete
  9. Thanks Cory. This is a great piece of code, and has prompted me to re-learn my SPlus skills and convert to R. There are two things that a relative newbie to R has got me stumped, could you help please. (1) Can I get a more refined timescale than 5 minutes for heart rate from get_intraday_data ? and (2) why won't the following work when steps and distance are okay ?
    > df<-get_intraday_data(cookie, what="heart-rate", date="2016-02-14")
    > ggplot(df) + geom_bar(aes(x=time, y=heart-rate, fill=heart-rate), stat="identity")

    ReplyDelete
    Replies
    1. I've made a change to the package that's working it's way through the CRAN gods now that will fix this... but until then, just add back-ticks around heart-rate...

      ggplot(df) + geom_bar(aes(x=time, y=`heart-rate`, fill=`heart-rate`), stat="identity")

      Delete
    2. Oh, and no, 5 minutes is the best we can do with heart rate from fitbit.

      Delete
  10. This is really cool. Do you know if Fitbit provides location information (like lat/long) as well?

    ReplyDelete
    Replies
    1. I don't have a fitbit that has a GPS, so I'm not sure. I'm pretty much limited to what I have in my profile page. My friend gave me his password so I was able to add heart rate, but I don't know anybody with a GPS enabled fitbit...

      Delete
  11. I wish I could understand R!
    Very nice way to get your data.

    ReplyDelete
  12. Hi Cory, this is a great package! I have been using this for last two weeks to analyse my daily activity.Is there any possibility to extract "daily goal" for steps and calories burned ? Also I would like to extract the "fat" data log. Looking forward to your response. Thanks in advance.

    ReplyDelete
    Replies
    1. I'll add it to my queue for the next update.

      Delete
    2. Thanks for your response. I'm eagerly waiting for your next update.

      Delete
  13. Does anyone know how to get the heart rate data for every minute? I am able to extract it using Cory Nissen's package but its giving me bpm every 5 minutes. Can I get it for every one minute or less?

    ReplyDelete

Note: Only a member of this blog may post a comment.