Edit 4/8/2015: Updated to use newer functions…
Edit 3/26/2015: fitbitScraper 0.1.2 is now on CRAN with new heart rate data capabilities.
Edit 2/2/2015: fitbitScraper 0.1.1 is now on CRAN
Fitbit is a device that tracks your daily activity. It's basically a pedometer, but it does a little more. It has an altimeter, so it can count flight of stairs climbed. It can detect your sleeping activity and give you a read on how often you are tossing and turning. Any how, I received a Fitbit One for Christmas and have been unexpectedly into this thing. I absolutely did not expect to be motivated by monitoring my fitness activity. It's been fun.
The problem is that Fitbit doesn't allow you to export your data under their “free” data tracking tier. It costs $50 a month to upgrade to “premium” to allow for data export. It's my data, I should be able to get it. So, I made an R package to do just that.
fitbitScraper is my new R package, hosted on Github for now, to download Fitbit data. It uses the internal API that fitbit uses to populate your dashboard. Here, they give data as granular as every 15 minutes. Also, there's daily aggregate data available. Basically, the package logs into fibit.com using your email / password, and is returned a cookie. Using that cookie, the internal API is accessible. For the time being, I can only get that cookie if you use email / password to login. Facebook and Google login are available, but I haven't put any effort into figuring out how to get a cookie if one of those methods are used to login. One could copy and paste their cookie information into R and use this package that way…
Here's a quick example of downloading and graphing step data for a given day. On this particular day, I played basketball from about 6am until 9am.
install.packages("fitbitScraper")
library("fitbitScraper")
# just reading from file to hide pw and to make .Rmd document to work...
mypassword <- readLines("pw.txt")
cookie <- login(email="corynissen@gmail.com", password=mypassword)
df <- get_intraday_data(cookie, what="steps", date="2015-01-10")
library("ggplot2")
ggplot(df) + geom_bar(aes(x=time, y=steps, fill=steps), stat="identity") +
xlab("") +ylab("steps") +
theme(axis.ticks.x=element_blank(),
panel.grid.major.x = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
panel.background=element_blank(),
panel.grid.major.y=element_line(colour="gray", size=.1),
legend.position="none")
Here's an example of the daily data.
df <- get_daily_data(cookie, what="steps", start_date="2015-01-13", end_date="2015-01-20")
library("ggplot2")
ggplot(df) + geom_bar(aes(x=time, y=steps), stat="identity") +
xlab("") +ylab("steps") +
theme(axis.ticks.x=element_blank(),
panel.grid.major.x = element_blank(),
panel.grid.minor.x = element_blank(),
panel.grid.minor.y = element_blank(),
panel.background=element_blank(),
panel.grid.major.y=element_line(colour="gray", size=.1),
legend.position="none")
I have to give a should out to this guy for inspiration. I don't know perl, so I couldn't really use anything here, but it gave me confidence that I could figure it out.
I hope this is of some use to some of you R hackers out there…
This is great! I've been wanting this, but have been too busy (lazy?) to figure out the details.
ReplyDeleteOne thing: I get a POSIX time zone warning, as the tz isn't defined in the `get_15_min_data()` call to `as.POSIXct()`.
> df <- get_15_min_data(cookie, what="steps", date="2015-01-10")
Warning messages:
1: In strptime(xx, f <- "%Y-%m-%d %H:%M:%OS", tz = tz) :
unknown timezone '%Y-%m-%d %H:%M:%S'
2: In as.POSIXct.POSIXlt(x) : unknown timezone '%Y-%m-%d %H:%M:%S'
3: In strptime(x, f, tz = tz) : unknown timezone '%Y-%m-%d %H:%M:%S'
4: In as.POSIXct.POSIXlt(as.POSIXlt(x, tz, ...), tz, ...) :
unknown timezone '%Y-%m-%d %H:%M:%S'
What timezone does the fitbit API return? Local? Perhaps the `get_*_data()` functions should have a call to specify timezone, with a default value of `Sys.timezone()`.
Cheers,
Clark
Weird... I wasn't getting that warning on Ubuntu. Any how, I've added the tz parameter as you've suggested. Thanks for the feedback.
DeleteCool -- my hack was to just load the lubridate package and force the time vector to `Sys.timezone` after extracting the data from the web.
DeleteI wonder if you have a default timezone set in an R startup file? Something like this:
http://www.r-bloggers.com/doing-away-with-%E2%80%9Cunknown-timezone%E2%80%9D-warnings/
I definitely don't set a standard tz, as (almost) everything I do I need to have in UTC, so I make sure that it gets set explicitly in my code.
Also, I think timezones are mostly handled by the system, so there are some instances where different OSes produce different results. I'm on OSX, so I would have expected it to be similar, but who knows ...
DeleteCool yeah. Sys.timezone() is NULL for me. But, Sys.time() has a timezone, oddly enough... so I added this to the package...
Deletetz <- Sys.timezone()
if(is.null(tz)){tz <- format(Sys.time(),"%Z")}
Hey, where is the documentation/source code? I think I must have missed it on Github and I don't feel like patiently waiting for CRAN ;)
ReplyDeletehttps://github.com/corynissen/fitbitScraper
DeleteCode to install this via the devtools package is in the blog post and in the github README.md file
Really cool! I've been using an IFTTT recipe to get my daily summaries for about 6 months now, but the more granular data will be much more fun.
ReplyDeleteThis is great! I was able to use your package to whip up a script to download the entirety of my 15 minute data (about 1.5 years worth).
ReplyDeleteIt's great, but an exception is raised,
ReplyDeleteError in fromJSON(content, handler, default.size, depth, allowComments, :
invalid JSON input
Looks like they made a slight change to the API. I fixed the package. Use the dev branch on github until I get a chance to submit the fix to CRAN... devtools::install_github("corynissen/fitbitScraper@dev")
DeleteI tried to use it. It seems like everything is working but get_intraday_data
DeleteIt returns:
Error in fromJSON(content, handler, default.size, depth, allowComments, :
invalid JSON input
I have R 3.2.1 on Windows. I used devtools to install fitbitScraper.
I can't replicate the issue. Can you please try updating the "httr" package and see if that helps?
DeleteSorry If this posts twice but I think it would be nice if you added a column next to the intraday HR data that specifies the type of activity being performed during that time window (e.g. "running" "sleeping" "weight lifting") this info is already available for the Surge on the online dashboard and would be a great addition to add context to the HR data.
ReplyDeleteFitBit recently took away the ability to see (or download) things like Blood Pressure, Glucose and Body Measurement data.
ReplyDeleteAccording to a moderator on the forums page, the data is still in their database, but they've got no way to display or
export the data :(
Do you know how we might figure out the database table name for these things (other than hit-or-miss hacking and trying, which I may do to get the body measurements back)?
I'll be trying this tool after work tonight - thank you!
I was unable to access the fitbitscraper package from CRAN for R version 3.2.3
ReplyDelete*************************
Warning message:
package ‘fitbitScraper’ is not available (for R version 3.2.3)
> library("fitbitScraper")
Error in library("fitbitScraper") :
there is no package called ‘fitbitScraper’
>
I don't know man... can you install other packages? Are you connected to the internet? Did you try downloading the zip files and installing from that? Do you have the following packages installed: httr, stringr, RJSONIO, methods, utils? Could be lots of things... I don't have enough information to help you.
DeleteThanks Cory. This is a great piece of code, and has prompted me to re-learn my SPlus skills and convert to R. There are two things that a relative newbie to R has got me stumped, could you help please. (1) Can I get a more refined timescale than 5 minutes for heart rate from get_intraday_data ? and (2) why won't the following work when steps and distance are okay ?
ReplyDelete> df<-get_intraday_data(cookie, what="heart-rate", date="2016-02-14")
> ggplot(df) + geom_bar(aes(x=time, y=heart-rate, fill=heart-rate), stat="identity")
I've made a change to the package that's working it's way through the CRAN gods now that will fix this... but until then, just add back-ticks around heart-rate...
Deleteggplot(df) + geom_bar(aes(x=time, y=`heart-rate`, fill=`heart-rate`), stat="identity")
Oh, and no, 5 minutes is the best we can do with heart rate from fitbit.
DeleteThanks for your replies
DeleteThis is really cool. Do you know if Fitbit provides location information (like lat/long) as well?
ReplyDeleteI don't have a fitbit that has a GPS, so I'm not sure. I'm pretty much limited to what I have in my profile page. My friend gave me his password so I was able to add heart rate, but I don't know anybody with a GPS enabled fitbit...
DeleteI wish I could understand R!
ReplyDeleteVery nice way to get your data.
Hi Cory, this is a great package! I have been using this for last two weeks to analyse my daily activity.Is there any possibility to extract "daily goal" for steps and calories burned ? Also I would like to extract the "fat" data log. Looking forward to your response. Thanks in advance.
ReplyDeleteI'll add it to my queue for the next update.
DeleteThanks for your response. I'm eagerly waiting for your next update.
DeleteDoes anyone know how to get the heart rate data for every minute? I am able to extract it using Cory Nissen's package but its giving me bpm every 5 minutes. Can I get it for every one minute or less?
ReplyDelete