Getting Good Data, Part II (of many)

Okay, so by now you’ve heard dozens and dozens of times that you guys produce really good data: your aggregated answers are 97% correct overall (see here and here and here). But we also know that not all images are equally easy. More specifically, not all species are equally easy. It’s a lot easier to identify a giraffe or zebra than it is to decide between an aardwolf and striped hyena.

The plot below shows the different error rates for each species. Note that error comes in two forms. You can have a “false negative” which means you miss a species given that it’s truly there. And then you can have a “false positive,” in which you report a species as being there when it really isn’t. Error is a proportion from 0 to 1.

Species specific error rates.

Species specific error rates.

We calculated this by comparing the consensus data to the gold standard dataset that Margaret collated last year. Note that at the bottom of the chart there are a handful of species that don’t have any values for false negatives. That’s because, for statistical reasons, we could only calculate false negative error rates from completely randomly sampled images, and those species are so rare that they didn’t appear in the gold standard dataset. But for false positives, we could randomly sample images from any consensus classification – so I gathered a bunch of images that had been identified as these rare species and checked them to calculate false positive rates.

Now, if a species has really low rates of false negatives and really low rates of false positives, then it’s one people are really good at identifying all the time. Note that in general, species have pretty low rates of both types of error. Furthermore, species with lower rates of false negatives have higher rates of false positives. There aren’t really any species with high rates of both types of error. Take rhinos, for example: folks often identify a rhino when it’s not actually there, but never miss a rhino if it is there.

Also: we see that rare species are just generally harder to identify correctly than common species. The plot below shows the same false negative and false positive error rates plotted against the total number of pictures for every species. Even though there is some noise, those lines reflect  significant trends: in general, the more pictures of an animal, the more often folks get it right!

Error rates vs. species commonness, measured by the total number of pictures of that species

Error rates vs. species commonness, measured by the total number of pictures of that species

This makes intuitive sense. It’s really hard to get a good “search image” for something you never see. But also folks are especially excited to see something rare. You can see this if you search the talk pages for “rhino” or “zorilla.” Both of these have high false positive rates, meaning people say it’s a rhino or zorilla when it’s really not. Thus, most of the images that show up tagged as these really rare creatures aren’t.

But that’s okay for the science. Because recall that we can assess how confident we are in an answer based on the evenness score, fraction support, and fraction blanks. Because such critters are so rare, we want to be really sure that those IDs are right — but because those animals are so rare, and because you have high levels of agreement on the vast majority of images, it makes it really easy to review any “uncertain” image that’s been ID’d as a rare species.

Pretty cool, huh?

Check out Mammal March Madness!

For some reason I missed this in 2013 and 2014. Maybe it was because I was finishing up my dissertation the first time and then recovering from a cross-country move the second time. But now I am totally excited about 2015’s

Mammal March Madness

What is Mammal March Madness? I’ll let organizer Katie Hinde explain:

In honor of the NCAA College Basketball March Madness Championship Tournament, Mammals Suck is featuring *simulated* combat competition among mammals. … Battle outcome is a function of the two species’ attributes within the battle environment. Attributes considered in calculating battle outcome include temperament, weaponry, armor, body mass, fight style, and other fun facts that are relevant to the outcome.

As a spectator to Mammal March Madness, you fill out a bracket and then follow along on Twitter or on the Mammals Suck … Milk! blog. The first game is on Monday, March 9, and direct elimination games continue until the championship on March 26.

I’ll note that the 2014 winner was Hyena who defeated Orca in the championship game. This year, we’ve got some Serengeti representation as well. But with Lion, Baboon, and Vervet monkey ranked just 8th, 12th, and 13th in the ‘Sexy Beasts’ division, they’re going to need all the cheering-on they can get.

So head on over, print out a bracket, and tell me who you think will make it all the way to the top this year.

(And just to be clear, I am not involved in Mammal March Madness in any way except as a participant. But it looks fun!)

Notes from the Lab

Meredith: Not all of our exciting research takes place in the field — there’s a lot going on behind the scenes in our lab, and we rely on an invaluable group of undergraduate research assistants to help us go through the massive amounts of Snapshot data you guys provide! Jess has been working with us for the last few semesters and has some insight on what it’s like to work with this Serengeti data set. 

Hello everyone! My name is Jessica Dewey, and I am currently an undergraduate student at the University of Minnesota working in the research lab that runs this project! Cool, right? I’m new around here so I thought my first post should be an introduction of myself and how I got involved in this lab.

Imagine me a few years ago: a young high school student, undoubtedly procrastinating in some way, suddenly stumbling upon a website called “Snapshot Serengeti”. At the time, I was only certain of two things — I loved animals and I loved research – so this discovery was perfect for me! I spent most of my evening identifying animals, and continued to go back to procrastinate even more.

Now flash forward to last semester, when I get an email from one of the university biology clubs saying that Dr. Craig Packer, head of the Serengeti Lion Project, will be giving a talk about his work. Well I HAVE to go! I sit and listen intently, eager to learn all about research being done with lions. Near the end of his talk he then mentions a website (Snapshot Serengeti, of course) where all of the images from the field get uploaded for the public to identify, and I’m immediately floored. How did I get so lucky, to go to the very University that uses those pictures I spent time identifying years ago in their research? The best moment was when the graduate students working with Craig said that they were looking for undergraduates to help them with their research. I took the opportunity to introduce myself to Meredith, and so far my experience in this lab has been amazing.

I’ve learned a lot about how field research is done, how data is collected and analyzed, and what it takes for someone to actually be a researcher in the field.  Not everything I do is as fun as going through tons of pictures a day, but all of the work in this lab is interesting and meaningful, and that’s what really matters to me. One of the major projects the lab has been working on with Meredith is trying to characterize changes in habitat at the camera trap sites by looking at the Snapshot pictures. We have been going through the giant list of data to find pictures to use for this characterization. We haven’t been going through the images themselves – rather the metadata, or the data ABOUT the data (it’s literally the biggest Excel sheet I’ve ever seen). It can get monotonous at times, but what keeps me going is the thought that when we finish picking out all of these random images, we will get to look at them and use them for this research project.

I hope that was a thorough enough introduction for you all, but let me say one last thing: THANK YOU!  Without the time you all put in to identifying these pictures, the research we are doing would not be happening at the pace it is.

Here is one of my favorite images I’ve seen so far:

pic1

 

Help us find animal selfies!

We’re partnering with National Geographic to put together a photo book of animal selfies from Snapshot Serengeti. We’ve got some selfies already from the first seven seasons, but because no one has looked through Season 8 yet, we don’t know what great selfies might be in there.

You can help! If you find an animal selfie, please tag it as #selfie in Talk. (Click the ‘Discuss’ button after you’ve classified the image and then enter #selfie below the image on the Talk page. You can get back to classifying using the button in the upper right.)

All proceeds from book sales will go to supporting Snapshot Serengeti. We’re planning for a fall 2016 publication date, so it will be a while. But we’re excited to get working on it.

tommie

Season 8 Movie Time

Looks like everyone is sinking their teeth into Season 8! As a reminder, feel free to ask questions or chat with us through the Snapshot Serengeti Discussion board or in the comments of any of our blog posts.

Now, there’s some data from this new season that hasn’t made it online — sometimes, instead of taking pictures, our cameras accidentally switch into “video” mode and capture 10-second clips of animals doing their Serengeti thing. While this isn’t very good for us in terms of data collection (although we’ve been tossing around the idea of setting up a Snapshot Serengeti: Video Edition!…), it gives you a unique perspective on the lives of these animals.

(Okay, so it’s mostly animals eating grass. They eat a lot of grass. Perhaps not the most “unique” insight on their behaviors, but they’re still pretty fun to watch). Here’s some of my favorite accidental movies from our new Season!

 

(If you want more videos, Margaret has blogged some of her favorite movie clips from past seasons here and here)

Season 8 Release!

And now, the moment you’ve all been waiting for …  Can I present to you:

IMAG0335

SEASON 8!

I’m particularly proud of this, the first season that I’ve helped to bring all the way from the field to your computers. We’ve got a lot of data here, and I can’t wait for you guys to discover a whole host of exciting things in this new season.

This season is accompanied by IMPORTANT changes to our interface!

There’s a few more bits of data we think we can pull out of the camera trap photos this time around, in addition to all the great information we already get. One thing we’re particularly interested in is the occurrence of fire. Now, fire is no fun for camera traps (because they tend to melt), but these wildfires are incredibly important to the cycle of ecosystem functioning in Serengeti. Burns refresh the soil and encourage new grass growth, which attracts herbivores and may in turn draw in the predators. We have added a fire checkbox for you to tick if things look hot. Now, because we’re looking for things other than just animals, we replaced your option to click on “nothing there” with “no animals visible“, just to avoid confusion.

Some of the more savvy creature-identifiers among you may have noticed that there are a few Serengeti animals that wander into our pictures that we didn’t have options for. For this new season, we’ve added six new animal choices: duiker, steenbok, cattle, bat, insect/spider, and vultures. Keep an eye out for the following:

duikersteenbokcattle

batspidervulture

This season runs all the way from September 2013 until July 2014, when I retrieved them this summer, my first field season. Our field assistants, Norbert and Daniel, were invaluable (and inhumanly patient) in helping me learn to navigate the plains, ford dry river beds, and avoid, as much as possible, driving the truck into too many holes. Together, we set out new cameras, patched up some holes in our camera trap grid, and spent some amazing nights camped out in the bush.

Once I got the hang of the field, I spend my mornings running around to a subset of the cameras conducting a pilot playback experiment to see if I could artificially “elevate” the predation risk in an area by making it seem as though it were frequented by lions (I’m interested in the reactions of the lion’s prey, and to see whether they change their behaviors in these areas and how long it takes them to go back to normal). I’m more than a bit camera-shy (and put a lot of effort into carefully sneaking up around the cameras’ blind spots) but perhaps you’ll catch a rare glimpse of me waving my bullhorn around blaring lion roars…

Back in the lab, there’s been a multi-continental collaboration to get these data cleaned up and ready for identification. We’ve been making some changes to the way we store our data, and the restructuring, sorting, and preparing process has been possible only through the substantial efforts of Margaret, over here with me in the States, and Ali, all the way across the pond, running things from the Zooniverse itself!

But for now, our hard work on this season is over – it’s your turn! Dig in!

P.S. Our awesome developers have added some fancy code, so the site looks great even on small phone and tablet screens. Check it out!

Getting good data: part 1 (of many)

Despite the site being quiet, there’s a lot going on behind the scenes on Snapshot Serengeti at the moment. Season 8 is all prepped and currently being uploaded and should be online next week! And on my end, I’ve been busily evaluating Snapshot Serengeti data quality to try and develop some generalisable guidelines to producing expert quality data from a citizen science project. These will actually be submitted as a journal article in a special section of the journal Conservation Biology, but as that is a slowwwwwww process, I thought I’d share them with you in the meanwhile.
So! Recall that we use a “plurality” algorithm to turn your many different answers into a “consensus dataset” — this has one final answer of what is in any given image, as well as various measures of certainty about that image. For example, back in 2013, Margaret described how we calculate an “evenness score” for each image: higher evenness means more disagreement about what is in an image, which typically means that an images is hard. For example, everyone who looks at this photo
would probably say there is 1 giraffe , but we’d expect a lot more diversity of answers for this photo:
(It’s a warthog, btw.)
To test how good the plurality algorithm answers were, we created a “gold-standard dataset” by asking experts to identify ~4,000 images. Overall, we found that consensus answers from your contributions agreed with experts nearly 97% of the time!  Which is awesome. But now I want to take a closer look.  So I took all the images that had gold standard data and I looked at Evenness, Number of “nothing here” responses, and %support for final species, and evaluated how those measures related to whether the answer was right or wrong (or, impossible). Even though we don’t have an “impossible” button on the SS interface, some images simply are impossible, and we let the experts tell us so so these wouldn’t get counted as just plain “wrong.”
A note on boxplots: If you’re not familiar with a boxplot, what you need to know is this: the dark line in the middle shows the median value for that variable; the top and bottom of the boxes shows the 25 and 75 percentiles; and the “whiskers” out the ends show the main range of values (calculated as 1.5 * interquartile range, details here). Any outliers are presented as dots.
Evenness: The boxplot below shows the mean “Evenness” score described above vs. how the consensus answer matched the gold standard answer.   What you can see below is that the average evenness score for “correct” answers is about 0.25, and the average evenness score for wrong and impossible answers is 0.75. Although there are some correct answers with high evenness scores, there are almost no wrong/impossible answers with evenness scores below 0.5.
 EvennessVsAccuracy
Percent Support: This number tells us how many people voted for the final answer out of the total number of classifiers. So, if 9 out of 10 classifiers said something was a giraffe, it would have 90% support. It’s similar to evenness, but simpler, and it shows essentially the same trend. Correct answers tended to have more votes for what was ultimately decided as the final species.
PercentSupport
NumBlanks: So with the evenness and percent support scores, we can do a decent job of predicting whether the consensus answer for an image is likely to be right or wrong. But with Number of Blanks we can get a sense of whether it is identifiable at all. Margaret noticed a while back that people sometimes say “nothing here” if they aren’t sure about an animal, so the number of “nothing here” votes for an image ultimately classified as an animal also reflects how hard it is. We see that there isn’t a huge difference in the number of “nothing here” answers on images that are right or wrong — but images that experts ultimately said were impossible have much higher average numbers of “nothing here” answers.
 NumBlanksVsAccuracy
So, what does this tell us? Well, we can these metrics on the entire dataset to target images that are likely to be incorrect. In any given analysis, we might limit our dataset to just those images with >0.50 evenness, or go back through all those images with >0.05% evenness to see if we can come up with a final answer. We can’t go through the millions of Snapshot Serengeti images on our own, but we can take a second look at a few thousand really tricky ones.
There’s all sorts of cool analyses still to come — what species are the hardest, and what’s most often mistaken for what. So stay tuned!

Good news, bad news, good news, 2015 edition

It’s been quiet here on the blog, but we’ve been busy behind the scenes. In 2014, we revamped our data management procedures and structures. Season 7 — the one you finished classifying most recently — was the first where images and metadata were fully pre-processed and vetted before being sent to the Zooniverse. This pre-processing makes things much easier on us after we get all your classifications back from Zooniverse. But it does add some lead time.

Season 8 is the first good news. We’ve been pre-processing all December, finding weirdnesses like 84 images in a row all with the same timestamp, miscellaneous video files, timestamps from the future, and so forth. We are just about to start sending the images to Zooniverse, a processes which takes a few days. You should see Season 8 up within a couple weeks. We’ve also tweaked the interface a tiny bit. More on that soon.

The bad news is bad. After waiting since August for a reply from the National Science Foundation about our most recent grant proposal, we finally got it at the very end of December: declined. That means that we are again scrambling to find funds to keep the cameras rolling for 2015. And this time without much warning.

Season 8 is the first half of 2014 and Season 9 is the second half of 2014. Those are already in the bag. The cameras are rolling right now, and so there will be at least something of a Season 10. Worst case scenario is that we have to shut everything down for a while until we get more funding. But Craig is working hard to find interim funds.

The other good news is that we’ve been talking with some other Serengeti researchers who have set up a small camera trap survey in the western part of the ecosystem. They have tons of images and we’re talking with them about putting their images up on Snapshot Serengeti for classification. These images would be of new locations in the Serengeti and potentially a few new animal species. Could be a lot of fun. So even if there’s a pause in our image collection, hopefully we’ll have these other images to classify from the Serengeti that will be useful for ecological research.

Mystery Animal

I’ve been slogging through some field data today, when this mysterious critter crawled across my screen. These are the only shots we got – check out those zebra-stripey tube socks in the first image and the strange feathery texture. Who here knows what we snapped a pic of?

IMAG3392 IMAG3393 IMAG3394

Follow

Get every new post delivered to your Inbox.

Join 4,850 other followers