Okay, so by now you’ve heard dozens and dozens of times that you guys produce really good data: your aggregated answers are 97% correct overall (see here and here and here). But we also know that not all images are equally easy. More specifically, not all species are equally easy. It’s a lot easier to identify a giraffe or zebra than it is to decide between an aardwolf and striped hyena.
The plot below shows the different error rates for each species. Note that error comes in two forms. You can have a “false negative” which means you miss a species given that it’s truly there. And then you can have a “false positive,” in which you report a species as being there when it really isn’t. Error is a proportion from 0 to 1.
We calculated this by comparing the consensus data to the gold standard dataset that Margaret collated last year. Note that at the bottom of the chart there are a handful of species that don’t have any values for false negatives. That’s because, for statistical reasons, we could only calculate false negative error rates from completely randomly sampled images, and those species are so rare that they didn’t appear in the gold standard dataset. But for false positives, we could randomly sample images from any consensus classification – so I gathered a bunch of images that had been identified as these rare species and checked them to calculate false positive rates.
Now, if a species has really low rates of false negatives and really low rates of false positives, then it’s one people are really good at identifying all the time. Note that in general, species have pretty low rates of both types of error. Furthermore, species with lower rates of false negatives have higher rates of false positives. There aren’t really any species with high rates of both types of error. Take rhinos, for example: folks often identify a rhino when it’s not actually there, but never miss a rhino if it is there.
Also: we see that rare species are just generally harder to identify correctly than common species. The plot below shows the same false negative and false positive error rates plotted against the total number of pictures for every species. Even though there is some noise, those lines reflect significant trends: in general, the more pictures of an animal, the more often folks get it right!
This makes intuitive sense. It’s really hard to get a good “search image” for something you never see. But also folks are especially excited to see something rare. You can see this if you search the talk pages for “rhino” or “zorilla.” Both of these have high false positive rates, meaning people say it’s a rhino or zorilla when it’s really not. Thus, most of the images that show up tagged as these really rare creatures aren’t.
But that’s okay for the science. Because recall that we can assess how confident we are in an answer based on the evenness score, fraction support, and fraction blanks. Because such critters are so rare, we want to be really sure that those IDs are right — but because those animals are so rare, and because you have high levels of agreement on the vast majority of images, it makes it really easy to review any “uncertain” image that’s been ID’d as a rare species.
Pretty cool, huh?
For some reason I missed this in 2013 and 2014. Maybe it was because I was finishing up my dissertation the first time and then recovering from a cross-country move the second time. But now I am totally excited about 2015’s
What is Mammal March Madness? I’ll let organizer Katie Hinde explain:
In honor of the NCAA College Basketball March Madness Championship Tournament, Mammals Suck is featuring *simulated* combat competition among mammals. … Battle outcome is a function of the two species’ attributes within the battle environment. Attributes considered in calculating battle outcome include temperament, weaponry, armor, body mass, fight style, and other fun facts that are relevant to the outcome.
As a spectator to Mammal March Madness, you fill out a bracket and then follow along on Twitter or on the Mammals Suck … Milk! blog. The first game is on Monday, March 9, and direct elimination games continue until the championship on March 26.
I’ll note that the 2014 winner was Hyena who defeated Orca in the championship game. This year, we’ve got some Serengeti representation as well. But with Lion, Baboon, and Vervet monkey ranked just 8th, 12th, and 13th in the ‘Sexy Beasts’ division, they’re going to need all the cheering-on they can get.
So head on over, print out a bracket, and tell me who you think will make it all the way to the top this year.
(And just to be clear, I am not involved in Mammal March Madness in any way except as a participant. But it looks fun!)
Meredith: Not all of our exciting research takes place in the field — there’s a lot going on behind the scenes in our lab, and we rely on an invaluable group of undergraduate research assistants to help us go through the massive amounts of Snapshot data you guys provide! Jess has been working with us for the last few semesters and has some insight on what it’s like to work with this Serengeti data set.
Hello everyone! My name is Jessica Dewey, and I am currently an undergraduate student at the University of Minnesota working in the research lab that runs this project! Cool, right? I’m new around here so I thought my first post should be an introduction of myself and how I got involved in this lab.
Imagine me a few years ago: a young high school student, undoubtedly procrastinating in some way, suddenly stumbling upon a website called “Snapshot Serengeti”. At the time, I was only certain of two things — I loved animals and I loved research – so this discovery was perfect for me! I spent most of my evening identifying animals, and continued to go back to procrastinate even more.
Now flash forward to last semester, when I get an email from one of the university biology clubs saying that Dr. Craig Packer, head of the Serengeti Lion Project, will be giving a talk about his work. Well I HAVE to go! I sit and listen intently, eager to learn all about research being done with lions. Near the end of his talk he then mentions a website (Snapshot Serengeti, of course) where all of the images from the field get uploaded for the public to identify, and I’m immediately floored. How did I get so lucky, to go to the very University that uses those pictures I spent time identifying years ago in their research? The best moment was when the graduate students working with Craig said that they were looking for undergraduates to help them with their research. I took the opportunity to introduce myself to Meredith, and so far my experience in this lab has been amazing.
I’ve learned a lot about how field research is done, how data is collected and analyzed, and what it takes for someone to actually be a researcher in the field. Not everything I do is as fun as going through tons of pictures a day, but all of the work in this lab is interesting and meaningful, and that’s what really matters to me. One of the major projects the lab has been working on with Meredith is trying to characterize changes in habitat at the camera trap sites by looking at the Snapshot pictures. We have been going through the giant list of data to find pictures to use for this characterization. We haven’t been going through the images themselves – rather the metadata, or the data ABOUT the data (it’s literally the biggest Excel sheet I’ve ever seen). It can get monotonous at times, but what keeps me going is the thought that when we finish picking out all of these random images, we will get to look at them and use them for this research project.
I hope that was a thorough enough introduction for you all, but let me say one last thing: THANK YOU! Without the time you all put in to identifying these pictures, the research we are doing would not be happening at the pace it is.
Here is one of my favorite images I’ve seen so far:
We’re partnering with National Geographic to put together a photo book of animal selfies from Snapshot Serengeti. We’ve got some selfies already from the first seven seasons, but because no one has looked through Season 8 yet, we don’t know what great selfies might be in there.
You can help! If you find an animal selfie, please tag it as #selfie in Talk. (Click the ‘Discuss’ button after you’ve classified the image and then enter #selfie below the image on the Talk page. You can get back to classifying using the button in the upper right.)
All proceeds from book sales will go to supporting Snapshot Serengeti. We’re planning for a fall 2016 publication date, so it will be a while. But we’re excited to get working on it.
Looks like everyone is sinking their teeth into Season 8! As a reminder, feel free to ask questions or chat with us through the Snapshot Serengeti Discussion board or in the comments of any of our blog posts.
Now, there’s some data from this new season that hasn’t made it online — sometimes, instead of taking pictures, our cameras accidentally switch into “video” mode and capture 10-second clips of animals doing their Serengeti thing. While this isn’t very good for us in terms of data collection (although we’ve been tossing around the idea of setting up a Snapshot Serengeti: Video Edition!…), it gives you a unique perspective on the lives of these animals.
(Okay, so it’s mostly animals eating grass. They eat a lot of grass. Perhaps not the most “unique” insight on their behaviors, but they’re still pretty fun to watch). Here’s some of my favorite accidental movies from our new Season!
And now, the moment you’ve all been waiting for … Can I present to you:
I’m particularly proud of this, the first season that I’ve helped to bring all the way from the field to your computers. We’ve got a lot of data here, and I can’t wait for you guys to discover a whole host of exciting things in this new season.
This season is accompanied by IMPORTANT changes to our interface!
There’s a few more bits of data we think we can pull out of the camera trap photos this time around, in addition to all the great information we already get. One thing we’re particularly interested in is the occurrence of fire. Now, fire is no fun for camera traps (because they tend to melt), but these wildfires are incredibly important to the cycle of ecosystem functioning in Serengeti. Burns refresh the soil and encourage new grass growth, which attracts herbivores and may in turn draw in the predators. We have added a fire checkbox for you to tick if things look hot. Now, because we’re looking for things other than just animals, we replaced your option to click on “nothing there” with “no animals visible“, just to avoid confusion.
Some of the more savvy creature-identifiers among you may have noticed that there are a few Serengeti animals that wander into our pictures that we didn’t have options for. For this new season, we’ve added six new animal choices: duiker, steenbok, cattle, bat, insect/spider, and vultures. Keep an eye out for the following:
This season runs all the way from September 2013 until July 2014, when I retrieved them this summer, my first field season. Our field assistants, Norbert and Daniel, were invaluable (and inhumanly patient) in helping me learn to navigate the plains, ford dry river beds, and avoid, as much as possible, driving the truck into too many holes. Together, we set out new cameras, patched up some holes in our camera trap grid, and spent some amazing nights camped out in the bush.
Once I got the hang of the field, I spend my mornings running around to a subset of the cameras conducting a pilot playback experiment to see if I could artificially “elevate” the predation risk in an area by making it seem as though it were frequented by lions (I’m interested in the reactions of the lion’s prey, and to see whether they change their behaviors in these areas and how long it takes them to go back to normal). I’m more than a bit camera-shy (and put a lot of effort into carefully sneaking up around the cameras’ blind spots) but perhaps you’ll catch a rare glimpse of me waving my bullhorn around blaring lion roars…
Back in the lab, there’s been a multi-continental collaboration to get these data cleaned up and ready for identification. We’ve been making some changes to the way we store our data, and the restructuring, sorting, and preparing process has been possible only through the substantial efforts of Margaret, over here with me in the States, and Ali, all the way across the pond, running things from the Zooniverse itself!
But for now, our hard work on this season is over – it’s your turn! Dig in!
P.S. Our awesome developers have added some fancy code, so the site looks great even on small phone and tablet screens. Check it out!
It’s been quiet here on the blog, but we’ve been busy behind the scenes. In 2014, we revamped our data management procedures and structures. Season 7 — the one you finished classifying most recently — was the first where images and metadata were fully pre-processed and vetted before being sent to the Zooniverse. This pre-processing makes things much easier on us after we get all your classifications back from Zooniverse. But it does add some lead time.
Season 8 is the first good news. We’ve been pre-processing all December, finding weirdnesses like 84 images in a row all with the same timestamp, miscellaneous video files, timestamps from the future, and so forth. We are just about to start sending the images to Zooniverse, a processes which takes a few days. You should see Season 8 up within a couple weeks. We’ve also tweaked the interface a tiny bit. More on that soon.
The bad news is bad. After waiting since August for a reply from the National Science Foundation about our most recent grant proposal, we finally got it at the very end of December: declined. That means that we are again scrambling to find funds to keep the cameras rolling for 2015. And this time without much warning.
Season 8 is the first half of 2014 and Season 9 is the second half of 2014. Those are already in the bag. The cameras are rolling right now, and so there will be at least something of a Season 10. Worst case scenario is that we have to shut everything down for a while until we get more funding. But Craig is working hard to find interim funds.
The other good news is that we’ve been talking with some other Serengeti researchers who have set up a small camera trap survey in the western part of the ecosystem. They have tons of images and we’re talking with them about putting their images up on Snapshot Serengeti for classification. These images would be of new locations in the Serengeti and potentially a few new animal species. Could be a lot of fun. So even if there’s a pause in our image collection, hopefully we’ll have these other images to classify from the Serengeti that will be useful for ecological research.