Some Results from Season 4

I was asked in the comments to last week’s blog post if I could provide some feedback about the results of Season 4. If you felt like you were seeing a lot of “nothing here” images, you’re right: of the 158,098 unique capture events we showed you, 70% were classified as having no animals in them. That left 47,320 with animals in them to classify, and the vast majority of these (94%) contained just one species. Here’s the breakdown of what was in all those images:

Maybe it won’t surprise you that Season 4 covered 2012’s wet season, when over a million wildebeest, zebra, and Thomson’s gazelle migrate through our study area. I find it interesting that hartebeest are also pretty numerous, but I wonder if it’s because of that one hartebeest that stood in front of the camera for hours on end.

This pie chart is based on the number of what we call “capture events,” which is the set of 1 or 3 pictures you see every time you make a classification. Once a camera has taken a set of pictures, we delay it from triggering again for about a minute. That way we don’t fill up the camera’s memory card with too many repeats of the same animals before we have a chance to replace them. But a minute isn’t a very long time for an animal that has decided to camp out in front of a camera, and so we frequently get sequences of many capture events that are all of the same animal. One of the things we’ll have to do in turning your classifications into valid research results is to figure out how to find these sequences in the data automatically.

Here’s a sequence of an elephant family hanging out around our camera for the night about a year ago. (Hat tip to dms246 who put together a collection of most of these images to answer the concerned question of some classifiers who saw just one image out of the whole sequence: is that elephant dead or just sleeping?)

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

If you’re interested in how I made the above pie chart, keep reading. But we’re going to get technical here, so if algorithms don’t interest you, feel free to stop.

The Nitty Gritty

The first thing I had to do with the data is to figure out what the “right” animal is based on all the volunteer classifications. Each capture event gets at least 10 pairs of eyeballs looking at it, so we always have 10 or more classifications to consider.

Now I have an admission to make. In doing this analysis, I used an old table of classifications from January 17, five days before we actually finished Season 4 and when we were about 90% through it. This means that not all capture events in my analysis table actually had 10 classifications. This minor discrepancy doesn’t matter for the big overall picture, like the pie chart above, but you’ll see it does matter for the details.

To figure out the correct species, I took the 47,320 capture events with animals in them and counted how many species were marked in each classification for each capture event. I used the median number of species classified for each capture event (rounded up) as the “real” number of species in that image. The result of this was that 44,471 capture events contained just one species, 2,811 contained two species, and 38 had three species.

Next, I decided to use a really simple strategy to assign the “correct” species to each capture event. For capture events with just one species, I determined the correct species as the one with at least 50% of the vote. If a species had exactly 50% of the vote, I declared it the winner if there wasn’t another species that also had 50% of the vote. So, for example, if a capture event had 10 votes for Thomson’s gazelle and another 10 votes for Grant’s gazelle, then neither would win. But if it had 10 votes for Thomson’s gazelle, 8 votes for Grant’s gazelle, and 2 votes for impala, I would declare Thomson’s gazelle the winner. This strategy left me with “correct” species identifications for 96% of the capture events! There remained just 1,604 capture events where no one species had at least half of the vote. I left these as “hard to figure out” in the pie chart above.

How certain am I that my simple algorithm to determine species identifications is correct? I obviously haven’t flipped through all the capture events by hand to check. But I did note that 25,279 (57%) of the single-species capture events were unanimous – in other words everyone who saw those images agreed on the species. And remember that’s at least ten people. Not bad. A total of 37,430 (84%) capture events had at least 75% agreement among classifiers. So I feel pretty good that this simple algorithm can do well on the majority of capture events.

What about those capture events with multiple species in them? I used the same simple logic to assign species to those. For two-species capture events, I looked for exactly two species that each had at least 25% of the vote. That would mean that together, they had at least 50% of the vote. If I ended up with 3 or 1 or 0 species with at least 25% of the vote, I considered the capture event “hard to figure out.”  For three-species capture events, I looked for exactly three species that each had at least 16.66% of the vote. The result was that I could determine “correct” identifications on 96% (2,700/2,811) of two-species capture events and 82% (31/38) of three-species capture events.

Below are the tabulated results. And now here is where it’s important to remind you that I didn’t use the fully-finished Season 4 data. I probably should have removed capture events that hadn’t yet accumulated ten classifications at the start, but I didn’t think to do that at the time. When I started looking at the actual images of supposed rarely captured animals I discovered that many of them were errors. If only one person had classified a capture event, then it was considered to be at 100% known identification, even if the classification was completely wrong.

So take the numbers at the bottom of the table with a grain of salt; many of them would decrease or turn to zero if I were to run this algorithm again on the fully classified data. For example, I found only one real rhino and one real wildcat in the several images that were putatively “rhino” and “wildcat” due to too few classifications, and no civets, genets, or zorillas. (Sorry for those of you trying to collect them all!) However, the multi-species rare combinations seem to be pretty good. The two female-lion-and-other-bird images (here and here) checked out, as did the one secretary-bird-wildebeest-zebra image, and several others that I spot-checked.