We need an ‘I don’t know’ button!
Okay, okay. I hear you. I know it’s really frustrating when you get an image with a partial flank or a far away beast or maybe just an ear tip. I recognize that you can’t tell for sure what that animal is. But part of why people are better at this sort of identification process than computers is that you can figure out partial information; you can narrow down your guess. That partial flank has short brown hair with no stripes or bars. And it’s tall enough that you can rule out all the short critters. Well, now you’ve really narrowed it down quite a lot. Can you be sure it’s a wildebeest and not a buffalo? No. But by taking a good guess, you’ve provided us with real, solid information.
We show each image to multiple people. Based on how much the first several people agree, we may show the image to many more people. And when we take everyone’s identifications into account, we get the right answer. Let me show you some examples to make this clearer. Here’s an easy one:
And if we look at how this got classified, we’re not surprised:
I don’t even have to look at the picture. If you hid it from me and only gave me the data, I would tell you that I am 100% certain that there is one moving giraffe in that image.
Okay, let’s take a harder image and its classifications:
This image is, in fact, of buffalo – at least the one on the foreground is, and it’s reasonable to assume the others are, too. Our algorithm would also conclude from the data table that this image is almost certainly of buffalo – 63% of classifiers agreed on that, and the other three classifications are ones that are easily confused with buffalo. We can also figure out from the data you’ve provided us that the buffalo are likely eating and moving, and that there is one obvious buffalo and another 2 or 3 ones that are harder to tell.
My point in showing you this example is that even with fairly difficult images, you (as a group) get it right! If you (personally) mess up an image here or there, it’s no big deal. If you’re having trouble deciding between two animals, pick one – you’ll probably be right.
Now what if we had allowed people to have an ‘I don’t know’ button for this last image? I bet that half of them would have pressed, ‘I don’t know.’ We’d be left with just 4 identifications and would need to send out this hard image to even more people. Then half of those people would click ‘I don’t know’ and we’d have to send it out to more people. You see where I’m going with this? An ‘I don’t know’ button would guarantee that you would get many, many more annoying, frustrating, and difficult images because other people would have clicked ‘I don’t know.’ When we don’t have an ‘I don’t know’ button, you give us some information about the image, and that information allows us to figure out each image faster – even the difficult ones.
“Fine, fine,” you might be saying. “But seriously, some of those images are impossible. Don’t you want to know that?”
Well, yes, we do want to know that. But it turns out that when you guess an animal and press “identify” on an impossible image, you do tell us that. Or, rather, as a group, you do. Let’s look at one:
Now I freely admit that it is impossible to accurately identify this animal. What do you guys say? Well…
Right. So there is one animal moving. And the guesses as to what that animal is are all over the place. So we don’t know. But wait! We do know a little; all those guesses are of small animals, so we can conclude that there is one small animal moving. Is that useful to our research? Maybe. If we’re looking at hyena and leopard ranging patterns, for example, we know whatever animal got caught in this image is not one we have to worry about.
So, yes, I know you’d love to have an ‘I don’t know’ button. I, myself, have volunteered on other Zooniverse projects and have wished to be able to say that I really can’t tell what kind of galaxy that is or what type of cyclone I’m looking at. But in the end, not having that button there means that you get fewer of the annoying, difficult images, and also that we get the right answers, even for impossible images.
So go ahead. Make a guess on that tough one. We’ll thank you.
23 responses to “We need an ‘I don’t know’ button!”
Trackbacks / Pingbacks
- - May 13, 2014
- - July 8, 2015
- - February 2, 2016
Snapshot Serengeti 
Excellent explanation of why a “I don’t know ” button is of no use to this project.
Hopefully it will slow the whining? 🙂
OK, I’m convinced. 🙂
Brilliantly argued!
Yeah, I’m convinced too. 🙂 I’d still like a “This is a complete guess” button though, just to make me more comfortable when I have to do a complete guess! 😀
Really good explanation thank you. What an amazing project and how exciting that I can be part of it. Has cheered up a cold and wet day in Wales!
Is it worth adding a ‘ flora movement’ button to help filter out images that were obviously triggered by such.
Why not have a requirement to choose something, as you explained, AND a confidence level measure, as a radio button, or a slider “Random guess – probably – pretty sure – certain”? This way you could assess user answers in a better way.
The trouble is that confidence doesn’t correlate with ability, and just asking about confidence makes people less careful, reducing the quality of classifications. Humans are complicated!
well said…explains it all very well…of course, I hope you know we were trying to be helpful (even though we were a little misguided now having read your response). There is, as at turns out, a bit of guilt associated with guessing on your data but we are now comforted that our efforts are actually contributory, despite some speculation. Thanks!
Thanks for the nice explaination. Now it all makes perfect sense.
And Andrew (comment 1)…we are not “whining”…we are trying to do a good job, and take an interest in this endevor.
Thanks for the feedback! It’s nice to know that wild guesses aren’t going to spoil the results and will feel much happier attempting to classify those random blobs. lol
Andrew, I was actually refreshing the page to get a new image rather than attempt a complete guess, which rather goes against the point of the project!!
So was I. I’ll do that a lot less now.
I may still refresh some because agonizing over an unindentifiable animal reduces my enjoyment and therefore makes me less likely to identify as many future pictures. But at least I know I can feel free to make a wild guess, particularly if I can narrow it down *at all*.
It soon became clear to me from the moment I became aware of this site, read about its intent and began enjoying my own HOURS of “classifying”, we the general public were enlisted because current complex computer algorithms just aren’t as competent as humans when it comes to identifying animals in photographs.
This fact was alluded to in the “About” section of this site, and excellently clarified in Margaret’s post.
We are an eclectic mix of the general populous, none of us are expected to (or dare I say few of us qualified to) identify with absolute certainty any of of the animals captured in the photos. I as an entrepreneur with a computer background am laughably the least qualified biologist here! 🙂 We are just contributing our time and best guesses to help the actual scientists and experts collecting and analyzing the data.
My original comment was based on reading the LONG blog post of people complaining about not being able to identify with any certainty many photos. I just had to laugh because the entire purpose of this project seemed lost on those individuals.
Trust me when I say I am just as equally frustrated as everyone else when I am confronted with a nearly impossible ID, or worse with one that gives me just enough a taste that I find my pseudo perfectionist personality can’t seem to give up on… 🙂
Can you put this explanation about why there is no such button in the instructions for people? I would’ve felt a lot more comfortable about guessing if I had known guessing was desirable.
Bit late to comment on this but I’d much rather, instead of having to pick an animal that it may or may not be, to have an option to pick all the features that you CAN identify (colour, size, pattern, tail etc…) and click a “features only” button so you’ve given as much info as possible without “identifying something wrong” which I think is putting some people off.
Also an “Other (write in)” or “Insect” option would be good. I saw a butterfly that was rather nice in one.
Yes, I agree, but Margaret’s purpose seems to be to get a rounded view, although some of us are likely to pick up something others may miss, at least sometimes. I agree completely about an “insect” category; an insect should not merit a “nothing here” yet there is no other category. Such a category
would probably decrease chat comments about them.
It is important to remember that the information to a researcher is specific to her objective. If additional categories or parameters of data are added, it may increase the amount of specific information that is given about a photo, but it may not necessarily be of added benefit to the researcher using the data, instead offering up even more data for them to sift through that is not relevant to their objective. Having to evaluate the self-value given by citizen scientists on their work by the researcher using it would not only increase workload unnecessarily, but also make the data collected less valid as a self-reflective evaluation would push the collection into the emotional/social rather than empirical realm; it would become data on how citizen scientist evaluate their contributions rather than on what they determine they perceive in a given image that can be be used as correlative data to determine identification.
Guess I’d better stop using the ‘nothing here’ button when I mean ‘nothing here that I can figure out’….. even if the photo could be showing a polar bear for all I know…
so is it better to guess and go into talk and say that it was a guess? or just guess and move on?
Many people interested in science are naturally rationalists. A rationalist knows that he makes mistakes, and in a scientific system where the “I don’t know” button is forbidden, this is not very rational. Personally I avoided the Zooniverse sites where I could not be sure. A “I don’t know” button seems more rational to me than forcing a false truth. It is a dilemma for a rationalist to know that we are being coerced into something false. So I think that the absence of this button must have put off many people from coming to help decipher the images.