Getting good data: part 1 (of many)

Despite the site being quiet, there’s a lot going on behind the scenes on Snapshot Serengeti at the moment. Season 8 is all prepped and currently being uploaded and should be online next week! And on my end, I’ve been busily evaluating Snapshot Serengeti data quality to try and develop some generalisable guidelines to producing expert quality data from a citizen science project. These will actually be submitted as a journal article in a special section of the journal Conservation Biology, but as that is a slowwwwwww process, I thought I’d share them with you in the meanwhile.
So! Recall that we use a “plurality” algorithm to turn your many different answers into a “consensus dataset” — this has one final answer of what is in any given image, as well as various measures of certainty about that image. For example, back in 2013, Margaret described how we calculate an “evenness score” for each image: higher evenness means more disagreement about what is in an image, which typically means that an images is hard. For example, everyone who looks at this photo
would probably say there is 1 giraffe , but we’d expect a lot more diversity of answers for this photo:
(It’s a warthog, btw.)
To test how good the plurality algorithm answers were, we created a “gold-standard dataset” by asking experts to identify ~4,000 images. Overall, we found that consensus answers from your contributions agreed with experts nearly 97% of the time!  Which is awesome. But now I want to take a closer look.  So I took all the images that had gold standard data and I looked at Evenness, Number of “nothing here” responses, and %support for final species, and evaluated how those measures related to whether the answer was right or wrong (or, impossible). Even though we don’t have an “impossible” button on the SS interface, some images simply are impossible, and we let the experts tell us so so these wouldn’t get counted as just plain “wrong.”
A note on boxplots: If you’re not familiar with a boxplot, what you need to know is this: the dark line in the middle shows the median value for that variable; the top and bottom of the boxes shows the 25 and 75 percentiles; and the “whiskers” out the ends show the main range of values (calculated as 1.5 * interquartile range, details here). Any outliers are presented as dots.
Evenness: The boxplot below shows the mean “Evenness” score described above vs. how the consensus answer matched the gold standard answer.   What you can see below is that the average evenness score for “correct” answers is about 0.25, and the average evenness score for wrong and impossible answers is 0.75. Although there are some correct answers with high evenness scores, there are almost no wrong/impossible answers with evenness scores below 0.5.
 EvennessVsAccuracy
Percent Support: This number tells us how many people voted for the final answer out of the total number of classifiers. So, if 9 out of 10 classifiers said something was a giraffe, it would have 90% support. It’s similar to evenness, but simpler, and it shows essentially the same trend. Correct answers tended to have more votes for what was ultimately decided as the final species.
PercentSupport
NumBlanks: So with the evenness and percent support scores, we can do a decent job of predicting whether the consensus answer for an image is likely to be right or wrong. But with Number of Blanks we can get a sense of whether it is identifiable at all. Margaret noticed a while back that people sometimes say “nothing here” if they aren’t sure about an animal, so the number of “nothing here” votes for an image ultimately classified as an animal also reflects how hard it is. We see that there isn’t a huge difference in the number of “nothing here” answers on images that are right or wrong — but images that experts ultimately said were impossible have much higher average numbers of “nothing here” answers.
 NumBlanksVsAccuracy
So, what does this tell us? Well, we can these metrics on the entire dataset to target images that are likely to be incorrect. In any given analysis, we might limit our dataset to just those images with >0.50 evenness, or go back through all those images with >0.05% evenness to see if we can come up with a final answer. We can’t go through the millions of Snapshot Serengeti images on our own, but we can take a second look at a few thousand really tricky ones.
There’s all sorts of cool analyses still to come — what species are the hardest, and what’s most often mistaken for what. So stay tuned!

Good news, bad news, good news, 2015 edition

It’s been quiet here on the blog, but we’ve been busy behind the scenes. In 2014, we revamped our data management procedures and structures. Season 7 — the one you finished classifying most recently — was the first where images and metadata were fully pre-processed and vetted before being sent to the Zooniverse. This pre-processing makes things much easier on us after we get all your classifications back from Zooniverse. But it does add some lead time.

Season 8 is the first good news. We’ve been pre-processing all December, finding weirdnesses like 84 images in a row all with the same timestamp, miscellaneous video files, timestamps from the future, and so forth. We are just about to start sending the images to Zooniverse, a processes which takes a few days. You should see Season 8 up within a couple weeks. We’ve also tweaked the interface a tiny bit. More on that soon.

The bad news is bad. After waiting since August for a reply from the National Science Foundation about our most recent grant proposal, we finally got it at the very end of December: declined. That means that we are again scrambling to find funds to keep the cameras rolling for 2015. And this time without much warning.

Season 8 is the first half of 2014 and Season 9 is the second half of 2014. Those are already in the bag. The cameras are rolling right now, and so there will be at least something of a Season 10. Worst case scenario is that we have to shut everything down for a while until we get more funding. But Craig is working hard to find interim funds.

The other good news is that we’ve been talking with some other Serengeti researchers who have set up a small camera trap survey in the western part of the ecosystem. They have tons of images and we’re talking with them about putting their images up on Snapshot Serengeti for classification. These images would be of new locations in the Serengeti and potentially a few new animal species. Could be a lot of fun. So even if there’s a pause in our image collection, hopefully we’ll have these other images to classify from the Serengeti that will be useful for ecological research.

Mystery Animal

I’ve been slogging through some field data today, when this mysterious critter crawled across my screen. These are the only shots we got – check out those zebra-stripey tube socks in the first image and the strange feathery texture. Who here knows what we snapped a pic of?

IMAG3392 IMAG3393 IMAG3394

Everybody dance now!

Formerly titled: I love the Zooniverse, and all its ridiculousness.

Some of you might be familiar with the silly manoeuvres we undertake to trigger the camera traps in the field. The sensors on the cameras are best at capturing movement side-to-side, hence ridiculousness such as this:

flcd4

The Stan Dance

and this…

flcgb

Craig surprising the camera

and this.

“The Ali Dance”

Well, fellow Zoo-ites Rob and Grant have now generously attempted their own version of this last one. Check it out on the Zooniverse Advent Calendar here!! And, if you’re feeling especially brave, send in your own recreation to team@zooniverse.org!

The predator of my predator is my friend.

If you’re a prey animal, you spend an awful lot of your time trying to not wind up like this:

lions_ngm_0813_006_brnd

 

As we’ve talked about an awful lot on this blog (here, here, and here, for example), the same holds true for a lot of predators. Just because you kill and eat other animals, doesn’t mean you don’t have to worry about being killed yourself (as this hyena so unceremoniously discovered).

But, what we haven’t talked so much about, is that the same holds true for plants. If you’re a plant, you get eaten by these terrifying animals:

But, just like prey animals and mesopredators can change their behaviour to minimise the risk of being killed, plants have a few tricks up their sleeves. They can spend a lot of energy growing big thorns, for example, that makes them less delectable.

Or! They can grow in places that their predators avoid — the places where their predators’ predators hang out. Got that?  It’s a trickle-down landscape of fear, which had, until now, only been really well documented in small experimental systems with critters like spiders and grasshoppers. But researcher Dr. Adam Ford and colleagues just published an elegant paper in Science showing that leopards and African wild dogs can make the Kenyan savanna less thorny through this cascade. Basically, leopards and wild dogs eat impala. Impala eat Acacia trees. Impala much prefer to eat acacias with fewer thorns (because really, who doesn’t?) – and, if given the opportunity, impala will can eat these small-thorned acacias so much that they can suppress the acacia population.

But! Leopards and wild dogs seem to be offering these tasty small-thorned acacias a refuge. Leopards and wild dogs spend most of their time in denser thickets, where they have more cover to hunt. Impala avoid these thickets and rarely venture in —  when they do, however, they have a much higher probability of being killed. And this creates this spiral – those tasty small-thorned trees survive and grow in these thickets because predators scare impala away.

So it’s a trickle down landscape of fear – a compelling and really exciting story. But, what sets Adam’s paper apart from many other attempts to document this effect in large predators, is the series of elegant experiments in which he and colleagues explicitly tested each step in this cascade.  Controlling for habitat use to confirm that impala aren’t getting killed in the woods simply because they spend more time there (and in fact, they get killed more even though the spend less time there). Adding and removing thorns to acacias to see if it was really the thorns that mattered. Creating herbivore exclosures to measure whether impala could really suppress acacia density. I spent my entire time reading the article alternating between saying “This is so cool!” and “I am so jealous!” It’s an amazing story. Read more about it here (or here, or or here)!

 

Zooniverse advent calendar

At the Zooniverse, we love any excuse to have fun, be festive, and highlight cool things from our awesome projects. Check out the Christmas countdown Advent Calendar at https://www.zooniverse.org/advent!

The How To Guide for Getting Field Experience

The Minnesota winter has finally come upon us and time is passing exasperatingly slowly, waiting to hear back from funding sources, plowing through homework, cleaning up data, and mostly daydreaming about heading back to Serengeti. Perhaps the dread of spending the next semester in the cold is stirring undergraduates into action, but I’ve been contacted by numerous students recently inquiring about something near and dear to my heart: field experience and how to get it.

Field work is what makes biology for me – I don’t think I could get by without that glimmer of hope, the promise of going out and getting dirty and experiencing ecology in the raw. The summers of my own undergraduate career and the two years before I entered graduate school were spent almost entirely out in the bush: measuring fishes and catching snakes and doing pretty much whatever kind of work I could come across that would let me mess around doing science in the great outdoors.

forblog1

“Doing Science” in the Great Outdoors

I lived for that work, but I can’t claim that it’s entirely glamorous. You won’t be picking up a brand new Ferrari any time soon, that’s for sure. My first field jobs could barely be called sustenance living, but after a few years of experience, I was picking up jobs that came with fancy, real-person benefits (oooh, like Dental).

And then there’s that whole “in the field” thing to consider — in all its glorious, treacherous, beautiful and exhausting majesty. I’ve been on field jobs where people have suffered through dengue and malaria, contracted parasites, twisted limbs, narrowly avoided encounters with venous snakes (on an almost daily basis), and quite literally passed out from exhaustion in the middle of the wilderness. “Sweat, blood, and tears” sums it up quite nicely. You’re stuck with the same old crew for weeks, or even months, on end, often with limited amenities. If isolation is not your thing, perhaps second thoughts may be in order. Also take into account the facilities you’ll be living in. I’ve been overwhelmed by the relative “luxury” of some field stations (electricity! food that isn’t rice and beans!), and enjoyed the struggle of situations at the opposite end of the spectrum (cold showers are good for you, and you didn’t need to check that Facebook this month anyway…).

forblog2

Just another day at work

 

Which isn’t to sell any aspect of fieldwork short. Doing fieldwork is an absolutely wonderful way to get your butt outdoors, see the world, enjoy nature, and it does wonders preparing you for a career in science. Techniques I’ve learned and people I’ve met along the way have been invaluable when it came to getting new jobs and heading back to school. I feel far more prepared to do my own research after having participated in such a diversity of projects. Plus, you get to be your down David Attenborough and live the things you’ve only ever seen on Nature documentaries or in the zoo. It’s a well worth-while experience.

So, the important part: where to find the job.

THE UNDERGRADUATE

For those still in an undergrad program looking for a summer position, the NSF Research Experiences for Undergraduates (REUs) are definitely the first place to hit up (NSF REU; NSF for BIOLOGY). These are great paying positions that are geared specifically towards getting you involved in your own research. I completed two REUs during my undergrad, spending one summer working in Panama studying developmental plasticity in Red-eyed tree frogs and another on the island of Puerto Rico filming the territorial behaviors of Anolis lizards. These experiences are wonderful because you are highly involved with the lab you work in, you get to meet and interact with a large body of scientists from various disciplines, and if you’re designing your own project, get invaluable input into the process of constructing an experiment. For me, both of my REU projects resulted in publications – an important factor for applying for graduate school.

LIST SERVS

List servs are beautiful, beautiful things, because job applications find their own way into your inbox and sit there waiting for you to read them. They’re also a great place to join in on scientific discussion and share ideas, articles, and even research equipment. Some of my favorite list-servs are:

  • ECOLOG-L: Run by the Ecological Society of America
  • MARMAM: For researchers working with marine mammals
  • MAMMAL-L: I believe this was set up by the American Society of Mammologists?

You can probably tell that I’m a bit biased towards mammal work, but ECOLOG runs job advertisements from everything ranging from forest ecology to herps and fishes through to hyena biology in Kenya.

 

WEBSITES

Biology job boards are the next place I turn when looking for the next field position. These update fairly regularly, so keep checking up on them:

  • Texas A&M biology: My absolute favorite – there are some really fantastic research opportunities that make their way to the Texas A&M job board
  • ConBio: Run by the Society for Conservation Biology
  • Primates: For those interested specifically in primates
  • AZA (Zoos):  If the field isn’t quite for you, but you’re still gung ho about working with animals, be sure to check out what’s going on at the zoos
  • USAjobs: Government jobs are some of the better-paying gigs in the biology business

Find the job applications is, like most things in life, just the first step in a Process. Next come the cover letters, the applications themselves, scrounging up enough references and actually getting them to submit letters for you on time (often, the most difficult part). But hopefully this provides as starting point for those ready to get out there and do some science.

Welcome Darren

Darren, our new Community Builder

Darren, our new Community Builder

Hey everyone – I just wanted to introduce you to one of the Zooniverse’s newest members, Darren McRoy, who is our new Community Builder. 

As Community Builder, Darren serves as the Zooniverse’s liaison with its citizen science community, including Snapshot Serengeti. He also assists with Zooniverse’s general communications efforts and is working closely with designers and developers on the next generation of the Talk discussion system.

Darren is a 2010 graduate of the Medill School of Journalism at Northwestern University and has a background in journalism and digital communications. He is a resident of the northern Chicago suburbs, and enjoys golf, volleyball, fiction writing, gaming, and participating in a variety of online communities.

Darren can be reached at darren@zooniverse.org, and posts on Talk under the handle “DZM.” Please feel free to get in touch with him if you have any thoughts, comments, questions, etc about how the Zooniverse communicates with its community — you!

Groovin’

Just for fun. I was hanging around on Talk today and stumbled across this grooving Kori Bustard. Kind of makes me want to dance…

dancin' kori bustard

dancin’ kori bustard

Follow

Get every new post delivered to your Inbox.

Join 4,784 other followers