NYU Data Science newsletter – October 5, 2015

GROUP CURATION: N/A

Digital storytelling can offer unprecedented insights into scientific data for both the lay public and scientific researchers, says Samuel Van Ransbeek.

Welcome to the Data Documentation Initiative

A metadata specification for the social and behavioral sciences

Find out how others have put DDI to work in their organizations, explore resources for learning more about and using the DDI, or join the DDI Community.

In this notebook, we cover the basics of probability theory, and show how to implement the theory in Python. (You should have a little background in probability and Python.) Then we show how to solve some particularly perplexing paradoxical probability problems.

Data scientist interviews can be complex and there is no definite recipe for the success. Understand the complications and processes of an interview and what you should be careful about before accepting the offer.

Scientists at Virginia refined a mathematical model that simulates the impact of genetic mutations on cell division, a step that could provide insight into errors that produce and sustain harmful cells, such as those found in tumors. In a study (“Experimental testing of a new integrated model of the budding yeast START transition”) in Molecular Biology of the Cell, the group detailed their findings in laboratory experiments, which examined how mutant cells moved through a series of processes to duplicate their genetic material and divide.

Using these results, the team says it developed more accurate techniques for predicting the effects that gene mutations will have on a cell’s ability to regulate its rate of division using natural checkpoints such as cell size and the availability of nutrients.

Volkswagen didn’t make a faulty car: They programmed it to cheat intelligently. The difference isn’t semantics, it’s game-theoretical (and it borders on applied demonology).

… Spotify has found a new way to tap the collective intelligence of its 75 million users, turning their taste into a data layer that can be used to better personalize everyone’s experience. I’m not the only one who noticed. “It’s good. It’s better than I thought it would be,” says tech entrepreneur and web pioneer Anil Dash. To Dash, Discover Weekly felt carefully tended, even though it was being produced by machines. “They’re as good as DJs — at scale.”

“At first I was a bit skeptical,” says Billy Chasen, founder of Turntable.fm. “I was definitely in the camp that believes you really need people to help find music for you, [that] algorithms can only take you so far.” After a few weeks using Discover, Chasen was a believer. “One [song] every week is a real gem. All of a sudden it’s like, this thing is awesome.”

If this forecast holds, Hurricane Joaquin will yield one clear winner: the model from the European Center for Medium-Range Weather Forecasts — or simply, the European model — which consistently forecast that Joaquin would head off to sea.

When the highly anticipated TechCrunch Disrupt competition drew to a close last week in San Francisco, a New York City-based, NYU-backed startup came out on top.

Agrilyst, an intelligence platform for indoor farms, was named the winner of TechCrunch Disrupt’s insanely competitive Startup Battlefield.

Machine learning, which is the basis for most commercial artificial-intelligence systems, is intrinsically probabilistic. An object-recognition algorithm asked to classify a particular image, for instance, might conclude that it has a 60 percent chance of depicting a dog, but a 30 percent chance of depicting a cat.

At the Annual Conference on Neural Information Processing Systems in December, MIT researchers will present a new way of doing machine learning that enables semantically related concepts to reinforce each other.

Actually the course is called Statistical Communication and Graphics, but I was griping about how few students were taking the class, and someone suggested the title Communicating Data and Statistics as being a bit more appealing. So I’ll go with that for now.

I love love love this class and everything that’s come from it (including statistics diaries and ShinyStan).

Broadly, I’m interested in the process of data analysis/science and how to make it easier, faster, and more fun. That’s what has lead to the development of my most popular packages like ggplot2, dplyr, tidyr, stringr. This year, I’ve been particularly interested in making it as easy as possible to get data into R. That’s lead to my work on the DBI, haven, readr, readxl, and httr packages. Please feel free to ask me anything about the craft of data science.

I’m also broadly interested in the craft of programming, and the design of programming languages. I’m interested in helping people see the beauty at the heart of R and learn to master it as easily as possible.