NYU Data Science newsletter – October 8, 2015

GROUP CURATION: N/A

The time when big data and data science were just buzzwords has definitely past. More and more companies and marketing professionals realize that these two specialisms evolved from their ‘hype’ status. However, that doesn’t mean that these companies can fully benefit from it. Besides best-of-breed technology, this also demands the brightest brains. Data expertise alone is not sufficient to truly understand your customer. Data scientists are important but you also need other experts on your team.

This introductory course in data science is built on three interrelated perspectives: inferential thinking, computational thinking, and real-world relevance. Given data arising from some real-world phenomenon, how does one analyze that data so as to understand that phenomenon? How does one collect data to answer questions that one is interested in? Inferential thinking refers to an ability to connect data to underlying phenomena and to the ability to think critically about the conclusions that are drawn from data analysis. Computational thinking refers to the ability to conceive of the abstractions and processes that allow inferential procedures to be embodied in computer programs, and to ensure that such programs are scalable, robust and understandable. In addition to teaching basic skills in computer programming and statistical inference, the course will also involve the hands-on analysis of a variety of real-world datasets, including economic data, document collections, geographical data and social networks, and it will delve into social and legal issues surrounding data analysis, including issues of privacy and data ownership.

Computational and Inferential Thinking is an introductory text for data science that explores foundational concepts in data processing and statistics using modern programming tools. Ideas are illustrated by real-world data sets and examples. While rigorous in presentation, this text does not expect prior experience in computing, calculus, or linear algebra.

Current infrastructure makes it difficult for data scientists to share analytical models with the software engineers who need to integrate them.

Yhat is an enterprise software company tackling the challenge of how data science gets done. Their products enable companies and users to easily deploy data science environments and translate analytical models into production code. [audio, 47:14]

Data scientist – a job that barely existed a decade ago – has become one of the hottest and best-paid professions in the U.S. Companies say they need people who have the skill sets – both business and technical – to analyze the rising tide of data produced by customers and operations.

RJMetrics, a software startup that itself is looking for data scientists to fill open positions, dove into LinkedIn data to gauge the field’s scope.

The value of social media analytics was long before proven. But the extent to which it is influencing real-time business decisions has made all predictions look miniscule. Today, social insights has become one of the primary inputs into marketing and product strategy formulation.

Dashboards by Radian6, Brandwatch and Sysomos have been ruling the monitoring space, but social analytics has evolved beyond monitoring in the last few years. From consumer insights to breaking news alerts, a bunch of companies have created their unique niches, and are providing unparalleled intelligence to their clients. Here is a list of seven of those and what makes them unique.

The Microsoft Ventures Accelerator in Seattle has opened applications for its newest batch of startups, and our focus this third time around is on machine learning and data science startups.

… Obama’s chief analytics officer, Dan Wagner, is hoping to bring the power of this tool to brands, non-profits, and of course, the 2016 presidential candidates, through his Chicago-based startup Civis Analytics.

“The lack of precision in television is really kind of astonishing,” Wagner says. “But people accept higher levels of waste, because there’s been no alternative. We’re hoping to provide that alternative.”

In 2013, five years after he co-authored a paper showing that Democratic candidates in the United States could get more votes by moving slightly to the right on economic policy, Andrew Gelman, a statistician at Columbia University in New York City, was chagrined to learn of an error in the data analysis. In trying to replicate the work, an undergraduate student named Yang Yang Hu had discovered that Gelman had got the sign wrong on one of the variables.

Gelman immediately published a three-sentence correction, declaring that everything in the paper’s crucial section should be considered wrong until proved otherwise.

Reflecting today on how it happened, Gelman traces his error back to the natural fallibility of the human brain: “The results seemed perfectly reasonable,” he says. “Lots of times with these kinds of coding errors you get results that are just ridiculous. So you know something’s got to be wrong and you go back and search until you find the problem. If nothing seems wrong, it’s easier to miss it.”

I spent the better part of the last three weeks working on an NSF-IOS Doctoral Dissertation Improvement Grant (DDIG) proposal. Pretty much daily, I consulted this list of publically available grant proposals in the biological sciences to look at other people’s proposals. It’s an awesome resource if you want to see how people write their project description, but there are no links to example data management plans, facilities, summaries, etc. Where does one go for examples of or advice on these supplementary documents?

At least part of the answer is “here.”

This is an user-friendly python package for interfacing with large collections of tweets. Developped at the SMaPP lab at New York University.

Last week, astronomers, astrophysicists, data scientists and programmers came together at New York University to try to solve some of astronomy’s toughest problems — in just five days.

The event, called Astro Hack Week, has only one rule: Everybody has to produce something. It might be a build of an astronomy data search algorithm, a series of programming tutorials or a bot that generates fake (and surprisingly plausible) tweets from one of the event creators. It might be planned from the get-go or something dreamed up based on a morning teaching session

… Could you talk about the CATT grant and your involvement?

CATT is a New York state sponsored program that works to improve existing technology and infrastructure through university and industry partnerships in the fields of economic development, technology transfer, workforce training, entrepreneurial support, and research & development. Data science is one of the three ares we are focusing on, in addition to wireless networking and cyber security. I got involved in the mid-1980s, and I’ve been the center’s director since 1996.