NYU Data Science newsletter – September 8, 2015

GROUP CURATION: N/A

… What’s interesting about Flickr’s approach is that they are leveraging as many “in house” tools to get the near-real-time service to sit up and hit the latency and availability levels users expect. For Yahoo, which has been the center of Hadoop development since the beginning, this means tapping into MapReduce frameworks and continuing to build out from all of the components that are part of that ecosystem (even if not part of Hadoop proper). This means cobbling together a framework that leverages Storm for the real-time needs, HBase for ultra-fast queries, MapReduce to meld when large-scale computations are needed, and a Lamba architecture to tie it all together.

In recent years the research community has accumulated overwhelming evidence for the emergence of complex and heterogeneous connectivity patterns in a wide range of biological and sociotechnical systems. The complex properties of real-world networks have a profound impact on the behavior of equilibrium and nonequilibrium phenomena occurring in various systems, and the study of epidemic spreading is central to our understanding of the unfolding of dynamical processes in complex networks. The theoretical analysis of epidemic spreading in heterogeneous networks requires the development of novel analytical frameworks, and it has produced results of conceptual and practical relevance. A coherent and comprehensive review of the vast research activity concerning epidemic processes is presented, detailing the successful theoretical approaches as well as making their limits and assumptions clear. Physicists, mathematicians, epidemiologists, computer, and social scientists share a common interest in studying epidemic spreading and rely on similar models for the description of the diffusion of pathogens, knowledge, and innovation. For this reason, while focusing on the main results and the paradigmatic models in infectious disease modeling, the major results concerning generalized social contagion processes are also presented. Finally, the research activity at the forefront in the study of epidemic spreading in coevolving, coupled, and time-varying networks is reported.

Toyota, the world’s largest automaker by sales, says it will establish two collaborative research centers at MIT and Stanford, with an investment of $50 million over the next five years. The initial focus will be on accelerating the development of AI with applications to smarter and safer vehicles, as well as robots that can make our lives better at home, especially as we age.

… What the products on display have in common is that they don’t solve problems people actually have. Technology is integrated not because it is necessary, but because the technology exists to integrate it — and because it will enable companies to sell you stuff you never knew you were missing.

Last Thursday, our article “Estimating the Reproducibility of Psychological Science” was published in Science. Coordinated by the Center for Open Science, we conducted 100 replications of published results in psychology with 270 authors and additional volunteers. We observed a substantial decline effect between the original result and the replications. This community-driven project was conducted transparently, and all data, materials, analysis code, and reports are available openly on the Open Science Framework.

Ask us anything about our process and findings from the Reproducibility Project: Psychology, or the initiatives to improve transparency and reproducibility in science more generally.

The School of Medicine has launched the new Center for Statistical Genetics and Genomics. The center, led by Andrew Allen, PhD, professor of Biostatistics and Bioinformatics, will bring together quantitatively-oriented scientists from various disciplines on the Duke campus to address the computational and statistical challenges associated with efforts to use genomics to improve patient care.

“This center, under Dr. Allen’s leadership, will help establish Duke as a leading center of genomic-focused statistical and computational methods development,” said Nancy C. Andrews, MD, PhD, Dean, Duke University School of Medicine.

… New York City is enlisting some big-league help to figure out what that plan will be. This week, the city held the first meeting of its newly formed Technology Advisory Group, a panel convened to consider the question of what to do about Uber. The group’s two dozen members include academics from New York University and Columbia, venture capitalists like Fred Wilson, and representatives from tech companies, Uber itself among them.

The goal is to reorient the city’s regulatory framework around the transportation industry as it exists today, rather than the way things used to be.

… Yet there is another force in the striking democratization of computing beyond hardware, one that is more subtle but still crucial. That is the steady stream of improvements in the design of computer products, mainly software, which have opened the door to new users by making computers easier to use. The term most used now is “user-interface design.” But that suggests a narrower, product focus than the field that stretches back several decades, called human-computer interaction, which embraces psychology, anthropology and other disciplines.

“I think human-computer interaction designs have had as much impact as Moore’s Law in bringing the web and mobile devices to the world,” said Ben Shneiderman, a professor at the University of Maryland, College Park.

Anomalies in data are everywhere. They appear in the performance of just about anything you can measure. Sometimes, if you’re lucky, they’re rare. Other times they aren’t. The problem in both cases, however, is figuring out what is truly an anomaly and what just appears to be one. This is why we need statistical methods to sort the WTF from the chaff, so to speak.

Now, using stats is fine but when you don’t know exactly how to characterize anomalies, you wind up wasting a lot of effort identifying the correct and then optimal methodologies to identify the stuff you need to find which is why handing over the problem to a computer is a much better solution.

But what computer technology to use? Numenta, founded by Jeff Hawkins, the founder of Palm Computing turned artificial intelligence researcher, argues that the best solution lies in artificially intelligent system that perform complex pattern detection, use automated modeling, and implement adaptive learning, all of which is the basis for the company’s new offering: Grok for IT.

In this post, we explore LDA an unsupervised topic modeling method in the context of twitter timelines. Given a twitter account, is it possible to find out what subjects its followers are tweeting about?

This shift towards personal use leads to challenging new research questions. This special issue of Human-Computer Interaction focuses on emerging research about how people might appropriate and use personal data for personal purposes such as:

Self-monitoring and self-understanding

Identity work, self-representation, reminiscing, and legacy

Behavior change that might promote physical and mental well-being

Developing and maintaining interpersonal and community relationships

Monitoring and managing their relationships with organizations and applications

Deadline for Proposals: Thursday, October 15