NYU Data Science newsletter – June 30, 2015

GROUP CURATION: N/A

Deep Learning is the hottest trend now in AI and Machine Learning. We review the popular software for Deep Learning, including Caffe, Cuda-convnet, Deeplearning4j, Pylearn2, Theano, and Torch.

@JennyBryan posted her slides from the 2015 R Summit and they are a must-read for instructors and even general stats/R-folk. She’s one of the foremost experts in R+GitHub and her personal and class workflows provide solid patterns worth emulation.

One thing she has mentioned a few times—and included in her R Summit talk—is the idea that you can lean on GitHub when official examples of a function are “kind of thin”. She uses a search for vapply as an example, showing how to search for uses of vapply in CRAN (there’s a read-only CRAN mirror on GitHub) and in GitHub R code in general.

I did manage to get myself out to San Diego for the 15th International Conference on Artificial Intelligence and Law. As mentioned in my short introductory post about the conference in early May that ICAIL 2015 was took place from June 8-12 at the University of San Diego. The view from the elevated USD campus was spectacular and made spending time in the Joan R. Kroc Institute for Peace and Justice and surrounding gardens all the more pleasurable. Congratulations to the organizers for providing a well-run and fruitful conference.

The era of big data is alive and well. It is leading to better and faster decisions in a diverse set of industries – from disease detection and insurance, to stock trading, crime prevention and election forecasting. Big data applications are powering self-driving cars and winning at Jeopardy. Today, businesses have access to an unprecedented pool of insights but face a growing challenge: data abundance. With data production expected to be 44 times greater in 2020 than it was in 2009, the question becomes how to make sense of it all.

Thankfully, advancements in analytics technology are keeping pace with the data explosion. Even so, data scientists are becoming more important than ever for businesses that want to stay ahead of the competition. It’s critical to have both the technology and a team that understands what data to collect, how to best collect it and how to unearth insights from all that data. Perhaps the greatest challenge is the ability to communicate data findings in a way that is compelling. You need a data analyst and a storyteller. Good luck with that.

One of our weapons in the fight against spam, malware, and other abuse on Facebook is a system called Sigma. Its job is to proactively identify malicious actions on Facebook, such as spam, phishing attacks, posting links to malware, etc. Bad content detected by Sigma is removed automatically so that it doesn’t show up in your News Feed.

We recently completed a two-year-long major redesign of Sigma, which involved replacing the in-house FXL language previously used to program Sigma with Haskell. The Haskell-powered Sigma now runs in production, serving more than one million requests per second.

The more connected we are, the more we are all just pixels on a screen. And we crave more pixels, pixels talking to us, responding to us, acknowledging us—as often as possible.

“It’s just a reality that social life is moving through screens,” says Eric Klinenberg, director of the Institute for Public Knowledge at NYU and the author of Going Solo: The Extraordinary Rise and Surprising Appeal of Living Alone. “There’s research showing that people who stay off of social media are more prone to isolation because they’re missing out on the place where the action is.”

… What I see is many business analysts that haven’t even got any understanding of big data technology or programming languages call themselves data scientists. Then there are programmers from the IT function who understand programming but lack the business skills, analytics skills or creativity needed to be a true data scientist.

Part of the problem here is simple supply and demand economics: There simply aren’t enough true data scientists out there to fill the need, and so less qualified (or not qualified at all!) candidates make it into the ranks.

… IBM is now training Watson to be a cancer specialist. The idea is to use Watson’s increasingly sophisticated artificial intelligence to find personalized treatments for every cancer patient by comparing disease and treatment histories, genetic data, scans and symptoms against the vast universe of medical knowledge.

Such precision targeting is possible to a limited extent, but it can take weeks of dedicated sleuthing by a team of researchers. Watson would be able to make this type of treatment recommendation in mere minutes.