NYU Data Science newsletter – September 21, 2015

GROUP CURATION: N/A

Everything today is being quantified, measured, and tracked — everything is generating data, and data is powerful. Businesses are using data in a variety of ways to improve customer satisfaction. For instance, data scientists are building machine learning models to generate intelligent recommendations to users so that they spend more time on a site. Analysts can use churn analysis to predict which customers are the best targets for the next promotional campaign. The possibilities are endless.

However, there are challenges in the machine learning pipeline. Typically, you build a machine learning model on top of your data. You collect more data. You build another model. But how do you know when to stop?

The data infrastructure space has seen tremendous growth and innovation over the last few years. With the rapid adoption of open source technologies, organizations are now able to process and analyze data at volumes that were unfathomable a decade earlier. However, given the rapid growth of the space, it can be difficult to keep up with all the new systems that have been created, and more difficult to understand what problems a system is great at solving. Two popular open source technologies, Druid and Spark, are often mentioned as viable solutions for large-scale analytics. Spark is a platform for advanced analytics, and Druid excels at low-latency, interactive queries. Although the high level messaging presented by both projects may lead you to believe they are competitors in the same space, the technologies are in fact extremely complementary solutions. By combining the rich query model of Spark with the powerful indexing technology of Druid, we can build a more powerful, flexible, and extremely low latency analytics solution.

What does the future of data science look like? If you’re Forrester analyst Mike Gualtieri, the future of data science is all about predictive models—lots of them running in semi-automated fashion at truly massive scale. But will that eliminate the need for data scientists?

Gualtieri laid out his vision on the future of data science and the role that data scientists will play in it during a recent webinar sponsored by Skytree, a developer of software for automating predictive analytics using machine learning models. That in itself should give you a hint as to where the Forrester analyst was going with his predictions.

Our first Galvanize Data Science hiring day is coming up at our Denver – Platte campus, and we’re excited to see students unveil what they’ve been working on for the past few weeks. If you’re not familiar with Hiring Day at Galvanize, think of it as a demo day for students where they show off technologies they’re learned, projects they’ve built, and get a chance to meet with potential hiring partners.

Sooner or later you’re going to find yourself looking for a data science job. Maybe it’s your first one or maybe you’re changing jobs. Even if you’re fully confident in your skills, have no impostor syndrome, and have tons of inside leads at great companies, it’s a tremendously stressful experience. The process of looking for a new job is often one that occurs secretly and confidentially and then is so exhausting that discussing the process is the last thing you want to do. I hope to change that.

I recently went through this myself and thought I’d record my thoughts on the process while they’re still fresh. I interviewed a lot. Some went well, some didn’t go well at all. The reason for this was sometimes me, sometimes them, often both. Sometimes I didn’t get selected for an on-site interview. Other times I withdrew from the process after seeing that it wouldn’t be a good fit for me. I took notes throughout, though, and here they are.

Deep learning algorithms have pushed image recognition and classification to new heights over the last few years, and those same approaches are now being moved into more complex image classification areas, including satellite imagery. Without making light of the complexity of say, picking out a single face in a crowd of many thousands, satellite data has its own diverse array of challenges—enough of them to outpace what is currently used for general image recognition at the likes of Facebook, Google, and elsewhere.

Spurred by the need for neural networks capable of tackling vast wells of high-res satellite data, a team from the NASA Advanced Supercomputing Division at NASA Ames have sought a new blend of deep learning techniques that can build on existing neural nets to create something robust enough for satellite datasets. The problem, however, is that to do so, they first had to create their own fully labeled training datasets—a terabyte-class problem, which was only one small step for deep image classification.

Hoping to avoid the potholes that recently wrecked a similarly ambitious study of children, a panel of human geneticists, medical researchers, and other experts today proposed a blueprint for the National Institutes of Health’s (NIH’s) plan to recruit 1 million Americans for a long-term study of genes and health. The study, which hopes to recruit its first volunteers next year and could last a decade or longer, may become the largest national study of this kind in the world.

For NIH Director Francis Collins, the project, known as the Precision Medicine Initiative (PMI) Cohort Program, brings to fruition an idea he first proposed 11 years ago. “I am so excited to see this dream come to life,” Collins said in a statement released after he accepted the blueprint. “[It] will be a broad, powerful resource for researchers working on a variety of important health questions.”

As Booz Allen gears up for the second Data Science Bowl (DSB), Dr. Bob Cowen, Director of Hatfield Marine Science Center, said he’d participate again “in a heartbeat.” Cowen said, “The algorithms developed in the Data Science Bowl helped move our work ahead leaps and bounds. Without the winning algorithm from Team Deep Sea, it might have taken marine researchers more than two lifetimes to manually complete the classification process.” He added, “Just as important as the algorithm itself are the inroads the DSB made in rallying citizen scientists to work for the greater good.”

Interdisciplinary work is considered crucial by scientists, policymakers and funders — but how widespread is it really, and what impact does it have? Scholars say that the concept is complex to define and measure, but efforts to map papers by the disciplines of the journals they appear in and by their citation patterns are — tentatively — revealing the growth and influence of interdisciplinary research.

With the help of A.I., America’s most famous doll tries to fulfill a timeless dream —
convincing little girls that she’s a real friend. What will happen if they believe her?

In the constantly shifting and often convoluted world of media, two major industry movements are currently under way that will shape how content is produced and business gets done: the rise of programmatic ad buying—you might have heard something about that over the last 18 months or so—and a growing openness on the part of sellers to work more closely and collaboratively with creative.

The number of big data projects enterprises are investing in or plan to invest in have increased to 76 percent in 2015 from 73 percent in 2014, according to a Gartner survey of 437 IT and business leaders. While the increase is small, what’s perhaps more telling is that such initiatives are increasingly originating from financial, marketing and other business unit leaders, who are pressuring CIOs to collaborate with them to make sure the technology aligns with the company’s strategy.

“People are becoming aware of the value of data, not just in IT but overall,” Gartner analyst and report co-author Nick Heudecker, who conducted the research in June, told CIO.com. “They’re creating data and using it as a competitive advantage.”

The “digital assistant” is proliferating, able to combine intelligent natural language processing, voice-operated control over a smartphone’s functions and access to web services. It can set calendar appointments, launch apps, and run requests. But if that sounds very clever – a computerised talking assistant, like HAL9000 from the film 2001: A Space Odyssey – it’s mostly just running search engine queries and processing the results.