NYU Data Science newsletter – May 6, 2016

GROUP CURATION: N/A

For the past few months, Google has been feeding text [from romance novels] to an AI engine — all of it taken from steamy romance novels with titles like Unconditional Love, Ignited, Fatal Desire, and Jacked Up. Google’s AI has read them all — every randy, bodice-ripping page — because the researchers overseeing its development have determined that parsing the text of romance novels could be a great way of enhancing the company’s technology with some of the personality and conversational skills it lacks.

As technology advances and our economic, social, and civic lives become increasingly digital, we are faced with ethical questions of great consequence. Big data and associated technologies create enormous new opportunities to revisit assumptions and instead make data-driven decisions. Properly harnessed, big data can be a tool for overcoming longstanding bias and rooting out discrimination.

The era of big data is also full of risk. The algorithmic systems that turn data into information are not infallible—they rely on the imperfect inputs, logic, probability, and people who design them. Predictors of success can become barriers to entry; careful marketing can be rooted in stereotype. Without deliberate care, these innovations can easily hardwire discrimination, reinforce bias, and mask opportunity.

Using building records as well as the data collected by water testers, scientists created a predictive algorithm that makes a good guess about the state of each of Flint’s service lines based on certain property indicators, like the age, size, and type of building.

With the information they do know, the scientists will be able to deploy this algorithm across the entire city using machine learning, creating a fairly accurate map which recommends how repairs should be prioritized based on the number of people they will benefit.

An international team of scientists has developed an algorithm that can put data with large time uncertainty into chronological order. After applying statistical techniques to data obtained with a 300 fs (300 × 10–15 s) timing uncertainty, the team was able to describe the laser-driven explosion of a nitrogen molecule with 1 fs resolution – an improvement in time resolution of two orders of magnitude. Because the algorithm is based on statistics, it could potentially be applied to other disciplines with timing uncertainty, such as climate science and astronomy.

It’s 2016, and as Clarifai “data scientist and NSFW enthusiast” Ryan Compton (yes, that’s his title) puts it: “the discovery of nude pictures has been a central problem in computer vision for over two decades. Potter Stewart’s “I know it when I see it” maxim doesn’t apply to technology. Devices don’t really see things—not in any conventional sense. Computers can recognize patterns, but that raises the question of what nudity looks like as a pattern. Humanity, as anyone with an Internet connection can tell you, has conjured all sorts of imaginative ways of being nude, so good luck with that one.

Picking the right words is a key component of many jobs, and most of the time, we rely on our own vocabulary. But a new feature from Seattle-based tech startup Textio uses the power of machine learning to find the most effective words, thanks to a vast database and the power of a supercomputer.

Textio’s new predictive engine, called Opportunities, was announced today. Designed for job posts and emails to candidates, Opportunities adds to the existing Textio analysis tool that helps companies find diverse employees. The tool already spots language that historically attracts more male candidates and suggests replacements that are more neutral.

In an earlier FDA Voice blog post, we discussed a pair of concepts – interoperability and connectivity – that are essential prerequisites for the creation of a successful national system for evidence generation (or “EvGen”). In this post, we take a look at how we would apply these constructs as we go about building such a system.

What I’m worried about and what I’m thinking about these days is if we’re really making progress in AI. I’m also interested in the same kind of question in neuroscience, which is that we feel like we’re making progress, but are we?

Let’s take AI first. There’s huge progress in AI, or at least huge interest in AI—a bigger interest than there’s ever been in my lifetime. I’ve been interested in AI since I was a little kid trying to program computers to play chess, and do natural language databases, and things like that, though not very well. [video, 39:01]

It turns out you can accomplish quite a lot with 4,709 lines of Go code! How about a full time-series database implementation, robust enough to be run in production for a year where it stored 2.1 trillion data points, and supporting 119M queries per second (53M inserts per second) in a four-node cluster? Statistical queries over the data complete in 100-250 ms while summarizing up to 4 billion points. It’s pretty space-efficient too, with a 2.9x compression ratio. At the heart of these impressive results, is a data structure supporting a novel abstraction for time-series data: a time partitioning, copy-on-write, version-annotated, k-ary tree.

Did more people see #thedress as blue and black or white and gold? How many Twitter users wanted pop star Katy Perry to take the #icebucketchallenge?

The power to explore online social media movements — from the pop cultural to the political — with the same algorithmic sophistication as top experts in the field is now available to journalists, researchers and members of the public from a free, user-friendly online software suite released today by scientists at Indiana University.

The Web-based tools, called the Observatory on Social Media, or “OSoMe” (pronounced “awesome”), provide anyone with an Internet connection the power to analyze online trends, memes and other online bursts of viral activity.

Edward is a Python library for probabilistic modeling, inference, and criticism. It enables black box inference for models with discrete and continuous latent variables, neural network parameterizations, and infinite dimensional parameter spaces. Edward serves as a fusion of three fields: Bayesian statistics and machine learning, deep learning, and probabilistic programming.