NYU Data Science newsletter – June 29, 2016

GROUP CURATION: N/A

Apache Spark, one of the Apache Foundation’s fastest-growing open source projects, delivers new levels of speed to computing clusters, combining in-memory computing and efficient parallelization. With Spark, Hadoop clusters and data lakes can achieve speeds far greater than available with Hadoop’s MapReduce framework.

We’re happy to announce this month that Microsoft has integrated support for Apache Spark into Microsoft R Server for Hadoop, bringing Spark’s speed advantages within the reach of R users.

But how much faster is it?

The White House Office of Science and Technology Policy is announcing a Request for Information soliciting public input on the subject of artificial intelligence.

Also in government:

Request for Information on Artificial Intelligence (June 27, Federal Register)

Researchers Sue the Government Over Computer Hacking Law (June 29, WIRED, Security)

Biden threatens funding cuts for researchers who fail to report clinical trial results (June 29, STAT)

USGS finds data fraud, closes chemistry lab (June 24, Chemical & Engineering News)

The Office of Science and Technology Policy is interested in developing a view of AI across all sectors for the purpose of recommending directions for research and determining challenges and opportunities in this field. The views of the American people, including stakeholders such as consumers, academic and industry researchers, private companies, and charitable foundations, are important to inform an understanding of current and future needs for AI in diverse fields. The purpose of this RFI is to solicit feedback on overarching questions in AI, including AI research and the tools, technologies, and training that are needed to answer these questions.

Also in government:

How to Prepare for the Future of Artificial Intelligence (June 27, whitehouse.gov, Ed Felten)

Researchers Sue the Government Over Computer Hacking Law (June 29, WIRED, Security)

Biden threatens funding cuts for researchers who fail to report clinical trial results (June 29, STAT)

USGS finds data fraud, closes chemistry lab (June 24, Chemical & Engineering News)

A team of scientists has developed a computational method to map cancer progression, an advance that offers new insights into the factors that spur this affliction as well as new ways of selecting effective therapies.

“Our work focuses on ‘causality-like’ relationships among several genes and their mutations that drive the cancer progression as the tumor environment reacts to changes, such as lack of oxygen, cell mobility, or immune response,” explains New York University Professor Bud Mishra, one of the study’s co-authors. “It then uses the model to predict how a tumor’s genomes will change over time.”

We are already living with deep learning and large-scale neural networks, as evidenced by the growing number of applications that rely on computer vision, language understanding, and robotics. What we now want most from machine learning, said Google Senior Fellow Jeff Dean to the audience at SIGMOD 2016 keynote today (Tuesday, June 28), is “understanding.”

“We now have sufficient computation resources, large enough interesting data sets,” Dean told SIGMOD attendees. “We can store tons of interesting data but what we really want is understanding about that data.”

In the field of artificial intelligence, the phrase deep learning applies to software that improves its model of reality with experience. Consider, for example, a project developed at Google in 2012, in which a neural network running on 16,000 computer processors, browsing through 10 million YouTube videos, began on its own to identify and seek out one of the most popular YouTube genres: cat videos.

The then director of that project, Andrew Ng, went on to become the founding chief scientist at Baidu Research, an innovation center run by the giant Web services company Baidu. The parent company owns the largest search engine in China, along with Chinese-language browsers, online encyclopedias, social networks, and other Web-based services. According to the company, Baidu responds to more than 6 billion search requests from more than 138 countries every day. Because search engines and advertising placement platforms (such as Baidu’s Phoenix Nest) depend on artificial intelligence (AI) to satisfy vague or ambiguous requests, the company — along with Google, Microsoft, and other providers of internet guidance — has a natural interest in machine learning. Thus, Baidu Research, formed in 2014 in Sunnyvale, Calif., is a nexus of leading global AI research; it contains three facilities: Big Data Lab, the Institute of Deep Learning, and the Silicon Valley AI Lab.

It happens all the time in Olympic and extreme sports: An athlete thinks he nailed it—a snowboarding jump, a gymnastics tumbling pass—but the judges don’t agree. Based on their naked-eye view or a slow-motion video, they dock him points for things like not rotating part of his body a full 360 degrees. The not-quite-perfect camera angle leaves viewers doubtful and the athlete trudges off, shaking his head.

For those moments, some numbers from highly precise wearable or on-board motion sensors could come in handy. Prototypes of such devices are popping up in freestyle sports such as skateboarding and bicycle motocross, or BMX. Now, researchers are making headway in snowboarding too; one group presented its system earlier this month at IEEE’s Body Sensor Network Conference in San Francisco.

Advanced machine learning, also known as artificial intelligence or just A.I., holds far greater promise than unsettling headlines about computers beating humans at games like Jeopardy!, chess, checkers, and Go. Ultimately, humans and machines will work together—not against one another. Computers may win at games, but imagine what’s possible when human and machine work together to solve society’s greatest challenges like beating disease, ignorance, and poverty.

Doing so, however, requires a bold and ambitious approach that goes beyond anything that can be achieved through incremental improvements to current technology. Now is the time for greater coordination and collaboration on A.I.

I caught a glimpse of what this might yield earlier this year while standing onstage with Saqib Shaikh, an engineer at Microsoft, who has developed technology to help compensate for the sight he lost at a very young age.

Is AI ready for primetime? Not according to Admiral Michael S. Rogers, Commander U.S. CYBER COMMAND. In a recent interview with Charlie Rose, he stated that machine learning showed great promise for cybersecurity, but that the necessary technology was probably five years out.

If machine learning is currently so successful in other areas of society, why isn’t it ready for cybersecurity? Or is it?

Late last week my Twitter and Instagram were blowing up with photos of President Obama, U.S. Secretary of State John Kerry and entrepreneur/”Shark Tank” star Daymond John on the Stanford campus. Those three were among the 1,500 or so people who came to the university for the 2016 Global Entrepreneurship Summit, and there was clearly a lot of excitement.

The global summit offered participants a wide range of workshops, panels, exhibitions and networking sessions, with one of the highlights being an in-depth panel discussion on the future — and societal benefits — of artificial intelligence.

“In my own area of health and biomedicine, the opportunities [for AI] are ridiculously exciting,” panel co-chair Russ Altman, MD, PhD, said during his opening remarks Thursday evening. He noted that the amount of biomedical data that electronic health records and “the little devices we wear” generate have become “far too big for us to interpret without intelligent assistance.”

After twice confirming it used location to suggest new friends, Facebook now says it doesn’t currently use “location data, such as device location and location information you add to your profile, to suggest people you may know.” The company says it ran a brief test using location last year. New story here.

Opting in gives you more granular control over how ads work across devices signed into your Google account. If a search for boat shoes (you know, the grey ones with white laces) haunts you across the web, you’ll be able to kill it everywhere, all at once, rather than going device by device.

Google’s also introducing My Activity, a page that bundles search history, videos you’ve watched, and pages you’ve visited that serve Google ads. Opt in to the new setting, and you’ll be able to comb through your online life in far finer detail.

Alleged misconduct and data manipulation at a U.S. Geological Survey (USGS) laboratory may have affected thousands of environmental quality measurements processed between 2008 and 2014, according to the Interior Department’s Office of Inspector General (OIG).

As many as 24 research projects, representing some $108 million in funding for the laboratory, may have been impacted, OIG said.

AMTA-2016 solicits original research papers that will advance the field of Machine Translation. We seek submissions across the entire spectrum of MT-related research activity. … This year, we are particularly interested in submissions discussing research on translation system design, implementation, and scaling.

Austin, TX Saturday-Wednesday, October 29-November 2.

Deadline for submissions is Monday, July 11.

Stefan Karpinski is one of the co-creators of the Julia programming language and a co-founder of Julia Computing, Inc., which provides support, consulting and training for commercial usage of Julia. He has previously worked as a software engineer and data scientist at Akamai, Citrix, and Etsy. He is currently a Research Engineer at NYU as part of the Moore-Sloan Data Science Initiative.

Last week, the Center for Data Science welcomed Kx Systems—a software company based in New York—into our office, where they gave two consecutive day-long workshops on their q programming language, and their kdb+ database. … We spoke with Simon Garland, a Senior Engineer at Kx Systems.

D3 is now modular, composed of many small libraries that you can also use independently. Each library has its own repo and release cycle for faster development. The modular approach also improves the process for custom bundles and plugins.

There are a lot of improvements in 4.0: there were about as many commits in 4.0 as in all prior versions of D3. Some changes make D3 easier to learn and use, such as immutable selections. But there are lots of new features, too!