How A.I. Is Creating Building Blocks to Reshape Music and Art

An artwork created using DeepDream, which researchers at Google developed in 2015. The newer work at Google, with Project Magenta, involves music, and has led to creation of a tool called NSynth.

MOUNTAIN VIEW, Calif. — In the mid-1990s, Douglas Eck worked as a database programmer in Albuquerque while moonlighting as a musician. After a day spent writing computer code inside a lab run by the Department of Energy, he would take the stage at a local juke joint, playing what he calls “punk-influenced bluegrass” — “Johnny Rotten crossed with Johnny Cash.” But what he really wanted to do was combine his days and nights, and build machines that could make their own songs. “My only goal in life was to mix A.I. and music,” Mr. Eck said.

It was a naïve ambition. Enrolling as a graduate student at Indiana University, in Bloomington, not far from where he grew up, he pitched the idea to Douglas Hofstadter, the cognitive scientist who wrote the Pulitzer Prize-winning book on minds and machines, “Gödel, Escher, Bach: An Eternal Golden Braid.” Mr. Hofstadter turned him down, adamant that even the latest artificial intelligence techniques were much too primitive. But over the next two decades, working on the fringe of academia, Mr. Eck kept chasing the idea, and eventually, the A.I. caught up with his ambition.

Last spring, a few years after taking a research job at Google, Mr. Eck pitched the same idea he pitched Mr. Hofstadter all those years ago. The result is Project Magenta, a team of Google researchers who are teaching machines to create not only their own music but also to make so many other forms of art, including sketches, videos and jokes. With its empire of smartphones, apps and internet services, Google is in the business of communication, and Mr. Eck sees Magenta as a natural extension of this work.

“It’s about creating new ways for people to communicate,” he said during a recent interview inside the small two-story building here that serves as headquarters for Google A.I. research.

The project is part of a growing effort to generate art through a set of A.I. techniques that have only recently come of age. Called deep neural networks, these complex mathematical systems allow machines to learn specific behavior by analyzing vast amounts of data. By looking for common patterns in millions of bicycle photos, for instance, a neural network can learn to recognize a bike. This is how Facebook identifies faces in online photos, how Android phones recognize commands spoken into phones, and how Microsoft Skype translates one language into another. But these complex systems can also create art. By analyzing a set of songs, for instance, they can learn to build similar sounds.

In the 1990s, at that juke joint in New Mexico, Mr. Eck combined Johnny Rotten and Johnny Cash. Now, he is building software that does much the same thing. Using neural networks, he and his team are crossbreeding sounds from very different instruments — say, a bassoon and a clavichord — creating instruments capable of producing sounds no one has ever heard.


Another work using DeepDream, which relies on neural networks to generate imagescapes from existing photography.

Much as a neural network can learn to identify a cat by analyzing hundreds of cat photos, it can learn the musical characteristics of a bassoon by analyzing hundreds of notes. It creates a mathematical representation, or vector, that identifies a bassoon. So, Mr. Eck and his team have fed notes from hundreds of instruments into a neural network, building a vector for each one. Now, simply by moving a button across a screen, they can combine these vectors to create new instruments. One may be 47 percent bassoon and 53 percent clavichord. Another might switch the percentages.And so on.

For centuries, orchestral conductors have layered sounds from various instruments atop one other. But this is different. Rather than layering sounds, Mr. Eck and his team are combining them to form something that didn’t exist before, creating new ways that artists can work. “We’re making the next film camera,” Mr. Eck said. “We’re making the next electric guitar.”

Called NSynth, this particular project is only just getting off the ground. But across the worlds of both art and technology, many are already developing an appetite for building new art through neural networks and other A.I. techniques. “This work has exploded over the last few years,” said Adam Ferris, a photographer and artist in Los Angeles. “This is a totally new aesthetic.”

In 2015, a separate team of researchers inside Google created DeepDream, a tool that uses neural networks to generate haunting, hallucinogenic imagescapes from existing photography, and this has spawned new art inside Google and out. If the tool analyzes a photo of a dog and finds a bit of fur that looks vaguely like an eyeball, it will enhance that bit of fur and then repeat the process. The result is a dog covered in swirling eyeballs.

At the same time, a number of artists — like the well-known multimedia performance artist Trevor Paglen or the lesser-known Adam Ferris — are exploring neural networks in other ways. In January, Mr. Paglen gave a performance in an old maritime warehouse in San Francisco that explored the ethics of computer vision through neural networks that can track the way we look and move. While members of the avant-garde Kronos Quartet played onstage, for example, neural networks analyzed their expressions in real time, guessing at their emotions.

The tools are new, but the attitude is not. Allison Parrish, a New York University professor who builds software that generates poetry, points out that artists have been using computers to generate art since the 1950s. “Much like as Jackson Pollock figured out a new way to paint by just opening the paint can and splashing it on the canvas beneath him,” she said, “these new computational techniques create a broader palette for artists.”

A year ago, David Ha was a trader with Goldman Sachs in Tokyo. During his lunch breaks he started toying with neural networks and posting the results to a blog under a pseudonym. Among other things, he built a neural network that learned to write its own Kanji, the logographic Chinese characters that are not so much written as drawn.

Soon, Mr. Eck and other Googlers spotted the blog, and now Mr. Ha is a researcher with Google Magenta. Through a project called SketchRNN, he is building neural networks that can draw. By analyzing thousands of digital sketches made by ordinary people, these neural networks can learn to make images of things like pigs, trucks, boats or yoga poses. They don’t copy what people have drawn. They learn to draw on their own, to mathematically identify what a pig drawing looks like.

Then, you ask them to, say, draw a pig with a cat’s head, or to visually subtract a foot from a horse or sketch a truck that looks like a dog or build a boat from a few random squiggly lines. Next to NSynth or DeepDream, these may seem less like tools that artists will use to build new works. But if you play with them, you realize that they are themselves art, living works built by Mr. Ha. A.I. isn’t just creating new kinds of art; it’s creating new kinds of artists.

Oracle v. Google

At issue in Oracle v. Google is whether Oracle can claim a copyright on Java APIs and, if so, whether Google infringes these copyrights. When it implemented the Android OS, Google wrote its own version of Java. But in order to allow developers to write their own programs for Android, Google’s implementation used the same names, organization, and functionality as the Java APIs. For non-developers out there, APIs (Application Programming Interfaces) are, generally speaking, specifications that allow programs to communicate with each other. So when you read an article online, and click on the icon to share that article via Twitter, for example, you are using a Twitter API that the site’s developer got directly from Twitter.

In May 2012, Judge William Alsup of the Northern District of California ruled that APIs are not subject to copyright. The court clearly understood that ruling otherwise would have impermissibly—and dangerously—allowed Oracle to tie up “a utilitarian and functional set of symbols,” which provides the basis for so much of the innovation and collaboration we all rely on today. Simply, where “there is only one way to declare a given method functionality, [so that] everyone using that function must write that specific line of code in the same way,” that coding language cannot be subject to copyright.

Oracle appealed Judge Alsup’s ruling to the U.S. Court of Appeals for the Federal Circuit. On May 30, 2013, EFF filed an amicus brief on behalf of many computer scientists asking the Federal Circuit to uphold that ruling and hold that APIs should not be subject to copyright.  On May 9, 2014, the Federal Circuit issued a disastrous decision reversing Judge Alsup and finding that the Java APIs are copyrightable, but leaving open the possibility that Google might have a fair use defense.

On October 6, 2014, Google filed a petition asking the U.S. Supreme Court to review the Federal Circuit’s decision.  On November 7, 2014, EFF filed an amicus brief on behalf of many computer scientists that asked the Supreme Court to grant Google’s petition for review, reverse the Federal Circuit, and reinstate Judge Alsup’s opinion. Unfortunately, in June 2015 the Supreme Court denied Google’s petition.

The case returned to the district court for a trial on Google’s fair use defense.  Fortunately, in May 2016, a jury unanimously agreed that Google’s use of the Java APIs was fair use. Oracle has filed another appeal. In May 2017, EFF (along with Public Knowledge) filed an amicus brief asking the Federal Circuit to affirm the jury’s verdict.

Justices Strike Down Law Banning Disparaging Trademarks

The court’s decision probably means that the Washington Redskins football team will win its fight to retain federal trademark protection.

WASHINGTON — In a decision likely to bolster the Washington Redskins’ efforts to protect their trademarks, the Supreme Court on Monday ruled that the government may not refuse to register potentially offensive names. A law denying protection to disparaging trademarks, the court said, violated the First Amendment.

The decision was unanimous, but the justices were divided on the reasoning.

The decision, concerning an Asian-American dance-rock band called the Slants, was viewed by a lawyer for the Washington Redskins as a strong indication that the football team will win its fight to retain federal trademark protection.

Lisa S. Blatt, a lawyer for the team, said the decision “resolves the Redskins’ longstanding dispute with the government.”

“The Supreme Court vindicated the team’s position that the First Amendment blocks the government from denying or canceling a trademark registration based on the government’s opinion,” she said.

The law at issue in both cases denies federal trademark protection to messages that may disparage people, living or dead, along with “institutions, beliefs or national symbols.”

Four justices said the law could not withstand even the fairly relaxed judicial scrutiny that the Supreme Court applies to commercial speech. Those justices rejected the two government interests that the law was said to advance: protecting disadvantaged groups from demeaning messages and the orderly flow of commerce.

The First Amendment protects offensive speech, Justice Samuel A. Alito Jr. wrote for this group of four justices. “Speech that demeans on the basis of race, ethnicity, gender, religion, age, disability, or any other similar ground is hateful; but the proudest boast of our free speech jurisprudence is that we protect the freedom to express ‘the thought that we hate,’ ” he wrote, quoting a classic 1929 dissent from Justice Oliver Wendell Holmes Jr.

Justice Alito added that the law’s disparagement clause was far too broad. “It is not an anti-discrimination clause; it is a happy-talk clause,” he wrote.

Chief Justice John G. Roberts Jr. and Justices Clarence Thomas and Stephen G. Breyer joined that part of Justice Alito’s opinion.

Four other justices would have struck down the law using the more searching First Amendment scrutiny that applies to viewpoint discrimination.

“The danger of viewpoint discrimination,” Justice Anthony M. Kennedy wrote, “is that the government is attempting to remove certain ideas or perspectives from a broader debate. That danger is all the greater if the ideas or perspectives are ones a particular audience might think offensive, at least at first hearing.”

“To permit viewpoint discrimination in this context is to permit Government censorship,” Justice Kennedy wrote.

Justices Ruth Bader Ginsburg, Sonia Sotomayor and Elena Kagan joined Justice Kennedy’s opinion.

Justice Neil M. Gorsuch did not participate in the case, which was argued in January, before he joined the court.

The competing opinions from the two four-justice blocs will mute the extent to which the decision sets precedent in other contexts.

All eight participating justices did agree on some points. They were unanimous in rejecting the argument that federal trademarks are the government’s own speech and thus immune from First Amendment scrutiny of any kind.

In 2015, in a 5-to-4 decision in Walker v. Sons of Confederate Veterans, the Supreme Court ruled that Texas could refuse to allow specialty license plates bearing the Confederate battle flag because the plates were the government’s speech.

Justice Alito, writing for eight justices on Monday, said trademarks are different.

“If the federal registration of a trademark makes the mark government speech, the federal government is babbling prodigiously and incoherently,” he wrote. “It is saying many unseemly things. It is expressing contradictory views. It is unashamedly endorsing a vast array of commercial products and services. And it is providing Delphic advice to the consuming public.”

The Slants said they did not intend to disparage anyone. Instead, they said, they sought to adopt and reform a disparaging term about Asians, much as some gay people have embraced the term “queer.”

That was significant, Justice Kennedy wrote. The band wanted, he wrote, “to supplant a racial epithet, using new insights, musical talents, and wry humor to make it a badge of pride.”

The government has applied the law inconsistently when faced with trademarks based on ethnic slurs. It has, for instance, both registered and rejected trademarks for the terms “Heeb,” “Dago,” “Injun” and “Squaw.”

Members of a band called the Slants said their was goal is to adopt and reform a disparaging term about Asians.

In the Redskins case, the trademark office registered the team’s trademarks in 1967, 1974, 1978 and 1990. In 2014, though, it reversed course and canceled six registrations, saying they disparaged Native Americans.

The team lost before a trial judge in Virginia and appealed to the United States Court of Appeals for the Fourth Circuit, also in Virginia. The appeals court put the case aside while the Supreme Court considered the Slants case, Matal v. Tam, No. 15-1293.

In a second First Amendment case decided Monday, the Supreme Court unanimously struck down a North Carolina law that made it a crime for registered sex offenders to use Facebook and many other websites.

The law was challenged by Lester Packingham, a registered sex offender who was convicted of violating it after posting an account of having a traffic ticket dismissed. “God is good,” he wrote on Facebook.

Mr. Packingham, who had pleaded guilty in 2002 to taking indecent liberties with a minor when he was a 21-year-old student, said the law violated the First Amendment.

Justice Kennedy, writing for a five-justice majority, said the internet is transforming American life and has turned into “the modern public square.” Denying access to it, he wrote, violates the First Amendment.

“The statute here enacts a prohibition unprecedented in the scope of First Amendment speech it burdens,” Justice Kennedy wrote. “Social media allows users to gain access to information and communicate with one another about it on any subject that might come to mind.”

“By prohibiting sex offenders from using those websites, North Carolina with one broad stroke bars access to what for many are the principal sources for knowing current events, checking ads for employment, speaking and listening in the modern public square, and otherwise exploring the vast realms of human thought and knowledge,” he wrote.

Justices Ginsburg, Breyer, Sotomayor and Kagan joined Justice Kennedy’s opinion. Justice Gorsuch did not participate in the case.

In a concurrence, Justice Alito, joined by Chief Justice Roberts and Justice Thomas, agreed with the result in the case but did not join what he called the “loose rhetoric” in Justice Kennedy’s opinion.

The North Carolina law was too broad, Justice Alito wrote, but states retain many legal tools to protect children on the internet.

Read more at NYTIMES

Gene Simmons is making a move that might have some rock fans shooting him a hand gesture — in the form of a one-finger salute.

The application, filed June 9, is described as “a hand gesture with the index and small fingers extended upward and the thumb extended perpendicular.” (See image from the application below.)

The gesture was first used by Simmons “At least as early as 11/14/1974,” according to the application, and has been used for “Entertainment, namely, live performances by a musical artist, personal appearances by a musical artist.”

A declaration filed by the musician with the U.S. Patent and Trademark Office states, “Gene Simmons … declares: That he is the applicant herein; he believes the applicant to be the owner of the mark sought to be registered; that to the best of his knowledge and belief no other person, firm, corporation or association has the right to use said mark in commerce, either in the identical form or in such near resemblance thereto as may be likely, when applied to the goods or services of such other person, to cause confusion, or to cause mistake, or to deceive; that the facts set forth in this application are true; and that all statements made herein of his knowledge are true, and all statements made on information and belief are believed to be true.”

Which doesn’t exactly roll off the tongue — even a jumbo-sized tongue such as Simmons’.

Copyright Office Releases an Updated Draft of the Compendium of U.S. Copyright Office Practices, Third Edition

Acting Register of Copyrights Karyn Temple Claggett today released a revised draft of the Compendium
of U.S. Copyright Office Practices, Third Edition. This draft includes the first proposed updates to the
Compendium since its release in December 2014. The public draft is available on the Office’s website at It will go into effect on July 3, 2017.

The Compendium is the administrative manual of the Register of Copyrights concerning the mandate
and statutory duties of the Copyright Office under Title 17 of the United States Code. See 37 CFR. §
 The proposed updates are the result of a comprehensive review of the Office’s practices
and procedures. The draft revisions to the registration chapters clarify how and when the Office
communicates with applicants and how it handles duplicate claims, deposit requirements, and claims
involving multiple works, among other improvements. The update also provides preliminary guidance
for claims involving useful articles based on the Supreme Court’s recent decision in Star Athletica v.
Varsity Brands.

Monkey Business, A Copyright Issue

Monkey Selfie

“Wildlife photographer David Slater, who was trekking through the Indonesian wilderness in 2011, when a group of monkeys came upon him and his equipment. “At first there was a lot of grimacing with their teeth showing because it was probably the first time they had ever seen a reflection,” Slater said at the time. “They were quite mischievous jumping all over my equipment, and it looked like they were already posing for the camera when one hit the button.”

A crested black macaque snuck off with one of his cameras and took, according to the Telegraph, “hundreds of selfies,” most of which were blurry and unusable. Save for this: the pristine, in-focus beauty of a photograph of the female macaque’s grinning face pictured above, which gave Slater a brief brush with fame and plastered the monkey’s selfie on websites everywhere.”

Read More: FastCompany

IP in a World Without Scarcity

“More recently, new technologies promise to do for a variety of physical goods and even services what the Internet has already done for information… While none of these technologies are nearly as far along as the Internet, they share two essential characteristics with the Internet: they radically reduce the cost of production and distribution of things, and they separate the informational content of those things (the design) from their manufacture. Combine these four developments – the Internet, 3D printing, robotics, and synthetic biology – and it is entirely plausible to envision a not-too-distant world in which most things that people want can be downloaded and created on site for very little money. The role of IP in such a world is both controverted and critically important.”

Read full article..