Machine Learning Applications in Arts & Urbanism



What is the essence of artistic style? What makes up the identity of the city? Are visual cues the primary means by which a city or work of art is distinguished?

For our first meeting of the semester, we explored machine learning applications in fine arts and urbanism. Elliot Creager, Research Scientist in Lyric Labs at Analog Devices walked us through A Neural Algorithm of Artistic Style, a seminal work that applies deep-learning to mining and reproducing "artistic style." Lehzi Li, an MDes candidate at Harvard presented her ongoing thesis work using machine learning to uncover the identity of cities.

 photo Poster_2.23.001_zpsdaxk2wbi.jpg

What is machine learning? Machine Learning is an increasingly popular form of pattern classification, in which an algorithm searches for structures in big data---complex patterns rarely discernable by the human eye alone. It then uses these patterns to make meaningful classifications for new data. For example, Netflix uses machine learning to "learn" your preferences, and suggest new movies based on patterns in your previous selections.

How can machine learning apply to something like artistic style or cities? As evidenced by Elliot and Lehzi, machine learning algorithms are increasingly being applied to understand "essences" of human creations like cities or paintings. Researchers in this area question whether computers can distinguish between higher-order concepts that imply a network of information-relations. These higher-order concepts are easily recognized by humans, even if we can't quite put our fingers on precisely what makes Harvard Square feel different from Inman Square, or what exactly comprises of Kandinsky's style. Machine learning has a way of recognizing differences by finding hidden patterns of relationships between (typically but not necessarily) visual features, such as texture, shape or color.

Interestingly enough, when fed into a machine learning algorithm, these same features used to distinguish paintings are also used to differentiate between urban landscapes. Do these visual features map onto deeper socioeconomic or demographic conditions? Mapping visual characteristics of neighborhoods or paintings, to experience with the historical or economic forces that shape them start sounding like the job-descriptions of urban planners or art historians. Indeed, some would argue that machine learning science replicates human intuition and therefore threatens expertise. How close to human intuition can a machine learning algorithm really get?

Ironically, the group discovered through both presentations that the "misclassification" by machine learning algorithms seemed to contain more information about the system than perfect classification. This appeared to be a result of the fact that perfect classification simply replicated existing knowledge: the algorithm tells us with great accuracy what we already know. What proved to be more interesting were the moments when the algorithm failed, indicating that something about this particular place, or painting did not quite fit the parameters we used to describe the system. The cases where Singapore was repeatedly mis-classified as Boston, seemed to indicate the presence of an interesting force shaping the condition of that location.

Ultimately, machine learning is an algorithm designed to mimic our perception that gives us insight into why we perceive difference the way we do. What kind of implications or applications does this have for our understanding of each other and the environments or objects we create? Is this a form of redundant knowledge? Can we gain deeper insight into the networks of forces that shape essences of artifacts? Can these insights be used as a design parameter to reproduce "Boston" or a Van Gogh? Or are paintings and cities the results of historically-networked and irreproducible forces? Can an algorithm ever truly create?


You can find an interview with our presenters from this week here:

Cover Image: Forged painting by Wolfgang Beltracchi. Labeled for reuse.