The art of style transfer

By Elliot Robinson and Mariah Rosales

Summer 2022 Center for Advanced Research Computing Graduate Assistant Jacob McCullough created a program capable of style transfer, where a trained algorithm identifies the style of an image and transfers it onto another.

One factor that distinguishes one piece of art from another is the art style, a unique set of design characteristics that make art identifiable. Each piece of art has its own unique set of elements that distinguish the style such as the medium it is created with, or the particular techniques used by the artist. For example, Van Gogh’s short-rounded brushstrokes, along with various other elements, make up his distinct art style.

The process of style transfer requires a program that is capable of digital manipulation to images or videos. McCullough made this possible using CARC resources by copying and then modifying the code of a program used to classify images. Due to the sheer number of different factors, the transferring process from a particular image, such as Starry Night, onto another image is difficult and time consuming to do by hand. The program that McCullough modified worked by identifying the art style of an image, for example Starry Night (figure 1), before applying it to a content image, such as the Mona Lisa (figure 2). This results in a processed image that is not exactly a combination of the two, but quite literally the transfer of the art style in Starry Night (figure 3).

McCullough first developed this program for use during an Albuquerque Public Schools summer program to demonstrate some of the interesting image results that coding can achieve. This program educated rising seventh graders on the basics of Python, a programming language that can be used to build a website, automate tasks, and conduct data analysis. Students in the program submitted images to McCullough, who ran the images through the program on several of CARC’s systems as either content or style images. The majority of the submitted images were content images and primarily came from popular media such as SpongeBob, Lord of the Rings, and Pokémon. In contrast, far fewer style images were submitted, but the most popular among the students were Van Gogh’s paintings, and one particular Lord of the Rings poster that featured Balrog.

McCullough states, “I came up with this project while teaching this course and I thought it would be a fun idea for the kids to help me. They ended up sending me around 200 photos that they wanted to style transfer, which I was not expecting, but they were all super excited about this whole project.”