money MIT algorithm finds subtle connections between art pieces
Researchers developed the system to compare works at the MET and Amsterdam’s Rijksmuseum.
A new system developed by MIT researchers called “MosAIc” is finding hard-to-spot similarities between art pieces at the Metropolitan Museum of Art and Amsterdam’s Rijksmuseum. MosAIc scans an image, then uses deep networks to find similarities in pieces that span cultures, artists and media that may not have otherwise been noticed, according to a blog post from MIT’s Computer Science & Artificial Intelligence Lab.
To use MosAIc, the user inputs an image and MosAIc’s algorithm finds similar art pieces. In one example, MosAIc linked Francisco de Zurbarán’s, The Martyrdom of Saint Serapion and Jan Asselijn’s The Threatened Swan. “These two artists did not have a correspondence or meet each other during their lives, yet their paintings hinted at a rich, latent structure that underlies both of their works,” said MIT CSAIL PhD student Mark Hamilton, lead author on a paper about MosAIc.
A particularly difficult aspect of developing MosAIc was creating an algorithm that could find not only similarities in color or style, but also in meaning and theme, Hamilton said. Researchers examined a deep network of “activations,” or features, for each image in the open access collections of both museums. The distance between the “activations” of the deep network was how researchers judged similarity.
Researchers also used a new image search data structure called a “KNN Tree,” which groups images together in a tree-like structure. To find one image’s closest match, the algorithm starts at the “trunk” of the grouping, then follows the most promising “branch” until it’s found the closest image. The data structure improves on itself by allowing the tree to “prune” itself based on characteristics of the image.
Hamilton said he hopes the work started on MosAIc can be expanded upon to other fields, like humanities, social sciences and medicine. “These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts,” he said.
