Carnegie Mellon University‘s Robotics Institute has welcomed a new artist-in-residence called FRIDA, which is a robotic arm with a paintbrush attached to it. Framework and Robotics Initiative for Developing Arts (FRIDA) is a way for robots and people to work together on works of art. The project is led by Peter Schaldenbrand, a Ph.D. student at the School of Computer Science, with RI faculty members Jean Oh and Jim McCann. The robot makes paintings using AI models similar to those used in OpenAI‘s ChatGPT and DALL-E 2. It does this by learning, simulating, and painting in the real world.

People can tell FRIDA what to do by typing in a description or uploading a picture. The team is experimenting with other inputs, including audio. In the end, the things that FRIDA makes are whimsical and impressionistic, with bold brushstrokes that are not very precise and include mistakes. The team is focused on exploring the intersection of human and robotic creativity.

FRIDA uses AI and machine learning several times during the artistic process. The first step involves the robot learning how to use its paintbrush. The robot then uses large vision-language models to understand the input, similar to those used in OpenAI’s DALL-E 2. The biggest technical problem with making a physical image is that the simulation and the real world are not the same. To address this, FRIDA uses a process known as “Real2Sim2Real,” where the robot’s actual brush strokes are used to train the simulator.

The team is also working to address limitations in current large vision-language models. They fed the models headlines from news articles to give them a sense of the world and further trained them on images and text from diverse cultures to avoid bias. The international group is led by master’s students from Carnegie Mellon and Dongguk University in Korea, and people from many different countries have contributed to it.

Lastly, once the user has told the robot what the painting will be about, it will use machine learning to create a simulation and a plan to help the user reach their goals. The robot displays a color palette on a computer screen so that a person can mix paint and give it to the robot. However, automatic paint mixing is being worked on right now. The robot works on a painting for a few hours, using an overhead camera to check its progress and make changes to its plan. With FRIDA, the team wants to help people be more creative. Its latest research will be shown at London’s 2023 IEEE International Conference on Robotics and Automation.

References

Aupperlee, A. (2023, February 7). Carnegie Mellon’s AI-Powered FRIDA Robot Collaborates With Humans To Create Art. Carnegie Mellon School of Computer Science; Carnegie Mellon University. https://www.cs.cmu.edu/news/2023/frida-robot

Lin, C. (2023, February 8). What happens when you mash generative AI with robotics? Carnegie Mellon scientists dream up an answer. Fast Company; Fast Company. https://www.fastcompany.com/90847320/generative-ai-art-robot-carnegie-mellon-frida-scientists