Artificial intelligence (AI) has revolutionized many aspects of our lives, and its potential in creative endeavors continues to be explored by researchers around the world. Recently, two researchers from the Alabama-based technology company PeopleTec developed a computer model that can create complex recipes with ingredients users have on hand, after analyzing images of what’s inside their fridge. This model combines a large language model and an image detector, and can generate recipes complete with instructions and serving sizes.
The project
Image: Noever and Noever, (2023).
The aim of the project was to show recent advances in the field of AI in a practical and useful way. To achieve the generation of recipes from images, the researchers used model application programming interfaces (APIs) for image analysis, as well as the text generator that underlies ChatGPT.
The model is based on the combination of a large language model and an image detector, which identifies objects in images. The main idea was to combine raw ingredients and recipe ingredients using image analysis and then ask the language model to build a plausible cooking recipe, including the expected title, proportions, and steps. Additionally, the model can accommodate different constraints such as food cost, portion size, and dietary restrictions.
The evaluation of the model
The model was evaluated in a series of tests, using more than 2,000 images of open refrigerators with different ingredients inside them. Using these images, the model generated a 100-page cookbook of interesting and unique recipes featuring the top 30 ingredients found in the input images.
The model has the ability to accommodate different constraints such as food cost, portion size, and dietary restrictions. Additionally, the model can generate recipes for different combinations of ingredients, making it useful for both amateur and expert cooks.
The future of the project
The next step for the researchers is to develop a mobile app that can take a picture of a real fridge and generate a list of available ingredients, without the restriction of using all the ingredients. This would allow users to mix and match the available ingredients to create a recipe that suits their needs.
The PeopleTec researchers’ project could inspire other teams around the world to use language models and other AI models in recipe generation tasks or other creative problems.
This project demonstrates the potential of AI to help with creative and practical tasks. The model’s ability to accommodate different constraints and generate recipes for different combinations of ingredients is impressive. These kinds of apps could also help people who have a hard time planning their meals, especially those with dietary restrictions.
