Specialists of the NUST MISIS Department of Engineering Cybernetics and the Sberbank Robotics Laboratory were the first ones in Russia to successfully integrate a large open-source language AI model into a robot manipulator control system. Young engineers demonstrated the ability of the language model to devise a plan for dealing with various tasks related to manipulating objects without programmer involvement and code writing.
A human only needs to articulate, in a natural language, what exactly has to be obtained as a result of the robot’s actions with the objects in front of it. This approach opens up new opportunities in robotics, rendering it possible to handle a wide spectrum of tasks that may now be posed in the form of a text query in a natural language.
Service robots have already become a part of everyday life, but they are still unable to reason upon anything. For example, a robot dog may be asked, “Bring me orange juice”. Our language model is already knowledgeable about the world and considers different options as acceptable: go to the kitchen, go to the store, go to the juice machine, etc. It evaluates them under the principle of whether it will affect the dealing with the global task “to bring orange juice” positively or negatively. Therefore, the “go to the bedroom” option is discarded. Then, using sorting algorithms, the robot evaluates the possibility of taking the action selected. As a result, we obtain an action plan for achieving the global goal, consisting of a set of simple robot actions.
“Practice has shown that our product makes it possible to apply the world knowledge of a large language model (LLM), already trained on a huge data set, to a physical robot without pre-training it. Furthermore, we are not limited to selecting only language models for code generation. We conducted our first experiments with a local open-source model — LLaMA-7B. However, the more powerful a language model is, the more logically complex applied tasks it is able to deal with,” Yaroslav Savelyev, a study author, a third-year student of the NUST MISIS Department of Engineering Cybernetics and an expert of the Sber Robotics Laboratory, said.
The work over the project started nine months ago at the initiative of Albert Efimov, Ph.D. (Philos.), Head of the NUST MISIS Department of Engineering Cybernetics, Vice President of Sberbank. The academic adviser was Alexander Semochkin, Ph.D. (Phys.-Math.), Chief Engineer of the Sberbank Robotics Laboratory. The goal was tomake use of the language model’s reasoning ability to help robots adapt to different tasks and environments.
This new AI capability opens up extremely broad prospects for practical application in industrial and service robotics. It will enable to significantly cut costs in the areas where programmers are expected to regularly change or adjust tasks at the place of production, for example, in the automotive sector, electrical engineering, electronics; construction, chemical industry, medicine and pharmaceuticals, etc. In addition, the integration of large language models and robots will prove useful where humans lack the speed of response, where hazardous conditions exist or where humans do not have the necessary skills and knowledge to interact with a robot.
The new ability of robots will help engineers avoid elaborating complex algorithms to set sequences of actions and set tasks through commands in a user-friendly form. Many research centers across the globe are conducting research in this area. In Russia, the Sberbank Robotics Laboratory, AIRI and NUST MISIS are consistently looking into the matter. The work of MISIS students has been recently recognized with a grant from the Innovation and Science Promotion Foundation.
One may observe the robot manipulator learning process.
