Application of Large Language Models to control robots in a simulation environment

Publication year: 2026   |   Status: open

Today, the role of large language models (LLMs) is undergoing a fundamental paradigm shift: from static text generators, they are becoming proactive, autonomous agents. These models are already capable of making decisions based on complex logic, interacting with software environments through purposeful tool-use, and interpreting the visual world through multimodal inputs. This development immediately raises the question of how this cognitive capacity can be applied to human movement in physical space for controlling robotic devices. Modern robot simulation frameworks provide an ideal basis for validating the technology, in which the decision-making mechanisms and control stability of LLM-based agents can be analyzed without physical constraints or costs, using individually configurable scenarios.

Potyók Csaba
Csaba Potyók

PhD student
potyok