Master thesis: Controlling industrial robots with gener-ative AI

Fraunhofer IEM Wie sieht das Engineering der Zukunft aus? Zu dieser Frage entwickelt das Fraunhofer IEM in Paderborn überzeugende Lösungen – von der Geschäftsidee über die Umsetzung bis zum Markterfolg. Im Fokus stehen intelligente Produkte, Produktionssysteme, Dienstleistungen und Softwareanwendungen. Unsere Wissenschaftlerinnen und Wissenschaftler arbeiten interdisziplinär an neuen Methoden, Werkzeugen sowie Prozessen und setzen innovative Technologien ein, um die Wettbewerbsfähigkeit unserer Kunden und Partner langfristig zu sichern.

Paderborn • Veröffentlicht vor 2 Jahren

Problem and Objective

Large Language Models (LLMs) are experiencing a significant surge in popularity. Following the release of ChatGPT, their adoption has dramatically increased, with a broad range of users reporting widespread usage and practical benefits. By leveraging Generative Artificial Intelligence (GenAI) in work environments, productivity can be enhanced, and results improved. While most tasks utilize LLMs’ extensive general knowledge, these models can be fine‑tuned to understand and solve complex problems. Examples include safety monitoring during production or generative design for improved material usage. Therefore, LLMs and GenAI have the potential to revolutionize entire industries. LLMs can generate multiple types of content, such as images, text, or source code. A notable example of the utilization of LLMs for source code generation is GitHub Copilot, which significantly impacted the field of software development. Source code generation can also be used to control industrial robots, like ABB’s YuMi series. Currently, programming such robots requires expert knowledge of the robot’s mechanics and a translation of the desired movements into code. This creates a barrier to using robots due to high training and implementation costs. Controlling the robot via natural language would significantly lower this barrier. However, it remains unclear whether and how LLMs can effectively solve this problem.

To address this uncertainty, the main objective of this thesis is to develop a concept for robot control using LLMs to translate natural language inputs into suitable code for the robot. This includes a comprehensive analysis of existing LLM‑based approaches for source code generation, evaluating the capabilities of LLMs for robot control, developing a concept for robot control using LLMs, designing a reference architecture, and implementing a proof of concept. Furthermore, the developed approach will be generalized and abstracted into a generic methodology that can be reused for future projects.

Work Program

• Familiarization with the task, preparation of a detailed outline – 130 h
• Problem analysis to determine requirements for the approach – 150 h
• Analysis of the state of the art – 120 h
• Development and validation of a systematic approach – 120 h
• Identification of relevant communication interfaces – 120 h
• Selection of suitable training data – 120 h
• Tuning of the selected LLM – 120 h
• Implementation and validation of the approach – 200 h
• Documentation of the results – 130 h
• Final thesis presentation – 20 h

Supervision

• M.Sc. Benjamin Tiggemann – E‑Mail:
• M.Sc. Ruslan Bernijazov – E‑Mail:
• Heinz Nixdorf Institut, Universität Paderborn, Fürstenallee 11, 33102 Paderborn