Application of mixed reality in robot manipulator programming

摘要

Abstract Purpose Mixed reality is expanding in the industrial market and several companies in various fields are adapting this set of technologies for various purposes, such as optimizing processes, improving the programming tasks and promoting the interactivity of their products with the users, or even improving teaching or training. Robotics is another area that can benefit from these recent technologies. In fact, most of the current and futuristic robotic applications, namely, the areas related to advanced manufacturing tasks (e.g. additive-manufacturing, collaborative robotics, etc.), require new technics to actually perceive the result of several actions, including programming tasks, anticipate trajectories, visualize the motion and related information, interface with programmers and users and several other human–machine interfaces. Consequently, this paper aims to explain a new concept of human–machine interfaces aiming to improve the interaction between advanced users and industrial robotic work cells. Design/methodology/approach The presented concept uses two different applications (apps) developed to explore the advanced features of the Microsoft HoloLens device. The objectives of the project reported in this paper are to optimize robot paths, just by allowing the advanced user to adjust the selected path through the mixed reality environment, and create new paths, just by allowing the advanced user to insert points in the mixed reality environment, correct them as needed, connect them using a certain type of motion, parametrize them (in terms of velocity, motion precision, etc.) and command them to the robot controller. Findings The solutions demonstrated in this paper show how mixed reality can be used to allow users, with limited programming experience, to fully use the robotics fields. They also show clearly that the integration of the mixed reality technology in the current robot systems will be a turning point in reducing the complexity for end-users. Research limitations/implications There are two challenges in the developed applications. The first relates to the robot tool identification, which is very sensitive to lighting conditions or to very complex robot tools. This can result in positioning errors when the software shows the path in the mixed reality scene. The paper presents solutions to overcome this problem. Another unattended challenge is associated with handling the robot singularities when adjusting or creating new paths. Ongoing work is concentrated in creating mechanisms that prevent the end-user to create paths that contain unreachable points or paths that are not feasible because of bad motion parameters. Practical implications This paper demonstrates the utilization of mixed reality device to improve the tasks of programming and commanding manufacturing work cells based on industrial robots [see video in (Pires et al., 2018)]. As the presented devices and robot cells are the basis for Industry 4.0 objectives, this demonstration has a vast field of application in the near future, positively influencing the way complex applications, that require much close cooperation between humans and machines, are thought, planned and built. The paper presents two different applications fully ready to use in industrial environments. These applications are scientific experiments designed to demonstrate the principles and technologies of mixed reality applied to industrial robotics, namely, for improving the programming task. Social implications Although the HoloLens device opens outstanding new areas for robot command and programming, it is still expensive and somehow heavy for everyday use. Consequently, this opens an opportunity window to combine these devices with other mobile devices, such as tablets and phones, building applications that take advantage of their combined features. Originality/value The paper presents two different applications fully ready to use in industrial environments. These applicatio