Reproduce the delicacy
of the human hand
“Manipulator”
Click here for the description of the video content.
Required responsiveness and
flexibility in changing environment
In the robotics industry, following the expansion of the industrial robots, service fields such as healthcare, communication, entertainment, and education are expected to drive the market growth in the future. The key factor for robots in these service fields is how the robot can safely perform interactions involving physical contact with people, objects and the environment while keeping force balance, which are different requirements for industrial robots.
In addition, “telepresence robots” which combine remote control and robotics technologies are already in practical use and show great effectiveness when interacting with people through voice communication as well as “see” and ”talk”. However, they are not adept at interacting with environment such as object manipulation. We believe that future requirements necessitate “force-controlled” robots that do not require instructions from robot experts or advanced environmental settings, but only require human interventions to make difficult or detailed decisions for autonomous control in an environment with various objects.
Three technologies supporting
a force-controlled robot
Sony is developing force-controlled robots that can detect more subtle forces and move softly and smoothly, allowing for greater affinity with people and the environment. Robots operating in human living spaces or unknown remote environments are controlled using information including errors and motion plans which require sufficient time to process. This makes it difficult to manipulate objects stably and quickly.
Sony is developing three technologies to tackle these issues.
The first technology is “hand” equipped with a group of sensors for stable grasping of unknown objects that contain uncertainties such as shape, weight and friction coefficient. In order to measure the position and shape of an object, an overhead camera mounted on the upper area of the robot such as a head is generally used. The arm and the hand are controlled based on the information from the overhead camera. However, it is difficult to measure position of the objects accurately. Likewise, it is difficult to accurately measure the position of the robot’s own arm and hand. All these position errors increase the failure rate of object reaching tasks. By implementing multiple sensors developed with our sensing technology onto the robotic hands, our robot is able to detect the accurate position of objects.
The second technology is grasp force control technology for unknown objects using such hands. (The details will be explained later on)
The third is fast motion planning technology that allows robots to move smoothly. Motion planning refers to technologies which generate a continuous collision-free trajectory toward the target position. As the number of joints (degrees of freedom) in a robot increases, it becomes a higher-dimensional search problem that requires a higher computational load. We aim to reduce the computational load and make motion planning faster by using the assets of our original machine learning frameworks and fast solving algorithm based on observed information.
By combining these three technologies, we aim to establish manipulation technology that can be safely and smartly used even under complicated conditions.
As a concept model, we developed a mobile manipulator that can stably grasp an unknown object based on simple instructions from humans. We will continue to challenge to realize task motions and remote operations in the scenes such as house work where smooth movements with sharing the same space with humans is required.
Realization of grasp force
control technology for
unknown objects
In order for robots to grasp objects stably, appropriate grasp force should be generated. However, the robot does not understand the object, how heavy it is, where the center of mass is, whether it is soft or hard, slippery or not. Robots are required to handle any objects flexibly.
Under conditions where the weight, hardness and friction coefficient are unknown, an object may break if the grasp force is too large. On the other hand, insufficient grasp force may result in slippage and drop of the object. In particular, precise and delicate grasp force control is necessary when a robot grasps soft objects without slipping and crushing it. This is still one of the major problems in the robotics field.
In order to address this problem, we got some inspirations from motion of human hands. Imagine when we try to grasp a fragile object carefully. We don’t grasp it with a large force just slightly less than what would be required to crushing the object, rather we grasp it with a small force just barely enough not to slip. Thus we hypothesized that adaptive and delicate grasping would be realized if we could estimate minimum necessary force preventing slip regardless of objects’ properties, by unraveling the human subconscious behavior.
In order for the robot to adaptively control minimum necessary grasp force, we focused on a physical phenomenon which is called “incipient slip”. Incipient slip is the physical phenomenon observed when only a partial area in the contact area slips before the robot drop an object by slip. Many researchers have suggested that the reason why humans can grasp objects with various shapes and textures without slipping them is that we can detect incipient slip which can be observed when making a contact with an object by using sensitive cutaneous sensing.
We hypothesized that the robot can keep minimum necessary grasp force without slipping and crushing an object if we could observe the incipient slip by using sensors.
World’s first mathematical
modeling of incipient slip
detection and control in arbitrary
direction for unknown objects
However, detecting and controlling the incipient slip is not trivial, as it has several challenges that have not been addressed in previous studies. There are two major challenges.
Firstly, stable detection of incipient slip. Unlike humans, robots do not have delicate cutaneous sensing. Therefore, we cannot directly implement the ideas that mimic humans’ mechanoreception. Rather, sensors and detection algorithms need to be realized in an engineering manner. Thus, we need to clarify the conditions that enable stable incipient slip detection and the sensor configurations.
Secondly, robust control of incipient slip. Controlling the incipient slip is recognized as one of difficult challenges as the incipient slip is a irreversible and discontinuous physical phenomenon in any direction. Previous studies have only focused on incipient slip detection in translational direction and failed to handle in rotational direction.
Solving the above two problems becomes even more difficult in the case of unknown objects whose weight, hardness and friction coefficient are unknown.
Therefore, we analyzed the mechanisms, detection algorithm and control algorithm of incipient slip using mathematical model, and derived the optimal sensor configuration and control algorithm based on the model. In particular, we focused on the theoretical explanation of incipient slip mechanisms in rotational direction that had not yet been solved. Moreover, we also theoretically and practically demonstrated that minimum necessary grasp force can be calculated by detecting incipient slip.
Figure 1 shows the incipient slip mathematical model.
Figure 1: Formulization of the stick ratio rotational direction
The round surface of the figure represents the surface of a robot finger, which is modeled as a generalized power function. Considering this contact model, we formulated the ratio (c/a) between the area where no slip occurs (c) and the whole contact area (a), which we refer to as “stick ratio”. A stick ratio reaching to 0 indicates that an object is about to slip and reaching to 1 indicates the grasp is stable. Fx represents tangential force (force which is parallel to the contact surface) and Fn represents normal force (force which is perpendicular to the contact surface). The smaller Fn or larger Fx would lead to slippage when the stick ratio is near zero.
We extended the equation regarding stick ratio in translational direction to the rotational direction. Unfortunately, both equations in the translational and the rotational direction cannot be solved because these refer the friction coefficient, which cannot be measured directly.
We decided to approach this problem from a different perspective. We assumed that the fingertip is covered with a flexible elastic material and we reformulated the equations with respect to shear displacements in the translational and the rotational direction. Ux represents the shear displacement in the translational direction and Uθ represents in the rotational direction. These reformulations of the equations reveal that the robot can continuously grasp an object without slipping it regardless of friction coefficient of the object.
Figure 2: Equations describing the relationships between the stick ratio and the translational/rotational shear displacement
This research is the first proposal of mathematical models to describe the mechanisms of rotational incipient slip and control algorithm in the world. We propose sensor configurations, sensor signal processing and adaptive grasp force algorithm to stably detect incipient slip.
Experiments are conducted to investigate our conceptual model of a manipulator that can grasp plastic bottle and fragile pastry such as an éclair without crushing and slipping it.
If the robot doesn’t control the grasp force based on the incipient slip detection, an éclair’s ( a pastry filled with cream) cream squeezes out or it slips if the grasp force is too large or too small. Also, if the robot only detects the translational incipient slip and cannot detect the rotational incipient slip as in conventional methods, the éclair can slip off during grasping it due to the rotational slip. In contrast, our method enables the robot to grasp and lift an éclair without crushing or slipping it. The figure at the bottom right shows the grasp force change controlled by our proposed method when the robot grasps an éclair. As this figure shows, grasp force is controlled immediately after grasping and lifting an éclair according to the amount of detected incipient slip. Adaptive and stable grasping is realized by controlling the grasp force according to the incipient slip detection by our original tactile sensor and adaptive control algorithm of grasp force.
Figure 3 Experimental setup and a part of results
From theory to practical
application
The experiments demonstrate that the algorithm based on the proposed mathematical model shows the consistency with the theory and the algorithm calculates an appropriate grasp force for objects with different friction coefficients or objects with different weights and moments.
However, in the development of robotics technology, there are many cases where experimental results using robots do not follow the theory, even if the results of simulations follow the theory. We believe it is important to be involved in the entire process from theory to real-life applications, in other words, firstly building a theory by deriving formula, then confirming the results on a simulator, and then implementing in hardware. For the model unveiled at Sony Technology Day (held in December 2021), our engineers are working tirelessly to provide ideas and improvements for the shape of the fingers as well as the specifications and placement of the pressure sensors.
Researchers
Toshimitsu Tsuboi
There are many unexplored technologies in robotics field. Many of them are realized only by integrating all of elements such as theory, sensors and control algorithm into hardware and software, as we introduce in this article. Sony is an attractive company as we have a great environment and broad-minded culture where we are encouraged to tackle such challenges from scratch. This is a workplace full of opportunities to create new world-changing technologies.
Kiyokazu Miyazawa
Robotics technologies are integrated technologies such as sensors, control, wiring, circuit boards, actuators, machine learning and other cutting-edge technology. Sony is unique in its environment where we can integrate all of these key technologies, from development to evaluation. In addition, the Sony Group’s business is connected to various business domains around the world. All you need is motivation and ideas. I think this is a truly wonderful environment.
