Reducing Uncertainty Using Placement and Regrasp Planning on a Triangular Corner Fixture

This paper presented a regrasp planning method to eliminate grasp uncertainty while considering the geometric constraints of a fixture. The method automatically finds the Stable Placement Poses (SPPs) of an object on a Triangular Corner Fixture (TCF), elevates the object from its SPPs to dropping poses and finds the Deterministic Dropping Poses (DDPs), builds regrasp graphs by using the SPP-DDP pairs and their associated grasp configurations, and searches the graph to find regrasp motion sequences for precise assembly. Since the SPPs and their associated regrasps are constrained by the TCF’s geometry and have high precision, the final object poses regrasped via it has low uncertainty and can be directly used for assembly by position control. In the experimental section, we study the performance of analytical and learning-based methods for estimating the DDPs of different objects and quantitatively examine the proposed method’s ability to suppress uncertainty using assembly tasks like peg-in-hole insertion and sheathing tubes, aligning holes, mounting bearing housings, etc. The results demonstrate the method’s robustness and efficacy. Note to Practitioners—In production lines, robots interact with peripheral devices to improve efficiency and reduce uncertainty. In this work, we focus on a particular peripheral device – a Triangular Corner Fixture (TCF) made by three inclined and mutually perpendicular plates. We study using the TCF to improve manipulation precision. The inclined plates of the TCF form a gravity bucket that holds dropped objects in stable states under gravity. In a real scenario, a robot picks up an object and releases it above the TCF. The released object will reach a stable state on the TCF. Then, the robot regrasps and moves the stabilized object to the target pose with reduced uncertainty. Using the method proposed in this paper, a robot can automatically finish the above procedure by finding all the object’s stable states in the TCF, planning grasp configurations, invalidating infeasible states and grasps, building regrasp graphs and searching the graph to find a regrasp motion sequence that moves the object to a goal pose with high precision for assembly. In industrial applications, the proposed method has the potential to improve the flexibility of robotic systems for high-precision tasks. In the research fields, it may promote the research on sensorless manipulation and extrinsic manipulation, and push forward the studies in robotic regrasp.


Reducing Uncertainty Using Placement and Regrasp
Planning on a Triangular Corner Fixture Zhengtao Hu , Weiwei Wan , Senior Member, IEEE, Keisuke Koyama , and Kensuke Harada , Fellow, IEEE Abstract-This paper presented a regrasp planning method to eliminate grasp uncertainty while considering the geometric constraints of a fixture.The method automatically finds the Stable Placement Poses (SPPs) of an object on a Triangular Corner Fixture (TCF), elevates the object from its SPPs to dropping poses and finds the Deterministic Dropping Poses (DDPs), builds regrasp graphs by using the SPP-DDP pairs and their associated grasp configurations, and searches the graph to find regrasp motion sequences for precise assembly.Since the SPPs and their associated regrasps are constrained by the TCF's geometry and have high precision, the final object poses regrasped via it has low uncertainty and can be directly used for assembly by position control.In the experimental section, we study the performance of analytical and learning-based methods for estimating the DDPs of different objects and quantitatively examine the proposed method's ability to suppress uncertainty using assembly tasks like peg-in-hole insertion and sheathing tubes, aligning holes, mounting bearing housings, etc.The results demonstrate the method's robustness and efficacy.

Note to Practitioners-In production lines, robots interact with peripheral devices to improve efficiency and reduce uncertainty.
In this work, we focus on a particular peripheral devicea Triangular Corner Fixture (TCF) made by three inclined and mutually perpendicular plates.We study using the TCF to improve manipulation precision.The inclined plates of the TCF form a gravity bucket that holds dropped objects in stable states under gravity.In a real scenario, a robot picks up an object and releases it above the TCF.The released object will reach a stable state on the TCF.Then, the robot regrasps and moves the stabilized object to the target pose with reduced uncertainty.Using the method proposed in this paper, a robot can automatically finish the above procedure by finding all the object's stable states in the TCF, planning grasp configurations, invalidating infeasible states and grasps, building regrasp graphs and searching the graph to find a regrasp motion sequence that moves the object to a goal pose with high precision for assembly.

I. INTRODUCTION
U CERTAINTY is a crucial problem in employing robotic manipulators for assembly tasks.Especially for autonomous manipulators that receive vision feedback and generate manipulation motion online, uncertainty is challenging to eliminate -They originate from a series of mutually coupled components like vision, control, contacts, etc. Overcoming them and achieving precise manipulation is tricky.
Contemporary literature tends to solve the uncertainty problem using multi-modal sensing and improved sensing algorithms.Related articles reported significant improvements in robotic perception [1].However, despite the achievements, the improvements in sensing technology still fail to provide sufficient qualifications for autonomous manipulators, as sensing is not the only reason for uncertainty.On the other hand, researchers in the robotic planning and control community developed sophisticated integral motion planning and control policies to enable robots to correct object poses during manipulation.The policies include but are not limit to scanning search, spiral research, impedance control, hybrid force/position control, etc. [2], [3] [4], which need force sensors [5], tactile sensors [6], [7], or current sensors for feedback.Compliant mechanisms are hardware alternatives of the policies [8], [9].They are effective and less expensive counterparts of the sensor-based implementation.The policy-based methods or the compliant mechanisms have advantages in regulated scenarios but tend to be influenced by environmental changes.Users need to adjust various parameters or key hardware components like springs for different applications.
Unlike methods that improve robotic perception and control or develop new compliant mechanisms, eliminating uncertainty through manipulation while considering geometric and physical constraints is more straightforward, robust, and cost-effective.The fundamental idea is deploying a fixture in the robot workspace.The pose of an object can be precisely aligned and determined by taking advantage of the geometric constraints induced by contacting the fixture and the physical constraints induced by gravity.The idea is not new.It is widely seen in factory automation for aligning randomly placed workpieces [10], [11], and has been practiced since the beginning of robotics.However, the object poses and the robot motion for dropping the objects in the previous systems were manually specified.Their stability relied on a system integration engineer's subjective adjustments and examination.After development, the systems were difficult to be altered or redeployed.
This paper reinspects the idea of employing a fixture to reduce uncertainty.Unlike conventional design and mechanical analysis, our focus is on planning.We assume object poses are unknown and detected online.There will be uncertainties in both position and rotation.We propose using a triangular corner fixture made of three mutually orthogonal planes to reduce uncertainty.We develop algorithms to compute an object's stable poses on a fixture and employ these poses as intermediate states to build regrasp graphs and plan robotic manipulation sequences.Driven by gravity, an object dropped from above the tilted corner fixture may first contact the inner surfaces and then slide to the bottom under gravity and the guides of the mutually orthogonal planes.The algorithms could find the object's stable intermediate poses at the bottom of a gravitational bucket formed by the fixture and use them to reduce the uncertainty in planned robotic manipulation sequences.The algorithms forced a robot to leverage an intermediate regrasping process to reduce uncertainty.Dropping an object to the triangular corner fixture and letting it converge to known stable placement poses help cancel the uncertainties in both position and rotation.
With the help of the fixture and the proposed algorithms, an object could be manipulated precisely to conduct difficult tasks like insertion.In detail, the method developed in this work automatically finds the Stable Placement Poses (SPPs) of an object on a Triangular Corner Fixture (TCF).Consequently, it enables auto-planned precise robotic regrasp and manipulation.It first computes the SPPs considering geometric contact constraints, physical feasibility, and static stability.Then, it elevates the object from its SPPs to dropping poses and finds the Deterministic Dropping Poses (DDPs) from them.When the object is released from the DPPs, it will rest at expected SPPs.Finally, the method computes the gripper configurations for grasping and regrasping the object considering the TCF, SPPs, and DDPs.The method outputs a pick-and-place sequence that manipulates the object under the TCF's help with high precision.In the experiments, we studied the performance of different methods for estimating the DDPs of different objects and quantitatively examined the proposed method's ability to eliminate uncertainty by inserting a peg into holes with different clearance.We also examined the method's practical performance using real-world assembly tasks like peg-in-hole insertion, sheathing tubes, aligning holes, mounting housings, etc.The results verified that the method enabled a robot to finish assembly tasks without using sensors, compliant control, or complicated mechanism, making the robot system more robust and flexible.
Our novelties and contributions are as follows.First, we extended the functions of autonomous regrasp from the conventional object reorientation to uncertainty elimination.Second, we developed the learning-based estimators to screen the DDP-SPP pairs, which could overcome the limitations of the analytical methods.Third, we integrated planning SPPs, estimating DDP-SPP pairs, and regrasp planning to provide an automatic framework for a robot to use external fixtures.
The paper is organized as follows.Related work is presented in Section II.background information about the TCF and an overall workflow of the proposed method are introduced in Section III.Section IV-VI present the details of the SPP sub-planner, DDP estimator, and release/regrasp subplanner, respectively.Experiments and analyses are shown in Section VII.Section VIII provides discussions.Conclusions and future work are presented in Section IX.

II. RELATED WORK
The research related to this study includes sensorless manipulation, placement estimation, and regrasp planning.

A. Sensorless Manipulation
Like its name, sensorless manipulation means manipulating objects without using sensors.It relies on the mechanic and geometric constraints of a task to pose objects and is simpler and more robust compared to sensor-based manipulation [12].Sensorless manipulation is widely seen in automation lines to eliminate uncertainty.The exemplary mechanism used for sensorless manipulation includes chutes, hoppers, bowl feeders and feed tracks, etc [10].For robotic applications, Mason initially discussed the basic concept of sensorless robotic manipulation in [13].After that, a variety of sensorless robotic manipulation approaches were studied.For example, Brost et al. [14] proposed using combined pushing and squeezing and flat finger pads to grasp an object with uncertainty.Nie et a. [15] and Hirata et al. [16] designed special-shape finger pads to align uncertain objects.Ha et al. [17] developed an automatic designer that finds finger pad shapes for robustly grasping various objects.Goldberg et al. [18], and Zhou et al. [19] used a sequence of parallel grasp actions for orienting and positioning uncertain objects to a specific pose.Maeda et al. [20] and Varkonyi et al. [21] developed caging-based methods to achieve in-hand manipulation and parts feeding, respectively.Erdmann et al. [22] and Schmidt et al. [23] used the active actions of palms and boundary walls to manipulate objects.Berretty et al. [24] and Akella et al. [24] studied the usage of passive settings like fences.Grossman et al. [25], Erdmann et al. [12], and Mannam et al. [26] respectively used robotic manipulators to move a tray attached to its tool center point.As the robotic manipulator moves, an object in the tray will be slid into a trihedral corner and stopped by the tray's walls.The final pose of the object can be determined by carefully planning and controlling the tray's tilting motion.Attractive regions [27] also stand as an important method for sensorless manipulation.It allowed low-accuracy robotic systems to perform high-precision manipulation.The previously explored tasks by researchers working on attractive regions included assembly [28], object location [29], and grasping with alignment [30].A drawback of the attractive region method was the curve of dimensionality.The method was well defined in R 3 but was difficult to be extended for compensating rotational uncertainty.Su et al. [31] developed online estimation methods to overcome the difficulty.Moreover, Shome et al. [32] proposed to rearrange objects with combined suction and non-prehensile manipulation policies.It leveraged online visual observation to determine manipulation policies that were most robust to errors.Anders et al. [33] proposed leveraged the belief state space of objects (determined by environmental geometries) to plan rearranging actions.The method allowed robustly moving objects to desired goals without sensors.
Similar to the conventional sensorless manipulation systems, our proposed method uses geometric constraints to hold objects.The objects are supposed to be dropped by a robotic manipulator onto a TCF and trapped by the tilted TCF inner surfaces under gravity.We assume that visual recognition is used to locate an object's initial pose, and allow recognition and other uncertainty.We develop algorithms to plan stable placement poses, estimate dropping poses, and plan grasp/regrasp poses to reduce uncertainty while taking advantage of the TCF's geometric constraints.Our process is fully automatic and applies to various objects provided that their model information is given.The proposed method can be interpreted under the framework of attractive regions.Evaluating an object's DDPs considering a TCF is similar to an attractive region problem.The corresponding SPPs form the bottom of the attractive regions.

B. Placement Estimation
We consider a placement estimation as a two-step process.In the first step, we find a set of stable placement poses for an object.In the second step, we infer the dropping or releasing poses based on the placement poses.The related work in placement estimation is reviewed surrounding the two steps.
The most fundamental problem of placement estimation is finding a stable placement on a horizontal plane.In this case, the object's stability can be determined by checking if its Center of Mass (CoM) projection passes through the convex supporting polygon [34], [35] [36].As an extension to the fundamental problem, Wan et al. studied the placement planning on a tilted plane [37], a support pin [35], and arbitrary support structures [38].Harada et al. [39] developed an algorithm to plan the stable object placement with nonflat contact considering the convexity of the paired contact surfaces.They assumed that the friction force was large enough to prevent sliding.The placement stability of rigid bodies and assemblies considering frictional contact was discussed in [40] and [41].Contact Wrench Space (CWS) was widely used for stability estimation.The radius of a maximum inscribed sphere in the contact wrench cone indicates how much external wrench or inertial wrench a grasp can tolerate.It can be used to evaluate the grasp qualities and find optimum grasp configurations [42], [43] [44], [45], and can also be used to estimate the stability of structures [46], [47].
Dynamic dropping simulation is also widely used for placement estimation [38], [48].However, to assure the reliability of the simulated results, various parameters need to be tuned, and repeated examinations must be performed, which makes the methods less credible and time-consuming.For this reason, many researchers studied fast alternatives for dynamic simulation.For example, Kriegman and David [49] proposed an algorithm that computed a maximal capture region of the desired stable pose in the configuration space where the object pose would converge into a desired one.Jorgensen et al. [50] presented to generate drop regions for stable poses and discussed two methods, the largest enclosing ellipsoid computation and the kernel density estimation, to determine optimal drop poses from them.Varkony [51] provided a statistics-based prediction method for estimating the resting poses of the dropped parts.Fekula et al. [52] used a similar method to perform the estimation, and based on the reasoned stable poses and the rendered top view images of them, they further positioned the objects using a vision-based method.Baumgartl et al. [53] developed a fast placement planner, which is capable of computing a stable position and orientation for a dropped object in complicated environments.In addition, learning-based methods also became popular for placement estimation and handling the uncertainties in manipulation processes.Lu et al. [54] proposed to train a probabilistic graphical model as a classifier to predict the appropriate grasp types (power grasp or precision grasp).Li et al. [55] developed a deep network that uses a single depth point cloud to estimate the pose of an articulated object.Cheng et al. [56] proposed a two-step learning framework to find an object's stable placements and then used the learned placements for regrasp planning.Newbury et al. [57] used two Convolutional Neural Networks (CNNs) to estimate both the placement rotations and stabilities and obtain the humanpreferred object placements and orientations.CNNs are also widely used to estimate the grasp configurations [58] and predict the grasp qualities [59] as well.Feng et al. [60] used a Support Vector Machine (SVM) and a Long Short-Term Memory (LSTM) model to analyze the features of tactile sensors to detect slip and unstable grasps.Additionally, learning methods improved the accuracy of multiple-view stereo based depth analysis [61], [62], which has the potential to be used for predicting the placement stability and the closeloop object placement with a visual servo.
In our proposed method, a TCF is used to hold the dropped object and constrain its final configuration.We first find the SPPs on the tray corner considering the geometric constraints at the contact.Then, we use analytical and learning methods to obtain the DDPs of the objects that lead to the found SPPs.
We compare the performances of the different estimation methods to understand the advantages and disadvantages.

C. Regrasp
Regrasp is a manipulation method for robots to reorient the grasped object.It refers to an intermediary fixture to help the robot overcome both robot kinematic constraints and environmental geometric interference.A robotic system, HANDEY [63], was designed with the regrasp ability based on the algorithm presented in [64].In some early work [65], [66] [67], the researchers decoupled the planning problem from the robot motion and concentrated on the regrasp sequences and online computation efficacy.More recent research paid attention to finding the optimal path, which combines the robot motion with regrasp [68], [69].Wan et al. [70] presented to use a relational database to manage the grasp data, which enabled reusing more than 10,000 grasps and their relationship to search the regrasp path.In [71], the dynamic regrasp graph was presented, and the planning method considering regrasping was extended to assembly tasks.Additionally, researchers also explored employing endto-end learning methods for regrasping.Balaguer et al. [72] proposed to achieve the dual-arm regrasp using a single stereo image as input.Gualtieri et al. [73] modeled the pick-and-place as a reinforcement learning problem and used it for regrasping objects with unknown geometries.In the most up-to-date literature, researchers proposed learning handover while considering receiving grasping poses and affordance [74], [75].The regrasps in these studies were from the robot or human side of human-robot interaction.The above end-to-end methods showed impressive flexibility.However, they focus less on precision, which limits their application in robotic assembly tasks.
In this work, we primarily focus on a tetrahedral intermediary fixture and use it to reduce uncertainty.Previously, Wan et al. [37] compared the efficacy of using a flat surface and a tilting surface in a work cell as the intermediary fixture [76].Cao et al. [35] presented to leverage a support pin as the intermediary fixture to provide stable placement.Ma et al. [38] discussed regrasping objects using general structures.The comparison of using different intermediary fixtures was presented in [77].The previous studies mainly concentrated on optimizing the motion sequence or improving the feasibility of orienting.Unlike them, we in this paper use regrasp to eliminate uncertainty and perform precise tasks.A TCF is used as the intermediary fixture considering that its crossing surfaces and tetrahedral shape can provide geometric constraints to align the dropped object.Together with the TCF, a dropping-regrasp process is implemented, and the estimation of robust dropping poses is studied.

III. PRELIMINARIES AND METHOD OVERVIEW
This section explains the background knowledge of a TCF, and presents the outline of the proposed method.

A. Background Knowledge of a TCF
A Triangular Corner Fixture (TCF) is made of three inclined plates (walls) intersecting at one bottom point, as shown in Fig. 2(a).The goal of the fixture is to hold an object like a cage [78] instead of immobilization [79], [80].The three walls of the fixture form a gravitational basket [81] that holds a dropped object at a stable pose.The inclined walls can always pull a dropped object into configurations with minimal potential energy in the gravitational field.
Especially, the walls of the TCF used in this paper are three congruent isosceles right triangles and are installed in a mutually perpendicular configuration.The angles between the triangular walls and a horizontal plane are the same and equal to 54.74 • .The reason we study this special TCF is that our goal is to assemble mechanical workpieces precisely.Although these workpieces have different shapes, they comprise geometric primitives like a cylinder, cuboid, ball, wedge, etc., and have three mutually perpendicular surfaces.We thus propose using a TCF made of three mutually perpendicular plates as an intermediary fixture to hold them.Figs.2(b) shows a real-world fabrication.The three triangular walls are acrylic and detachable from the base.We use the notation l e to denote the isosceles length of the triangular walls.Since the walls are detachable, the l e can be changed and the TCF dimensions can be adapted for parts of different scales.We use an abbreviation, TCF-l e , to denote a specific TCF.For instance, TCF-100 represents the TCF with 100.0 mm isosceles inner edge length.The TCF base is mounted on a 3-axes rotational platform for fine adjustment.The platform bottom has an adapter plate for connecting with other fixtures. 1

B. Method Overview
We develop a planner that estimates the robust dropping poses and stable placements of an object in the mentioned TCF and hence finds the regrasp motion that leads to precise assembly.Fig. 3 shows the workflow of our proposed planner.It receives the meshed models of the target object, the robotic gripper, and the TCF as the input.Three sub-modules will use the input: the Stable Placement Pose (SPP) planning submodule, the Deterministic Dropping Pose (DDP) estimation sub-module, and the grasp configuration planning sub-module, respectively, to find stable placement poses, estimate robust Fig. 3. Workflow of the proposed planner.The "Available?" or "Sovable?"condition diamonds check if obtained data sets were empty.If no, the data will be sent to the "Yes" branch.
dropping poses, and plan gripper configurations.Specifically, the SPP sub-module computes a set of stable placement candidates of a given object that satisfies geometry constraints and is statically stable in the TCF.The DDP estimation submodule uses a classifier to predict if an object dropped from an elevation position can be aligned to the expected SPP and finds a set of SPP-DDP pairs.The grasp configuration planning sub-module computes the gripper configurations for releasing an object at the DDPs and regrasping the object at the SPPs in the found SPP-DDP pairs.These computed releasing and regrasping gripper configurations are used to build a regrasp graph and reason a robot motion sequence.The details of the three sub-modules will be explained in the following Sections IV-VI, respectively.

IV. PLAN STABLE PLACEMENT POSES
We define an SPP as follows: An object is at an SPP when it stays in the triangular corner fixture in a balanced static condition.We use the algorithm shown in Fig. 4 to plan the SPPs.The algorithm receives an object and a TCF model as input and returns all satisfying SPPs as output.It comprises three steps which are highlighted using diamonds and blocks of different colors in Fig. 4. The first step includes the blue diamonds and blocks.In this step, the algorithm clusters the faces of the object's mesh model into facets and computes their triple combinations.The triplets of facets are the candidate contact faces with the inner surface of the TCF.The second step includes the yellow diamonds and blocks.In this step, the algorithm uses the triplets of facets to compute the object poses in the TCF.The triplets of facets must be in contact with the TCF's inner surface during the computation.Meanwhile, the object models are required to be not penetrating the TCF and the surrounding environments.The third step includes the pink diamonds and blocks.The algorithm in this step examines the static stability of the object poses computed in the second step and discards the unstable ones.The details of these three steps are presented below.We only consider face-to-face contact between the object and the TCF as effective contact candidates in the algorithm.Although a pointto-face or line-to-face contact can also stabilize an object, they are less reliable and ignored to simplify the planning algorithm.

A. Step 1: Facets and Their Triplet Combinations
In this step, the algorithm clusters triangle faces of an object's mesh model M o into facets and then uses the facets to find mutually perpendicular triplet combinations.
Using conventional segmentation methods to cluster facets may lead to uneven area [83].Instead of the conventional methods, we use superimposed segmentation [84] to generate uniform facets.The method especially has better performance when handling curved surfaces.Take the T-shape pipe junction object shown in Fig. 5 for example.The mesh model of the junction is shown in Fig. 5(a).The segmented superimposed facets are shown in Fig. 5(b).For a mesh model M o , we denote its superimposed facet set using S o = {s i } (i = 1, 2, . . .m), where each s i indicates a facet.
After getting the facets, we find the mutually perpendicular triplet facet combinations.Here, we assume to only consider the face-to-face contacts between an object and the TCF, and thus ignore the edge and vertex contact.The details of our workflow is as follows.With all segmented facets, we combine every three surfaces into a triplet and get a collection of triplets Then, we examine the orthogonality of each triplet's facet normal.The one that has three mutual orthogonal normal is considered as a feasible candidate, as illustrated in Fig. 5(c).We use g (i ) ∈ S g } to denote the feasible candidate collection.The workflow can be accelerated using linear programming to avoid repeatedly examining the impossible combinations.

B. Step 2: Computing Transformations
In the second step, the algorithm computes the transformation that fits the triplet facets onto the inner surfaces of the TCF.We use {C o } and {C f } to respectively represent the object frame and the TCF frame, and use {C o } to denote the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.local frame of S g (i ).The intersection point of S g (i )'s three orthogonal facets are selected as {C o }'s origin.Its coordinate axes are determined considering the inverted normal directions of the facets (The exact x, y, and z choices are free, as long as they meet the right-hand rule).Fig. 5(d) illustrates a {C o } defined considering the S g (i ) shown in Fig. 5(c).Next, we compute the placement pose of an object by transforming its {C o } onto TCF.We define two coordinate systems for the TCF.One is {C f }.Its origin is at the bottom point of the TCF, and its orientation is the same as the world coordinate system.The other one is {C f }, which has the same origin as {C f } but the x, y, z axes are along the intersection edges of the TCF's perpendicular surfaces.The placement poses of the object can be computed by superposing An object may have many S g (i ) and thus many Note that the potential C o T i may not be logically feasible since we did not check interference and contact.The object may penetrate the TCF, as shown in Fig. 5(g).It may also be floating in the air as the size of the TCF is limited and the contact is phantom (Fig. 5(h)).Thus, at the end of the second step, we screen the PPPs by detecting collisions and the existence of contact and removing the logically infeasible ones.We get a set of Logically Feasible Poses (LFPs) after the screening, as illustrated by Fig. 5(i).

C. Step 3: Examining the Static Stability
In the third step, we further use Contact Wrench Space (CWS) analysis to examine the static stability of the LFPs and obtain the SPPs.For a clear illustration, we use an Lshape object instead of the T-junction to exemplify this step.The workflow is as follows.
First, we extract the contact polygons between the object and the TCF's inner surfaces.Each of the three TCF inner surfaces has a contact polygon set, which may have a single or multiple elements.We compute the convex hull of the contact polygons in each set to get three support polygons for the three inner surfaces.The SP1-3 in Fig. 6(a) illustrate the three support polygons of the L-shape object.Second, we consider the vertices of the three support polygons as the effective contact points that provide supporting forces for the object, compute a wrench cone formed by the wrenches exerted on them and the object's gravity, and judge the stability of the object using the relation between the wrench cone and the origin of the wrench space.The yellow spheres in Fig. 6(a) illustrate the effective contact points.Assume there are in total k effective contact points p i = [x i , y i , z i ], (i = 1, 2 . . ., k).We build a local frame at each of the p i s to describe the contact force.The x and y axes of the local frame compose a tangent plane on the contact point, and the z axis aligns with the normal of the TCF's inner surfaces, as shown in Fig. 6(d).The contact force at p i can be represented by the components along the three axes as The effect wrench exerted on p i can be computed using , where τ i indicates the exerted torque.Considering the frictional constraints at the contact point, f i must be in a friction cone and f xi , f yi , and where μ is the friction coefficient.Since f i is inside a cone and is not deterministic, directly using the equation to compute the wrench cone is difficult.To overcome the difficulty, we approximate the friction cone with a pyramid [85], as shown in Fig. 6(b.2-3).The lateral edges of the pyramid represent the extreme f i choices.They are named as f j i , where the granularity of the approximation determines j .A linear combination of the f j i could approximate a freely chosen f i in the friction cone.With this consideration, we represent w i using multiple values w i = {w j i } and use all w j i to compute the wrench cone.Each w j i will be the wrench from one f j i .Considering all of them for wrench cones is the same as considering a linear combination of the f j i .The wrench set W born by the object comprises the {w j i } at every p i and the object's center of mass.It can be expressed as W = {w j 1 } ∪ . . .∪ {w j k } ∪ {w g }, where w g denotes the gravitational wrench.The wrench cone W s spanned by the wrench is essentially a convex hull of the elements in W [43].The stability of LFPs is judged by examining the relationship between the origin of the wrench space and W s . 2 If the origin is inside the W s of an LFP, the LFP is considered to be stable  Visualization of the elevation height h e and its effective range [h − , h + ].The d f is the depth of the TCF.It is not mentioned in the main text.We choose h − to be a bit lower than d f and choose h + to be outside d f to take into account various possibilities.and will be counted as an SPP.The planner will look over all LFPs and find a set of SPPs (S spp ).

V. ESTIMATE DETERMINISTIC DROPPING POSES
If an object released from a pose on top of the TCF has a deterministic and expectable SPP when it gets stabilized inside the TCF, we call the releasing pose a Deterministic Dropping Pose (DDP).This section presents methods to estimate if the SPPs obtained in the last section have correspondent DDPs.The methods are based on the assumption that a candidate DDP is linearly elevated from an SPP.The elevation height h e ranges from h − to h + , as shown in Fig. 7.We elevate an SPP to a random height in [h − , h + ] to get a candidate releasing pose and use the methods presented in the following subsections to estimate if the object deterministically stabilizes at the SPP after being dropped from the releasing pose.If the algorithms suggest a positive predicted result, we save the SPP and the releasing pose as an SPP-DDP pair.All saved SPP-DDP pairs will be used for reasoning and planning the regrasp sequences to improve grasping precision.
Specifically, we propose two classes of methods, an analytical and a learning-based, for the estimation.

A. Analytical Methods
In the first class of methods, we predict the SPP by considering a static stability criterion and screen an SPP considering its capability of resisting external disturbance wrenches.The method is based on an assumption that an SPP with larger stability is more likely to have a DDP than a less stable one.
We have three candidate implementations for methods in this class.First, similar to the third step of planning SPPs, we use CWS to evaluate static stability.However, instead of directly generating the convex hull of contact wrenches in the wrench space, we construct the wrench cone by computing the convex hull of W's Minkowski sum.We use the notation W mkv to differ the wrench cone in this section from the W s used before.Compared to W s , W mkv can quantify the resistible external wrench, thus make it easier to decide an evaluation criteria [43].In particular, we compute the shortest distance from the origin of the wrench space to the hyperplanes that constitute W mkv and use the shortest distance as the stability quality.Then, we the SPPs that have enough stability quality from the obtained SPP set and elevate them to get DDPs.The second implementation is a "projection method".It uses the shortest distance from the projected CoM to the boundary of the support surfaces as the stability index.The third implementation is an optimization-based method published in [86].It computes the minimal external disturbance that breaks the balance and uses the magnitude of the minimal disturbance as the stability index.
The analytical methods may find the DDPs with significant determinism from the SPPs of an object.However, the stability quality of different objects cannot be measured on a unified scale, making it difficult to set a unique threshold for a general estimation.Also, the DDPs may have uncertainty (positional and rotational noises and bouncing) in the real world, which is challenging to consider analytically.For these reasons, more advanced methods need to be explored.

B. Learning-Based Methods
In the second class of methods, we use machine learning to predict DDPs.Especially, we use a sim-to-real method [87] to obtain the training data and train different classifiers to judge if an SPP has a correspondent DDP.

1) Training Data:
The training data comprises a data section and a label section.The data part comprises the contact polygons, the position of the object's CoM, and the support surfaces of the TCF.They are projected onto a horizontal plane and formulated as a 2D grayscale image shown in Fig. 8 to simplify numerical computation.In detail, we assume a grayscale image with 224 × 224 pixels.The background of the image is white (grayscale value: 255).The regions of the projected support polygons and the contact polygons are set to 220 (support surfaces) and 0 (contact polygons).The projected CoM is formulated as a circular patch.Its color is computed using v grey = φ • h com , where φ is the ratio between a real-world distance and the numbers of image pixels used to represent it, h com indicates the vertical distance from the CoM to the bottom point of a TCF.The v grey essentially normalizes h com considering the dimensions of the TCF and the image.
The label section is collected by physical simulation.We place a work table and a TCF in simulation and generate the candidate releasing poses by randomly elevating an object from their SPPs, dropping the object from the candidate poses, and examining the finally stabilized poses.Unlike the analytical method, we add noises to the releasing poses to simulate uncertainty.The object falls from the releasing poses with random noises, and we compare the object's stabilizing CoM with the CoM of the expected SPP when Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.it gets stabilized.If the two CoMs coincide, a successful trial is recorded.Otherwise, a failure is recorded.Here we use the CoM as the reference to avoid misjudging symmetric objects with small support surfaces (e.g., balls and cylinders).The configurations of these symmetric objects are considered identical when moving around the symmetry center.By comparing the CoMs instead of the configurations, we may avoid misjudging the identical configurations.We run 100 trials for each releasing pose and compute a success rate.If the success rate is more significant than a given threshold, the releasing pose will be labeled as a positive sample.
2) Classifiers: Using the training data collected in the last section, we train classifiers to predict if the object dropped from a releasing pose can rest at an expected SPP.The classification is a simple binary one since there are only two labels.Various methods like Support Vector Machine (SVM), Fully Connected Network (FCN), and Convolutional Neural Network (CNN) can be used to model the classifier.Specifically, we implement and compare a linear SVM, a fourlayer FCN, an AlexNet, and a ResNet50.The detailed results and discussions about the implementation and comparison will be presented in Section VII.A.

VI. PLAN GRASP CONFIGURATIONS
AND REGRASP SEQUENCES This section presents detailed releasing and regrasp planning algorithms for adjusting grasping precision.The algorithms are based on our previous work published in [84] and [88].First, we plan grasps configurations for an object without considering any obstacles.The methods presented in [84] are directly applied.Then, based on the planned grasp configurations, we generate two sets of grasp configurations for the SPP and DDP in each SPP-DDP pair while considering different levels of collisions.Finally, we build a regrasp graph by reasoning and connecting the grasp sets associated with all SPP-DDP pairs, and search the graph to obtain regrasp sequences.We make a minor change to the original algorithm published in [88] when building the regrasp graph as the concepts of SPPs and DDPs were not available in the previous implementation: We additionally define SPP subgraphs and DDP subgraphs and connected the corresponding nodes and edges to the regrasp graph.The influences of target shapes, surrounding environments, and task goals are automatically considered, and the regrasp sequences obtained by searching the graph are optimized towards these constraints.
Fig. 9 and 10 exemplify the above workflow using the L-shape object.Fig. 9(a) shows the planned grasp configurations when there are no surrounding obstacles and the object pose is aligned with the global frame.Fig. 9(b-c) show an SPP and its associated grasp configuration set.These grasps in the set are transformed from (a) along with the object pose.The grasp configurations that collide with the TCF after the transformation are removed.Fig. 9(d-j) show the DDP paired with the SPP and the procedure for generating its associated grasp configuration set.Fig. 9(d) is the DDP.Fig. 9(e) is the grasps transformed from (a), with the ones in collision with the TCF removed.Fig. 9(f) shows the swept volume of the released object.The grasps in (e) are further examined considering the swept volume.If an opening hand collides with the swept volume, the released object will collide with the hand when it falls onto the TCF, leading to an unexpected resting pose.Thus, we further examine the collision between the grasp configurations in (e) and the swept volume, and remove the collided ones.Fig. 9(g.1-2) and 9(h.1-2) show a collision-free and a collided examples respectively.The grasp configuration in Fig. 9(g.1)does not collide with the swept volume after releasing in Fig. 9(g.2).Contrarily, the grasp configuration in Fig. 9(h.1)get collided in Fig. 9(h.2).Fig. 9(i) highlights all collided grasp configurations in (e) with red color.Fig. 9(j) shows the remaining collision-free grasps.Fig. 10 shows the regrasp graph built using the two grasp configuration sets in Fig. 9(c) and (j).The black maximally connected graphs in Fig. 10  connected graphs in Fig. 10(b) and (c) show the transit relations among the grasp configurations associated with the DDPs and their pairing SPPs.The blue edges among the maximally connected graphs show the transfer relations among the grasp configurations associated with different object states.The yellow edges represent strong connections between the black maximally connected graphs in Fig. 10(b) and (c).They indicate that a grasp configuration associated with a DDP can transit to another grasp configuration associated with its pairing SPP.To precisely regrasp the L-shape object, our planner will search a path on the graph by starting from one node in (a) (the initial grasp) and ending at another node in (b) (the final grasp).The path essentially represents a sequence where the object will be grasped from the initial pose (a node in (a)), moved to a DDP (a node in (b)), dropped down onto the TCF and regrasped (a node in (c)), and finally moved to the goal (a node in (d)).

VII. EXPERIMENTS AND ANALYSIS
This section includes three parts.First, we compare the methods for estimating the DDP-SPP pairs.Second, we use the most satisfying method to perform regrasp and examine the regrasp precision.Third, we validate the benefits of the proposed method using real-world assembly tasks.

1) Simulation Results With Random Objects:
We proposed three analytical and four learning-based methods in Section V) for estimating the DDP.In this subsection, we compare their performance using physical simulation.For the learning-based methods, we used the 14 primitives shown in Fig. 11(a) to obtain the training data.The primitives were scaled from 50% to 150% with 10% granularity, as shown in Fig. 11(b), to obtain 154 objects.The primitives and their scaled shapes were designed considering our expected target categories.Using these objects, we got 4464 SPPs.We collected training data using these SPPs in a PyBullet-based physical simulator.According to the real-world model, the friction coefficient and the bounce rate between the object and the TCF in the simulator were set to 0.3 and 0.2, respectively.We assumed a TCF with a 50.0 mm inner edge length (TCF-50) and a 28.0 mm depth (d f = 28.0 mm), and collected the training data by repeatedly elevating the objects from the SPPs to random start positions between h − = 0.8d f and h + = 1.5d f with maximally 3.0 mm positional and 3.0 • rotational noises, and dropping them from the start position.When the objects got stabilized, we compare their CoMs with that of the source SPPs and label the results.Through the physical simulation, we collected 1770 positive samples and 2674 negative samples.During training, we randomly selected 80% of the data for learning the estimators and 20% of the data for cross-validation [89].The datasets were restrained to the given primitives.
The results using the different methods are shown in Fig. 12.Here, we used a linear kernel for the SVM.The performance of SVM relies on selecting an appropriate kernel.It is not easy to make the selection general for all objects, and we made the most primary choice: a linear kernel.We used the cross-entropy loss function, the SGD optimizer, and a learning rate of 10e −4 to train the FCN and AlexNet.Each training batch size was 16.The whole training process was carried out for 50 epochs.To train the Resnet50, we used the cross-entropy loss function and the Adam optimizer.We kept the batch size to 16 but reduced the number of epochs to 20 to avoid over-fitting.The results imply that the learning methods had better performance.The ResNet exhibited the highest success rate (90.7%).The AlexNet also exhibited a competitive performance (90.1%).The CWS had poor performance because "finding the SPPs with enough stability quality" needed a threshold.For practical purposes, we only used the most stable configuration, which easily leads to ignored DDPs.Note that even if one configuration has the most stable stability, there is no guarantee that its elevated counterpart is a DDP.The DDPs found by the method are not necessarily robust.
2) Real-World Results: Besides the simulated data, we also validated the various methods using four real objects shown in   Objects used for obtaining the data.By scaling the 14 primitive objects (a) using the rules shown in (b) (resize the object from 50% to 150% with 10% granularity), in total 154 objects are prepared.in the discussion section (Section VIII.E).For the learningbased methods, we used the classifiers trained above to judge DDP-SPP pairs.For the analytical methods, we used the most stable configuration.The results are shown in Table I.The table's ground truth values (Denominators of the "Unpaired" and "DDP-SPP" columns) were obtained by repeated physical simulation.There were 54 SPPs for the L-shape object.6 of them had DDP-SPP pairs, as shown by the denominators of the L-shape object's "DDP-SPP" column.The remaining 48 did not have counterpart DDPs, as shown by the denominators of the L-shape object's "Unpaired" column.The numerators of the "Unpaired" column show the actual number of SPPs that did not have a DDP.The numerators of the "DDP-SPP" column show the actual number of SPPs that had a DDP.The T-junction had 72 SPPs, where 12 of them had DDP-SPP The bearing housing had 72 DDP-SPP pairs and 72 unpaired ones.The results show that the three analytical methods had similar performance.They worked effectively for the L-shape object and the bracket, but provided poor estimation for the T-junction object and the bearing housing.Especially, although they exhibited high success rates in estimating the unpaired candidates, they performed poorly in finding DDP-SPP pairs.The results are the same as our expectations.The learning-based methods are better on average but may have shortages for specific objects (i.e., the bracket).The AlexNet method has the best performance on the L-shape object and T-junction.The ResNet50 shows advantages in the Bracket and Bearing housing.The results are consistent with Fig. 12.
We also carried out experiments to particularly examine individual estimation performance.First, we placed the object with a selected SPP on the TCF.Then, the robot grasped the object, elevated it to a DDP with random offset noises, and opened the gripper to release the object.We observed the dropping process, checked if the object gets stabilized at the selected SPP, and recorded the results.The process was repeated 30 times for each SPP to get a statistical view.Fig. 14 shows the results.It is impossible to show all SPPs due to page limits, and we only present some representative DDP-SPP results for readers' convenience.The results show that the estimation mostly accords with the real world.
Since the analytical method had an extremely bad performance on the T-junction object and the bearing housing, we further analyzed the detailed contact between these objects and the TCF surfaces to understand the reason.We found that the DDP-SPP pairs with the best SPP stability were like the ones shown in Fig. 12(b.3)and (d.2).These SPPs had high static stability qualities, but their contact areas were distributed around the objects' CoMs (as shown by the 2D grayscale images in the figure).A large section of an object was not in contact with the inner surface of the TCF.The object thus had a low chance to stably "stand" on the distributed contact when being dropped.It was easy to get stuck by the edges of the TCF.

B. Performance on Eliminating Uncertainty
In this subsection, we analyze the uncertainty sources and carry out two experiments to study the performance of our method for eliminating uncertainty.
An assembly sequence found based on our method comprises three steps: Object pose recognition, grasp and motion planning, and real-world execution.Uncertainty may happen in each of the steps as follows.In the first step, a depth sensor captures the point clouds of the target object and the surrounding environment.The recognition algorithm clusters the point clouds, transforms them to the robot frame, and registers the object pose by matching its model with the processed point clouds.The uncertainty in this step depends on sensor measuring qualities, robot-sensor calibration algorithms, and pose recognition algorithms.For our system, we used the Phoxi 3D Scanner M (Photoneo S.R.O.) depth sensor for capturing point clouds.The sensor has a 0.1 mm measuring uncertainty.Its calibration uncertainty with the robot coordinate system is affected by multiple factors like the calibration board's fabrication precision, board installation precision, etc.The calibration uncertainty is much larger than the measuring quality.Its value is coupled with pose recognition uncertainty to influence final pose estimation results.After discussing the pose recognition uncertainty, we will show their total maximal values.We used DBSCAN and Iterative Closest Point (ICP) algorithms for object pose recognition.The selected convergence criteria of these algorithms determine the algorithm precision and thus recognition uncertainty.There could be a total ±3.25 mm uncertainty at maximum (experimentally measured) coupled with calibration.In the second step, the grasp planner finds grasp candidates for holding an object at the initial and goal poses, and the motion planner generates trajectories for moving the held object.The uncertainty in this part originates from the modeling error in the simulation, which is about ±0.72 mm (experimentally measured).In the third step, the robot moves following the planned trajectories and gripper actions.The robot control accuracy determines the uncertainty in this step.The UR3e robot arm used in our work has a ±0.03 mm control accuracy.Thus, the uncertainty in the third step is also ±0.03 mm.In summary, there are five uncertainty sources in our study: (i) Sensor, (ii) sensor-robot calibration, (iii) object recognition, (iv) modeling in simulation, and (v) robot control.The factors related to the vision system are the most problematic.The total uncertainty could reach as much as ±4.0 mm.
We conducted two experiments to analyze the proposed method's performance in eliminating the above uncertainty.In the first experiment, we recorded the goal position of an object moved by the robot, and observed the deviations of the recorded results to understand the reducible uncertainty.In the second experiment, we performed robotic peg-in-hole insertion tasks, and observed the affordable minimal shaft-tohole clearance to understand the method's usefulness.
In either experiment, we considered a conventional method and two varied implementations of the proposed method.They are shown in Fig. 15.All of them assume position control.The conventional method directly plans to move a picked object to a goal pose, as shown in Fig. 15(a).It is abbreviated as DPM in the following context.For our proposed method, we build and search a regrasp graph to find a regrasp sequence.We have two particular implementations for it: Regrasp with All Grasps (RAG) and Regrasp with a Prescribed Grasp (RPG).In the RAG implementation, all grasp configurations for the goal pose and SPP are considered when building the regrasp graph, as shown in Fig. 15(b).The built graph is exactly the same as Fig. 10.In the RPG implementation, a goal SPP and a goal grasp configuration are prescribed manually, as shown in Fig. 15(c).The motion between the prescribed goal SPP and the goal pose using the prescribed grasp configuration is taught instead of planned.The built graph is different from Fig. 10 due to the prescribed data: The connections in Fig. 10(c) and Fig. 10(d) are replaced with a given path and the goal pose subgraph collapses to a single node.Our expectations for them are as follows: The DPM implementation will be influenced by all uncertainties, as the TCF is not used to reduce uncertainty; The RAG implementation will reduce the uncertainties from sensors, sensor-robot calibration, and object recognition.The RPG implementation will further reduce modeling uncertainty.
1) Goal Deviation Measurements: In this experiment, we spread a grid paper on the work table to record the final position of a shaft (diameter: 10.0 mm) and asked the robot to grasp the shaft from different initial poses, move it to a defined goal pose above the grid paper with different methods and implementations, and lower it down to touch the paper.The setups of our experiments are shown in Fig. 16(a.1).When the shaft touched the paper, we drew the contact profile of the shaft on the grid paper to record the result.With multiple initial shaft poses, we could obtain a collection of goal profiles to understand the uncertainty range.The results with the three different methods/implementations mentioned in the last part are shown in Fig. 16(b).Here, the manually recorded goal positions are redrawn using computer graphics software for better visual effects.The yellow circles show all the recorded goal positions.The red circles are the minimum circumscription.Fig. 16(b.3) is the result of the RPG implementation.Its uncertainty was minimal and was not observable using the proposed method (less than ±0.03 mm, the robot control accuracy).Considering the maximum uncertainty was ±4.0 mm, the RAG and RPG implementations could reduce the uncertainty by ±3.25 mm and ±3.97 mm, respectively.The results were the same as our expectations about the methods/implementations.
2) Peg-in-Hole Insertion: In this experiment, we prepared a series of holes by cutting an acrylic board with a laser cutter.The holes had different clearance, as shown in Fig. 17   10.0 mm.The clearance of the peg-in-hole tasks (difference between the peg and hole diameters) thus ranged from 0.1 mm to 8.0 mm.We ran the insertion for each hole by 15 repetitions with each of the three methods/implementations to obtain average success rates and got the methods' performance by observing the smallest clearance with 100% success rate. 3n each repetition, we placed the object in a random initial position on a table.A robot detected it using the depth sensor and moved it to a common goal pose with or without regrasp using the TCF.At the goal pose, the robot inserted the peg by moving along a straight line with position control.
The results are summarized in Fig. 17 using a bar chart.The horizontal axis of the chart is the different hole diameters.The vertical axis is the average success rate.The results showed that the minimally affordable clearance of the DPM, RAG, and RPG methods were 7.0 mm, 1.0 mm, and 0.1 mm, respectively.Compared with the goal deviation measurements in the first experiment, the robot could successfully insert the shaft even if the expected deviation was larger than the hole clearance.The reason was that the chamfer of the shaft helped to partially afford the uncertainty.

C. Time Costs
Our search for DDPs and SPPs was carried out in a brute force style.The time complexity was O(n 3 ) where n denoted the number of planar segments on a mesh surface.Despite the cubic asymptotic time complexity, the method met our requirements because we assumed industrial workpieces as the target objects.These workpieces tended to have a small number of planar surface segments and could be solved in a practically limited time.This subsection demonstrates the claimed practicality by analyzing the time costs of the method applied to the four objects used in our real-world experiments and discusses possible improvements based on the results.Table II shows the time costs.Each value in the table was obtained by averaging five repeatable observations.The number of candidate grasp poses for each object was unified as 316 to avoid influences from the grasp planner.The results showed that the method took tens to hundreds of seconds to plan for each object.The T-junction was the most costly object, which took 190.69+24.25=214.95s to find DDPs and SPPs and 5.43 s to generate a manipulation sequence.The cost of planning SPP was more expensive than estimating the DDP-SPP pair.The result was observable for all objects.When an object had curved surfaces, i.e., the T-junction and housing, a pre-processing step was needed to discretize curved surface to piece-wise segments.The time costs of such an object depended on the discretization granularity and were usually much higher than objects without curved surfaces.Selecting a proper granularity hyper-parameter for discretizing curved surfaces is important to balance the computational cost and planning accuracy.In our experiments, we chose the granularity hyper-parameter by considering a planarity angle and an overlapping angle.Detailed definitions of these angles and their influence on discretization granularity can be found in [84]. 4

D. Performance in Practical Real-World Tasks
Finally, we test the proposed method using four realworld assembly tasks: (1) Inserting the L-shape object into a rectangular groove; (2) Sheathing the T-junction with a tube; (3) Aligning the holes of the bracket and a base plate; (4) Mounting the bearing housing on a bracket.These tasks are frequently seen at industrial manufacturing sites.
1) Inserting the L-Shape Object: In this task, we fixed an acrylic board with a rectangular groove on a table, and asked the robot to insert the L-shape object into the rectangular groove.Fig. 18(a.1)shows the sizes of the object and the groove.The clearance between them was 2.0 mm.

2) Sheathing the T-Junction:
The goal of this task was to sheathe a tube into the T-junction.The tube was vertically fixed on the table, and the robot was asked to manipulate the T-junction to perform the sheathing action.3) Aligning the Holes: In this task, a base plate with thread holes was fixed on a table.The two through-holes on the short side of the bracket were required to be aligned with the thread holes on the base plate.If a screw bolt could be fastened in the thread holes across the through-holes, we judge the alignment to be successful.Fig. 18(c.1)shows the sizes of the bracket and  the thread holes.The difference between the inner thread-hole diameter and through-hole diameter is the task's clearance.Its value was 1.7 mm.

4) Mounting the Bearing Housing:
This task required the robot to mount the bearing housing on a fixed bracket.The sizes of the bearing housing and mounting hole are shown in Fig. 18(d.1).The clearance between them was less than 0.1 mm.
Like the previous experiments, the environment model, object models, and the configuration of the TCF were pregiven and pre-calibrated.Also, the goal poses of them in the assembly tasks were known.The initial poses of the objects were random.The conventional method (DPM) and the two implementations of our method (RAG and RPG) were tested.For each of the above tasks, we ran ten times of experiments using different methods.Table III shows the experiment results.Using the DPM method, only two successful attempts were observed in inserting the L-shape object.All other tasks failed.Using the RAG method, all attempts to insert the L-shape object and align the holes succeeded, but no success was observed in the tasks of sheathing a T-junction and mounting a bearing housing.All tasks were successfully performed when the RPG method was used.The results showed that the proposed method could provide reliable and robust performance for these tasks, especially when the RPG method was used.Fig. 18(a.2-d.2) show execution pictures of some successful results in Table III.Readers may also refer to the video supplementary attached to this manuscript to observe the detailed robotic actions.

A. Weighted Regrasp Graph Instead of Thresholding
We used thresholds to judge placement poses and corresponding stability of dropping poses, and then used a non-weighted regrasp graph to plan a regrasp path considering a small number of edges.Besides the threshold-based method, one may consider building a weighted and directed regrasp graph to plan the path.Instead of thresholding, the weighted and directed regrasp graph will include all dropping poses and placements.Placements with high SPP scores or superior learning results will be associated with high weights.The planner will discover a path with large total weights to assure stability and action efficiency.The idea is advisable.In this article, however, we did not practice it since designing such a graph can be tricky and will inevitably introduce an unclear interplay between weight values and the number of edges along the path (the number of regraps).When the number of regrasps is the priority, the current non-weighted implementation stands as a handy choice.

B. Number of Regrasp Using the Proposed Method
The proposed regrasp graph and searching method could have flexible results, i.e., a long regrasp sequence that repeats the release-regrasp actions multiple times.The number of repetitions depends on task settings.It is influenced by many factors like the initial object poses, the goal object poses, the obstacles in the environment, the pose of the TCF, etc.For a simple task where the objects and TCF have shared grasp configurations, only one time of regrasp is needed.Contrarily, if the grasp configurations are highly constrained, i.e. the initial or goal object poses are near the boundary of the robot workspace, the planner may require more regrasps.We statistically measured the required number of regrasps by asking a robot to move the L-shape objects from four initial poses to one hundred random goal poses (400 tests in total).The results show that our planner successfully solved 73 of the 400 tests, where 69 of them can be done using a single regrasp.The remaining four required two regrasp.None of the tests required more than three regrasp.Fig. 19 shows two examples.In Fig. 19(a), the initial and goal poses have shared grasp configurations, and the robot used a single regrasp to precisely repose it.In Fig. 19(b), the initial and goal poses are near the workspace boundary, and the robot regrasped twice to finish the task.

C. Using Compliant Poking Instead of Throwing
In the proposed method, we considered throwing an object onto the TCF, and the object would reach a stable pose under the influence of gravity.An alternative solution could be taking advantage of manipulation compliance to guide the object to a belief state [90].Manipulation compliance may either be implemented actively or passively.Active compliance relies on sensors and delicate control algorithms.Since there is rich contact between the TCF and objects, using sensor data to actively judge contact states and stability is challenging and is beyond our scope.Passive compliance takes advantage of gravity and spring-dumper mechanisms or soft materials.For our robot hand, we may attach soft tips to the fingers to implement passive compliance and align objects to the TCF by poking them using the soft tips.Then, it would be interesting to explore designs and methods to overcome the following problems that may arise: (1) Modifying the grippers considering various constraints; (2) Determining which point to poke when the object has an uncertain pose and an arbitrary mesh shape; (3) Predicting bounce and uncertainty caused by deformation and energy storage, and thus improve the releasing and regrasping accuracy in the TCF.

D. Limitations of the Proposed TCF
We studied this particular TCF because our goal was to stably hold objects for industrial assembly tasks.Although these objects have different shapes, they comprise basic geometric elements like a cylinder, a cuboid, a ball, a wedge, etc., and have three mutually perpendicular surface segments.Such properties inspired us to use a TCF made of three mutually perpendicular plates as an intermediary fixture to hold them.Meanwhile, from theoretical robotic grasping studies, we understood that at least three contact points with unparalleled contact normals were needed to form a form closure together with gravity [91], [92] and thus assure stable poses.The condition also motivated us to choose three mutually intersecting Despite the deliberate selection, the proposed TCF has a fatal problem that it cannot eliminate the rotational uncertainty of centrosymmetric objects.Fig. 20 exemplifies such an object.The fixture and the proposed method cannot differentiate rotation angles.Also, we assumed rigid objects, and the proposed method cannot deal with deformation.

E. Other Fixtures
The proposed method can be adapted to other types of fixtures, given that the fixture comprises three supporting walls.Fig. 21 exemplifies several such fixtures.To adapt the current method to them, we need the following adjustments: (1) While planning the SPPs, our method finds the candidate poses that can match the triplet contact of an object with the three walls of the fixture.The proposed TCF's walls are mutually perpendicular.Thus the matching condition was defined as that the normals of the triplet contact were orthogonal.For other types of three-wall fixtures, we need to define different matching conditions following the changes in the walls.After finding the candidate poses, our method will screen the found candidate poses, considering stability and collision.The screening results change with the pose of the fixture.For the proposed TCF, the screening was carried out considering a horizontal pose, as explained in Section V of the main manuscript.When a fixture is installed with Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.a different pose (different inclination angles), the screening must be performed considering varied collision and stability criteria.(2) While planning the DDPs, the proposed method estimates the DDP-SPP pairs.The estimator is trained using simulation data.The simulation data should be recollected for different fixtures or installation poses, and the estimator should be retrained.(3) While planning grasp poses, the proposed method generates collision-free grasp poses.The same algorithms can be reused for different fixtures, given that the CAD models of the objects and fixtures are known.
On the other hand, the proposed method cannot handle fixtures with arbitrary numbers of supporting walls or other supporting elements.For example, adapting to the V-shape, pyramid, and pentagonal pyramid fixtures shown in Fig. 22, or fixtures with point and line contacts [35], [38], is beyond the scope of the current method: The V-shape fixture shown in Fig. 22(a) is widely used for alignment.However, it relies on friction to stop sliding along the intersection axis; Fixtures with more than three supporting walls have redundant contacts and lower flexibility (Fig. 22(b)).They are usually designed as special-purpose fixtures and require careful placement design; Lines and points contacts are helpful to improve the flexibility of placements.They are however ignored as balancing with edge and point contacts are either difficult to be planned and predicted.

F. Handling Objects With Large Size Variations
A single fixture is not enough to handle objects with large size variations as the fixture's dimension may obstruct small objects from regrasp or may not provide enough support for large objects.One solution is provide a set of fixtures to account for different contact surfaces and object sizes.In this part, we present a fixture screening algorithm to find an optimal set of fixtures for handling a set of objects with large size variations.We represent the set of target objects using S obj = {obj i } and a set of TCFs with varying sizes but the same configuration as our particular one using S t = {tc f i }.
For each object obj i , we traverse all fixtures tc f i in S t to find a set of qualified candidates S qt (obj i ).The optimal set of fixtures for S obj is a minimum set S ot that has intersection with all S qt (obj i ).Fig. 23 shows an example.The row header and the column header respectively illustrate the candidate set of fixtures S t and target objects S obj .Each cell in the body shows the representative SPP and the regrasp hands.The cells with green regrasp hands indicate the corresponding TCFs are the qualified elements of S qt (obj i ).The minimal S ot that has intersection with all S qt (obj i ) is {TCF-100, TCF-71, TCF-50}.Consequently, we may select them for handling the four objects.Note that this algorithm is a brute-force one.It could be very consumptive for traversing a large S t set.Also, it could be challenging to take into account other varied threewall fixtures or fixtures comprising other elements (lines [35], points [38], etc.).These problems deserve further exploration.

IX. CONCLUSION AND FUTURE WORK
This paper presented a regrasp planning method to eliminate grasp uncertainty.The proposed method first computed SPPs on a TCF, then estimated DDPs to find DDP-SPP pairs, and finally generated grasp configurations for releasing and regrasping an object and eliminating uncertainty.In particular, analytical and learning-based algorithms were proposed for the DDP estimation.Experimental results verified that the learning-based algorithm is more reliable than the analytical one.The regrasp sequences planned by the proposed method was demonstrated to be table to reduce uncertainty to robot control using an RPG implementation, which was way more robust than a conventional regrasp sequence without taking into account a TCF.Several real-world applications were also presented to show the proposed method's pracality in assembly tasks.
In the future, we are interested in building a large deep neural network that generalizes to many common materials and developing flexible features and planners to consider more complicated object shapes.APPENDIX See Table IV.

Fig. 1 .
Fig. 1.Using the geometric constraints of a fixture to reduce uncertainty.(a-c) Drop and regrasp.(d) Insertion.

Fig. 4 .
Fig. 4. Plan the SPPs.The diagram is a close-up view of the "Plan SPP" sub-module in Fig. 3.

Fig. 5 .
Fig. 5. (a) A raw meshed model.(b) Segmented surfaces.(c) A candidate triplet of facets.It has three mutual orthogonal surfaces.(d) The object's frame ({C o }) and the object-to-TCF transformation coordinate described in it ({C o }).(e) The TCF's frame ({C f }) and the object-to-TCF transformation coordinate described in it ({C f }).(f) A placement pose of the object on the TCF.(g) The first failure case: Penetration.(h) The second failure case: Phantom contact.(i) All planned LFPs.
Fig. 5(f) illustrates one PPP of the T-juction object.

Fig. 6 .
Fig. 6.(a) Support polygons and contact points formed by the contact between the object and the TCF.(b) Friction cones and gravity exerted on the object.The friction cones are approximated using hexagonal pyramids.(c) Wrench cone W s in the wrench space.

Fig. 7 .
Fig. 7.Visualization of the elevation height h e and its effective range [h − , h + ].The d f is the depth of the TCF.It is not mentioned in the main text.We choose h − to be a bit lower than d f and choose h + to be outside d f to take into account various possibilities.

Fig. 8 .
Fig. 8.An SPP shown in (a) is converted into a grayscale image in (c) by using the top view shown in (b).

Fig. 9 .
Fig. 9. (a) Grasp configurations planned without considering any surrounding obstacles or pose changes.(b) An SPP.(c) The grasp configuration set associated with the SPP in (b).(d) A DDP. (e) Transformed collision-free grasps from (a), with those in collision with the TCF removed.(f) Swept volume of the dropped object.(g-h) Releasing hands may collide with the swept volume.(i) Grasp configurations that collide with the swept volume are highlighted in red and will be removed.(j) Grasp configurations associated with the DDP.
(a) and (d) show the transit relations among the grasp configurations associated with the initial and goal object poses.The black maximally

Fig. 10 .
Fig. 10.A regrasp graph.(a) Initial pose and its subgraph.(b) DDP and its subgraph.(c) SPP and its subgraph.(d) Goal pose and its subgraph.Each black node in the graph indicates one grasp pose.Each circle indicates an object pose.The nodes inside the circle are the grasp configurations associated with the corresponding pose.First, the nodes in (a), (b), (c), and (d) are connected separately to represent transit relations.Second, the shared grasp configurations between (a) and (b), and between (c) and (d) are connected for transfer relations.Third, the nodes in (b) and (c) are connected to represent transit relations between DDPs and SPPs.

Fig. 13 .
They included: (a) an L-shape object; (b) a T-junction; (c) a bracket; (d) a bearing housing.We used TCF-50 for the L-shape object and bearing housing, and used TCF-71 for the bracket and the T-junction.The reason we used TCFs with different sizes and how we determined them could be found Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

Fig. 11 .
Fig. 11.Objects used for obtaining the data.By scaling the 14 primitive objects (a) using the rules shown in (b) (resize the object from 50% to 150% with 10% granularity), in total 154 objects are prepared.

Fig. 12 .
Fig. 12. Success rates of various DDP estimation methods.The learningbased ones had higher performance.ResNet50 is the most effective.

Fig. 14 .
Fig. 14.The results of testing the estimator using different objects.(a) L-shape object.(b) T-junction.(c) bracket.(d) bearing housing.The SPPs of each object are selectively shown.The left column shows the real-world photos, the middle column shows the placements in the simulator, and the right column illustrates the projected images.

Fig. 15 .
Fig. 15.Illustrations of the task process using different methods.The initial pose, the intermediate DDP, and their associated grasps are illustrated in a yellow color.The intermediate SPP, the goal pose, and their associated grasp poses are illustrated in a cyan color.(a) DPM: Directly plan to move the object to the goal pose.(b) RAG: An implementation of the proposed method that plans regrasp using all grasp configurations for the goal pose and the regrasp.(c) RPG: Another implementation that plans regrasp using a prescribed grasp configuration for the goal pose and the regrasp.The cyan object poses and grasp configurations (a single one for each object pose) indicate the prescribed items.
(a).The diameters of the holes ranged from 10.1 mm to 18.0 mm.The length and diameter of the peg were respectively 75.0 mm and

Fig. 17
Fig. 17.The success rates (%) of inserting a peg into holes with different diameters using DPM, RAG, and RPG.
Fig. 17.The success rates (%) of inserting a peg into holes with different diameters using DPM, RAG, and RPG.

Fig. 18 (
b.1) shows the sizes of the T-junction and tube.The maximum clearance between the inner circle of the T-junction and the outer circle of the tube was 0.3 mm.

Fig. 18 .
Fig. 18.(a) Inserting the L-shape object.(b) Sheathing the T-junction.(c) Aligning the holes.(d) Mounting the bearing housing.The left column shows the main boundary dimensions of the components and also points the assembly direction.The right column shows the demonstration of the successful task.All metrics are in mm.

Fig. 19 .
Fig. 19.(a) A task done by a single regrasp.The found path goes directly from the initial pose to the goal pose while passing the DDP and SPP subgraphs.(b) A task that required 2 times of regrasp.In this case, the initial and goal object poses are near the robot arm's workspace boundary.The object was dropped two times and passed the DDP and SPP subgraphs twice before finally reaching the goal pose.

Fig. 21 .
Fig. 21.Other fixtures.(a) Shapes of supporting walls are changed.(b) Angles between the supporting walls are changed.(c) Installation poses are changed.

Fig. 23 .
Fig. 23.Selecting an optimal set of TCFs for handling objects with large size variations.The column headers show the set of TCFs S t .The row headers show the set of target objects S obj .The planned SPP and the regrasp configurations are shown in the cells.The S qt (obj i ) are marked with shadows.The red objects denote the unstable placement poses.The translucent red regrasp hands are in collision.The translucent green regrasp hands are accessible.

TABLE I COMPARING
THE SUCCESS RATES OF DIFFERENT ESTIMATION METHODS

TABLE II TIME
CONSUMPTION OF THE PROPOSED METHOD

TABLE III RESULTS
OF REAL-WORLD TASKS

TABLE IV GLOSSARY
OF ACRONYMS AND NOTATIONS