T²SQNet: A Recognition Model for Manipulating Partially Observed Transparent Tableware Objects

Superquadrics, parametrized by only a few parameters, can represent a relatively wide range of geometric shapes. We employ two kinds of superquadrics: superellipsoids, which have been used for object manipulation, and superparaboloids, which are newly introduced. Superellipsoids and superparaboloids are implicit surfaces with the following implicit functions with size parameters \((a_1, a_2, a_3) \in \mathbb{R}_+^3\) and shape parameters \((e_1, e_2) \in \mathbb{R}_+^2\): for \(\textbf{x} = (x, y, z)\), $$ \begin{equation*} \overbrace{f_{se}(\textbf{x})=\left(\left|\frac{x}{a_1}\right|^{\frac{2}{e_2}} + \left|\frac{y}{a_2}\right|^{\frac{2}{e_2}}\right)^{\frac{e_2}{e_1}} + \left|\frac{z}{a_3}\right|^{\frac{2}{e_1}} = 1 }^{\text{Superellipsoid}}, \:\:\:\:\:\: \overbrace{f_{sp}(\textbf{x})= \left(\left|\frac{x}{a_1}\right|^{\frac{2}{e_2}} + \left|\frac{y}{a_2}\right|^{\frac{2}{e_2}}\right)^{\frac{e_2}{e_1}} - \left(\frac{z}{a_3}\right) = 1}^{\text{Superparaboloid}} \label{eq:sq} \end{equation*} $$ Deformable superquadrics extend superquadrics by incorporating global deformations, including tapering, bending, and shearing transformations. By adjusting the parameters, various surfaces can be represented, as shown below.

We combine deformable superquadrics to define templates representing seven types of tableware: wine glasses, bottles, beer bottles, bowls, dishes, handleless cups, and mugs. By adjusting parameters, we can generate diverse 3D tableware meshes. Spawning these meshes in a user-defined environment (e.g., table or shelf) within a physics simulator allows us to generate cluttered scenes. Using Blender, a photorealistic renderer, with transparent textures, we obtain RGB images of the scenes from arbitrary camera poses.

Overall, our method consists of four steps: (1) mask prediction in 2D images, (2) prediction of 3D bounding boxes, (3) computation of a smoothed visual hull through voxel carving, and (4) prediction of tableware parameters (i.e., a set of superquadric parameters). We apply these modules sequentially during inference, which can lead to the accumulation of prediction errors. To address this, we develop techniques to train each module accurately and robustly against noise and sim-to-real gaps.

The figure below shows the ground-truth shapes of the transparent TRansPose objects alongside the inferred implicit surfaces from T²SQNet. Although capturing surface details, such as the curvature of a water bottle, is challenging due to the nature of superquadric surfaces, we can confirm that T²SQNet infers the overall shapes to a considerable extent.

T²SQNet offers several practical advantages for downstream object manipulation tasks. For example, it allows for the easy design of an effective 6-DoF grasp sampler based on deformable superquadric representations, enables rapid collision checks through implicit function representations of deformable superquadric surfaces, and facilitates target-driven manipulation with instance-wise object recognition.

We demonstrate the effectiveness of our model, T²SQNet, on two object manipulation tasks: (i) sequential decluttering, which involves sequential grasping in a cluttered environment, and (ii) target retrieval, which involves object rearrangement planning to retrieve an initially non-graspable target object. The target object is indicated by a specific tableware class name (e.g., wineglass). Real-world manipulation videos can be found below.