the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 07 Sep 2026
A visual grasping method for collaborative robots in unstructured scenes based on object detection and grasp pose estimation
Abstract. In unstructured multi-object scenes, unclear target regions and interference from backgrounds and adjacent objects can affect grasp point selection. To address these problems, this paper proposes a visual grasping method for collaborative robots based on improved object detection and grasp pose estimation networks. In the object detection stage, the corresponding convolution and upsampling modules in YOLOv8n are replaced with RFAConv and DySample, respectively, to improve the detection of target boundaries and local features. In the grasp pose estimation stage, a residual attention structure and a region-guidance mechanism are incorporated into GR-ConvNet to enhance the stability of grasp prediction in complex backgrounds. To further reduce interference from non-target regions, the detected bounding box is used as a bounding box prompt for MobileSAM to generate a target mask. The target mask is then used to constrain the grasp quality map output by the improved GR-ConvNet and restrict grasp point search to the target region. The proposed method is validated on a collaborative robot visual grasping experimental platform using RGB-D images as input. Experimental results show a target detection success rate of 98 % and an overall grasp success rate of 95 %. The proposed method reduces interference from non-target regions during grasp point selection and improves the stability of target grasping in unstructured multi-object scenes.
- Preprint
(7136 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on ms-2026-165', Anonymous Referee #1, 12 Sep 2026
-
AC1: 'Reply on RC1', Junxiao Liu, 12 Oct 2026
We appreciate the reviewer’s careful review and valuable suggestions. We have carefully considered all comments and provide detailed responses to each comment below.
Response to Comment 1:
We agree that some components adopted in our method are derived from existing studies. Therefore, the contribution of this work does not lie in proposing these individual network modules themselves, but in effectively integrating object detection, target-region constraint, and grasp pose estimation for robotic grasping in unstructured multi-object scenes.
The main contribution of this study is to propose a vision-based robotic grasping method that addresses the unclear target regions and interference from backgrounds and adjacent objects during grasp point selection in unstructured multi-object scenes.
Existing detection-assisted grasping methods usually locate target objects using bounding boxes. However, bounding boxes only provide rectangular regions and cannot accurately describe the actual contour of target objects. Meanwhile, background regions and adjacent objects contained within bounding boxes may interfere with the subsequent grasp point searching process. Although segmentation-assisted methods can provide more accurate target-region information, how to effectively incorporate segmentation results into grasp pose estimation in complex multi-object scenes still requires further investigation.
In the proposed method, the improved YOLOv8n is first employed to identify target objects and determine their positions. Subsequently, the detected bounding boxes are used as prompts for MobileSAM to generate target masks, providing pixel-level target-region constraints for grasp point searching. Compared with methods that only use detection bounding boxes, the generated target masks can reduce the influence of background regions and adjacent objects within the bounding boxes on the grasping process. Based on the constrained target regions, the improved GR-ConvNet performs grasp pose prediction, enabling target grasping in unstructured multi-object scenes.
According to the reviewer’s suggestion, we have revised the Introduction section to further clarify the research motivation of this work, the differences between our method and existing detection-assisted and segmentation-assisted grasping methods, as well as the main contributions of this study.
Response to Comment 2:
We agree that the description of the experimental settings in the original manuscript was insufficient, particularly regarding the dataset information and training configurations, which may affect the reproducibility of the proposed method.
According to the reviewer’s suggestion, we have further clarified and supplemented the experimental settings in the revised manuscript.
For the object detection experiments, this paper uses a synthetic object detection dataset for training and validation. The dataset contains 16,242 images, including 12,347 images for training and 3,895 images for validation. No separate test set was established, and the detection metrics were calculated based on the validation set.
To evaluate the adaptability of the proposed method in real-world scenarios, we further collected a small-scale real-world object detection dataset for fine-tuning. This dataset contains 428 images, including 256 images for training and 172 images for validation. The model parameters obtained from training on the synthetic dataset were used as pretrained weights, and the model was further fine-tuned on the real-world dataset.
In addition, we have supplemented the detailed training configurations, including the optimizer, learning rate, batch size, training epochs, and hardware platform. The improved YOLOv8n was trained using the Adam optimizer with an initial learning rate of 1×10-4, a batch size of 16, and 100 epochs.
All experiments were conducted on a computing platform equipped with an Intel Core i7-10700 CPU, an NVIDIA GeForce RTX 3060 GPU, and 16 GB RAM. The software environment was PyTorch with CUDA 11.1.
Response to Comment 3:
We agree that the description of the coarse-to-fine prediction mechanism and Gaussian region guidance strategy in the improved GR-ConvNet was insufficient in the original manuscript. The explanations of some parameter settings and the training process were not detailed enough. According to the reviewer’s suggestion, we have further supplemented the relevant parameter settings, prediction branch descriptions, and loss function definitions in the revised manuscript.
Specifically, in the coarse-to-fine prediction module, a 25×25 sliding window is used to traverse the grasp probability map with a stride of 8, and the top K=3 high-response windows are selected for subsequent prediction. In the Gaussian region guidance process, the weighting coefficients of the overall response and the maximum response in each window are set to γ1=0.5 and γ2=0.5, respectively. The standard deviation of the Gaussian weights is set to σ=10. These parameters are kept fixed in all experiments.
During the training process, the coarse prediction branch estimates the probability of a valid grasp pose at each pixel, and the resulting probability map is used to generate Gaussian region guidance to reweight the decoder features. The fine prediction branch predicts the grasp quality, grasp width, and grasp angle to achieve the final grasp pose estimation.
In addition, we have supplemented the loss function definition of the improved GR-ConvNet:
L=0.3Lprob+Lscore+Lwidth+Langle
where Lprob is used to supervise the coarse prediction branch, while Lscore , Lwidth ,and Langle are used to supervise the fine prediction branch.
Regarding the parameter sensitivity issue, the parameters K,γ1, γ2, σ and the sliding window parameters are all fixed in this study and remain consistent across all experiments to ensure a fair comparison of experimental results.
Response to Comment 4:
We agree that the target-region IoU and background redundancy metrics alone cannot directly demonstrate the influence of the target-region constraint strategy on the final robotic grasping performance. The previous evaluation was mainly used to verify that the target mask generated by MobileSAM provides more accurate target-region information than the YOLO bounding box.
According to the reviewer’s suggestion, we have further added comparative robotic grasping experiments under the same experimental conditions to directly verify the effects of the bounding-box-constrained strategy and the MobileSAM-mask-constrained strategy on the final grasping performance.
Specifically, while keeping the robotic platform, target objects, scene arrangement, and grasp success criterion consistent, grasping experiments were conducted using the YOLO bounding-box constraint strategy and the MobileSAM mask constraint strategy, respectively. The experiments selected marker pen and lemon as target objects, and each method configuration was tested with 20 grasping attempts.
The results of the newly added comparative experiments show that, under the same YOLOv8n and improved GR-ConvNet configuration, replacing the bounding-box constraint with the MobileSAM mask constraint increased the number of successful grasps from 14/20 to 17/20, corresponding to an improvement in the grasp success rate from 70% to 85%. These results demonstrate that the pixel-level target-region constraint can effectively reduce the interference from background regions and adjacent objects in the detection region during grasp point prediction, thereby improving robotic grasping performance.
Response to Comment 5:
We agree that reporting only the robotic grasping results of the complete method in the original manuscript is insufficient to fully demonstrate the contribution of different modules to the final grasping performance. According to the reviewer’s suggestion, we have further added comparative robotic grasping experiments with different method configurations to analyze the contribution of each module to the final grasping performance.
Specifically, under the same robotic experimental conditions, the following method configurations were compared:
- Original YOLOv8n + GR-ConvNet;
- YOLOv8n + improved GR-ConvNet;
- YOLOv8n + improved GR-ConvNet + MobileSAM mask;
- Improved YOLOv8n + improved GR-ConvNet + MobileSAM mask.
All methods were evaluated under the same robotic platform, target objects (marker pen and lemon), scene arrangement, and grasp success criterion. Each method configuration was tested with 20 grasping trials, and the number of successful grasps and grasp success rate were recorded. The successful grasp trials of the above four configurations were 12, 14, 17, and 19 out of 20 trials, respectively, corresponding to grasp success rates of 60%, 70%, 85%, and 95%.
In addition, we have further clarified the grasp success criterion and the calculation method of the grasp success rate in the revised manuscript. A grasp is considered successful when the robot correctly grasps the specified target object, lifts it from the support surface, and maintains a stable grasp without dropping the object during the lifting process. The grasp success rate is calculated as follows:
Success rate=Ns/Nt ×100%
where Ns represents the number of successful grasps, and Nt represents the total number of grasping trials.
Response to Comment 6:
According to the reviewer’s suggestion, we have further improved the related work review in the Introduction section by adding recent studies on feature processing, target-region constraint, and robotic visual grasping in complex scenes.
Specifically, we have added recent studies on segmentation-assisted robotic grasping methods, including SISG-Net proposed by Yan et al. (2023) and GraspSAM proposed by Noh et al. (2025), to supplement the research progress of target-region-aware visual grasping methods in complex scenarios. In addition, we have further incorporated related studies on feature processing and feature information utilization to improve the completeness of the related work discussion in this paper.
Citation: https://doi.org/10.5194/ms-2026-165-AC1
-
AC1: 'Reply on RC1', Junxiao Liu, 12 Oct 2026
-
RC2: 'Comment on ms-2026-165', Anonymous Referee #2, 12 Oct 2026
The manuscript presents a promising visual grasping framework supported by dataset evaluation and physical robot experiments. However, important methodological details and experimental comparisons are missing, limiting reproducibility and assessment of the reported improvements. I recommend major revision to address the following concerns:
-
Dataset sizes, data partitions, and training configurations are insufficiently documented. Please report preprocessing, augmentation, hyperparameters, and checkpoint selection; clarify how data leakage was prevented; and provide results across multiple runs or folds.
-
The training of the coarse-to-fine prediction module remains unclear. Please specify the coarse-branch supervision, complete loss function, Gaussian guidance parameters, window size, and stride. Explain gradient handling through the discrete top-\(K\) and argmax operations, and include a parameter sensitivity analysis.
-
Improved segmentation quality does not directly demonstrate improved grasping performance. Please compare unconstrained, bounding-box-constrained, and MobileSAM-mask-constrained grasping under identical conditions. Include separate ablations of CBAM and Gaussian guidance to establish their contributions to physical grasp success.
-
The conversion from image-space predictions to executable robot poses requires further detail. Please describe camera intrinsics, hand–eye calibration, coordinate transformations, orientation conventions, gripper-width conversion, and grasp-rectangle height. Explain how invalid depth and potential gripper collisions are handled.
-
The physical evaluation protocol needs clarification. Please define successful grasping, report trial counts and results by object class and scene difficulty, and distinguish overall grasp success from success conditional on detection. Include uncertainty estimates and clarify whether trials involve repeated objects or scene configurations.
-
The reported inference times lack hardware specifications and measurement details. Please state the computing platform, numerical precision, batch size, and timing procedure. Report complete perception latency, including MobileSAM and preprocessing/postprocessing, and discuss the accuracy–latency trade-off.
-
The novelty beyond integrating existing components should be established more clearly. Please distinguish the approach from closely related target-oriented grasping methods and include relevant baselines, such as GR-ConvNet v2, evaluated using the same data and protocol.
Citation: https://doi.org/10.5194/ms-2026-165-RC2 -
AC2: 'Reply on RC2', Junxiao Liu, 02 Nov 2026
We appreciate the reviewer’s careful review and valuable suggestions. We have carefully considered all comments and provide detailed responses to each comment below.
Response to Comment 1:
We agree that the description of the dataset splitting, training parameters, and model evaluation process in the original manuscript was insufficient. According to the reviewer’s suggestion, we have supplemented the descriptions of data preprocessing, data augmentation, training configurations, model selection, and the experimental results of multiple independent training runs.
Regarding dataset splitting, the YOLO object detection experiments used a synthetic object detection dataset containing 16,242 images, of which 12,347 images were used for training and 3,895 images were used for validation. No separate test set was established; therefore, all detection metrics were calculated based on the validation set. In addition, we collected 428 images from real-world scenarios, including 256 images for training and 172 images for validation. The model was first trained on the synthetic dataset, and the weights obtained from this training were then used as pretrained weights for further fine-tuning on the real-world dataset.
For the GR-ConvNet grasp pose estimation experiments, the Cornell Grasping Dataset, which contains 885 images, was adopted. Considering that objects not encountered during training may be encountered in practical grasping scenarios, an object-wise (OW) split was adopted. The dataset was divided into training and validation sets at an approximate ratio of 9:1, with 796 images used for training and 89 images used for validation. Images of the same object captured from different viewpoints were assigned to the same subset to prevent the same object from appearing in both the training and validation sets.
Regarding data preprocessing and augmentation, the input resolution of YOLO was set to 640 × 640. During training, data augmentation techniques including HSV color augmentation, random translation, random scaling, horizontal flipping, and Mosaic augmentation were employed. Mosaic augmentation was disabled during the final 10 training epochs. For GR-ConvNet, the RGB images and corresponding depth maps were cropped and normalized to construct a four-channel RGB-D input with a resolution of 224 × 224. Random rotation and scaling were employed for data augmentation. The rotation angle was randomly selected from 0°, 90°, 180°, and 270°, while the scaling factor was randomly selected from the range of 0.5–1.0. The above random data augmentation techniques were applied only to the training sets, while no random augmentation was applied to the validation sets.
Regarding the training configurations, both the improved YOLOv8n and the improved GR-ConvNet were trained using the Adam optimizer, with an initial learning rate of 1 × 10⁻⁴. The batch sizes were set to 16 and 8, respectively, and the numbers of training epochs were 100 and 50, respectively. All experiments were conducted on a computing platform equipped with an Intel Core i7-10700 CPU, an NVIDIA GeForce RTX 3060 GPU, and 16 GB RAM. The operating system was Ubuntu 18.04, and the software environment was based on PyTorch and CUDA 11.1.
Regarding model selection, the best YOLO model was selected according to the fitness score on the validation set. For GR-ConvNet, the predictions on the validation set were evaluated according to the grasp rectangle evaluation criteria defined by Eqs. (12) and (13), and the model with the best performance was selected. The validation sets were used only for model evaluation and checkpoint selection and were not involved in gradient updates.
Furthermore, to investigate the stability of the training results, three independent training runs were conducted for both the improved YOLOv8n and the improved GR-ConvNet. The mAP50 values of the improved YOLOv8n in the three experiments were 94.943%, 94.876%, and 94.988%, respectively, while the corresponding mAP50:95 values were 87.596%, 87.482%, and 87.641%, respectively. The improved GR-ConvNet achieved a mean object-wise grasp pose estimation accuracy of 98% across the three independent training runs.
Response to Comment 2:
We agree that the description of the training strategy and parameter settings of the coarse-to-fine prediction module in the original manuscript was insufficient. To address this issue, we have supplemented the descriptions of the supervision strategy for the coarse prediction branch, the loss function, the Gaussian region guidance parameters, and the gradient handling of discrete operations.
In the coarse-to-fine prediction module, a sliding window of size 25 × 25 with a stride of 8 is used to traverse the coarse prediction probability map, and the top K = 3 candidate windows with the highest responses are selected. During the Gaussian region guidance process, the weighting coefficients of the overall window response and the maximum response within each window are set to γ₁ = 0.5 and γ₂ = 0.5, respectively, and the standard deviation of the Gaussian weights is set to σ = 10. These parameters remain fixed across all experimental configurations.
Regarding training supervision, the improved GR-ConvNet is optimized using the grasp prediction loss function. The complete loss function is defined as follows:
L = 0.3Lprob + Lscore + Lwidth + Langle
where Lprob is the auxiliary supervision loss for the coarse prediction branch, while Lscore , Lwidth, and Langle correspond to the grasp quality, grasp width, and grasp angle losses of the fine prediction branch, respectively.
For the discrete region selection process, a stop-gradient strategy is adopted. The top-K operation is used to select high-response windows, and argmax is employed to determine the local maximum response position within each window. The Gaussian guidance map is then generated based on these positions. The discrete region selection and Gaussian guidance map generation processes are excluded from gradient backpropagation. Meanwhile, the coarse prediction branch receives direct supervision through its auxiliary loss, while the fine prediction loss can still be backpropagated through the feature modulation path to the shared feature extraction network and the fine prediction head, thereby enabling network parameter optimization.Regarding parameter sensitivity, the experiments in the original study employed a fixed parameter configuration to ensure consistency in comparisons among different models.
Response to Comment 3:
We agree that an improvement in segmentation quality does not directly demonstrate an improvement in the actual grasping performance of the robot. Therefore, in addition to evaluating the object segmentation performance of MobileSAM, we conducted real-robot grasping experiments using different method configurations to investigate the effects of target-region constraints and the improved grasp prediction network on the actual grasp success rate.
Specifically, four method configurations were compared under the same experimental conditions:
YOLOv8n + GR-ConvNet;
YOLOv8n + improved GR-ConvNet;
YOLOv8n + improved GR-ConvNet + MobileSAM mask;
Improved YOLOv8n + improved GR-ConvNet + MobileSAM mask.
All four configurations used the same robot platform, target objects (marker pens and lemons), scene arrangement, and grasp success criterion. Each configuration was evaluated through 20 real-robot grasping trials. The numbers of successful grasps were 12, 14, 17, and 19, respectively, corresponding to grasp success rates of 60%, 70%, 85%, and 95%.
This study focuses on designated-target grasping in unstructured multi-object scenes, where the robot is required to detect and grasp a specified target object rather than arbitrarily selecting an object from the scene. For a method that completely omits target-region constraints, the network can only select high-response locations from the grasp quality map of the entire image, without ensuring that the predicted locations belong to the specified target object. Therefore, such a method is not suitable for the designated-target grasping task considered in this study.
Regarding target-region constraints, a detection bounding box can approximately localize the target object, but it may still contain background regions and adjacent objects. In contrast, the target mask generated by MobileSAM can more accurately represent the actual region of the target object, thereby constraining grasp candidate locations at the pixel level. The experimental results show that, while keeping YOLOv8n and the improved GR-ConvNet unchanged, introducing the MobileSAM mask increased the grasp success rate from 70% to 85%, representing an improvement of 15 percentage points. This indicates that the target mask can reduce interference from the background and adjacent objects during grasp point selection, thereby helping improve the actual grasp success rate for the designated target.
Regarding the CBAM and Gaussian region guidance strategies, the objective of this study is not to propose these two modules independently, but to integrate them into the improved GR-ConvNet to construct a coarse-to-fine grasp prediction method for complex multi-object scenes. Specifically, CBAM enhances the network's attention to relevant features, while the Gaussian region guidance strategy uses spatial guidance information generated by the coarse prediction branch to modulate the features during fine prediction. The two mechanisms operate from the perspectives of feature representation and spatial guidance, respectively, and jointly contribute to grasp pose estimation.
The results of the first and second experimental configurations show that the grasp success rate increased from 60% to 70% after adopting the improved GR-ConvNet, representing an improvement of 10 percentage points. This indicates that the overall configuration of the improved grasp prediction network can improve actual grasping performance.
Furthermore, a unified grasp success criterion was adopted in this study. A grasp was considered successful if the robot correctly grasped the specified target object, lifted it from the supporting surface, and maintained a stable grasp without dropping the object during lifting. The grasp success rate was calculated as follows:
Success rate= (Ns / Nt) × 100%
where Ns denotes the number of successful grasps and Nt denotes the total number of grasping trials.
Response to Comment 4:
We agree that the description of the process of converting image-space grasp predictions into executable robot poses in the original manuscript was insufficient. To address this issue, we have supplemented the descriptions of the camera intrinsic parameters, hand–eye calibration, coordinate transformation, grasp pose, gripper opening control, grasp height, and the handling of invalid depth values and collision risks.
Regarding camera calibration, the experiments were conducted using an Intel RealSense D435i RGB-D camera mounted on the end effector of a UR5 robot arm in an Eye-in-hand configuration. The color image resolution was 640 × 480. The camera intrinsic parameters were fx = 614.320, fy = 613.190, cx = 321.433, and cy = 245.623. The camera intrinsic parameters were obtained through camera calibration, while the coordinate transformation relationship between the camera and the robot was determined through hand–eye calibration. Since images were consistently acquired at a fixed observation pose of the robot during the experiments, the transformation matrix corresponding to this observation pose, which maps coordinates from the camera frame to the robot base frame, was used for grasp point localization. The corresponding transformation matrix and depth scale factor were stored in the files camera_pose.txt and camera_depth_scale.txt, respectively.
Regarding coordinate transformation, the grasp center pixel coordinates (u, v) output by GR-ConvNet, the camera intrinsic parameters, and the valid depth Zc estimated from the local region around the grasp center were first used to convert the grasp position from the image space into a three-dimensional position in the camera coordinate frame:
Xc = (u − cx) × Zc / fx
Yc = (v − cy) × Zc / fy
where Zc denotes the estimated depth based on valid depth samples in the local region around the grasp center, and Xc and Yc represent the horizontal and vertical coordinates of the grasp point in the camera coordinate frame, respectively. Subsequently, the grasp point was transformed into the robot base frame using the calibrated coordinate transformation relationship:
pB = RBC × pC + tBC
where pC and pB represent the three-dimensional positions of the grasp point in the camera coordinate frame and the robot base frame, respectively, while RBC and tBC denote the rotation matrix and translation vector of the transformation from the camera coordinate frame to the robot base frame, respectively.
The above process is consistent with the two-stage coordinate transformation relationship described by Eq. (11) in the manuscript:
Gr = Trc(Tci(qp))
where Tci represents the transformation from the image coordinate frame to the camera coordinate frame, Trc represents the transformation from the camera coordinate frame to the robot base frame, qp denotes the grasp pose in the image coordinate frame, and Gr represents the resulting spatial grasp pose in the robot base frame. It should be noted that pB = RBC × pC + tBC describes the transformation of the position component of the grasp pose, whereas the complete grasp pose also includes the end-effector orientation.
Regarding the grasp orientation, GR-ConvNet outputs the grasp direction angle in the image plane. The predicted angle is adjusted according to the gripper mounting orientation, and the yaw angle of the robot end effector is determined through coordinate transformation. The roll and pitch angles of the end effector are assigned predefined values to determine the final grasp pose.
Regarding gripper opening control, the program adjusts the gripper control parameter using a piecewise empirical mapping strategy based on the longer side of the grasp rectangle predicted by GR-ConvNet. The resulting parameter is normalized and used as the open_size input to the Robotiq gripper control function. This mapping is an empirical control strategy developed for the current robot and gripper configuration, rather than an exact physical conversion between pixel length and the actual gripper opening width.
Regarding invalid depth handling, the valid depth range was set to 0.05–1.2 m, and the grasp point depth was estimated using valid depth samples in the local region around the grasp center. If no valid depth values were available, the current grasp attempt was canceled. The actual grasp height was determined by considering the target object height, the table surface height, and the safety clearance.
Regarding collision risk mitigation, YOLOv8n was used to detect the target objects, while the target masks generated by MobileSAM were used to constrain the grasp candidate regions. In addition, robot workspace constraints were imposed, and a pre-grasp position 0.1 m above the target grasp position was established. The robot then sequentially executed the approaching, grasping, and lifting motions.
Response to Comment 5:
We agree that the description of the evaluation protocol for the real-robot grasping experiments in the original manuscript was insufficient. Following the reviewer's suggestion, we have further clarified the success criterion, number of trials, experimental results for different object combinations, statistical uncertainty, and experimental scene configurations.
Regarding the grasp success criterion, a grasp was considered successful when the robot successfully grasped the specified target object, lifted it from the supporting surface, and maintained a stable grasp without dropping the object during lifting.
In the comparative experiments involving different methods, each of the four method configurations was evaluated through 20 grasping trials. The corresponding overall grasp success rates were 60%, 70%, 85%, and 95%, with 12, 14, 17, and 19 successful grasps, respectively.
Furthermore, 100 real-robot grasping trials were conducted using the complete method, involving five groups of object combinations, with 20 trials for each group. The objects included marker pens and lemons, apples and bananas, toothpaste boxes and cola cans, rhinoceroses and elephants, and balls and giraffes. Different multi-object scene configurations were generated by randomly arranging the objects. The numbers of successful grasps in the five groups were 19, 20, 18, 18, and 20, respectively, corresponding to success rates of 95%, 100%, 90%, 90%, and 100%.
Regarding grasping performance evaluation, the overall grasp success rate and the grasp success rate conditioned on successful target detection were calculated separately. In the 100 real-robot grasping trials conducted using the complete method, target detection was successful in 98 trials, and 95 trials resulted in successful grasps. Accordingly, the target detection success rate was 98.0%, the overall grasp success rate was 95.0%, and the grasp success rate conditioned on successful target detection was 96.9%.
Regarding experimental scene configurations, this study primarily focuses on designated-target grasping in unstructured multi-object scenes, in which a target object is specified in advance and the robot performs target detection, grasp pose prediction, and actual grasping. Five groups of different object combinations were used to construct multi-object scenes, and the object arrangements were varied through random placement to evaluate the grasping performance of the proposed method under different target objects and interference conditions caused by surrounding objects.
Response to Comment 6:
We thank the reviewer for this valuable comment. To address the concerns regarding the testing conditions for inference time and the overall perception latency, we have further supplemented the relevant descriptions and experimental results.
First, the experiments were conducted on a computing platform equipped with an Intel Core i7-10700 CPU, an NVIDIA GeForce RTX 3060 GPU, and 16 GB RAM. The operating system was Ubuntu 18.04, the deep learning framework was PyTorch, and the CUDA version was 11.1. All tests were performed using FP32 numerical precision with a batch size of 1. After model loading and warm-up were completed, CUDA synchronization was performed during timing, and the results of repeated tests were recorded to calculate the average inference latency.
Second, to comprehensively evaluate the computational efficiency of the proposed method, the latency of the complete perception pipeline was measured. The measurement included image preprocessing, YOLOv8n object detection, MobileSAM object segmentation, GR-ConvNet grasp prediction, and the corresponding post-processing operations. The results showed that the average latency of the complete perception pipeline was 119.08 ms, of which MobileSAM segmentation and mask processing accounted for 51.41 ms. The reported latency represents the computational overhead of the perception process from input image acquisition to the generation of grasp prediction results.
Finally, to analyze the trade-off between grasping performance and computational latency, the results were evaluated in conjunction with the real-robot grasping experiments. Under the same experimental conditions, introducing the MobileSAM mask constraint increased the grasp success rate from 70% to 85%, representing an improvement of 15 percentage points. With the further introduction of the improved YOLOv8n, the complete method achieved a grasp success rate of 95%. These results indicate that the target-region constraints provided by MobileSAM help reduce interference from the background and adjacent objects during grasp location selection, thereby improving actual grasping performance, while also increasing the computational overhead of perception. The average latency of the complete perception pipeline was 119.08 ms, indicating that the proposed method has a certain degree of practical feasibility for the static-target grasping tasks evaluated in this study.
Response to Comment 7:
We agree that the original manuscript needs to provide a clearer explanation of the differences between the proposed method and existing target-oriented grasping methods, as well as the roles of the individual components in practical grasping tasks.
Some of the network modules adopted in this study are derived from existing studies. Our work does not aim to propose entirely new and independent network modules. Instead, it investigates an approach that combines target-region constraints with grasp pose estimation for designated-target grasping in unstructured multi-object scenes, with the aim of reducing interference from background regions and adjacent objects during grasp point selection.
Existing detection-assisted grasping methods generally use object detection bounding boxes to constrain the grasping region. However, rectangular bounding boxes may contain background regions and adjacent objects, making it difficult to accurately represent the actual contours of target objects. Segmentation methods can provide more precise target-region information, but how to utilize segmentation results to filter grasp candidates still requires specific designs in combination with the corresponding grasp prediction methods.
To address this issue, the proposed method first employs the improved YOLOv8n to detect target objects and uses the detection bounding boxes as prompts for MobileSAM to generate target masks, thereby constraining the selection of grasp candidates at the pixel level. On this basis, GR-ConvNet is improved by incorporating a CBAM-based residual module and a coarse-to-fine Gaussian region guidance strategy to enhance feature extraction and use coarse prediction results to guide fine-grained grasp pose prediction.
Following the reviewer's suggestion, GR-ConvNet v2 was added as a comparative baseline and evaluated using the same Cornell Grasping Dataset split and evaluation protocol. Under the object-wise split, GR-ConvNet v2 achieved a grasp pose estimation accuracy of 95%, an inference time of 19.50 ms, and a parameter count of 1.90 M.
Citation: https://doi.org/10.5194/ms-2026-165-AC2
-
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 399 | 98 | 35 | 532 | 58 | 54 |
- HTML: 399
- PDF: 98
- XML: 35
- Total: 532
- BibTeX: 58
- EndNote: 54
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
This paper presents a visual grasping method for collaborative robots in unstructured multi-object scenes by combining an improved YOLOv8n, MobileSAM, and GR-ConvNet. The method is evaluated on a UR5 collaborative robot platform. The topic has practical engineering relevance, and the manuscript provides experiments on grasp pose estimation, object detection, target-region constraint, and physical robotic grasping. The reported results indicate that the proposed method is effective to some extent. However, several issues concerning the novelty of the method, reproducibility of the experiments, and the actual contribution of individual components to the final grasping performance need to be clarified. I suggest the following modifications:
(1) The proposed method introduces RFAConv and DySample into YOLOv8n, incorporates CBAM and Gaussian region guidance into GR-ConvNet, and further uses MobileSAM for target-region constraint. Most of these components are based on existing studies. The authors should therefore clarify the main problem addressed by this work and explain more clearly where the main novelty lies compared with existing detection-assisted and segmentation-assisted robotic grasping methods. In its current form, the method can easily be understood as a combination of several existing modules.
(2) The experimental settings are not sufficiently described. The manuscript does not clearly report the size and training/validation/test split of the synthetic object-detection dataset, while the real dataset is only described as a "small-scale real-world object detection dataset". The authors should provide the dataset size and split, together with basic training settings such as the optimizer, learning rate, batch size, number of training epochs, pretrained weights, and hardware platform.
(3) The coarse-to-fine prediction and Gaussian region-guidance mechanism in the improved GR-ConvNet requires further explanation. The values and selection criteria of K, γ1, γ2, σ, and the sliding-window parameters in Eqs. (4)-(8) are not provided. The training procedure and loss functions of the coarse and fine prediction branches should also be described. It would also be useful to clarify whether the performance is sensitive to these parameters.
(4) Table 4 shows that the MobileSAM mask provides a higher target-region IoU and less background redundancy than the YOLO bounding box. However, this result does not directly demonstrate an improvement in robotic grasping performance. A direct comparison between the bounding-box-constrained strategy and the MobileSAM-mask-constrained strategy under the same robotic experimental conditions is suggested. This experiment would more clearly show whether the proposed target-region constraint improves the final grasp success rate.
(5) Table 5 only reports the robotic grasping results of the complete method. The manuscript reports 95 successful grasps in 100 trials, but no direct comparison with the basic pipeline is provided. It is suggested to compare several representative configurations, such as the original YOLOv8n + GR-ConvNet, the improved GR-ConvNet without MobileSAM, and the complete proposed method. The criterion for a successful grasp and the calculation of the grasp success rate should also be stated explicitly.
(6) The coverage of recent related work could be further improved. It is suggested to include more studies from the past two years on feature processing and robotic visual grasping. For example, the paper "Density cluster-based feature selection: An information theory approach" investigates feature relevance and redundancy from an information-theoretic perspective and can be considered as a recent reference on feature selection and redundancy reduction. Recent studies on target-region constraint and robotic grasping in complex scenes should also be considered.