the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 07 Sep 2026
A visual grasping method for collaborative robots in unstructured scenes based on object detection and grasp pose estimation
Abstract. In unstructured multi-object scenes, unclear target regions and interference from backgrounds and adjacent objects can affect grasp point selection. To address these problems, this paper proposes a visual grasping method for collaborative robots based on improved object detection and grasp pose estimation networks. In the object detection stage, the corresponding convolution and upsampling modules in YOLOv8n are replaced with RFAConv and DySample, respectively, to improve the detection of target boundaries and local features. In the grasp pose estimation stage, a residual attention structure and a region-guidance mechanism are incorporated into GR-ConvNet to enhance the stability of grasp prediction in complex backgrounds. To further reduce interference from non-target regions, the detected bounding box is used as a bounding box prompt for MobileSAM to generate a target mask. The target mask is then used to constrain the grasp quality map output by the improved GR-ConvNet and restrict grasp point search to the target region. The proposed method is validated on a collaborative robot visual grasping experimental platform using RGB-D images as input. Experimental results show a target detection success rate of 98 % and an overall grasp success rate of 95 %. The proposed method reduces interference from non-target regions during grasp point selection and improves the stability of target grasping in unstructured multi-object scenes.
- Preprint
(7136 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on ms-2026-165', Anonymous Referee #1, 12 Sep 2026
-
AC1: 'Reply on RC1', Junxiao Liu, 12 Oct 2026
We appreciate the reviewer’s careful review and valuable suggestions. We have carefully considered all comments and provide detailed responses to each comment below.
Response to Comment 1:
We agree that some components adopted in our method are derived from existing studies. Therefore, the contribution of this work does not lie in proposing these individual network modules themselves, but in effectively integrating object detection, target-region constraint, and grasp pose estimation for robotic grasping in unstructured multi-object scenes.
The main contribution of this study is to propose a vision-based robotic grasping method that addresses the unclear target regions and interference from backgrounds and adjacent objects during grasp point selection in unstructured multi-object scenes.
Existing detection-assisted grasping methods usually locate target objects using bounding boxes. However, bounding boxes only provide rectangular regions and cannot accurately describe the actual contour of target objects. Meanwhile, background regions and adjacent objects contained within bounding boxes may interfere with the subsequent grasp point searching process. Although segmentation-assisted methods can provide more accurate target-region information, how to effectively incorporate segmentation results into grasp pose estimation in complex multi-object scenes still requires further investigation.
In the proposed method, the improved YOLOv8n is first employed to identify target objects and determine their positions. Subsequently, the detected bounding boxes are used as prompts for MobileSAM to generate target masks, providing pixel-level target-region constraints for grasp point searching. Compared with methods that only use detection bounding boxes, the generated target masks can reduce the influence of background regions and adjacent objects within the bounding boxes on the grasping process. Based on the constrained target regions, the improved GR-ConvNet performs grasp pose prediction, enabling target grasping in unstructured multi-object scenes.
According to the reviewer’s suggestion, we have revised the Introduction section to further clarify the research motivation of this work, the differences between our method and existing detection-assisted and segmentation-assisted grasping methods, as well as the main contributions of this study.
Response to Comment 2:
We agree that the description of the experimental settings in the original manuscript was insufficient, particularly regarding the dataset information and training configurations, which may affect the reproducibility of the proposed method.
According to the reviewer’s suggestion, we have further clarified and supplemented the experimental settings in the revised manuscript.
For the object detection experiments, this paper uses a synthetic object detection dataset for training and validation. The dataset contains 16,242 images, including 12,347 images for training and 3,895 images for validation. No separate test set was established, and the detection metrics were calculated based on the validation set.
To evaluate the adaptability of the proposed method in real-world scenarios, we further collected a small-scale real-world object detection dataset for fine-tuning. This dataset contains 428 images, including 256 images for training and 172 images for validation. The model parameters obtained from training on the synthetic dataset were used as pretrained weights, and the model was further fine-tuned on the real-world dataset.
In addition, we have supplemented the detailed training configurations, including the optimizer, learning rate, batch size, training epochs, and hardware platform. The improved YOLOv8n was trained using the Adam optimizer with an initial learning rate of 1×10-4, a batch size of 16, and 100 epochs.
All experiments were conducted on a computing platform equipped with an Intel Core i7-10700 CPU, an NVIDIA GeForce RTX 3060 GPU, and 16 GB RAM. The software environment was PyTorch with CUDA 11.1.
Response to Comment 3:
We agree that the description of the coarse-to-fine prediction mechanism and Gaussian region guidance strategy in the improved GR-ConvNet was insufficient in the original manuscript. The explanations of some parameter settings and the training process were not detailed enough. According to the reviewer’s suggestion, we have further supplemented the relevant parameter settings, prediction branch descriptions, and loss function definitions in the revised manuscript.
Specifically, in the coarse-to-fine prediction module, a 25×25 sliding window is used to traverse the grasp probability map with a stride of 8, and the top K=3 high-response windows are selected for subsequent prediction. In the Gaussian region guidance process, the weighting coefficients of the overall response and the maximum response in each window are set to γ1=0.5 and γ2=0.5, respectively. The standard deviation of the Gaussian weights is set to σ=10. These parameters are kept fixed in all experiments.
During the training process, the coarse prediction branch estimates the probability of a valid grasp pose at each pixel, and the resulting probability map is used to generate Gaussian region guidance to reweight the decoder features. The fine prediction branch predicts the grasp quality, grasp width, and grasp angle to achieve the final grasp pose estimation.
In addition, we have supplemented the loss function definition of the improved GR-ConvNet:
L=0.3Lprob+Lscore+Lwidth+Langle
where Lprob is used to supervise the coarse prediction branch, while Lscore , Lwidth ,and Langle are used to supervise the fine prediction branch.
Regarding the parameter sensitivity issue, the parameters K,γ1, γ2, σ and the sliding window parameters are all fixed in this study and remain consistent across all experiments to ensure a fair comparison of experimental results.
Response to Comment 4:
We agree that the target-region IoU and background redundancy metrics alone cannot directly demonstrate the influence of the target-region constraint strategy on the final robotic grasping performance. The previous evaluation was mainly used to verify that the target mask generated by MobileSAM provides more accurate target-region information than the YOLO bounding box.
According to the reviewer’s suggestion, we have further added comparative robotic grasping experiments under the same experimental conditions to directly verify the effects of the bounding-box-constrained strategy and the MobileSAM-mask-constrained strategy on the final grasping performance.
Specifically, while keeping the robotic platform, target objects, scene arrangement, and grasp success criterion consistent, grasping experiments were conducted using the YOLO bounding-box constraint strategy and the MobileSAM mask constraint strategy, respectively. The experiments selected marker pen and lemon as target objects, and each method configuration was tested with 20 grasping attempts.
The results of the newly added comparative experiments show that, under the same YOLOv8n and improved GR-ConvNet configuration, replacing the bounding-box constraint with the MobileSAM mask constraint increased the number of successful grasps from 14/20 to 17/20, corresponding to an improvement in the grasp success rate from 70% to 85%. These results demonstrate that the pixel-level target-region constraint can effectively reduce the interference from background regions and adjacent objects in the detection region during grasp point prediction, thereby improving robotic grasping performance.
Response to Comment 5:
We agree that reporting only the robotic grasping results of the complete method in the original manuscript is insufficient to fully demonstrate the contribution of different modules to the final grasping performance. According to the reviewer’s suggestion, we have further added comparative robotic grasping experiments with different method configurations to analyze the contribution of each module to the final grasping performance.
Specifically, under the same robotic experimental conditions, the following method configurations were compared:
- Original YOLOv8n + GR-ConvNet;
- YOLOv8n + improved GR-ConvNet;
- YOLOv8n + improved GR-ConvNet + MobileSAM mask;
- Improved YOLOv8n + improved GR-ConvNet + MobileSAM mask.
All methods were evaluated under the same robotic platform, target objects (marker pen and lemon), scene arrangement, and grasp success criterion. Each method configuration was tested with 20 grasping trials, and the number of successful grasps and grasp success rate were recorded. The successful grasp trials of the above four configurations were 12, 14, 17, and 19 out of 20 trials, respectively, corresponding to grasp success rates of 60%, 70%, 85%, and 95%.
In addition, we have further clarified the grasp success criterion and the calculation method of the grasp success rate in the revised manuscript. A grasp is considered successful when the robot correctly grasps the specified target object, lifts it from the support surface, and maintains a stable grasp without dropping the object during the lifting process. The grasp success rate is calculated as follows:
Success rate=Ns/Nt ×100%
where Ns represents the number of successful grasps, and Nt represents the total number of grasping trials.
Response to Comment 6:
According to the reviewer’s suggestion, we have further improved the related work review in the Introduction section by adding recent studies on feature processing, target-region constraint, and robotic visual grasping in complex scenes.
Specifically, we have added recent studies on segmentation-assisted robotic grasping methods, including SISG-Net proposed by Yan et al. (2023) and GraspSAM proposed by Noh et al. (2025), to supplement the research progress of target-region-aware visual grasping methods in complex scenarios. In addition, we have further incorporated related studies on feature processing and feature information utilization to improve the completeness of the related work discussion in this paper.
Citation: https://doi.org/10.5194/ms-2026-165-AC1
-
AC1: 'Reply on RC1', Junxiao Liu, 12 Oct 2026
-
RC2: 'Comment on ms-2026-165', Anonymous Referee #2, 12 Oct 2026
The manuscript presents a promising visual grasping framework supported by dataset evaluation and physical robot experiments. However, important methodological details and experimental comparisons are missing, limiting reproducibility and assessment of the reported improvements. I recommend major revision to address the following concerns:
-
Dataset sizes, data partitions, and training configurations are insufficiently documented. Please report preprocessing, augmentation, hyperparameters, and checkpoint selection; clarify how data leakage was prevented; and provide results across multiple runs or folds.
-
The training of the coarse-to-fine prediction module remains unclear. Please specify the coarse-branch supervision, complete loss function, Gaussian guidance parameters, window size, and stride. Explain gradient handling through the discrete top-\(K\) and argmax operations, and include a parameter sensitivity analysis.
-
Improved segmentation quality does not directly demonstrate improved grasping performance. Please compare unconstrained, bounding-box-constrained, and MobileSAM-mask-constrained grasping under identical conditions. Include separate ablations of CBAM and Gaussian guidance to establish their contributions to physical grasp success.
-
The conversion from image-space predictions to executable robot poses requires further detail. Please describe camera intrinsics, hand–eye calibration, coordinate transformations, orientation conventions, gripper-width conversion, and grasp-rectangle height. Explain how invalid depth and potential gripper collisions are handled.
-
The physical evaluation protocol needs clarification. Please define successful grasping, report trial counts and results by object class and scene difficulty, and distinguish overall grasp success from success conditional on detection. Include uncertainty estimates and clarify whether trials involve repeated objects or scene configurations.
-
The reported inference times lack hardware specifications and measurement details. Please state the computing platform, numerical precision, batch size, and timing procedure. Report complete perception latency, including MobileSAM and preprocessing/postprocessing, and discuss the accuracy–latency trade-off.
-
The novelty beyond integrating existing components should be established more clearly. Please distinguish the approach from closely related target-oriented grasping methods and include relevant baselines, such as GR-ConvNet v2, evaluated using the same data and protocol.
Citation: https://doi.org/10.5194/ms-2026-165-RC2 -
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 309 | 61 | 28 | 398 | 43 | 37 |
- HTML: 309
- PDF: 61
- XML: 28
- Total: 398
- BibTeX: 43
- EndNote: 37
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
This paper presents a visual grasping method for collaborative robots in unstructured multi-object scenes by combining an improved YOLOv8n, MobileSAM, and GR-ConvNet. The method is evaluated on a UR5 collaborative robot platform. The topic has practical engineering relevance, and the manuscript provides experiments on grasp pose estimation, object detection, target-region constraint, and physical robotic grasping. The reported results indicate that the proposed method is effective to some extent. However, several issues concerning the novelty of the method, reproducibility of the experiments, and the actual contribution of individual components to the final grasping performance need to be clarified. I suggest the following modifications:
(1) The proposed method introduces RFAConv and DySample into YOLOv8n, incorporates CBAM and Gaussian region guidance into GR-ConvNet, and further uses MobileSAM for target-region constraint. Most of these components are based on existing studies. The authors should therefore clarify the main problem addressed by this work and explain more clearly where the main novelty lies compared with existing detection-assisted and segmentation-assisted robotic grasping methods. In its current form, the method can easily be understood as a combination of several existing modules.
(2) The experimental settings are not sufficiently described. The manuscript does not clearly report the size and training/validation/test split of the synthetic object-detection dataset, while the real dataset is only described as a "small-scale real-world object detection dataset". The authors should provide the dataset size and split, together with basic training settings such as the optimizer, learning rate, batch size, number of training epochs, pretrained weights, and hardware platform.
(3) The coarse-to-fine prediction and Gaussian region-guidance mechanism in the improved GR-ConvNet requires further explanation. The values and selection criteria of K, γ1, γ2, σ, and the sliding-window parameters in Eqs. (4)-(8) are not provided. The training procedure and loss functions of the coarse and fine prediction branches should also be described. It would also be useful to clarify whether the performance is sensitive to these parameters.
(4) Table 4 shows that the MobileSAM mask provides a higher target-region IoU and less background redundancy than the YOLO bounding box. However, this result does not directly demonstrate an improvement in robotic grasping performance. A direct comparison between the bounding-box-constrained strategy and the MobileSAM-mask-constrained strategy under the same robotic experimental conditions is suggested. This experiment would more clearly show whether the proposed target-region constraint improves the final grasp success rate.
(5) Table 5 only reports the robotic grasping results of the complete method. The manuscript reports 95 successful grasps in 100 trials, but no direct comparison with the basic pipeline is provided. It is suggested to compare several representative configurations, such as the original YOLOv8n + GR-ConvNet, the improved GR-ConvNet without MobileSAM, and the complete proposed method. The criterion for a successful grasp and the calculation of the grasp success rate should also be stated explicitly.
(6) The coverage of recent related work could be further improved. It is suggested to include more studies from the past two years on feature processing and robotic visual grasping. For example, the paper "Density cluster-based feature selection: An information theory approach" investigates feature relevance and redundancy from an information-theoretic perspective and can be considered as a recent reference on feature selection and redundancy reduction. Recent studies on target-region constraint and robotic grasping in complex scenes should also be considered.