Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN)

Jing Zhang, Long He, Manoj Karkee, Qin Zhang, Xin Zhang, Zongmei Gao

Research output: Contribution to conferencePaper

2 Citations (Scopus)

Abstract

Due to the rising cost and decreasing availability of labor, manual picking is becoming increasingly challenging for apple and other tree fruit growers. A targeted shake-and-catch apple harvesting machine is under development at Washington State University to address this challenge. This machine is showing a promise for harvesting some varieties of apples. However, the performance and productivity of such a harvesting system could greatly be increased if the shaking process could be automated. First step towards automated shaking is the detection of branches and localization of shaking points according to the position of branches in real world. A b r a n c h detection method was developed in this work for apple trees trained in fruiting wall architecture using a stereo vision system and a Regions-Convolutional Neural Network (R-CNN). A stereo vision camera was used to acquire RGB images, depth images as well as index images in natural orchard environment. The R-CNN composed of improved AlexNet, which was trained to detect the apple tree branches. In this study, a fusion detection method called Depth & Index (D&I) was proposed to fuse the detection results of branches from both depth images and index images. The results showed that the value of average recall and Average Accuracy (AA) from the D&I method was 70.5% and 63.3% when the R-CNN confidence of depth image was 50.0%. However, in the same conditions, the average recall and AA was only 62.7% and 59.2% using the depth images alone. Furthermore, the D&I method also had better performance in terms of the morphology fitting of apple tree branches. This study showed a great potential using both of depth and index images to detect and fit apple tree branches in real-time.

Original languageEnglish (US)
DOIs
StatePublished - Jan 1 2017
Event2017 ASABE Annual International Meeting - Spokane, United States
Duration: Jul 16 2017Jul 19 2017

Other

Other2017 ASABE Annual International Meeting
CountryUnited States
CitySpokane
Period7/16/177/19/17

Fingerprint

Stereo vision
neural networks
fruiting
apples
Neural networks
Orchards
Electric fuses
Fruits
Fusion reactions
Productivity
Cameras
Availability
Personnel
tree fruits
computer vision
harvesters
Costs
methodology
growers
labor

All Science Journal Classification (ASJC) codes

  • Bioengineering
  • Agronomy and Crop Science

Cite this

Zhang, J., He, L., Karkee, M., Zhang, Q., Zhang, X., & Gao, Z. (2017). Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN). Paper presented at 2017 ASABE Annual International Meeting, Spokane, United States. https://doi.org/10.13031/aim.201700427
Zhang, Jing ; He, Long ; Karkee, Manoj ; Zhang, Qin ; Zhang, Xin ; Gao, Zongmei. / Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN). Paper presented at 2017 ASABE Annual International Meeting, Spokane, United States.
@conference{41800df438f046b4b3d144515ab94be2,
title = "Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN)",
abstract = "Due to the rising cost and decreasing availability of labor, manual picking is becoming increasingly challenging for apple and other tree fruit growers. A targeted shake-and-catch apple harvesting machine is under development at Washington State University to address this challenge. This machine is showing a promise for harvesting some varieties of apples. However, the performance and productivity of such a harvesting system could greatly be increased if the shaking process could be automated. First step towards automated shaking is the detection of branches and localization of shaking points according to the position of branches in real world. A b r a n c h detection method was developed in this work for apple trees trained in fruiting wall architecture using a stereo vision system and a Regions-Convolutional Neural Network (R-CNN). A stereo vision camera was used to acquire RGB images, depth images as well as index images in natural orchard environment. The R-CNN composed of improved AlexNet, which was trained to detect the apple tree branches. In this study, a fusion detection method called Depth & Index (D&I) was proposed to fuse the detection results of branches from both depth images and index images. The results showed that the value of average recall and Average Accuracy (AA) from the D&I method was 70.5{\%} and 63.3{\%} when the R-CNN confidence of depth image was 50.0{\%}. However, in the same conditions, the average recall and AA was only 62.7{\%} and 59.2{\%} using the depth images alone. Furthermore, the D&I method also had better performance in terms of the morphology fitting of apple tree branches. This study showed a great potential using both of depth and index images to detect and fit apple tree branches in real-time.",
author = "Jing Zhang and Long He and Manoj Karkee and Qin Zhang and Xin Zhang and Zongmei Gao",
year = "2017",
month = "1",
day = "1",
doi = "10.13031/aim.201700427",
language = "English (US)",
note = "2017 ASABE Annual International Meeting ; Conference date: 16-07-2017 Through 19-07-2017",

}

Zhang, J, He, L, Karkee, M, Zhang, Q, Zhang, X & Gao, Z 2017, 'Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN)', Paper presented at 2017 ASABE Annual International Meeting, Spokane, United States, 7/16/17 - 7/19/17. https://doi.org/10.13031/aim.201700427

Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN). / Zhang, Jing; He, Long; Karkee, Manoj; Zhang, Qin; Zhang, Xin; Gao, Zongmei.

2017. Paper presented at 2017 ASABE Annual International Meeting, Spokane, United States.

Research output: Contribution to conferencePaper

TY - CONF

T1 - Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN)

AU - Zhang, Jing

AU - He, Long

AU - Karkee, Manoj

AU - Zhang, Qin

AU - Zhang, Xin

AU - Gao, Zongmei

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Due to the rising cost and decreasing availability of labor, manual picking is becoming increasingly challenging for apple and other tree fruit growers. A targeted shake-and-catch apple harvesting machine is under development at Washington State University to address this challenge. This machine is showing a promise for harvesting some varieties of apples. However, the performance and productivity of such a harvesting system could greatly be increased if the shaking process could be automated. First step towards automated shaking is the detection of branches and localization of shaking points according to the position of branches in real world. A b r a n c h detection method was developed in this work for apple trees trained in fruiting wall architecture using a stereo vision system and a Regions-Convolutional Neural Network (R-CNN). A stereo vision camera was used to acquire RGB images, depth images as well as index images in natural orchard environment. The R-CNN composed of improved AlexNet, which was trained to detect the apple tree branches. In this study, a fusion detection method called Depth & Index (D&I) was proposed to fuse the detection results of branches from both depth images and index images. The results showed that the value of average recall and Average Accuracy (AA) from the D&I method was 70.5% and 63.3% when the R-CNN confidence of depth image was 50.0%. However, in the same conditions, the average recall and AA was only 62.7% and 59.2% using the depth images alone. Furthermore, the D&I method also had better performance in terms of the morphology fitting of apple tree branches. This study showed a great potential using both of depth and index images to detect and fit apple tree branches in real-time.

AB - Due to the rising cost and decreasing availability of labor, manual picking is becoming increasingly challenging for apple and other tree fruit growers. A targeted shake-and-catch apple harvesting machine is under development at Washington State University to address this challenge. This machine is showing a promise for harvesting some varieties of apples. However, the performance and productivity of such a harvesting system could greatly be increased if the shaking process could be automated. First step towards automated shaking is the detection of branches and localization of shaking points according to the position of branches in real world. A b r a n c h detection method was developed in this work for apple trees trained in fruiting wall architecture using a stereo vision system and a Regions-Convolutional Neural Network (R-CNN). A stereo vision camera was used to acquire RGB images, depth images as well as index images in natural orchard environment. The R-CNN composed of improved AlexNet, which was trained to detect the apple tree branches. In this study, a fusion detection method called Depth & Index (D&I) was proposed to fuse the detection results of branches from both depth images and index images. The results showed that the value of average recall and Average Accuracy (AA) from the D&I method was 70.5% and 63.3% when the R-CNN confidence of depth image was 50.0%. However, in the same conditions, the average recall and AA was only 62.7% and 59.2% using the depth images alone. Furthermore, the D&I method also had better performance in terms of the morphology fitting of apple tree branches. This study showed a great potential using both of depth and index images to detect and fit apple tree branches in real-time.

UR - http://www.scopus.com/inward/record.url?scp=85035327628&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85035327628&partnerID=8YFLogxK

U2 - 10.13031/aim.201700427

DO - 10.13031/aim.201700427

M3 - Paper

AN - SCOPUS:85035327628

ER -

Zhang J, He L, Karkee M, Zhang Q, Zhang X, Gao Z. Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN). 2017. Paper presented at 2017 ASABE Annual International Meeting, Spokane, United States. https://doi.org/10.13031/aim.201700427