AI Model Falls Short in Identifying Wild Oysters: Comparing Human Annotators to Deep Learning Algorithm

Strategic Insights -

Despite the growing interest in using artificial intelligence (AI) in monitoring oyster reefs, a recent study conducted at the University of Delaware has found that a deep learning algorithm, ODYSSEE, is not sufficiently accurate in identifying live oysters. The study, published in Frontiers in Robotics and AI, compared the performance of ODYSSEE to that of human annotators and found that while the AI model can make inferences faster than humans, it overpredicts the number of live oysters and achieves lower accuracy. The researchers identified image quality as a critical factor in determining the accuracy of the model and annotators, suggesting that future training on higher-quality images and additional annotation classes may improve the model's predictive power.

Key Takeaways:

  • The ODYSSEE model, a deep learning algorithm, was developed to identify live oysters using video or images taken in the field of oyster reefs.
  • The model was compared to expert and non-expert annotators in terms of accuracy in identifying live oysters, with the AI model achieving lower accuracy (63%) compared to experts (74%) and non-experts (75%).
  • Image quality was found to be critical in determining the accuracy of the model and annotators, with better quality images improving human accuracy and worsening model accuracy.
  • The researchers identified potential sources of prediction error and suggested that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes may improve the model's predictive power.
  • The study highlights the importance of human expert evaluation in monitoring oyster reefs, particularly in small-scale or sensitive environments.
  • The research was supported by the U.S. Army Corps of Engineers, National Institute of Food And Agriculture, and National Oceanic And Atmospheric Administration.

Statistics:

  • The AI model (ODYSSEE) achieved 63% accuracy in identifying live oysters compared to experts (74%) and non-experts (75%).
  • The model made inferences significantly faster than human annotators, taking 39.6 seconds compared to 2.34±0.61 hours for non-expert annotators and 4.50±1.46 hours for expert annotators.
  • The study identified potential sources of prediction error, including image quality and additional annotation classes.

Sources:

  • Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE. Frontiers in Robotics and AI, 2025,12.
  • University of Delaware
  • Frontiers Media S.A.
  • VerticalNews
  • NewsRx LLC
  • U.S. Army Corps of Engineers
  • National Institute of Food And Agriculture
  • National Oceanic And Atmospheric Administration