PDPC: Plant disease phenotype captioning via semantic correction and trait description dependency grammar based on LLM


Introduction

PDPC is a framework designed to describe plant disease images, addressing several key issues of existing models: first, they tend to produce overly simplistic descriptions; second, they overlook important secondary features due to biased attention mechanisms; and third, they lack adequate generalization across diverse plant species. The PDPC framework consists of three core components: Support Text Acquisition (STA), which uses CLIP technology to retrieve relevant texts from an expanded plant disease database; Concept Graph Constructor (CGC), which transforms text data into graphical representations for enhanced processing; and Concept Injection Refinement (CIR), which refines descriptions by adding missing details, correcting errors, and integrating plant pathology terminology. Together, these components ensure comprehensive, accurate, and domain-specific descriptions, greatly enhancing the quality of plant disease descriptions and supporting intelligent agriculture diagnosis and management. Experimental results show that our framework effectively addresses the generalization issue of current plant disease description methods and outperforms existing zero-shot learning approaches.

Dependencies

  • CUDA 12.2
  • Python 3.10 (or later)
  • spaCy==3.7.5
  • torch==2.0.0+cu118
  • Pillow==10.3.0
  • torchvision==0.15.1+cu118
  • modelscope==1.15.0
  • transformers==4.41.2
  • numpy==1.23.5
  • opencv-python==4.5.5
  • matplotlib==3.9.0

GitHub: https://github.com/GZU-SAMLab/PDPC-main.git

Data

We propose a plant disease diagnostic description dataset named PDDC to address the inadequacies of existing datasets in describing disease characteristics. The PDDC dataset includes 20,943 images sourced from the PDDD and PlantVillage repositories, covering 60 crop types and over 300 disease characteristics. We use the deep learning model PlantText to automatically generate initial labels, which are then reviewed and corrected by domain experts to ensure each image has eight precise labels. Finally, we integrate these labels to generate detailed text descriptions for each image.

The datasets of PDDC Dataset can be download from there.

Get Started

Predict

  python PDPC.py
Validation

  python val.py

Results

Performance Comparison of different models.

We compare the PDPC model with GPT-4o and GPT-4 to validate its ability to describe plant pathological phenotypic traits. The results show that the PDPC model consistently scores higher across various traits, such as lesion color and pathological morphology, due to its advanced framework and specialized plant pathology terminology. This unique design not only improves description accuracy but also enhances model robustness, demonstrating the PDPC model's superior performance in plant disease description.


Table 1: Comparison between the frontier model and our model on multiple phenotypic traits.



Comparison of Baseline Models

We conduct a comparative analysis between baseline and PDPC-enhanced models to assess the enhancement in plant disease image caption capabilities. The results show significant performance improvements with the PDPC framework, particularly in BLEU-1 and CIDEr metrics. The rise in BLEU-1 scores highlights the framework's ability to capture key details and translate them into precise word choices, while the improvement in CIDEr suggests enhanced diversity and detail in image captions. Overall, the PDPC framework significantly enhances the baseline model's capability in describing plant disease images.


Table 2: The evaluation results on baseline experiments, reveal that PDPC consistently outperforms the baseline method across all indicators.