Zero-Shot Semantic Segmentation for Robots in Agriculture

Y. L. Chong, L. Nunes, F. Magistri, X. Zhong, J. Behley, and C. Stachniss

Conventional crop production, which is essential for providing food, feed, fuel, and fiber for our society, relies heavily on harmful herbicides to control weeds. Instead, agricultural robots could remove weeds more sustainably. However, these robots require a generalizable perception system that can locate weeds, enabling automatic removal of weeds. Specifically, they need to perform crop-weed semantic segmentation, which locates and distinguishes between the crop and the weed plants with pixel-level resolution. However, most existing crop-weed semantic segmentation methods are fully supervised and require expensive and labor-intensive pixel-wise labeling of the training data. To avoid the costly labeling process, we address the problem of unsupervised crop-weed segmentation in this paper. Unlike previous approaches, we leverage the idea that weeds are “weird” plants that occur less frequently and are highly variable in appearance, and reframe the problem as an anomaly segmentation problem. We propose an approach to segment weeds as anomalous plants by categorizing plants in the feature space of a pretrained foundation model. Our approach curates a bag-of-features representation of crop features and models the manifold of crop plants as hyperspheres. During inference, it classifies vegetation segments of the image with features within this manifold as crop plants and all other plants as weeds. Our experiments show that our zero-shot anomaly segmentation method can perform crop-weed segmentation on several datasets from real crop fields.

@inproceedings{chong2025iros,
author = {Y.L. Chong and L. Nunes and F. Magistri and X. Zhong and J. Behley and C. Stachniss},
title = {Zero-Shot Semantic Segmentation for Robots in Agriculture},
booktitle = iros,
year = 2025}