Humans have a strong class-agnostic object segmentation ability and canoutline boundaries of unknown objects precisely, which motivates us to proposea box-supervised class-agnostic object segmentation (BoxCaseg) based solutionfor weakly-supervised instance segmentation. The BoxCaseg model is jointlytrained using box-supervised images and salient images in a multi-task learningmanner. The fine-annotated salient images provide class-agnostic and preciseobject localization guidance for box-supervised images. The object maskspredicted by a pretrained BoxCaseg model are refined via a novel merged anddropped strategy as proxy ground truth to train a Mask R-CNN forweakly-supervised instance segmentation. Only using $7991$7991 salient images, theweakly-supervised Mask R-CNN is on par with fully-supervised Mask R-CNN onPASCAL VOC and significantly outperforms previous state-of-the-artbox-supervised instance segmentation methods on COCO. The source code,pretrained models and datasets are available at\url{https://github.com/hustvl/BoxCaseg}.