Object detection in 3D with stereo cameras is an important problem incomputer vision, and is particularly crucial in low-cost autonomous mobilerobots without LiDARs. Nowadays, most of the best-performing frameworks for stereo 3D objectdetection are based on dense depth reconstruction from disparity estimation,making them extremely computationally expensive. To enable real-world deployments of vision detection with binocular images,we take a step back to gain insights from 2D image-based detection frameworksand enhance them with stereo features. We incorporate knowledge and the inference structure from real-time one-stage2D/3D object detector and introduce a light-weight stereo matching module. Our proposed framework, YOLOStereo3D, is trained on one single GPU and runsat more than ten fps. It demonstrates performance comparable tostate-of-the-art stereo 3D detection frameworks without usage of LiDAR data.The code will be published in https://github.com/Owen-Liuyuxuan/visualDet3D.