Intro

The goal of this project was to learn and implement different aspects of Autonomous Driving into one modular and robust stack of packages. The end result is to build isolated ros2 packages for simulation, sensor fusion, localization, planning and control such that they can execute on any input data. The input data should be able to be changed between simulation data, rosbag data or live data from real sensors, and be able to work effectively.

Currently, I have two packages, mike_av_stack and mike_av_stack_sensor_fusion. 

Running on the host machine, the base code mike_av_stack is responsible for the input data pipeline. Carla is used for simulation and carla ros bridge is used to format the sensor and position data into ros2 topics. 

Included in the base mike_av_stack python package are a few helper features.  A launch script and ros launch files are used to start all nodes simultaneously. A point cloud stacker makes it possible to compute BEV on a full scan, instead of the partial scans coming from Carla. A visualizer node adds output boxes from the sensor fusion package to input data images to visualize in rviz. 

Self contained in a Docker container, the mike_av_stack_sensor_fusion ros2 foxy package is able to detect objects in lidar and camera data.

Yolov8 is used to detect vehicles, pedestrians and signs in camera images. I am currently working on a fork of yolov4 to train a model using lidar BEV data to output 3d bounding boxes. My main goal is to develop a new model that combines BEV data and image data in the input step resulting in a 416x416x6 shaped input where the 6 channels are Red, Green, and Blue camera matrix combined with Intensity, Density, Height Birds-Eye-View lidar matrix to result in accurate bounding boxes and classes for each object. I also want to attempt changing the backbone such that the inputs are separated into a split backbone architecture, then combined at a lower level convolution step, or possibly even a Fully Connected layer. I am also looking into other models and papers that already do this.