Design and Implementation of an End-to-End Autonomous Driving System in the CARLA Simulation Environment

Abstract

In the quest for Level 5 autonomous driving, a critical cognitive bottleneck exists between high-frequency mechanical reflexes and low-frequency semantic reasoning. While end-to-end Vision-Language-Action (VLA) models provide semantic reasoning, they introduce inference latency incompatible with real-time vehicle control; current approaches struggle to simultaneously preserve sub-millisecond reflexes while embedding interpretable semantic arbitration without relying on massive, data-center-grade hardware. This thesis introduces a novel Neuro-Symbolic Co-Pilot operating within the CARLA simulator. We engineered a Hierarchical Perception, Decision, and Control Pipeline that decouples driving tasks by abstraction level. A custom multi-modal perception pipeline explicitly fuses 2D neural inferences (YOLOv8s, SegFormer-B0) with 3D LiDAR to construct a deterministic Bird’s-Eye-View (BEV) and Object Vector state. For decision-making, an unsupervised Dynamic Time Warping (DTW) KMeans clustering algorithm extracts distinct behavioral modes directly from a Soft Actor-Critic (SAC) agent’s trajectory history. A GRU sequence classifier then enforces class-conditional safety overrides, while a Small Language Model (Qwen 0.8B) acts as an asynchronous semantic governor. Experimental results demonstrate that this hybrid architecture successfully preserves sub-millisecond physical reflexes while embedding higher-level human-like reasoning, offering a robust, latency-efficient solution to the Vision-Language-Action (VLA) crisis