The experiment had a deliberately narrow goal: use a camera image to predict a steering angle and keep a virtual car on a simulated track.
This is a retrospective of the project documented in my May 20, 2022 report. It combined an image-processing exercise with a learned steering model. The report, source notebook, and recorded demonstration are linked below so the result can be examined in context. [1] [2]
Start by reducing the problem
A complete autonomous-driving system has to address far more than steering: other road users, changing conditions, localization, speed, and safety all matter. My project did not attempt that full problem. A simulator made it possible to collect examples and test one restricted behavior without a physical vehicle.
The first exercise used a conventional lane-detection pipeline. I converted images to grayscale, smoothed noise with Gaussian blur, detected edges with Canny, masked a region of interest, and used a Hough transform to find candidate lines. Averaging left and right line candidates produced a cleaner lane overlay.
This was a separate learning exercise, not a claim that the eventual neural network consumed those line overlays. It made the visual problem concrete before moving to learned steering. [1]
Collect examples of the behavior
For behavioral cloning, I used the Udacity Unity-based simulator to record camera images alongside human steering angles. I drove three laps in one direction and three in the other, aiming to stay near the center of the lane. Driving both ways helped reduce a directional turning bias.
Each example connected an observation to an action. The model’s task was to approximate that relationship: given an image like this, what steering angle should it produce?
The first useful discovery was in the data
The steering-angle histogram contained many more straight-driving examples than turns. That is natural when driving, but an imbalanced dataset can reward a model for predicting “straight” too often.
I grouped angles into 25 bins and limited the dominant bins to 200 examples. The examples removed from an overrepresented bin were selected after shuffling, so an entire contiguous portion of a drive was not simply cut away.
The report records approximately 1,010 training samples and 253 validation samples after preparation. Those are small experiment-scale numbers, not evidence of broad driving coverage. [1]
Prepare the image for the task
The preprocessing pipeline cropped away irrelevant image regions, converted the color representation to YUV, applied Gaussian smoothing, and resized each image to 66 × 200 pixels with three channels.
Reducing the input size made the training problem more manageable. Cropping also focused the input on the region relevant to steering. These choices were part of the experiment; they should not be assumed to be a universal recipe for every camera or road.
Learn a steering function
The model followed an NVIDIA-style convolutional architecture: convolutional layers extracted visual features, then fully connected layers produced a single steering-angle prediction. This approach was inspired by the end-to-end driving work of Bojarski and colleagues, which learned a mapping from camera pixels to steering commands. [3]
I trained with mean squared error and the Adam optimizer, using 30 epochs and batches of 100. During the experiment, the initial ReLU-based configuration did not converge satisfactorily. Replacing those activations with ELU improved the observed training behavior. I also introduced 50% dropout at selected layers to address overfitting.
These are observations from this run. The report’s discussion of inactive ReLU units was a possible explanation for the initial difficulty, rather than a separate experiment proving the cause.
Close the loop in the simulator
Training happens on stored examples. Driving requires repeated inference. A Python service connected the trained model to the simulator: receive an image, apply the same preprocessing, predict a steering command, send it back, and repeat.
The demonstration shows the model steering in the simulated environment. The useful lesson was seeing the whole loop work, from data collection through training to an action that changed the next image the model would see.
What the result does, and does not, establish
My original conclusion was candid: the results were not exemplary, and further tuning was needed. A demonstration on a familiar simulated track is not a road-safety evaluation or proof of reliable real-world autonomy.
It also highlights a limitation of imitation learning: small errors can move a vehicle into views that were rare in the recorded demonstrations. Good validation loss is valuable, but testing the closed-loop behavior remains essential.
What I took away was an engineering process: inspect the data before trusting a model, keep training and evaluation distinct, and test the behavior of the system around the model. Those habits matter well beyond self-driving experiments.
References & project materials
- Sahil Bambulkar. Self Driving Car Project, May 20, 2022. Lane detection: pp. 7–13. Data, architecture, training, and conclusions: pp. 13–18.
- Project notebook and simulated-driving demonstration, linked from the project repository.
- Mariusz Bojarski et al. End to End Learning for Self-Driving Cars, 2016. arXiv:1604.07316.
- CS677 course materials, for additional background on deep learning and autonomous agents.