Abstract— Psychological and physiological status has a big impact on the driver‟s behavior. It affects the driver‟s visual scanning behavior which helps drivers maintain visual attention. This paper proposes a system for detecting the abnormal driving behavior from the sequential pattern of the driver‟s peripheral visual scanning. The system continuously monitors the driver‟s activities through an in-vehicle camera to measure the driver‟s visual distraction. Feature descriptors of both the transition and rotation vectors of the driver‟s head pose and eye gaze are extracted and provided to a linear support vector machine (SVM) classifier to output one of six driver‟s common gaze zones. Then, a reservoir computing (RC) based on echo state networks (ESNs) is used for driver behavior classification from the sequence of the driver‟s gaze zones. The system is implemented on NVIDIA Jetson Nano to execute the processing of all the data since it has a Maxwell graphics processing unit (GPU) with 128 compute unified device architecture (CUDA) cores. The obtained results show that the driver‟s behavior can be classified to normal or abnormal based on his visual scanning activities with high accuracy. They also demonstrate the efficiency of both SVM and ESN in detecting the abnormal driver‟s behavior from a sequence of driver‟s gaze zones. Moreover, the results show that the proposed monitoring system is capable of detecting the driver‟s abnormal behavior with a detection accuracy of 98%, making it an appropriate candidate for successful deployment in both driver assistant systems (DAS) and driving safety support systems (DSSS).