Dynamic traffic signals play a pivotal role in reducing traffic congestion. In current adaptive traffic signal control systems, the vehicle traffic is detected by inductive loop sensors to adjust the signal timings. To control the other modes such as pedestrians (in city centres with greater walkability), the system is only informed of the presence of pedestrians by push buttons. To this end, traffic signal optimizations are mainly relied on improving vehicular delays without being sensitive to the volume of pedestrians. With significant advancements in ITS and computer vision, advanced sensors such as video cameras are enabled to detect individual vehicles and pedestrians at the intersection. Moreover, AI has become a major area of interest for a variety of applications, especially traffic signal control. Hence, including pedestrian volumes in controlling traffic signals is imperative. In this study, we present a novel deep reinforcement learning-based traffic signal model to control vehicle and pedestrian flow. The objective (i.e., reward function) of the proposed model is designed to improve the total user delays. To consider the real-world challenges, the vehicle-to-vehicle, vehicle-to-pedestrian, and pedestrian-to-pedestrian interactions are modelled in a simulation environment. Then the simulation model is calibrated using video camera data at the intersection. Our method is tested and evaluated for a comprehensive set of scenarios even in the presence of pedestrian jaywalking. The experimental results have shown the superiority of our model compared to the adaptive traffic signal control (i.e., actuated) in terms of total user delays.