Uplink power control plays a significant role in maintaining a good signal quality at the serving cell while minimizing interference to neighboring cells, thus maximizing the system performance. Traditionally, a single value open-loop power control (OLPC) parameter, P0, is configured for all the user equipments (UEs) in a cell, and often same setting is used for similar cells. Recent studies have demonstrated that optimal P0 depends on many factors, which yields a complex multidimensional optimization problem and there are no efficient approaches known to solve it under practical system-level settings. In this paper, we propose a solution based on reinforcement learning (RL) where each BS autonomously adjusts its P0 setting to maximize its throughput performance. As compared to conventional sub-optimal approach, our solution encompasses a smart clustering of UEs, where each cluster specifies its own P0. The proposed solution is evaluated by extensive system level simulations, where our results demonstrate a potential performance enhancement as compared to the baseline proposals.