Modern distribution systems with high penetration of distributed energy resources face multiple sources of uncertainty. This transforms the traditional Optimal Power Flow problem into a problem of sequential decision-making under uncertainty. In this framework, the solution concept takes the form of a \textit{policy}, i.e., a method of making dispatch decisions when presented with a real-time system state. Reasoning over the future uncertainty realization and the optimal online dispatch decisions is especially challenging when the number of resources increases and only a small dataset is available for the system’s random variables. In this paper, we present a data-driven distributed policy for making dispatch decisions online and under uncertainty. The proposed policy is guaranteed to satisfy the system’s constraints while experimentally shown to achieve a performance close to the optimal-in-hindsight solution, significantly outperforming state-of-the-art policies based on stochastic programming and plain machine-learning approaches.