From the most known Gaussian mixture to the cutting-edge multi-Bernoulli mixture of various forms, mixture offers a fundamental means to deal with uncertainties, which has led to a variety of appealing applications in the state estimation realm based on a single sensor or a sensor network. Like noise is often used to model unknown system input, one may use various hypotheses to deal with the uncertain state space model or data association. Meanwhile, consensus may be sought over the cross-correlated sensors. These all drive a need for representing the probability distribution by a mixture of properly weighted component distributions, which fuse the information gained from different models/hypotheses or from different sensors. This technical note presents information-theoretical results which answer how the averaging/mixture approach makes sense and how the fusing weights should be designed.