Forming solid solutions has long been considered an effective approach for good thermoelectrics because the lattice thermal conductivities are lower than those of the constituent compounds due to phonon scattering from disordered atoms. However, this effect could also be compensated by a reduction in carrier mobility due to electron scattering from the same disorder. Using a detailed study of n-type (PbTe)1x (PbSe)x solid solution (0 x 1) as a function of composition, temperature, and doping level, quantitative modeling of transport properties reveals the important parameters characterizing these effects. Based on this analysis, a general criterion for the improvement of zT due to atomic disorder in solid solutions is derived and can be applied to several thermoelectric solid solutions, allowing a convenient prediction of whether better thermoelectric performance could be achieved in a given solid solution. Alloying is shown to be most effective at low temperatures and in materials that are unfavorable for thermoelectrics in their unalloyed forms: high lattice thermal conductivity (stiff materials with low Gruneisen parameters) and high deformation potential.