The present work investigated diffusion interactions between nanoparticles and polymers during freezing colloidal suspensions. The size effects of nanoporous media formed by packed nanoparticles on the diffusion instability of the polymer solution were explored. It is found that small particles under low pulling speeds will obstruct the diffusion of polymers and the corresponding morphology will be banded structures. The intrinsic reason is the inhibited tube-like motion of polymer chains in the nanoporous particle layer. The increased particle size or the decreased solute size will solve the diffusion problem. On the other hand, the small pulling speed constructs an increased length of the particle layer in front of the freezing interface, which presents a longer diffusion path to impede the polymer diffusion. Instead, an increased pulling speed shortens the length of the particle layer so that it is easy for polymers to go through a short porous media. Hence, the diffusion of polymers will control the freezing morphology of the suspension and create dendrites. These results imply that a relatively larger particle size and a moderately higher pulling speed are beneficial for well-developed microstructures in the production of porous ceramics with the freeze-casting method.