A mass transfer model based on unsteady state mass balance over concentration boundary layer coupled with gradual development of so called "gel" or "cake" layer has been formulated in this study. The model considers the boundary layer problem simultaneously with the film theory of mass transfer and resistance in series model. The resulting equations are solved first by reducing the set of partial differential equations to nonlinear equations utilizing orthogonal collocation technique and then applying multidimensional Newton-Raphson method with suitable seeding scheme to solve the above-mentioned nonlinear equations together with some additional constraints. The problem of determination of membrane surface concentration is eliminated by this simultaneous solution of boundary layer equation together with gel layer. The model is found to be capable of predicting permeate flux and rejection under different experimental conditions. The main feature of this model is that it takes into account realistically the effect of gel layer formation within concentration boundary layer whereas in most of the previous studies this effect has not been considered. The dependency of concentration profile, gel thickness, and permeate concentration with different operating variables are also studied and the variations are found to be quite reasonable.