This paper examines the nonconvex quadratically constrained quadratic programming (QCQP) problems using an iterative method. A QCQP problem can be handled as a linear matrix programming problem with a rank-one constraint on the to-be-determined matrix. One of the existing approaches for solving nonconvex QCQPs relaxes the rank one constraint on the unknown matrix into a semidefinite constraint to obtain the bound on the optimal value without finding the exact solution. By reconsidering the rank one matrix, the iterative rank penalty (IRP) method is proposed to gradually approach the rank one constraint. Each iteration of IRP is formulated as a convex problem with semidefinite constraints. Furthermore, an augmented Lagrangian method, called an extended Uzawa algorithm, is developed to solve the sequential problem at each iteration of IRP for improved scalability and computational efficiency. Simulation examples are presented using the proposed method, and comparative results obtained from the other methods are provided and discussed.