We report a study of self-assembled GaN/AlN quantum dots (QDs) grown by metalorganic vapor phase epitaxy. We have investigated the impact of the AlN template quality on the nucleation of QDs obtained by the so-called Stranski-Krastanov growth mode transition. It is shown that the AlN epilayer deposited on GaN exhibits different relaxation steps. A rather inefficient plastic relaxation first occurs in the early few nanometers. Then, as the growth proceeds cracks are generated to further release the elastic energy. However, Raman spectroscopy indicates that strain relaxation takes place only nearby the cracks. Consequently, a third relaxation process occurs as the AlN layer thickness still increases. It is characterized by V-shape pits at the AlN surface likely originating from opening of threading dislocation terminations. Atomic force microscopy and micro-photoluminescence studies indicate that these topological defects act as preferential nucleation centers competing with the GaN QD formation. (c) 2007 Elsevier B.V. All rights reserved.