Asymmetric arrangement of metal atoms is crucial for understanding the chirality origin of chiral metal nanoclusters and facilitating the design and development of new chiral catalysts and chiroptical devices. Here, we describe the construction of four asymmetric gold and gold silver clusters by chirality transfer from diimido ligands. The acquired metal clusters show strong circular dichroism (CD) response with large anisotropy factors of up to 6 X 10(-3), larger than the values of most reported chiral gold nanoclusters. Regardless of the same absolute configuration of the applied three diimido ligands, sigmoidal and reverse-sigmoidal arrangements of gold atoms both can be achieved, which resultantly produce an opposite Cotton effect within a specific absorption range. On the basis of the detailed structural characterization via X-ray crystallography and contrast experiments, the chirality contribution of the imido ligand, the asymmetrically arranged metal cluster, and the chiral arrangement of aromatic rings of phosphine ligands have been qualitatively evaluated. Time-dependent DFT calculations reveal that the chiroptical property of the acquired metal clusters is mainly influenced by the asymmetrically arranged metal atoms. Correlation of asymmetric arrangements of metal atoms in clusters with their chiroptical response provides a viable means of fabricating a designable chiral surface of metal nanoclusters and opens a broader prospect for chiral cluster application.