We studied the effect of nonaffine displacement of micellar junctions on the linear viscoelasticity of transient networks formed with telechelic associating polymers. For a junction of small aggregation number, the recoil motion is large when a hydrophobic end chain dissociates from it and, hence, leads to large nonaffine fluctuations. We show that diffusive motion around the mean position of the junction gives rise to nonsymmetric deviation of the loss modulus from that of Maxwell fluid with a single relaxation time. We calculate the high-frequency shoulder and the degree of low-frequency softening in the loss modulus as functions of the parameter D beta(-1)(0)/na(2), the mean radius of fluctuations during the end-chain lifetime divided by the mean-square end-to-end distance of a bridge chain. The degree of softening in the storage modulus at low frequencies is also calculated. We find that nonlinearity in the tension-elongation curve of a chain significantly affects the linear viscoelasticity due to fluctuations of the end-to-end distance.