Polycrystalline diamond films with systematic change in microstructure that varies from microcrystalline to nanocrystalline structure are synthesized on Si by hot filament chemical vapor deposition. The morphology and structural properties of the grown diamond films are analyzed using field emission scanning electron microscope (FESEM), atomic force microscope (AFM), X-ray diffraction and Raman spectroscopy. The average roughness and grain size of the diamond films decrease with increase in CH4 to H-2 ratio from 0.5 to 3%. Also, structural disorder in these diamond films increases with decrease in grain size as evidenced from Raman spectroscopy. The coefficient of friction (CoF) is found to be very low for all the films. However, the average CoF is found to increase from 0.011 +/- 0.005 to 0.03 +/- 0.015 as the grain size decrease from similar to 1 mu m down to similar to 20 nm. Post analysis of wear track by FESEM, AFM based nanoscale friction and Raman spectroscopy reveal that microcrystalline diamond undergoes shear induced amorphization with negligible wear rate while nanocrystalline diamond films undergo shear induced plastic deformation without amorphization. A comprehensive mechanism for the observed CoF is discussed in the framework of microstructure, structural disorder and shear induced tribo-chemical reactions at the sliding interface.