Lipid molecules can bind to specific sites on integral membrane proteins, modulating their structure and function. We have undertaken coarse-grained simulations to calculate free energy profiles for glycolipids and phospholipids interacting with modulatory sites on the transmembrane helix dimer of the EGF receptor within a lipid bilayer environment. We identify lipid interaction sites at each end of the transmembrane domain and compute interaction free energy profiles for lipids with these sites. Interaction free energies ranged from ca. -40 to -4 kJ/mol for different lipid species. Those lipids (glycolipid GM3 and phosphoinositide PIP2) known to modulate EGFR function exhibit the strongest binding to interaction sites on the EGFR, and we are able to reproduce the preference for interaction with GM3 over other glycolipids suggested by experiment. Mutation of amino acid residues essential for EGFR function reduce the binding free energy of these key lipid species. The residues interacting with the lipids in the simulations are in agreement with those suggested by experimental (mutational) studies. This approach provides a generalizable tool for characterizing the interactions of lipids that bind to specific sites on integral membrane proteins.