As peak demand for electricity continues to rise, distributors have begun charging small and medium-sized users for their short term demand rather than just their energy use. This is not only to meet the political aspirations for increased demand-side flexibility that now exist in many corners of the world, but to make sure that users are charged for the costs they incur. As it is only until recently that this type of users have come to face demand charges, there are however very few studies on what the actual effects of such pricing policies are, and those studies that do exist suffer from different methodological shortcomings that reduce their validity as a basis for real-world policy evaluations. This study provides the first state-of-the-art causal analysis of the demand response effects of a billing demand charge involuntarily introduced to small and medium sized users (35-63 A), using novel two-level time series models on retrospective observational consumption and survey data. Our analyses suggest that the tariff has induced an average response of -0.32 kWh/day per user over a two year long posttreatment period in comparison to a matched control group, equal to 7.4% of their daily average use during the pretreatment period. The response seems to have increased over time and to be greater during wintertime: around -0.70 kWh/day or 16.2% of the treated users' average daily use during the pretreatment period. Comparing the individual users' response to the size of their financial incentive to respond given the new tariff as well as their self-reported perception of the relative importance of electricity expenditures, we did not find any support for the common assumption that users with a higher financial incentive to respond do so to a greater extent. This might suggest that small and medium-sized commercial users, just as residential users, may exhibit non-financial drivers and barriers for engaging in demand response that may be vital to understand as policy makers and industry continue to seek increased demand-side flexibility.