Key performance indicators (KPIs) are widely used as evaluation indices of plant alarm systems in chemical industries. However, it has been pointed out that KPIs are not effective for evaluation of a system's qualitative characteristics. In this study, we propose a novel method for evaluating the alarm system performance from the viewpoints of relevance, uniqueness, and timeliness. This method does not depend on the knowledge and experience of engineers. First, similarities between all event pairs are first calculated by event correlation analysis of the plant operation log data. Then, the three characteristics are evaluated according to similarities and time lags between all combinations of alarm and operation events. The proposed method was applied to the plant operation log data of an azeotropic distillation column. The evaluation results were helpful for finding weak points in the objective plant alarm system. Using the proposed method, we can effectively improve the qualitative characteristics of alarm systems in chemical industries.