This paper presents an application of a linear quadratic stochastic differential game to a model in finance, which describes trading behaviors of different types of players in a high frequency stock market. Stability of the high frequency market is a central issue for financial markets. Building a model that expresses the trading behaviors of the different types of players and the price actions in turmoil is important to set regulations to maintain fair markets. Firstly, we represent trading behaviors of the three types of players, algorithmic traders, general traders, and market makers as well as the mid price process of a risky asset by a linear quadratic stochastic differential game. Secondly, we obtain a Nash equilibrium by solving a forward-backward stochastic differential equation (FBSDE) derived from the stochastic maximum principle. Finally, we present numerical examples of the Nash equilibrium and the corresponding price action of the risky asset, which agree with the empirical findings on trading behaviors of players in high frequency markets. This model can be used to investigate the impact of regulation changes on the market stability as well as trading strategies of the players. (C) 2019 Elsevier Ltd. All rights reserved.