Bilinear systems emerge in a wide variety of fields as natural models for dynamical systems ranging from robotics to quantum dots. Analyzing controllability of such systems is of fundamental and practical importance, for example, for the design of optimal control laws, stabilization of unstable systems, and minimal realization of input-output relations. Tools from the Lie theory have been adopted to establish controllability conditions for bilinear systems, and the most notable development was the Lie algebra rank condition (LARC). However, the application of the LARC may be computationally expensive for high-dimensional systems. In this article, we present an alternative and effective algebraic approach to investigate controllability of bilinear systems. The central idea is to map Lie bracket operations of the vector fields governing the system dynamics to permutation multiplications on a symmetric group, so that controllability and the controllable submanifold can be characterized by permutation orbits. The method is further shown to be applicable to characterize controllability of systems defined on undirected graphs, such as multiagent systems with controlled couplings between agents and Markov chains with tunable transition rates between states, which, in turn, reveals a graph representation of controllability through graph connectivity.