Drilling faces many complex design and multiobjective optimization problems. Solving these problems is also a critical and complicated part of drilling optimization as part of well trajectory design and optimization. Many researchers have developed many algorithms, but they have some disadvantages. We take the shortest total borehole length, the highest target shooting accuracy, the lowest cost, and the minimum friction as the multiobjective function, and we use a fish swarm algorithm for trajectory optimization. In this paper, we present the idea of using a nondominant relation for sorting in the algorithm and we also use an optimization program in the Matlab software to obtain all numerical solutions satisfying the constraints. Therefore it is quite adaptable for introducing the idea of nondominant sorting into appropriate multiobjective optimization problems based on a fish swarm algorithm. We give an example of the calculation, and also show that the algorithm and the calculation procedure are accurate and reliable. The algorithm has a simple structure, a small number of calculations, and good convergence.