Replica exchange is a widely used sampling strategy in molecular simulation. While a variety of methods exist to optimize parameters for temperature replica exchange, less is known about how to optimize parameters for more general Hamiltonian replica exchange simulations. We present an algorithm for the online optimization of total acceptance for both temperature and Hamiltonian replica exchange simulations using stochastic gradient descent. We optimize the total acceptance, a heuristic objective function capturing the efficiency of replica exchange. Our approach is general and has several desirable properties, including: (1) it makes few assumptions about the system of interest, (2) optimization occurs online without the requirement of presimulation, and (3) most importantly, it readily generalizes to systems where there are multiple control parameters (e.g., temperatures, force constants, etc.) that determine the Hamiltonian of each replica. We explore some general properties of the algorithm on a simple harmonic oscillator system, and demonstrate its effectiveness on a more complex data-guided protein folding simulation.