A rectangular prism has a base area of $24$ cm$^2$ and a total surface area of $88$ cm$^2$. If the base is $4$ cm by $6$ cm, what is the height?