As part of an ongoing program to study $Sp(2N)$ gauge theories as potential realisations of composite Higgs models, we consider the case of $Sp(4)$ on the lattice, both as a pure gauge theory, and with two Dirac fermion flavors in the fundamental representation. In order to compare results between these two cases and maintain control of lattice artefacts, we make use of the gradient flow to set the scale of the simulations. We present preliminary results for the scale setting in the two cases, and also include results for the topological charge history.