Soft materials including elastomers and gels are widely used in applications of energy absorption, soft robotics, bioengineering, and medical instruments. For many soft materials subject to loading and unloading cycles, the stress required on reloading is often less than that on the initial loading, known as Mullins effect. Meanwhile, soft materials usually exhibit rate-dependent viscous behavior. Both effects were recently reported on a new kind of synthesized tough gel, with capability of large deformation, high strength, and extremely high toughness. In this work, we develop a coupled viscoelastic and Mullins-effect model to characterize the deformation behavior of the tough gel. We modify one of the elastic components in Zener model to be a damageable spring to incorporate the Mullins effect and model the viscous effect to behave as a Newtonian fluid. We synthesized the tough gel described in the literature (Sun et al., Nature 2012) and conducted uniaxial tensile tests and stress relaxation tests. We also investigated the two effects on three other soft materials, polyacrylate elastomer, Nitrile-Butadiene Rubber, and polyurethane. We find that our presented model is so robust that it can characterize all the four materials, with modulus ranging from a few tens of kilopascal to megapascal. The theory and experiment for all tested materials agree very well.