In this paper, the vibrational behavior of double-walled carbon nanotubes (DWCNTs) is studied by a nonlocal elastic shell model. The nonlocal continuum model accounting for the small scale effects encompasses its classical continuum counterpart as a particular case. Based upon the constitutive equations of nonlocal elasticity, the displacement field equations coupled by van der Waals forces are derived. The set of governing equations of motion are then numerically solved by a novel method emerged from incorporating the radial point interpolation approximation within the framework of the generalized differential quadrature method. The present analysis provides the possibility of considering different combinations of layerwise boundary conditions. The influences of small scale factor, layerwise boundary conditions and geometrical parameters on the mechanical behavior of DWCNTs are fully investigated. Explicit expressions for the nonlocal frequencies of DWCNTs with all edges simply supported are also analytically obtained by a nonlocal elastic beam model. Some new intertube resonant frequencies and the corresponding noncoaxial vibrational modes are identified due to incorporating circumferential modes into the shell model. A shift in noncoaxial mode numbers, not predictable by the beam model, is also observed when the radius of DWCNTs is varied. The results generated also provide valuable information concerning the applicability of the beam model and new noncoaxial modes affecting the physical properties of nested nanotubes.