Carbon nanotube (CNT) reinforced composites have been drawing intense attentions of researchers due to their good mechanical and physical properties as well as potential applications. The diameter, as an important geometric parameter of CNTs, significantly affects the performance of CNTs in the reinforced composites, not only in a direct way but also in an indirect way by influencing the effective modulus and strength of reinforcing CNTs. This paper investigates the comprehensive effect of CNT diameter on the fracture toughness of CNT reinforced composites by accounting for both direct and indirect influences of CNT diameter based on the three-level failure analysis. The criteria for failure modes are established analytically, and the types of failure mode transition with the corresponding optimal CNT diameter are obtained. It is found that reducing CNT diameter can cause a sudden drop in fracture toughness of composites due to the transition of dominant failure mode. Therefore, the CNTs with smaller diameter do not definitely confer a better fracture toughness on their reinforced composites, and the optimal CNT diameter may exist in the transition between failure modes, especially from interfacial debonding to CNT break. In addition, according to the results, the failure mode of CNT break is suggested to be avoided in the composite design because of the low fracture toughness enhancement of CNTs in this mode. This study can provide guiding reference for CNT reinforced composite design.