Abstract This article demonstrates the relationship between a decrease in energy efficiency and the load-carrying capacity of worm gears with an increase in the amplitude of torque oscillations as the coupling load increases. An analysis is conducted of the causes of the increase in the amplitude of torque oscillations with increasing gear load. It is shown that the disruption of the self-oscillation mode occurring in a worm gear should occur due to an increase in the friction coefficient, which depends on the lubrication conditions, in particular, on the thickness of the lubricant film. The study demonstrates that in the case of lubrication of the gear with standard mineral oil with a nano-modified additive, the loss of stability of self-oscillations occurs at significantly higher loads (290 N m) than in the case of lubrication with standard mineral oil without the additive (230 N m), indicating greater resistance of this option to disturbing effects and can be explained by the greater thickness of the lubricant film. The study obtained graphs of the dependence of the worm gear efficiency on the braking torque for various lubricant options. It is shown that the increase in efficiency when using standard mineral oil with a nanomodified additive (a nanodispersed suspension of serpentine in a solution of fatty acid salts) compared to standard mineral oil is associated with improved antifriction properties of the coupling and a reduction in the amplitude of torque oscillations. The study demonstrates that the stability of a worm gear under load can be experimentally assessed by measuring the amplitude of torque oscillations on the high-speed shaft of the gear. To expand the range of permissible loads, it is necessary to improve the antifriction properties of the lubricant by using a nanomodified additive. This leads to a reduction in the amplitude of self-oscillations of the worm shaft, a reduction in friction path, and a reduction in power losses.