Abstract

Reinforcement learning (RL) has potential to provide innovative solutions to existing challenges in estimating joint moments in motion analysis, such as kinematic or electromyography (EMG) noise and unknown model parameters. Here, we explore feasibility of RL to assist joint moment estimation for biomechanical applications. Forearm and hand kinematics and forearm EMGs from four muscles during free finger and wrist movement were collected from six healthy subjects. Using the proximal policy optimization approach, we trained two types of RL agents that estimated joint moment based on measured kinematics or measured EMGs, respectively. To quantify the performance of trained RL agents, the estimated joint moment was used to drive a forward dynamic model for estimating kinematics, which was then compared with measured kinematics using Pearson correlation coefficient. The results demonstrated that both trained RL agents are feasible to estimate joint moment for wrist and metacarpophalangeal (MCP) joint motion prediction. The correlation coefficients between predicted and measured kinematics, derived from the kinematics-driven agent and subject-specific EMG-driven agents, were 98% ± 1% and 94% ± 3% for the wrist, respectively, and were 95% ± 2% and 84% ± 6% for the metacarpophalangeal joint, respectively. In addition, a biomechanically reasonable joint moment-angle-EMG relationship (i.e., dependence of joint moment on joint angle and EMG) was predicted using only 15 s of collected data. In conclusion, this study illustrates that an RL approach can be an alternative technique to conventional inverse dynamic analysis in human biomechanics study and EMG-driven human-machine interfacing applications.

References

1.
Zajac
,
F. E.
,
1989
, “
Muscle and Tendon: Properties, Models, Scaling, and Application to Biomechanics and Motor Control
,”
Crit. Rev. Biomed. Eng.
,
17
(
4
), pp.
359
411
.https://pubmed.ncbi.nlm.nih.gov/2676342/
2.
Crouch
,
D. L.
, and
Huang
,
H.
,
2016
, “
Lumped-Parameter Electromyogram-Driven Musculoskeletal Hand Model: A Potential Platform for Real-Time Prosthesis Control
,”
J. Biomech.
,
49
(
16
), pp.
3901
3907
.10.1016/j.jbiomech.2016.10.035
3.
Holzbaur
,
K. R. S.
,
Murray
,
W. M.
, and
Delp
,
S. L.
,
2005
, “
A Model of the Upper Extremity for Simulating Musculoskeletal Surgery and Analyzing Neuromuscular Control
,”
Ann. Biomed. Eng.
,
33
(
6
), pp.
829
840
.10.1007/s10439-005-3320-7
4.
Delp
,
S. L.
,
Anderson
,
F. C.
,
Arnold
,
A. S.
,
Loan
,
P.
,
Habib
,
A.
,
John
,
C. T.
,
Guendelman
,
E.
, and
Thelen
,
D. G.
,
2007
, “
OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement
,”
IEEE Trans. Biomed. Eng.
,
54
(
11
), pp.
1940
1950
.10.1109/TBME.2007.901024
5.
Kuo
,
A. D.
,
1998
, “
A Least-Squares Estimation Approach to Improving the Precision of Inverse Dynamics Computations
,”
ASME J. Biomech. Eng.
,
120
(
1
), pp.
148
159
.10.1115/1.2834295
6.
Otten
,
E.
,
2003
, “
Inverse and Forward Dynamics: Models of Multi–Body Systems
,”
Philos. Trans. R. Soc. London. Ser. B Biol. Sci.
,
358
(
1437
), pp.
1493
1500
.10.1098/rstb.2003.1354
7.
Sartori
,
M.
,
Llyod
,
D. G.
, and
Farina
,
D.
,
2016
, “
Neural Data-Driven Musculoskeletal Modeling for Personalized Neurorehabilitation Technologies
,”
IEEE Trans. Biomed. Eng.
,
63
(
5
), pp.
879
893
.10.1109/TBME.2016.2538296
8.
Farina
,
D.
, and
Negro
,
F.
,
2012
, “
Accessing the Neural Drive to Muscle and Translation to Neurorehabilitation Technologies
,”
IEEE Rev. Biomed. Eng.
,
5
, pp.
3
14
.10.1109/RBME.2012.2183586
9.
Boostani
,
R.
, and
Moradi
,
M. H.
,
2003
, “
Evaluation of the Forearm EMG Signal Features for the Control of a Prosthetic Hand
,”
Physiolog. Meas.
,
24
(
2
), pp.
309
319
.10.1088/0967-3334/24/2/307
10.
Hoang
,
H. X.
,
Pizzolato
,
C.
,
Diamond
,
L. E.
, and
Lloyd
,
D. G.
,
2018
, “
Subject-Specific Calibration of Neuromuscular Parameters Enables Neuromusculoskeletal Models to Estimate Physiologically Plausible Hip Joint Contact Forces in Healthy Adults
,”
J. Biomech.
, 80, pp.
111
120
.10.1016/j.jbiomech.2018.08.023
11.
Lloyd
,
D. G.
, and
Besier
,
T. F.
,
2003
, “
An EMG-Driven Musculoskeletal Model to Estimate Muscle Forces and Knee Joint Moments In Vivo
,”
J. Biomech.
,
36
(
6
), pp.
765
776
.10.1016/S0021-9290(03)00010-1
12.
Garner
,
B. A.
, and
Pandy
,
M. G.
,
2003
, “
Estimation of Musculotendon Properties in the Human Upper Limb
,”
Ann. Biomed. Eng.
,
31
(
2
), pp.
207
220
.10.1114/1.1540105
13.
Wu
,
W.
,
Lee
,
P. V. S.
,
Bryant
,
A. L.
,
Galea
,
M.
, and
Ackland
,
D. C.
,
2016
, “
Subject-Specific Musculoskeletal Modeling in the Evaluation of Shoulder Muscle and Joint Function
,”
J. Biomech.
,
49
(
15
), pp.
3626
3634
.10.1016/j.jbiomech.2016.09.025
14.
Kober
,
J.
,
Bagnell
,
J. A.
, and
Peters
,
J.
,
2013
, “
Reinforcement Learning in Robotics: A Survey
,”
Int. J. Rob. Res.
, 32(11), pp.
1238
1274
.10.1177/0278364913495721
15.
Nguyen
,
H.
, and
La
,
H.
,
2019
, “
Review of Deep Reinforcement Learning for Robot Manipulation
,” Third IEEE International Conference on Robotic Computing (
IRC
), Naples, Italy, Feb. 25–27, pp.
590
595
.10.1109/IRC.2019.00120
16.
Kormushev
,
P.
,
Calinon
,
S.
, and
Caldwell
,
D. G.
,
2013
, “
Reinforcement Learning in Robotics: Applications and Real-World Challenges
,”
Robotics
,
2
(
3
), pp.
122
148
.10.3390/robotics2030122
17.
Silver
,
D.
,
Huang
,
A.
,
Maddison
,
C. J.
,
Guez
,
A.
,
Sifre
,
L.
,
Van Den Driessche
,
G.
,
Schrittwieser
,
J.
,
Antonoglou
,
I.
,
Panneershelvam
,
V.
,
Lanctot
,
M.
,
Dieleman
,
S.
,
Grewe
,
D.
,
Nham
,
J.
,
Kalchbrenner
,
N.
,
Sutskever
,
I.
,
Lillicrap
,
T.
,
Leach
,
M.
,
Kavukcuoglu
,
K.
,
Graepel
,
T.
, and
Hassabis
,
D.
,
2016
, “
Mastering the Game of Go With Deep Neural Networks and Tree Search
,”
Nature
,
529
(
7587
), pp.
484
489
.10.1038/nature16961
18.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Rusu
,
A. A.
,
Veness
,
J.
,
Bellemare
,
M. G.
,
Graves
,
A.
,
Riedmiller
,
M.
,
Fidjeland
,
A. K.
,
Ostrovski
,
G.
,
Petersen
,
S.
,
Beattie
,
C.
,
Sadik
,
A.
,
Antonoglou
,
I.
,
King
,
H.
,
Kumaran
,
D.
,
Wierstra
,
D.
,
Legg
,
S.
, and
Hassabis
,
D.
,
2015
, “
Human-Level Control Through Deep Reinforcement Learning
,”
Nature
,
518
(
7540
), pp.
529
533
.10.1038/nature14236
19.
Silver
,
D.
,
Schrittwieser
,
J.
,
Simonyan
,
K.
,
Antonoglou
,
I.
,
Huang
,
A.
,
Guez
,
A.
,
Hubert
,
T.
,
Baker
,
L.
,
Lai
,
M.
,
Bolton
,
A.
,
Chen
,
Y.
,
Lillicrap
,
T.
,
Hui
,
F.
,
Sifre
,
L.
,
Van Den Driessche
,
G.
,
Graepel
,
T.
, and
Hassabis
,
D.
,
2017
, “
Mastering the Game of Go Without Human Knowledge
,”
Nature
,
550
(
7676
), pp.
354
359
.10.1038/nature24270
20.
Kidziński
,
Ł.
,
Mohanty
,
S. P.
,
Ong
,
C.
,
Huang
,
Z.
,
Zhou
,
S.
,
Pechenko
,
A.
,
Stelmaszczyk
,
A.
,
Jarosik
,
P.
,
Pavlov
,
M.
,
Kolesnikov
,
S.
,
Plis
,
S.
,
Chen
,
Z.
,
Zhang
,
Z.
,
Chen
,
J.
,
Shi
,
J.
,
Zheng
,
Z.
,
Yuan
,
C.
,
Lin
,
Z.
,
Michalewski
,
H.
,
Miłoś
,
P.
,
Osiński
,
B.
,
Melnik
,
A.
,
Schilling
,
M.
,
Ritter
,
H.
,
Carroll
,
S.
,
Hicks
,
J.
,
Levine
,
S.
,
Salathé
,
M.
, and
Delp
,
S.
,
2018
, “
Learning to Run Challenge Solutions: Adapting Reinforcement Learning Methods for Neuromusculoskeletal Environments
,”
The NIPS'17 Competition: Building Intelligent Systems
, Springer, Berlin, pp.
121
153
.10.1007/978-3-319-94042-7_7
21.
Kidziński
,
Ł.
,
Mohanty
,
S. P.
,
Ong
,
C.
,
Hicks
,
J. L.
,
Carroll
,
S. F.
,
Levine
,
S.
,
Salathé
,
M.
, and
Delp
,
S. L.
,
2018
, “
Learning to Run Challenge: Synthesizing Physiologically Accurate Motion Using Deep Reinforcement Learning
,”
The NIPS'17 Competition: Building Intelligent Systems
, Springer, Berlin, pp.
101
120
.10.1007/978-3-319-94042-7_6
22.
Pavlov
,
M.
,
Kolesnikov
,
S.
, and
Plis
,
S. M.
,
2017
, “
Run, Skeleton, Run: Skeletal Model in a Physics-Based Simulation
,” arXiv preprint
arXiv:1711.06922
.https://arxiv.org/abs/1711.06922
23.
Wen
,
Y.
,
Si
,
J.
,
Brandt
,
A.
,
Gao
,
X.
, and
Huang
,
H.
,
2020
, “
Online Reinforcement Learning Control for the Personalization of a Robotic Knee Prosthesis
,”
IEEE Trans. Cybern.
,
50
(
6
), pp.
2346
2356
.10.1109/TCYB.2019.2890974
24.
Wen
,
Y.
,
Si
,
J.
,
Gao
,
X.
,
Huang
,
S.
, and
Huang
,
H. H.
,
2017
, “
A New Powered Lower Limb Prosthesis Control Framework Based on Adaptive Dynamic Programming
,”
IEEE Trans. neural Networks Learn. Syst.
,
28
(
9
), pp.
2215
2220
.10.1109/TNNLS.2016.2584559
25.
Jagodnik
,
K. M.
,
Thomas
,
P. S.
,
van den Bogert
,
A. J.
,
Branicky
,
M. S.
, and
Kirsch
,
R. F.
,
2017
, “
Training an Actor-Critic Reinforcement Learning Controller for Arm Movement Using Human-Generated Rewards
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
25
(
10
), pp.
1892
1905
.10.1109/TNSRE.2017.2700395
26.
Juliani
,
A.
,
Berges
,
V.-P.
,
Vckay
,
E.
,
Gao
,
Y.
,
Henry
,
H.
,
Mattar
,
M.
, and
Lange
,
D.
,
2018
, “
Unity: A General Platform for Intelligent Agents
,” arXiv preprint
arXiv:1809.02627
. https://arxiv.org/abs/1809.02627
27.
Ramachandran
,
P.
,
Zoph
,
B.
, and
Le
,
Q. V.
,
2017
, “
Swish: A Self-Gated Activation Function
,”
6th International Conference on Learning Representations
, Vancouver, BC, Canada, Apr. 30–May 3.https://arxiv.org/pdf/1710.05941v1.pdf?source=post_page
28.
Schulman
,
J.
,
Wolski
,
F.
,
Dhariwal
,
P.
,
Radford
,
A.
, and
Klimov
,
O.
,
2017
, “
Proximal Policy Optimization Algorithms
,” arXiv preprint
arXiv1707.06347
.https://arxiv.org/abs/1707.06347
29.
Formica
,
D.
,
Charles
,
S. K.
,
Zollo
,
L.
,
Guglielmelli
,
E.
,
Hogan
,
N.
, and
Krebs
,
H. I.
,
2012
, “
The Passive Stiffness of the Wrist and Forearm
,”
J. Neurophysiol.
,
108
(
4
), pp.
1158
1166
.10.1152/jn.01014.2011
30.
Delp
,
S. L.
,
Grierson
,
A. E.
, and
Buchanan
,
T. S.
,
1996
, “
Maximumisometric Moments Generated by the Wrist Muscles in Flexion-Extension and Radial-Ulnar Deviation
,”
J. Biomech.
,
29
(
10
), pp.
1371
1375
.10.1016/0021-9290(96)00029-2
31.
Pan
,
L.
,
Crouch
,
D. L.
, and
Huang
,
H.
,
2019
, “
Comparing EMG-Based Human-Machine Interfaces for Estimating Continuous, Coordinated Movements
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
27
(
10
), pp.
2145
2154
.10.1109/TNSRE.2019.2937929
32.
Pan
,
L.
,
Crouch
,
D. L.
, and
Huang
,
H.
,
2018
, “
Myoelectric Control Based on a Generic Musculoskeletal Model: Toward a Multi-User Neural-Machine Interface
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
26
(
7
), pp.
1435
1442
.10.1109/TNSRE.2018.2838448
33.
Chen
,
C.
,
Seff
,
A.
,
Kornhauser
,
A.
, and
Xiao
,
J.
,
2015
, “
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
,”
Proceedings of the IEEE International Conference on Computer Vision
, Santiago, Chile, Dec. 7–13, pp.
2722
2730
.10.1109/ICCV.2015.312
34.
Knutson
,
J. S.
,
Kilgore
,
K. L.
,
Mansour
,
J. M.
, and
Crago
,
P. E.
,
2000
, “
Intrinsic and Extrinsic Contributions to the Passive Moment at the Metacarpophalangeal Joint
,”
J. Biomech.
,
33
(
12
), pp.
1675
1681
.10.1016/S0021-9290(00)00159-7
35.
Quitadamo
,
L. R.
,
Cavrini
,
F.
,
Sbernini
,
L.
,
Riillo
,
F.
,
Bianchi
,
L.
,
Seri
,
S.
, and
Saggio
,
G.
,
2017
, “
Support Vector Machines to Detect Physiological Patterns for EEG and EMG-Based Human–Computer Interaction: A Review
,”
J. Neural Eng.
,
14
(
1
), p.
011001
.10.1088/1741-2552/14/1/011001
36.
Iqbal
,
N. V.
,
Subramaniam
,
K.
, and
P
,
S. A.
,
2018
, “
A Review on Upper-Limb Myoelectric Prosthetic Control
,”
IETE J. Res
,
64
(
6
), pp.
740
752
.10.1080/03772063.2017.1381047
37.
Scheme
,
E.
, and
Englehart
,
K.
,
2011
, “
Electromyogram Pattern Recognition for Control of Powered Upper-Limb Prostheses: State of the Art and Challenges for Clinical Use
,”
J. Rehabil. Res. Dev.
,
48
(
6
), pp.
643
660
.10.1682/JRRD.2010.09.0177
38.
Pradhan
,
A.
,
Jiang
,
N.
,
Chester
,
V.
, and
Kuruganti
,
U.
,
2020
, “
Linear Regression With Frequency Division Technique for Robust Simultaneous and Proportional Myoelectric Control During Medium and High Contraction-Level Variation
,”
Biomed. Signal Process. Control
,
61
, p.
101984
.10.1016/j.bspc.2020.101984
39.
Pan
,
L.
,
Harmody
,
A.
, and
Huang
,
H.
,
2018
, “
A Reliable Multi-User EMG Interface Based on a Generic-Musculoskeletal Model Against Loading Weight Changes
,” 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (
EMBC
), Honolulu, HI, July 18–21, pp.
2104
2107
.10.1109/EMBC.2018.8512685
40.
Hahne
,
J. M.
,
Schweisfurth
,
M. A.
,
Koppe
,
M.
, and
Farina
,
D.
,
2018
, “
Simultaneous Control of Multiple Functions of Bionic Hand Prostheses: Performance and Robustness in End Users
,”
Sci. Robot.
,
3
(
19
), p.
eaat3630
.10.1126/scirobotics.aat3630
You do not currently have access to this content.