- Author: Turhan Can Kargın
- Topic: Master's Thesis
Please Click The Picture Below For the Project Video
- Create the environment from the
environment.yml
file:conda env create -f environment.yml
- Activate the new environment:
conda activate continuum-rl
- git clone
https://github.com/turhancan97/RL-based-Control-of-a-Soft-Continuum-Robot.git
- Run
cd RL-based-Control-of-a-Soft-Continuum-Robot/Tests
on the terminal - Run
python DDPG_keras_test.py
orpython DDPG_pytorch_test.py
on the terminal
Note: Another way is to create your own environment, activate the environment and run pip install -r requirements.txt
. Then continue from step 3 above.
This project was prepared for my thesis project in the Automatic Control and Robotics master program at Poznan University of Technology.
In this repository, a detailed formulation and explanation of the forward kinematics and velocity kinematics model for continuum robots is presented. In addition, reinforcement learning algorithm has been applied for the control of the three section continuum robot. Since the robot control problem is continuous, traditional algorithms of reinforcement learning cannot produce a solution. Deep Deterministic Policy Gradient (DDPG) algorithm, which is one of the reinforcement learning algorithms, has been chosen for the continuum robot control problem within the scope of this project, since it provides continuous state and continuous action space. Finally, the algorithm developed in Python programming language and some of the common frameworks such as OpenAI gym, Keras, Pytorch and Numpy.
Kinematics is a sub-branch of physics that deals with the motion of objects. It examines the position, velocity, acceleration and jerk values of rigid objects according to a reference coordinate system. Kinematics is not concerned with the forces and moments that cause motion; that is studied in dynamics. In this project, forward and velocity kinematics calculations were made for continuum robots using the constant curvature method. The inverse kinematics of the continuum robot is not calculated. Forward and inverse kinematics of continuum robot can be seen in Figure below.
X. Wang, Y. Li, and K. Kwok. A survey for machine learning-based control of continuum robots. Frontiers in Robotics and AI, 2021In order to calculate the forward and velocity kinematics of the three-section continuum robot, we employed the methods of (Hannan and Walker, 2003). This section will focus on the discussion of forward kinematics.
The authors utilized a modified Denavit-Hartenberg method to perform forward kinematic analysis of continuum robots. This method is widely recognized and utilized. Unlike the original D-H approach, which assumes each joint is independent, the modified D-H approach links three motions by curvature.
Therefore, using the D-H method, the homogeneous transformation matrix for the curve from frame 0 to frame 3 can be written in terms of the curvature
The frames for a three section continuum robot can be setup as a combination of three sections. So, the transformation matrix from frame 0 to frame 9 is:
The previously undefined matrix elements are given as:
where
In Figure below, you can see the simulated forward kinematics model of our robot with
The velocity kinematics of a continuum robot can be expressed in a way that is similar to conventional kinematic analysis (Spong and Vidyasagar, 1989).
For our three-section planar robot, the task space vector
The matrix
If we apply the chain rule, we obtain the following equation:
Let's apply this to our case, as shown in the equation below:
The red elements in the equation above represent the elements of our Jacobian Matrix. To define the Jacobian Matrix, we can use the equation below:
As a result, the final form of equation is:
Jacobian matrices are a useful tool in robotics and control theory that explain the dynamic relationship between different system representations, such as position and curvature derivatives. In velocity kinematics, these matrices are used to describe the highly constrained motion of the end-effector (tip of the continuum robot).
In the algorithm described above, we applied it to depict the motion of the continuum robot. You can view the simulated velocity kinematics motion of our robot below by applying the algorithm. It portrays solely the motion of the continuum robot’s tip point. For this project, we utilized forward and velocity kinematics models to design a reinforcement learning environment that describes the three section continuum robot. The next section will delve into the specifics of the environment designed for reinforcement learning.
We simulated a continuum robot with a section length of 0.1
Our system's aim is to take the three segment continuum robot from a random starting point to a random target by using the forward kinematics and velocity kinematics to describe the continuum robot.
-
$x-y$ : cartesian coordinates of the robot's tip point$m$ . We don't have z axis since planar case for continuum robot was considered. -
$\kappa$ : Curvatures$\frac{1}{m}$ . -
$\dot{\kappa}$ : Derivative of curvatures$\frac{1}{\frac{m}{s}}$ . -
$l$ - section length$[m]$
Action refers to the artificial agent selecting an option from a provided list of actions. In this thesis, the agent's action is represented by a data structure consisting of an ndarray
with a shape of (3,)
that represents the curvature derivative of each segment. While the agent is navigating the environment, it will generate three different values for the rate of change of curvature that are continuous at each step. Based on the rate of change, the robot must move towards the intended target.
The state that is being designed for this project includes the current x and y coordinates of the robot in the cartesian system, as well as the x and y coordinates of the target point.
Task Space of three section continuum robotTo finalize the environment, we need to design the reward. The state space has been intentionally limited to match the robot's task space (check Figure above), making it easier to describe the reward function. We define the Euclidean distance to
The reward can be defined in different ways to showcase its significant impact on the learning process. The first reward is given as:
The second reward function is defined as:
The third reward is calculated based on the weighted distance:
The final reward is:
Some of the results can be seen in this section. Each demonstration has a different reward function implemented.
This project discusses the use of reinforcement learning algorithms to solve the control problem of continuum robots, which have high degrees of freedom and unmodelable dynamics. The superiority of reinforcement learning over classical model-based approaches is that the reward function is created specifically for the problem. The project focuses on the curvature control problem of the continuum robot, which is solved using the Deep Deterministic Policy Gradient Algorithm, a model-free deep reinforcement learning algorithm.
- Add different reward functions than distance based.
- Take into account the robot's orientation.
- For high-level control, the problem of the robot creating a path planning in the environment by adding obstacles to its environment can be solved with this algorithm.
- 2D moving target point can be designed and tracked.
- 3D position reaching problem can be solved.
- 3D position tracking problem can be solved.
- Try different RL algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO).
- A real continuum robot can be designed and the same algorithm can be tested on this robot.
You can check my thesis report, presentation and LaTeX files in here.
You can also click on the image below if you want to see my thesis defense presentation video.
** Please feel free to contribute to this project. **