Human 3D Pose Reconstruction
- mohamedabdulgafoor
- Dec 26, 2020
- 4 min read
Updated: Jan 7, 2021
Introduction
TASK-1
In this lab we are going to do a 3D reconstruction from multi-views of human pose on a calibrated system. OpenPose deep net has been used to estimate the joints of human body with a reliability score. There are 25 joints like depicted in the figure below;
We are going to use Structure From Motion (sfm) to compete the task. What does it mean by Structure from Motion? In computer vision, for a given displacement vector (also known as optical flow fields) from a moving camera, we have to estimate the sequence of camera poses and consequently reconstruct the 3D scene. For example, like in the figure below, there are several images that are taken from different perspective and reconstructed to create a 3D object.
The key idea is to trace the key points from a sequence of image data/movie. From that we can deduce the location of the cameras & the 3D coordinates of the key points. Then we can obtain the 3D geometry of the scene.
2D key-points matching
As we said there are 25 joints and the 2D coordinates of the output is in the form of;
xJ1 yJ1 rJ1 xJ2 yJ2 rJ2 xJ3 yJ3 rJ3....... xJ25 yJ25 rJ25
here xJ1 yJ1 are the x & y coordinates of the joints and the r is the reliability score. For example, following is the dataset of squat_1_0.0.txt. Here 333.318, 98.4255 are the x & y coordinate, and the 0.906826 is the reliability score.
This lab was run in the Google Colab. The following function was implemented to extract the 2D coordinate positions.
This function takes three argument. The location of the text file where the coordinate information are saved, video path and the name of the output file to be generated. For example, we can set the text & video paths as;
textPath = "/content/drive/MyDrive/ACV/LAB1/Lab1_3DReconstruction/data/OP2DTXT/Lea/" videoPath = "/content/drive/MyDrive/ACV/LAB1/Lab1_3DReconstruction/data/Videos/Lea/"
We also define another function to get the data. In our case this function will choose only the "squat" text data or related video files.
Finally, we will execute it as follow;
In the following video, we can see that the 2D points are embedded.
TASK-2
In this task let us use the configuration file of cameras to shift the origin of 3D joints from the reference zero camera to the another reference cameras, may be 1 or 2 and project the 3D joints back into the image. The below is the output of the origin shift matrix from 0 to 1. You can see this is a 3x4 matrix.
We can use this matrix along with the intrinsic parameter of the cameras, the 2D pose on view 2 for example, 3D pose after triangulation (reference at view 0), and retrieve the 3D points at view 0. The we can recall the origin shifting method which was define like below to shift the origin from view 0 to view 1or 1 to 2 etc.
Finally after the normalization, it can be project into 2D point on view2 (or any other), using the intrinsic of camera 2 (or any other). The below is a generated video by the script;
The video above shows the OpenPose (by big color circle) & the projected point after origin shifting with smaller red circle.
TASK-3
3D pose reconstruction
Now let us try to reconstruct the 3D skeleton. The following steps must be taken in order to reconstruct the 3D scene.
We must estimate the fundamental matrix from key points in two images
If we have the fundamental matrix and the camera intrinsic matrix, we can calculate the essential matrix.
If we have the essential matrix, we can calculate the rotation and the translation using single value decomposition (SVD) .
We can compute the projection matrices using the rotation matrices and the camera calibration matrices.
With all these information, we can cv2.triangulatePoints to get the 3D coordinates (world coordinates).
Finally we can use the bundle adjustment to improve the algorithm.
Just for fun let us try to visualize the 3D points that we get from the camera coordinate system. Let us project two different views in the 3D Cartesian axis. Note that this is not the actual reconstruction! We have simply projected the camera coordinates on to the 3D space.
We can see the two skeleton are simply shifted by a distance & also rotated.
The following code snippet helps us to reconstruct the 3D world coordinates.
The following video is a reconstructed one between the view 0 & view 1. In this reconstruction you can notice that I have used P01, when I reconstruct the P2 projection matrix.
But I faced some problems when I try to implement it using the following way in order to obtain the projection matrices. It appears it creates some kind of spherical aberration.
The following is the video that has been animated. Even though it is not very visible in the following video, it can be seen when we rotate the coordinate axis.
Because of the time limit, I could not implement the bundle adjustment.
References:
Comments