Tips: (v1.7.0b and older ver) Using Hand Tracking
NOTE
This page is about hand tracking supported in v1.7.0b and older version.
If you have v1.8.0 or later, please see the latest Hand Tracking page instead.
The hand tracking system in VMagicMirror runs with low CPU cost, but there are some limitations.
There also exists some preferred environments for the stability.
Limitations
This system does not aim to capture the precise hand position.
The finger state check is rough and limited to detect “rock, paper, scissors” pattern.
The orientation to move hand is up-down or left-right. Forward and backward motion is not detected.
Recommended Environments
Light up the room so your face and hands are captured clearly from the web camera.
Please put away the objects behind you whose colors are similar to your skin color.
For the clothing, please avoid to expose your shoulders and elbows.
Also the clothing with different color from your skin color is preferred.
Cf: How the Hand Tracking Works
In VMagicMirror, the hand tracking system is realized with following steps, on the base of face tracking technology. You do not have to understand the process perfectly, but it would help to understand the background of limitations.
Step1: Detect face area with deep neural network system.
Step2: Get the mean color value at the center of face area, to estimate user’s skin color.
Step3: In the left and right area of the face, try to pick up the similar color area (Blob).
Step4: If the blob area is enough large, then treat them as hand area.
Step5: Check the shape of the blob to count stretched fingers.
Step6: From the Step4 and Step5, the system get the hand area size and finger stretching count, so it can estimate the hand shape.
The following tweet shows the image.
#VMagicMirror
— 獏星(ばくすたー) / Megumi Baxter (@baku_dreameater) March 10, 2020
ひじょーーーーーーに今更なんですが、画像ベースのハンドトラッキング機能を作成中です。
ウェブカメラのみを使って顔検出とセットで動く機能で、CPU負荷が低いのがポイントです。
手の向きとかグーチョキパーの反映くらいまで出来るようになったらリリースしたいな~という感じです pic.twitter.com/QWOhDRbDYG
The black painted area is detected face area. This black paint is a pre-process to prevents the incorrect hand detection from neck or other skin colored areas around the face.
Pink circles on the hand means the area between fingers detected. This part shows that, the detection is done though it is not so highly precised.