Get Started
Let’s extend our default Guru Schema to counts reps and gives feedback on squats during a workout. You can clone this template here and run this app in your console.
We’ll walk through the code below:
Imports
First, let’s import some common utilities and data types from the standard Guru library guru/stdlib. These will help us with our video analysis.
import {Color, Keypoint, MovementAnalyzer, Position} from "guru/stdlib";
Constructor
Let’s initalize the state we want to track throughout the video, such as the frames that contain a person (personFrames), the reps data for when squats begin and end (reps), and the angles between the hip and knee joints so we can analyze the quality of the movement (hipKneeAngles).
We can do this with the constructor field in GuruSchema.
constructor() {
this.personFrames = [];
this.reps = [];
this.hipKneeAngles = [];
}
Process Video
Now we can start writing the code that applies our AI models and processes each frame. This logic goes into processFrame. The code below:
- Identifies objects labeled as “person” within the frame.
- Stores the first identified person in the personFrames array.
- Uses the MovementAnalyzer to determine repetitions based on keypoint distances.
- Calculates the angle between the right knee and right hip keypoints for each repetition.
- Returns the calculated outputs.
async processFrame(frame) {
const people = await frame.findObjects("person");
const person = people[0];
this.personFrames.push(person);
this.reps = MovementAnalyzer.repsByKeypointDistance(this.personFrames, Keypoint.rightHip, Keypoint.rightAnkle);
this.hipKneeAngles = this.reps.map((rep) => {
return MovementAnalyzer.angleBetweenKeypoints(rep.middleFrame, Keypoint.rightKnee, Keypoint.rightHip);
});
return this.outputs();
}
Render
Now that we’ve extracted information from the frames using AI, we’ll want to analyze that information and render it onto the video. To do this, we’ll use renderFrame. We’ll transform the state we’ve collected in processFrame with standard Javascript functions and use guru/stdlib functions to draw on the frame. The code below:
- Checks if there are any frames with a person identified.
- Draws bounding boxes and skeletons for the identified person.
- Displays a triangle between the right knee and right hip keypoints.
- Identifies and displays the current rep.
- Displays the hip-knee angle value. If the angle indicates bad form (less than -2°), a feedback message is shown to get the hips lower.
renderFrame(frameCanvas) {
if (this.personFrames.length > 0) {
frameCanvas.drawBoundingBox(this.personFrames, new Color(93, 236, 201));
frameCanvas.drawSkeleton(this.personFrames, new Color(97, 49, 255), new Color(255, 255, 255));
const person = this.personFrames.find((frameObject) => frameObject.timestamp >= frameCanvas.timestamp);
const kneeLocation = person.keypoints[Keypoint.rightKnee];
const hipLocation = person.keypoints[Keypoint.rightHip];
frameCanvas.drawTriangle(
kneeLocation,
hipLocation,
new Position(hipLocation.x, kneeLocation.y),
{
backgroundColor: new Color(93, 236, 201),
alpha: 0.75
}
);
if (this.reps && this.reps.length > 0) {
let repIndex = this.reps.findIndex((rep) => {
return frameCanvas.timestamp >= rep.startFrame.timestamp && frameCanvas.timestamp <= rep.endFrame.timestamp;
});
if (frameCanvas.timestamp > this.reps[this.reps.length - 1].endFrame.timestamp) {
repIndex = this.reps.length - 1;
}
if (repIndex >= 0) {
frameCanvas.drawText(`Rep ${repIndex + 1}`, new Position(0.1, 0.1), new Color(255, 255, 255), {
fontSize: 36,
backgroundColor: new Color(94, 49, 255),
padding: 4
});
const rep = this.reps[repIndex];
const repAlpha = 1.0 - Math.abs(rep.middleFrame.timestamp - frameCanvas.timestamp) / (rep.endFrame.timestamp - rep.startFrame.timestamp);
const hipKneeAngle = Math.round(this.hipKneeAngles[repIndex]);
const badForm = hipKneeAngle < -2;
frameCanvas.drawText(
`${hipKneeAngle}°`,
kneeLocation,
new Color(255, 255, 255),
{
fontSize: 36,
alpha: repAlpha,
backgroundColor: badForm ? new Color(232, 92, 92) : new Color(94, 49, 255),
padding: 4
}
);
if (badForm) {
frameCanvas.drawText(
`Get your hips lower`,
hipLocation,
new Color(255, 255, 255),
{
fontSize: 18,
alpha: repAlpha,
backgroundColor: new Color(232, 92, 92),
padding: 4
}
);
}
}
}
}
}
Outputs
Finally, we can use the outputs method to specify what JSON output we want from our schema once the video is finished processing. In this case, we care about the number of reps and the hip-knee joint angle throughout the video.
async outputs() {
return {
reps: this.reps,
hipKneeAngles: this.hipKneeAngles,
}
}
Full Code
Putting it altogether, our GuruSchema looks like this.
import {Color, Keypoint, MovementAnalyzer, Position} from "guru/stdlib";
export default class GuruSchema {
constructor() {
this.personFrames = [];
this.reps = [];
this.hipKneeAngles = [];
}
async processFrame(frame) {
const people = await frame.findObjects("person");
const person = people[0];
this.personFrames.push(person);
this.reps = MovementAnalyzer.repsByKeypointDistance(this.personFrames, Keypoint.rightHip, Keypoint.rightAnkle);
this.hipKneeAngles = this.reps.map((rep) => {
return MovementAnalyzer.angleBetweenKeypoints(rep.middleFrame, Keypoint.rightKnee, Keypoint.rightHip);
});
return this.outputs();
}
renderFrame(frameCanvas) {
if (this.personFrames.length > 0) {
frameCanvas.drawBoundingBox(this.personFrames, new Color(93, 236, 201));
frameCanvas.drawSkeleton(this.personFrames, new Color(97, 49, 255), new Color(255, 255, 255));
const person = this.personFrames.find((frameObject) => frameObject.timestamp >= frameCanvas.timestamp);
const kneeLocation = person.keypoints[Keypoint.rightKnee];
const hipLocation = person.keypoints[Keypoint.rightHip];
frameCanvas.drawTriangle(
kneeLocation,
hipLocation,
new Position(hipLocation.x, kneeLocation.y),
{
backgroundColor: new Color(93, 236, 201),
alpha: 0.75
}
);
if (this.reps && this.reps.length > 0) {
let repIndex = this.reps.findIndex((rep) => {
return frameCanvas.timestamp >= rep.startFrame.timestamp && frameCanvas.timestamp <= rep.endFrame.timestamp;
});
if (frameCanvas.timestamp > this.reps[this.reps.length - 1].endFrame.timestamp) {
repIndex = this.reps.length - 1;
}
if (repIndex >= 0) {
frameCanvas.drawText(`Rep ${repIndex + 1}`, new Position(0.1, 0.1), new Color(255, 255, 255), {
fontSize: 36,
backgroundColor: new Color(94, 49, 255),
padding: 4
});
const rep = this.reps[repIndex];
const repAlpha = 1.0 - Math.abs(rep.middleFrame.timestamp - frameCanvas.timestamp) / (rep.endFrame.timestamp - rep.startFrame.timestamp);
const hipKneeAngle = Math.round(this.hipKneeAngles[repIndex]);
const badForm = hipKneeAngle < -2;
frameCanvas.drawText(
`${hipKneeAngle}°`,
kneeLocation,
new Color(255, 255, 255),
{
fontSize: 36,
alpha: repAlpha,
backgroundColor: badForm ? new Color(232, 92, 92) : new Color(94, 49, 255),
padding: 4
}
);
if (badForm) {
frameCanvas.drawText(
`Get your hips lower`,
hipLocation,
new Color(255, 255, 255),
{
fontSize: 18,
alpha: repAlpha,
backgroundColor: new Color(232, 92, 92),
padding: 4
}
);
}
}
}
}
}
async outputs() {
return {
reps: this.reps,
hipKneeAngles: this.hipKneeAngles,
}
}
}
