This repo contains the official PyTorch implementation of our paper:

GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras
Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, Jan Kautz
CVPR 2022 (Oral)
website | paper | video

• [08/10/22]: Demos for multi-person videos are added (Thanks to Haofan Wang)!
• [08/01/22]: Demos for dynamic and static videos are released!

• Installation
• Demo
  • Dynamic Videos
  • Static Videos
  • Multi-Person Videos
• Datasets
• Motion Infiller
• Evaluation
• Trajectory Predictor
• Joint Motion Infiller and Trajectory Predictor
• Citation

• Tested OS: MacOS, Linux
• Python >= 3.7
• PyTorch >= 1.8.0
• HybrIK (used in demo)

1. Clone this repo recursively:

   git clone --recursive https://github.com/NVlabs/GLAMR.git
   

   This will fetch the submodule HybrIK.
2. Follow HybrIK's installation instructions and download its models.
3. Install PyTorch 1.8.0 with the correct CUDA version.
4. Install system dependencies (Linux only):

   source install.sh
   

5. Install python dependencies:

   pip install -r requirements.txt
   

6. Download SMPL models & joint regressors and place them in the data folder. You can obtain the model following SPEC's instructions here.
7. Download pretrained_w_cam.pth from Google Drive

• You can download third-party pretrained models from Google Drive or BaiduYun.
• Once the glamr_models.zip file is downloaded, unzipping it will create the results folder:

  unzip glamr_models.zip
  

  Note that the pretrained models directly correspond to the config files for the motion infiller and trajectory predictor.

We provide demos for single- and multi-person video with both dynamic and static cameras.

Run the following command to test GLAMR on a single-person video with dynamic camera:

python global_recon/run_demo.py --cfg glamr_dynamic \
                                --video_path assets/dynamic/running.mp4 \
                                --out_dir out/glamr_dynamic/running \
                                --save_video

This will output results to out/glamr_dynamic/running. Results videos will be saved to out/glamr_dynamic/running/grecon_videos. Additional dynamic test videos can be found in assets/dynamic. More video comparison with HybrIK are available here.

Run the following command to test GLAMR on a single-person video with static camera:

python global_recon/run_demo.py --cfg glamr_static \
                                --video_path assets/static/basketball.mp4 \
                                --out_dir out/glamr_static/basketball \
                                --save_video

This will output results to out/glamr_static/basketball. Results videos will be saved to out/glamr_static/basketball/grecon_videos. Additional static test videos can be found in assets/static. More video comparison with HybrIK are available here.

Use the --multi flag and the glamr_static_multi config in the above demos to test GLAMR on a multi-person video:

python global_recon/run_demo.py --cfg glamr_static_multi \
                                --video_path assets/static/basketball.mp4 \
                                --out_dir out/glamr_static_multi/basketball \
                                --save_video \
                                --multi

This will output results to out/glamr_static_multi/basketball. Results videos will be saved to out/glamr_static_multi/basketball/grecon_videos.

We use three datasets: AMASS, 3DPW, and Dynamic Human3.6M. Please download them from the official website and place them in the dataset folder with the following structure:

${GLAMR_ROOT}
|-- datasets
|   |-- 3DPW
|   |-- amass
|   |-- H36M

First, run GLAMR on the test set of the dataset you want to evaluate. For example, to run GLAMR on the 3DPW test set:

python global_recon/run_dataset.py --dataset 3dpw --cfg glamr_3dpw --out_dir out/3dpw

Next, evaluate the results generated by GLAMR:

python global_recon/eval_dataset.py --dataset 3dpw --results_dir out/3dpw

Similarly, to evaluate on Dynamic Human3.6M, you can replace the 3dpw to h36m for the dataset and config.

The following command processes the original AMASS dataset into a processed version used in the code:

python preprocess/preprocess_amass.py

The following command processes the original 3DPW dataset into a processed version used in the code:

python preprocess/preprocess_3dpw.py

Please refer to this doc for generating the Dynamic Human3.6M dataset.

To train the motion infiller:

python motion_infiller/train.py --cfg motion_infiller_demo --ngpus 1

where we use the config motion_infiller_demo.

To visualize the trained motion infiller on test data:

python motion_infiller/vis_motion_infiller.py --cfg motion_infiller_demo --num_seq 5

where num_seq is the number of sequences to visualize. This command will save results videos to out/vis_motion_infiller.

To train the trajectory predictor:

python traj_pred/train.py --cfg traj_pred_demo --ngpus 1

where we use the config traj_pred_demo.

To visualize the trained trajectory predictor on test data:

python traj_pred/vis_traj_pred.py --cfg traj_pred_demo --num_seq 5

where num_seq is the number of sequences to visualize. This command will save results videos to out/vis_traj_pred.

For ease of use, we also define a joint (wrapper) model of motion infiller and trajectory predictor, i.e., the model merges the motion infilling and trajectory prediction stages. The joint model composes of pretrained motion infiller and trajectory predictor and is just a convenient abstraction. We can define the joint model using config files such as joint_motion_traj_demo. The joint model will also be used in the global optimization stage.

To visualize the joint model's results:

python motion_infiller/vis_motion_traj_joint_model.py --cfg joint_motion_traj_demo --num_seq 5

where num_seq is the number of sequences to visualize. This command will save results videos to out/vis_motion_traj_joint_model.

If you find our work useful in your research, please cite our paper GLAMR:

@inproceedings{yuan2022glamr,
    title={GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras},
    author={Yuan, Ye and Iqbal, Umar and Molchanov, Pavlo and Kitani, Kris and Kautz, Jan},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2022}
}

Please see the license for further details.