More info: Visual ball tracking

Note

Page is only useful to persons having access to the robot lab in the Max Planck Institute for Intelligent Systems

Operators

Description

The robot lab is equipped with four RGB cameras, which can be used to track the (3D) position of a table tennis ball. For the server, the software used for this is tennicam which has been developed internally by Sebastian Gomez-Gonzalez. The desktop “rodau” is installed with the server software, and the cameras are plugged to it, ready to be used. For the client, the software used is tennicam_client, which implements an o80 standalone. The client is install along the PAM software, as described here.

How-to

How to start the server?

  • Turn on the “bright” light above the table tennis.

  • Login to rodau using the ball_tracking user. The password is written on the desktop.

  • Open a terminal.

  • Call tennicam_start. Two terminals should open. If they do not show any error message, the tracking is active.

How to fix the server?

If you experience any issue, you can try to run in a terminal:

  • tennicam_list_cameras: check if the drivers detect the four cameras

  • tennicam_ping_cameras: check if the desktop can ping the four cameras

  • tennicam_capture_indep*: check if the desktop can use the drivers to capture pictures from the cameras

  • tennicam_capture*: check if the desktop can use the drivers of the cameras to capture synchronized pictures

* The “capture” commands only work if the tennicam tracker server (started by tennicam_start) is not running. If it is running, you need to stop it first.

How to start the client?

The client can be started on any Ubuntu desktop installed with the PAM software. In a first terminal (you may use the default parameters):

tennicam_client

Optionally, if you want to see a log of the ball position as received from the server, in another terminal:

tennicam_client_print

In your python code, you may then use an o80 frontend to access ball informations.

import tennicam_client

TENNICAM_CLIENT_DEFAULT_SEGMENT_ID = "tennicam_client"

frontend = tennicam_client.FrontEnd(TENNICAM_CLIENT_DEFAULT_SEGMENT_ID)
obs = frontend.latest()
ball = obs.get()
print(ball.to_string())

# other useful methods:
# position = obs.get_position()
# velocity = obs.get_velocity()
# iteration = obs.get_iteration()
# time_stamp = obs.get_time_stamp()
# ball_id = obs.get_ball_id()

For an example, see the source code of tennicam_client_print here.

How to display the tracked ball in MuJoCo?

In a terminal:

pam_mujoco tennicam_client_display
# or: pam_mujoco_no_xterms tennicam_client_display

and in another terminal:

tennicam_client_display

A MuJoCo simulation will start, displaying the ball.

Vicon

To set table pose in the visualisation based on information provided by the Vicon system, run tennicam_client_display with the --vicon flag. Note that in this case vicon_o80_standalone needs to be run as well (see More info: Vicon).

To stop the mujoco simulation, type in any terminal pam_mujoco_stop tennicam_client_display.

How to fix the transform of the ball?

The ball is detected in a given frame, and then its position goes through a transformation (translation and rotation). The transformation applied is specified in the configuration file ~/.mpi-is/pam/tennicam_client/config.toml (or /opt/mpi-is/pam/tennicam_client/config.toml) which has, for example, the content:

[transform]
translation = [0.5, 0.5, 0]
# rotation is given as extrinsic xyz-Euler angles in radian
rotation = [0.2, 0, 0]
[server]
hostname = "rodau"
port = 7660

To update the parameters of the transform:

  • when you start tennicam_client, use the dialog to set active_transform to True. Note that this slow down the client, so set active_transform to True only when tuning the transform.

  • start tennicam_client_display as described above.

  • in yet another terminal, run:

tennicam_client_transform_update

which will start a dialog allow you to tune transform. The updated transform is applied to the active client, so may see the result in the mujoco simulation popped up by tennicam_client_display.

The dialog allows you to save the transform in the configuration file (i.e. to overwrite the config.toml). This ensure the next time tennicam_client is started (without active_transform set to True), the desired transform is applied.

How to log ball information?

After starting tennicam_client, start in another terminal:

tennicam_client_logger

The dialog will propose to you to save data in an not-already-existing file. The dump of the data in the file starts as soon as the logger start. ctrl+c to stop the logging.

Once ball information has been dumped into a file, it is possible to replay the recorded ball behavior into a mujoco simulation.

In a first terminal:

pam_mujoco tennicam_client_replay

In another terminal:

tennicam_client_replay

A dialog will allow you to enter the path to the file to replay.

To parse a dumped file, you may use the parser:

import tennicam_client
for ball_info in tennicam_client.parse(path_to_file):
    ball_id,time_stamp,position,velocity = ball_info

See the source code of tennicam_client_replay for an example.

Installation of the server

In case the installation on rodau has been compromized, and you need to reinstall.

Basics requirements

  • Ubuntu 20.04

  • you must be part of the Intelligent Soft Robots GitHub domain

  • your SSH key must be set on GitHub

Requirement: Cuda 10.0

If you are at MPI-IS, add to /etc/bash.bashrc:

echo "- setup CUDA 10.0 to /is/software/nvidia/cuda-10.0/lib64"
export LD_LIBRARY_PATH=/is/software/nvidia/cuda-10.0/lib64
export PATH=$PATH:/is/software/nvidia/cuda-10.0/bin
export CUDA_HOME=/is/software/nvidia/cuda-10.0

Then open a new terminal or source:

source /etc/bash.bashrc

Alternatively, to install Cuda:

sudo apt install -y cuda-libraries-dev-10-1 cuda-misc-headers-10-1 cuda-minimal-build-10-1

Clone the treep project

Install tree:

pip3 install treep

and clone the treep configuration:

mkdir ~/Tennicam # or any other directory of your choice
cd ~/Tennicam
git clone git@github.com:intelligent-soft-robots/treep_isr.git
treep --clone BALL_TRACKING

Install additional dependencies

cd ~/Tennicam/workspace/src/tennicam/
sudo ./install_dependencies

As, among other things, this compiles OpenCV, this runs for a while

Compile

cd ~/Tennicam/workspace
colcon build

Source the workspace

Add to ~/.bashrc:

echo "-sourcing tennicam"
source ~/Tennicam/workspace/setup.bash

and start a new terminal

Global installation of configuration files

Tennicam requires some configuration file. To install them:

cd ~/Tennicam/workspace/src/tennicam/extras
sudo ./install_config