Real-time Reconstruction

Table of Contents

• Description
• Overview
• Local Development Package for Windows
• Docker Deployment for Linux
• Parameter Description
• Server Program realtime_service
• API Interface Description
• Client Development Guide

0 Description

Real-time reconstruction supports images collected from drone orthomosaic flight paths with overlap requirements of 70% along-track and 70% cross-track.

Recommended to use DJI mapping drones M3E/M4E.

🚁 Drone Orthomosaic Flight Path Diagram

Orthomosaic Flight Path Planning Points

• Path Spacing: Calculate based on camera field of view and flight altitude to ensure cross-track overlap
• Shooting Interval: Set based on flight speed and along-track overlap requirements
• Flight Altitude: Affects ground resolution and shooting range
• Weather Conditions: Choose uniform lighting, windless or light wind weather

1 Overview

Real-time reconstruction is provided as a gRPC server program: realtime_service. Starting it launches the real-time reconstruction service. We will provide a proto file for developers to develop client programs. The client program performs the following operations:

(1) Initialize - Initialize the real-time reconstruction service

(2) AddFrame - Add image frames

(3) EndTask - End the current real-time reconstruction task

Real-time reconstruction service is implemented based on gRPC communication protocol. It is recommended that developers have gRPC client development experience for smooth integration and calling of related interfaces.

Below is the basic process diagram of the real-time reconstruction service:

2 Local Development Package for Windows

Please contact MipMap for the local development package.

3 Docker Deployment for Linux

For environment preparation before deployment, please refer to: Linux Deployment Guide download image section.

Download Image

docker pull registry.mipmap3d.com/docker/mipmap/realtime:v1.0-ubuntu22.04

Start Container from Image

# Start Docker container
docker run -it --rm \
  -v /tmp:/tmp \             # Mount host directory /tmp to container [Required for license access, must execute and mount as per this command]
  -v /mnt:/mnt \             # Mount host directory to container [Example, adjust according to actual situation]
  --cpus 8 \                 # Limit container to use 8 CPUs
  --gpus all \               # Use all GPUs (requires NVIDIA Container Toolkit)
   --name mipmap_rt \            # Container name is mipmap_rt
  registry.mipmap3d.com/docker/mipmap/realtime:v1.0-ubuntu22.04 \
  /bin/bash                  # Start bash after entering container

Important Notes:

• /mnt directory mapping: mnt is an example, actual caller can flexibly mount any directory
• /tmp directory mapping: Required for license access, must be mounted
• --cpus 8: Limit container to use 8 CPU cores
• --gpus all: Enable all GPU support

Run Real-time Reconstruction Service

# Run real-time reconstruction service inside container
./mipmap_engine/realtime_service 

---

4 Parameter Description

Real-time reconstruction needs to be initialized with a JSON file. The parameter description table is as follows:

Field Name	Type	Required	Description
working_dir	string	Yes	Working directory, saves intermediate files and final result files. UTF-8 encoding
gdal_folder	string	Yes	GDAL data path, MipMap provided gdal_data directory (movable), must be English directory
do_realtime_2d_map	bool	No	Whether to calculate real-time 2D map, i.e., 2D orthomosaic map
do_realtime_3d_map	bool	No	Whether to calculate real-time 3D map, i.e., 3D mesh (currently available on Windows)
target_max_size	int	No	Maximum side pixel count after image frame scaling. Real-time tasks recommend scaling image frames. This value controls the scaled image size. Larger values provide better real-time reconstruction results but longer single frame processing time. When 0, caller scales externally and passes image metadata string (see details below)
rt_mesh_save_type	int	No	Default 0
rt_mesh_reso_scale_to_gsd	int	No	Real-time 3D generated mesh resolution = rt_mesh_reso_scale_to_gsd * image gsd. Smaller values provide higher mesh resolution and clearer colors but lower efficiency. Default 6

Server Program realtime_service

realtime_service is a gRPC real-time reconstruction server program. Clients can complete the following tasks through this service:

(1) Initialize service and start real-time task.

(2) Pass image paths frame by frame for real-time SLAM positioning and 2D map | 3D mesh calculation.

(3) Execute post-processing and automatically end task.

(4) Actively end task.

The following is a detailed description of how clients use this server program:

The server listens on port 50051 by default, and the port number can be freely configured through startup parameters.

API Interface Description

The service defines the following gRPC interfaces:

Initialize

Function: Initialize the server and start a real-time task. Clients must call this interface before calling other interfaces, passing task configuration.

Request Format (InitRequest)

Field Name	Type	Description
task_json	string	Task JSON content string

Response Format (InitResponse)

Field Name	Type	Description
state	int32	Initialization status (0 indicates success, other values indicate failure)

AddFrame

Function: Pass image paths frame by frame for real-time processing. Clients continuously add image frames through this interface, and the server performs real-time computation.

Request Format (AddFrameRequest)

Field Name	Type	Description
image_path	string	Image file path, UTF-8 encoding
image_meta_data	string	Image metadata JSON content string. When set to empty, the real-time service will automatically parse image metadata (caller ensures image file contains camera focal length and GPS position information)

Image Metadata Description (Image Metadata)

Field Name	Type	Required	Description
width	int	Yes	Image width (pixels)
height	int	Yes	Image height (pixels)
camera_id	int	No	Corresponding camera ID
focal_length_in_35mm	double	No	35mm equivalent focal length (mm), required when pre_calib_param is not set
pos	array[3]	Yes	Position [longitude/X, latitude/Y, elevation/Z]
pos_sigma	array[3]	No	Position accuracy (meters), RTK:[0.03,0.03,0.06], Normal GPS: [2.0,2.0,5.0]
orientation	array[9]	No	Rotation matrix (9 values, row-major, unit vectors. If external parameters are available, fill in the rotation matrix from world coordinate system to camera coordinate system, Cartesian coordinate system)
position_constant	bool	No	Whether position is fixed (true/false)
relative_altitude	double	No	Relative flight altitude (meters)
pre_calib_param	array	No	Pre-calibration parameters (if available, fill in camera internal parameters)
dewarp_flag	bool	No	Distortion correction flag (if from DJI XMP data, true means distortion has been removed)

Example:

{
    "width": 5280,
    "height": 3956,
    "camera_id": 1,
    "pos": [
        121.11150580555555,
        31.788278305555558,
        21.226
    ],
    "pos_sigma": [
        0.03,
        0.03,
        0.06
    ],
    "orientation": [
        -0.3583679497241974,
        0.9335803985595703,
        0,
        0.7533622980117798,
        0.2891887128353119,
        -0.5906056761741638,
        -0.5513778924942017,
        -0.21165414154529572,
        -0.8069602847099304
    ],
    "relative_altitude": 6.012,
    "focal_length_in_35mm": 24,
    "pre_calib_param": [
        3713.29,
        3713.29,
        2647.02,
        1969.28,
        -0.11257524,
        0.01487443,
        -0.00008572,
        1e-7,
        -0.02706411
    ],
    "dewarp_flag": false
}

2D Real-time Results:

After executing AddFrame, Web Mercator (EPSG:3857) coordinate system orthomosaic tile format is generated in the 2D/dom_tiles directory of the working directory. Use third-party GIS libraries to read and display tiles.

3D Real-time Results:

After executing AddFrame, the 3d_update.json file will be generated and updated in real-time in the working directory.

3D models are stored in binary format (ply or 3dtiles) and are chunked according to geographic extent. Clients can update rendering data on demand to improve rendering efficiency. The 3d_update.json stores an array of model chunks, with each model chunk storing name, path, and last update time.

After each AddFrame execution, clients read this JSON file to get the current frame's 3D model update chunks and last update time. If new chunks are added or chunk update times are updated, load new mesh data from corresponding paths for rendering. If the last time hasn't been updated, skip corresponding chunks to reduce I/O and improve rendering efficiency.

Stability Note

To ensure stability, the first N frames (N≈10) generally won't produce positioning and mesh results. After positioning reaches a stable state, positioning and results begin updating to disk.

EndTask

Function: End the current real-time task.

Request Format (EndTaskRequest)

Field Name	Type	Description
No fields	None	End task request

Response Format (EndTaskResponse)

Field Name	Type	Description
state	int32	Task end status (0 indicates success, other values indicate failure)

5 Client Development Guide

Prerequisites

1. Install gRPC and Protocol Buffers

   Please ensure that gRPC and Protocol Buffers (protoc compiler) are correctly installed in your development environment. It is recommended to refer to the official documentation to complete the installation and ensure version compatibility.

2. Generate Client Code

   Use the .proto file provided by MipMap to generate client code for the corresponding language through the protoc tool. For specific commands and parameters, please refer to the gRPC official documentation or project instructions.

3. Prepare Task Configuration JSON File

   The real-time reconstruction service initialization requires a task configuration JSON file. Please ensure the file content is complete, the path is correct, and the server has access permissions.

Path Access

Please ensure that the image paths involved in the client configuration file are local or network paths directly accessible by the server to avoid task failures due to unreachable paths.

Standard Real-time Reconstruction Service Usage Process

1. Start Server (realtime_service)

   Start the server program and keep it in listening state. You can specify the listening port through the -port_number parameter, for example:

   realtime_service -port_number 50051

   Supports starting multiple server instances simultaneously, distinguished by different port numbers.

2. Start Client and Establish Connection

   Start the client program, specify the server port number, and establish gRPC communication connection. Call interfaces in the following order:

   • Initialize: Initialize real-time reconstruction task
   • AddFrame: Add image data frame by frame
   • EndTask: End current real-time task

3. Task Management and Multiple Calls

   The server process can run continuously and supports multiple tasks. After the client calls EndTask to end a task, it automatically exits. If you need to start a new task, you can restart the client and establish a new connection with the server, repeating the above process.

Client Program realtime_client

The client program realtime_client is provided for users who are not familiar with gRPC calls and have single usage patterns (referred to as callers below). No need to write client code, just start the realtime_client executable and server program.

Steps:

(1) The initialization JSON parameters need to include image directory and target image size for real-time computation. Please ensure the directory is valid.

Field Name	Type	Description
image_dir	string	Image folder directory, UTF-8 encoding
target_max_size	int	Maximum side pixel count after image frame scaling. Real-time tasks recommend scaling image frames. This value controls the scaled image size. Larger values provide better real-time reconstruction results but longer single frame processing time. When 0, caller scales externally and passes image metadata string (see details below)

(2) Console command to start realtime_service. The optional startup parameter is port number. Different port numbers represent different service ports. If no parameter is passed, port 50051 is used by default.

realtime_service -port_number 50051

(3) Console command to start realtime_client. The optional startup parameter is port number. Different port numbers represent different service ports. If no parameter is passed, port 50051 is used by default.

realtime_client -port_number 50051

After client starts, the console requires input of the parameter JSON path to initialize the service. Please have the caller input the parameter JSON file path in the console (without double quotes). After input, the server starts running real-time reconstruction and prints logs.

(4) Caller periodically accesses 2D real-time results and 3D real-time results files for rendering refresh.

(5) After the real-time task starts, the client will wait for the caller to input the end command. After inputting end, the server will stop adding frames and end the current real-time task.