Real-time Reconstruction

Table of Contents

• Description
• Overview
• Local Development Package for Windows
• Docker Deployment for Linux
• Parameter Description
• Server Program realtime_service
• API Interface Description
• Client Development Guide

0 Description

Real-time reconstruction supports images collected from drone orthophoto flight routes with overlap rates of 70% or higher for both forward and lateral directions.

Recommended use of DJI surveying drones M3E/M4E.

🚁 Drone Orthophoto Flight Route Diagram

Orthophoto Route Planning Key Points

• Route Spacing: Calculate based on camera field of view and flight altitude to ensure lateral overlap
• Capture Interval: Set according to flight speed and forward overlap requirements
• Flight Altitude: Affects ground resolution and coverage area
• Weather Conditions: Choose uniform lighting, windless or light wind conditions

1 Overview

Real-time reconstruction is provided as a gRPC server program: realtime_service. Starting it launches the real-time reconstruction service. We will provide a proto file for developers to develop client programs. The client program performs the following operations:

(1) Initialize - Initialize the real-time reconstruction service

(2) AddFrame - Add image frames

(3) EndTask - End the current real-time reconstruction task

The real-time reconstruction service is implemented based on the gRPC communication protocol. It is recommended that developers have experience with gRPC client development to successfully integrate and call related interfaces.

Download page for the real-time reconstruction service proto file: Proto File Download

Below is a basic flow diagram of the real-time reconstruction service:

2 Local Development Package for Windows

Please contact MipMap to request the local development package.

3 Docker Deployment for Linux

For environment preparation before deployment, please refer to: Linux Deployment Environment Preparation.

Download Image

docker pull registry.mipmap3d.com/docker/mipmap/realtime:v1.0-ubuntu22.04

Start Container from Image

# Start Docker container
docker run -it --rm \
  -v /tmp:/tmp \             # Mount host directory /tmp to container [License access required, must execute and mount as per this command]
  -v /mnt:/mnt \             # Mount host directory to container [Example, adjust according to actual situation]
  --cpus 8 \                 # Limit container to use 8 CPUs
  --gpus all \               # Use all GPUs (requires NVIDIA Container Toolkit)
   --name mipmap_rt \            # Container name is mipmap_rt
  registry.mipmap3d.com/docker/mipmap/realtime:v1.0-ubuntu22.04 \
  /bin/bash                  # Start bash after entering container

Important Notes:

• /mnt directory mapping: mnt is an example, the actual caller can flexibly mount whichever directory they want
• /tmp directory mapping: Required for license access, must be mounted
• --cpus 8: Limit container to use 8 CPU cores
• --gpus all: Enable all GPU support

Run Real-time Reconstruction Service

Start the real-time reconstruction service by launching realtime_service from the command line. The startup parameters are explained as follows:

Field Name	Type	Required	Description
port_number	number	No	Server port number, default 50051
task_type	string	No	Reconstruction type, 2D - stitching via GPS and gimbal angle, only performs 2D real-time fast stitching; 3D - stitching via SLAM, supports 2D real-time fast stitching and 3D real-time mesh, default 3D

# Run real-time reconstruction service in container
./mipmap_engine/realtime_service -port_number 50051(optional) -task_type 3D(optional)

---

4 Parameter Description

Real-time reconstruction requires a JSON for initialization. The content description table is as follows:

Field Name	Type	Required	Description
license_id	int	Yes	License ID, must be set to 9200
working_dir	string	Yes	Working directory, saves intermediate files and final output files
gdal_folder	string	Yes	GDAL directory path, /mipmap_engine/data directory in container
do_realtime_2d_map	bool	No	Default false, whether to compute real-time 2D map, i.e., 2D orthophoto map, and do_realtime_3d_map must have at least one set to true
do_realtime_3d_map	bool	No	Default false, whether to compute real-time 3D map, i.e., 3D mesh (currently available on Windows), and do_realtime_2d_map must have at least one set to true
target_max_size	int	No	Default 0, the longest side pixel count of the image frame after scaling. Real-time tasks recommend scaling image frames. This value controls the size of the scaled image. Larger values result in better real-time reconstruction effects but longer per-frame processing time. When 0, the caller needs to scale the image themselves before AddFrame and pass the scaled image metadata string (details below). Recommended values: 2048/1920/1280
tile_2d_save_type	int	No	Default 0, real-time 2D output format, 0: webp 1: png
mesh_3d_save_type	int	No	Default 0, real-time 3D output format, 0: ply 1: 3dtiles
rt_mesh_reso_scale_to_gsd	int	No	Default 6, real-time 3D generated mesh resolution = rt_mesh_reso_scale_to_gsd*image gsd. The smaller this value, the higher the mesh resolution, clearer colors, but efficiency will also decrease accordingly
save_dom_tiff	bool	No	Default false, whether to generate TIF format DOM large image for real-time frames. Generating large images will increase per-frame processing time. Large images are fixedly stored in working_dir/2D/geotiffs/dom.tif
coordinate_descriptor_2D	object	No	Default null, coordinate system of TIF format DOM large image. Format reference Coordinate System

Server Program realtime_service

realtime_service is a gRPC real-time reconstruction server program. Clients can use this service to complete the following tasks:

（1）Initialize service and start real-time task.

（2）Pass image paths frame by frame for real-time SLAM localization and 2D map|3D mesh computation.

（3）Execute post-processing and automatically end task.

（4）Actively end task.

The following is a detailed description of how clients use this server program:

The server listens on port 50051 by default, and the port number can also be freely configured through startup parameters

API Interface Description

The service defines the following gRPC interfaces:

Initialize

Function: Initialize the server and start a real-time task. Clients must call this interface before calling other interfaces, passing task configuration.

Request Format (InitRequest)

Field Name	Type	Description
task_json	string	Task JSON content string

Response Format (InitResponse)

Field Name	Type	Description
state	int32	Initialization state (0 indicates success, other values indicate failure) code

AddFrame

Function: Pass image paths frame by frame for real-time processing. Clients continuously add image frames through this interface, and the server performs real-time computation.

Request Format (AddFrameRequest)

Field Name	Type	Description
image_path	string	Image file path, UTF-8 encoded
image_meta_data	string	Image metadata JSON content string. When set to empty string, the real-time service will automatically parse image metadata internally (caller must ensure that the EXIF information in the image file header includes the camera 35mm equivalent focal length value and GPS location information)

Image Metadata Description (Image Metadata)

Field Name	Type	Required	Description
width	int	Yes	Image width (pixels)
height	int	Yes	Image height (pixels)
camera_id	int	No	Corresponding camera ID
focal_length_in_35mm	double	No	35mm equivalent focal length (mm), required when pre_calib_param is not set
pos	array[3]	Yes	Position [Longitude/X, Latitude/Y, Elevation/Z]
pos_sigma	array[3]	No	Position accuracy (meters), RTK:[0.03,0.03,0.06], Normal GPS: [2.0,2.0,5.0]
orientation	array[9]	No	Rotation matrix (9 values, row-major, unit vectors, if external parameters are available, fill in the rotation matrix from world coordinate system to camera coordinate system, Cartesian coordinate system)
position_constant	bool	No	Whether position is fixed (true/false)
relative_altitude	double	No	Relative flight altitude (meters)
pre_calib_param	array	No	Pre-calibration parameters (if available, fill in camera intrinsic parameters)
dewarp_flag	bool	No	Distortion correction flag (if from DJI XMP data, true indicates distortion has been removed)

Example:

{
    "width": 5280,
    "height": 3956,
    "camera_id": 1,
    "pos": [
        121.11150580555555,
        31.788278305555558,
        21.226
    ],
    "pos_sigma": [
        0.03,
        0.03,
        0.06
    ],
    "orientation": [
        -0.3583679497241974,
        0.9335803985595703,
        0,
        0.7533622980117798,
        0.2891887128353119,
        -0.5906056761741638,
        -0.5513778924942017,
        -0.21165414154529572,
        -0.8069602847099304
    ],
    "relative_altitude": 6.012,
    "focal_length_in_35mm": 24,
    "pre_calib_param": [
        3713.29,
        3713.29,
        2647.02,
        1969.28,
        -0.11257524,
        0.01487443,
        -0.00008572,
        1e-7,
        -0.02706411
    ],
    "dewarp_flag": false
}

Response Format (AddFrameResponse)

Field Name	Type	Description
update_json	string	JSON string storing the current frame update amount, see below for meaning

update_json field meanings:

Field Name	Type	Description
tile2d_folder	string	2D orthophoto fast stitching tile output folder path
updated_2d_tiles	array	List of tile paths updated by current frame, relative paths, with tile2d_folder as root directory
tile3d_folder	string	3D mesh output tile output folder path
updated_3d_tiles	array	List of mesh model paths updated by current frame, relative paths, with tile3d_folder as root directory

2D Real-time Output:

The 2D/dom_tiles directory in the working directory stores orthophoto tile format in Web Mercator (EPSG:3857) coordinate system. Use third-party GIS libraries to read and display tiles. Clients can incrementally update rendered tile data based on the returned update_2d_tiles collection to improve rendering efficiency.

3D Real-time Output:

The 3D/mesh in the working directory stores 3D mesh models. 3D models are stored in binary format (ply or 3dtiles) and are chunked according to geographic range. Clients can incrementally update rendered chunk data based on the returned update_3d_tiles collection to improve rendering efficiency.

Stability Note

To ensure stability, the first N frames (N≈10) generally do not produce localization and mesh output. After localization reaches a stable state, localization and output begin to update to disk.

EndTask

Function: End the current real-time task.

Request Format (EndTaskRequest)

Field Name	Type	Description
No fields	None	End task request

Response Format (EndTaskResponse)

Field Name	Type	Description
state	int32	Task end state (0 indicates success, other values indicate failure)

5 Client Development Guide

Prerequisites

1. Install gRPC and Protocol Buffers

   Please ensure that gRPC and Protocol Buffers (protoc compiler) are correctly installed in your development environment. It is recommended to refer to the official documentation to complete the installation and ensure version compatibility.

2. Generate Client Code

   Use the .proto file provided by MipMap Proto File Download to generate client code for the corresponding language through the protoc tool. For specific commands and parameters, please refer to the gRPC official documentation or project instructions.

3. Prepare Task Configuration JSON File

   The real-time reconstruction service requires a task configuration JSON file during initialization. Please ensure that the file content is complete, the path is correct, and the server has access permissions.

Path Access

Please ensure that the image paths involved in the client configuration file are local or network paths that the server can directly access, to avoid task failure due to unreachable paths.

Real-time Reconstruction Service Standard Usage Flow

1. Start Server (realtime_service)

   Start the server program and keep it in listening state. The listening port can be specified through the -port_number parameter, for example:

   realtime_service -port_number 50051

   Multiple server instances can be started simultaneously, distinguished by different port numbers.

2. Start Client and Establish Connection

   Start the client program, specify the server port number, and establish a gRPC communication connection. Call interfaces in the following order:

   • Initialize: Initialize real-time reconstruction task
   • AddFrame: Add image data frame by frame
   • EndTask: End current real-time task

3. Task Management and Multiple Calls

   The server process can run continuously and support multiple tasks. The client automatically exits after calling EndTask to end the task. If you need to initiate a new task, you can restart the client and establish a new connection with the server, repeating the above process.

Client Program realtime_client

The client program realtime_client is provided for users who are not familiar with gRPC calls and have a single usage pattern (referred to as callers below). No client code needs to be written. Just start the realtime_client executable and server program.

Steps:

(1) The initialization JSON parameters need to add image directory and target image size for real-time computation. Please ensure the directory is valid.

Field Name	Type	Description
image_dir	string	Image folder directory, UTF-8 encoded
target_max_size	int	Longest side pixel count of image frame after scaling. Real-time tasks recommend scaling image frames. This value controls the size of the scaled image. Larger values result in better real-time reconstruction effects but longer per-frame processing time. When 0, the caller scales externally and passes the image metadata string (details below)

(2) Start realtime_service from console command. The optional startup parameter is port number. Different port numbers represent different service ports. If no parameter is passed, port 50051 is used by default.

realtime_service -port_number 50051

(3) Start realtime_client from console command. The optional startup parameter is port number. Different port numbers represent different service ports. If no parameter is passed, port 50051 is used by default.

realtime_client -port_number 50051

After the client starts, the console will prompt for the path to the parameter JSON to initialize the service. Please enter the path to the parameter JSON file in the console (without quotes). After input, the server will start running real-time reconstruction and print logs.

(4) The caller periodically accesses 2D real-time output and 3D real-time output files to refresh rendering.

(5) After the real-time task starts, the client will wait for the caller to input the "end" command. After entering "end", the server will end frame addition and end the current real-time task.