一个新的 UI 系统：GGUI

分类	先决条件
操作系统	Windows / Linux / Mac OS X
后端	x64 / CUDA / Vulkan

从 v0.8.0开始，Taichi 添加了一个新的 GGUI系统。 新的 GUI 系统使用 GPU 进行渲染，使得 3 维场景的渲染更快。 这就是为什么这一新系统被命名为 GGUI。 本文档将介绍其提供的 API。

IMPORTANT

If you choose Vulkan as backend, ensure that you install the Vulkan environment.

note

It is recommended that you familiarize yourself with GGUI through the examples in examples/ggui_examples.

note

The variables referenced in code snippets below are define like this:

vertices         = ti.Vector.field(2, ti.f32, shape=200)
vertices_3d      = ti.Vector.field(3, ti.f32, shape=200)
indices          = ti.field(ti.i32, shape=200 * 3)
normals          = ti.Vector.field(3, ti.f32, shape=200)
per_vertex_color = ti.Vector.field(3, ti.f32, shape=200)

color  = (0.5, 0.5, 0.5)

创建一个窗口

ti.ui.Window(name, res) 可以创建一个窗口。

window = ti.ui.Window(name='Window Title', res = (640, 360), fps_limit=200, pos = (150, 150))

• The name parameter sets the title of the window.
• The res parameter specifies the resolution (width and height) of the window.
• The fps_limit parameter sets the maximum frames per second (FPS) for the window.
• The pos parameter specifies the position of the window with respect to the top-left corner of the main screen.

A ti.ui.Window can display three types of objects:

• 2D Canvas, which is used to draw simple 2D geometries like circles and triangles.+ 3D Scene, which is used to render 3D meshes and particles, and provides configurable camera and light sources.
• Immediate mode GUI components, such as buttons and textboxes.

2D 画布

创建画布

以下代码获取了一个覆盖整个窗口的 Canvas 对象。

canvas = window.get_canvas()

在画布上绘画

canvas.set_background_color(color)
canvas.triangles(vertices, color, indices, per_vertex_color)

radius = 5
canvas.circles(vertices, radius, color, per_vertex_color)

width = 2
canvas.lines(vertices, width, indices, color, per_vertex_color)
canvas.set_image(window.get_image_buffer_as_numpy())

参数 vertices, indices, per_vertex_color, 和 image 必须是 Taichi field。 如果提供了 per_vertex_color ，color 将会被忽略。

画布中几何图案的位置/中心的相对位置以在 0.0 和 1.0 之间的浮点数表示。 对于 circles() and lines()，radius 和 width 参数都是窗口高度的相对值。

画布在每帧后被清空。 务必在渲染循环中调用这些方法。

3D 场景

创建场景

scene = ti.ui.Scene()

配置相机

camera = ti.ui.Camera()
camera.position(1, 2, 3)  # x, y, z
camera.lookat(4, 5, 6)
camera.up(0, 1, 0)
camera.projection_mode(ti.ui.ProjectionMode.Perspective)
scene.set_camera(camera)

配置光源

添加点光源

调用 point_light() 将点光源添加到场景中。

scene.point_light(pos=(1, 2, 3), color=(0.5, 0.5, 0.5))

请注意，你需要为每一帧调用 point_light() 方法。 与 canvas() 方法类似，请在你的渲染循环中调用该方法。

3D 几何图形

scene.lines(vertices, width, indices, color, per_vertex_color)
scene.mesh(vertices_3d, indices, normals, color, per_vertex_color)
scene.particles(vertices, radius, color, per_vertex_color)

参数 vertices, indices, per_vertex_color 和 image 都被设计为 Taichi fields 类型。 如果提供了 per_vertex_color ，color 将会被忽略。

几何体的位置/中心应设在世界空间的坐标上。

note

If a mesh has num triangles, the indices should be a 1D scalar field with a shape (num * 3), not a vector field.

normals 是 scene.mesh() 的一个可选参数。

1. An example of drawing 3d-lines

import taichi as ti

ti.init(arch=ti.cuda)

N = 10

particles_pos = ti.Vector.field(3, dtype=ti.f32, shape = N)
points_pos = ti.Vector.field(3, dtype=ti.f32, shape = N)

@ti.kernel
def init_points_pos(points : ti.template()):
    for i in range(points.shape[0]):
        points[i] = [i for j in ti.static(range(3))]

init_points_pos(particles_pos)
init_points_pos(points_pos)

window = ti.ui.Window("Test for Drawing 3d-lines", (768, 768))
canvas = window.get_canvas()
scene = ti.ui.Scene()
camera = ti.ui.Camera()
camera.position(5, 2, 2)

while window.running:
    camera.track_user_inputs(window, movement_speed=0.03, hold_key=ti.ui.RMB)
    scene.set_camera(camera)
    scene.ambient_light((0.8, 0.8, 0.8))
    scene.point_light(pos=(0.5, 1.5, 1.5), color=(1, 1, 1))

    scene.particles(particles_pos, color = (0.68, 0.26, 0.19), radius = 0.1)
    # Draw 3d-lines in the scene
    scene.lines(points_pos, color = (0.28, 0.68, 0.99), width = 5.0)
    canvas.scene(scene)
    window.show()

Advanced 3d Geometries

scene = ti.ui.Scene()
width = 2
radius = 5

scene.lines(vertices, width, indices, color, per_vertex_color, vertex_offset=0, vertex_count=10, index_offset=0, index_count=10)

scene.mesh(vertices_3d, indices, normals, color, per_vertex_color, vertex_offset=0, vertex_count=10, index_offset=0, index_count=10, show_wireframe=True)

scene.particles(vertices, radius, color, per_vertex_color, index_offset=0, index_count=10)

scene.mesh_instance(vertices_3d, indices, normals, color, per_vertex_color, vertex_offset=0, vertex_count=10, index_offset=0, index_count=10, show_wireframe=True)

The additional arguments vertex_offset, vertex_count, index_offset and index_count control the visible part of the particles and mesh. For the mesh() and mesh_instance() methods, set whether to show wireframe mode through setting show_wireframe.

:::example

1. Example of drawing a part of the mesh/particles

scene = ti.ui.Scene()

center = ti.Vector.field(3, ti.f32, shape=10)

# For particles
# draw the 2-th to 7-th particles
scene.particles(center, radius=1, index_offset = 1, index_count = 6)

# For mesh
# 1. with indices
scene.mesh(
    vertices_3d, indices, index_offset=1, index_count=3,
    # vertex_offset is set to 0 by default, and it is not necessary
    # to assign vertex_offset a value that otherwise you must.
    vertex_offset = 1
    )

# usually used as below:
# draw the 11-th to 111-th mesh vertexes
scene.mesh(vertices_3d, indices, index_offset=10, index_count=100)

# 2. without indices (similar to the particles' example above)
scene.mesh(
    vertices_3d,
    vertex_offset=2,  # user defined first vertex index
    vertex_count=3,  # user defined vertex count
    )

2. An example of drawing part of lines

import taichi as ti

ti.init(arch=ti.cuda)

N = 10

particles_pos = ti.Vector.field(3, dtype=ti.f32, shape = N)
points_pos = ti.Vector.field(3, dtype=ti.f32, shape = N)
points_indices = ti.Vector.field(1, dtype=ti.i32, shape = N)

@ti.kernel
def init_points_pos(points : ti.template()):
    for i in range(points.shape[0]):
        points[i] = [i for j in range(3)]
        # points[i] = [ti.sin(i * 1.0), i * 0.2, ti.cos(i * 1.0)]

@ti.kernel
def init_points_indices(points_indices : ti.template()):
    for i in range(N):
        points_indices[i][0] = i // 2 + i % 2

init_points_pos(particles_pos)
init_points_pos(points_pos)
init_points_indices(points_indices)

window = ti.ui.Window("Test for Drawing 3d-lines", (768, 768))
canvas = window.get_canvas()
scene = ti.ui.Scene()
camera = ti.ui.Camera()
camera.position(5, 2, 2)

while window.running:
    camera.track_user_inputs(window, movement_speed=0.03, hold_key=ti.ui.RMB)
    scene.set_camera(camera)
    scene.ambient_light((0.8, 0.8, 0.8))
    scene.point_light(pos=(0.5, 1.5, 1.5), color=(1, 1, 1))

    scene.particles(particles_pos, color = (0.68, 0.26, 0.19), radius = 0.1)
    # Here you will get visible part from the 3rd point with (N - 4) points.
    scene.lines(points_pos, color = (0.28, 0.68, 0.99), width = 5.0, vertex_count = N - 4, vertex_offset = 2)
    # Using indices to indicate which vertex to use
    # scene.lines(points_pos, color = (0.28, 0.68, 0.99), width = 5.0, indices = points_indices)
    # Case 1, vertex_count will be changed to N - 2 when drawing.
    # scene.lines(points_pos, color = (0.28, 0.68, 0.99), width = 5.0, vertex_count = N - 1, vertex_offset = 0)
    # Case 2, vertex_count will be changed to N - 2 when drawing.
    # scene.lines(points_pos, color = (0.28, 0.68, 0.99), width = 5.0, vertex_count = N, vertex_offset = 2)
    canvas.scene(scene)
    window.show()

3. Details of mesh instancing

scene = ti.ui.Scene()

num_instance = 100
m_transforms = ti.Matrix.field(4, 4, dtype = ti.f32, shape = num_instance)


# For example: An object is scaled by 2, rotated by rotMat, and translated by t = [1, 2, 3], then
#
# The ScaleMatrix is:
# 2, 0, 0, 0
# 0, 2, 0, 0
# 0, 0, 2, 0
# 0, 0, 0, 1
#
# The RotationMatrix is:
# https://en.wikipedia.org/wiki/Rotation_matrix#General_rotations
#
# The TranslationMatrix is:
# 1, 0, 0, 1
# 0, 1, 0, 2
# 0, 0, 1, 3
# 0, 0, 0, 1
#
# Let TransformMatrix = TranslationMatrix @ RotationMatrix @ ScaleMatrix, then the final TransformMatrix is:
#   2 * rotMat00,     rotMat01,       rotMat02, 1
#       rotMat10, 2 * rotMat11,       rotMat12, 2
#       rotMat20,     rotMat21,   2 * rotMat22, 3
#              0,            0,              0, 1
...

# Draw mesh instances (from the 1st instance)
scene.mesh_instance(vertices_3d, indices, transforms = m_transforms, instance_offset = 1)

4. Example of setting wireframe mode

window = ti.ui.Window("Display Mesh", (1024, 1024), vsync=True)
canvas = window.get_canvas()
scene = ti.ui.Scene()
camera = ti.ui.Camera()

# slider_int usage
some_int_type_value = 0
def show_options():
    global some_int_type_value

    window.GUI.begin("Display Panel", 0.05, 0.1, 0.2, 0.15)
    display_mode = window.GUI.slider_int("Value Range", some_int_type_value, 0, 5)
    window.GUI.end()

while window.running:

    ...
    # if to show wireframe
    scene.mesh_instance(vertices_3d, indices, instance_count = 100 , show_wireframe = True)

    canvas.scene(scene)
    show_options()
    window.show()

note

If indices is not provided, consider using like this:

scene = ti.ui.Scene()
scene.mesh(vertices_3d, normals, color, per_vertex_color, vertex_offset=0, vertex_count=50, show_wireframe=True)

If indices is provided, consider using like this:

scene.mesh(vertices_3d, indices, normals, color, per_vertex_color, vertex_offset=0, index_offset=0, index_count=50, show_wireframe=True)

渲染场景

您可以在画布上渲染场景。

window = ti.ui.Window(name='Title', res=(640, 360))
canvas = window.get_canvas()
canvas.scene(scene)

Fetching Color/Depth information

img = window.get_image_buffer_as_numpy()
window.get_depth_buffer(scene_depth)
depth = window.get_depth_buffer_as_numpy()

After rendering the current scene, you can fetch the color and depth information of the current scene using get_image_buffer_as_numpy() and get_depth_buffer_as_numpy(), which copy the gpu data to a NumPy array(cpu). get_depth_buffer() copies the GPU data to a Taichi field (depend on the arch you choose) or copies data from GPU to GPU.

:::example

1. Example of fetching color information

window = ti.ui.Window("Test for getting image buffer from ggui", (768, 768), vsync=True)
video_manager = ti.tools.VideoManager("OutputDir")

while window.running:
    # render_scene()
    img = window.get_image_buffer_as_numpy()
    video_manager.write_frame(img)
    window.show()

video_manager.make_video(gif=True, mp4=True)

2. An example of fetching the depth data

window_shape = (720, 1080)
window = ti.ui.Window("Test for copy depth data", window_shape)
canvas = window.get_canvas()
scene = ti.ui.Scene()
camera = ti.ui.Camera()

# Get the shape of the window
w, h = window.get_window_shape()
# The field/ndarray stores the depth information, and must be of the ti.f32 data type and have a 2d shape.
# or, in other words, the shape must equal the window's shape
scene_depth = ti.ndarray(ti.f32, shape = (w, h))
# scene_depth = ti.field(ti.f32, shape = (w, h))

while window.running:
    # render()
    canvas.scene(scene)
    window.get_depth_buffer(scene_depth)
    window.show()

GUI 组件

GGUI 的 GUI 组件设计遵循 Dear ImGui API。

window = ti.ui.Window("Test for GUI", res=(512, 512))
gui = window.get_gui()
value = 0
color = (1.0, 1.0, 1.0)
with gui.sub_window("Sub Window", x=10, y=10, width=300, height=100):
    gui.text("text")
    is_clicked = gui.button("name")
    value = gui.slider_float("name1", value, minimum=0, maximum=100)
    color = gui.color_edit_3("name2", color)

显示窗口

调用 show() 来显示一个窗口。

window.show()

Call this method only at the end of the render loop for each frame.

用户输入处理

如需获取上一个方法调用后发生的事件：

events = window.get_events()

在 events 中每一个 event 是 ti.ui.Event 的一个实例。 它支持如下属性：

• event.action, 可以是 ti.ui.PRESS, ti.ui.RELEASE, 或 ti.ui.MOTI
• event.key: 与此事件相关的键。

如需获取光标位置：

• window.get_cursor_pos()

如需检查一个键是否被按下：

• window.is_pressed(key)

以下是一个来自 mpm128 的用户输入处理的示例：

gravity = ti.Vector.field(2, ti.f32, shape=())
attractor_strength = ti.field(ti.f32, shape=())

while window.running:
    # keyboard event processing
    if window.get_event(ti.ui.PRESS):
        if window.event.key == 'r': reset()
        elif window.event.key in [ti.ui.ESCAPE]: break
    if window.event is not None: gravity[None] = [0, 0]  # if had any event
    if window.is_pressed(ti.ui.LEFT, 'a'): gravity[None][0] = -1
    if window.is_pressed(ti.ui.RIGHT, 'd'): gravity[None][0] = 1
    if window.is_pressed(ti.ui.UP, 'w'): gravity[None][1] = 1
    if window.is_pressed(ti.ui.DOWN, 's'): gravity[None][1] = -1

    # mouse event processing
    mouse = window.get_cursor_pos()
    # ...
    if window.is_pressed(ti.ui.LMB):
        attractor_strength[None] = 1
    if window.is_pressed(ti.ui.RMB):
        attractor_strength[None] = -1

    window.show()

图像 I/O

如需在窗口中将当前帧写入图像文件：

window.save_image('frame.png')

Note that you must call window.save_image() before calling window.show().

离屏渲染

GGUI 支持在不显示窗口的情况下保存图像帧。 这也被称为“无头”渲染。 如需启用此模式，请在初始化窗口时将参数 show_window 设置为 False。

window = ti.ui.Window('Window Title', (640, 360), show_window = False)

Then you can call window.save_image() as normal and remove the window.show() call at the end.