/images/hugo/avatar.png

pregel 重要流程总结

本节我们总结回顾一下 Pregel 重要流程,包括

  1. PregelLoop 的更新循环
  2. PregelLoop 的崩溃恢复

1. PregelLoop 的更新循环

1.1 任务生成

prepare_next_tasks 会生成两种类型的 task:

  • PUSH: 是直接触发 Node 执行,并传入自定义参数
  • PULL: 是让 Node 发起检查,自己是否被触发,参数从 Node 监听的 channel 中读入

最终生成的任务包含如下信息:

pregel 的可配置项

1. configurable 配置

pregel 的实现在 RunnableConfig configurable 配置段中预定义了很多配置项。这些配置项对应我们理解 pregel 有很重要的作用。

以下是 LangGraph 中预定义的 configurable 配置键(CONFIG_KEY_*)的详细说明表格,按如下维度整理:

pregel loop - 2

上一节我们介绍了 pregel loop 的初始化、tick 函数。tick 函数其实也没有介绍完,并标记了我们要理解的其他方法,包括:

  1. _emit
  2. _put_checkpoint
  3. put_writes

这一节我们来学习这些函数的实现。

pregel algo - 1

1. Pregel Algo

pregel 有关任务生成的代码位于 langgraph\pregel\_algo.py。这个应该算是 pregel 最核心的部分了。_algo.py 内有如下几个函数: 2. prepare_next_tasks - 用于生成下一个 Pregel step 中的任务。 - 内部会调用 prepare_single_task 3. apply_writes - 把对 channel 的写入应用到 channel 中,并返回 updated_channels

pregel algo - 2

前面我们介绍了 _algo.py 中关联的对象,这一节我们来介绍 _algo.py 的这几个核心函数:

  1. prepare_single_task
  2. prepare_next_tasks
  3. local_read
  4. apply_writes

1. prepare_next_tasks

1.1 prepare_next_tasks 入参

prepare_next_tasks 函数的定义如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def prepare_next_tasks(
    checkpoint: Checkpoint,
    pending_writes: list[PendingWrite],
    processes: Mapping[str, PregelNode],
    channels: Mapping[str, BaseChannel],
    managed: ManagedValueMapping,
    config: RunnableConfig,
    step: int,
    stop: int,
    *,
    for_execution: bool,
    store: BaseStore | None = None,
    checkpointer: BaseCheckpointSaver | None = None,
    manager: None | ParentRunManager | AsyncParentRunManager = None,
    trigger_to_nodes: Mapping[str, Sequence[str]] | None = None,
    updated_channels: set[str] | None = None,
    retry_policy: Sequence[RetryPolicy] = (),
    cache_policy: CachePolicy | None = None,
) -> dict[str, PregelTask] | dict[str, PregelExecutableTask]:
    """Prepare the set of tasks that will make up the next Pregel step.

    Args:
        checkpoint: The current checkpoint.
        pending_writes: The list of pending writes.
        processes: The mapping of process names to PregelNode instances.
        channels: The mapping of channel names to BaseChannel instances.
        managed: The mapping of managed value names to functions.
        config: The runnable configuration.
        step: The current step.
        for_execution: Whether the tasks are being prepared for execution.
        store: An instance of BaseStore to make it available for usage within tasks.
        checkpointer: Checkpointer instance used for saving checkpoints.
        manager: The parent run manager to use for the tasks.
        trigger_to_nodes: Optional: Mapping of channel names to the set of nodes
            that are can be triggered by that channel.
        updated_channels: Optional. Set of channel names that have been updated during
            the previous step. Using in conjunction with trigger_to_nodes to speed
            up the process of determining which nodes should be triggered in the next
            step.

    Returns:
        A dictionary of tasks to be executed. The keys are the task ids and the values
        are the tasks themselves. This is the union of all PUSH tasks (Sends)
        and PULL tasks (nodes triggered by edges).
    """

下面是入参说明列表:

pregel executor

executor 是 langgraph 中对任务池的抽象。分为同步和异步两个实现。

1. BackgroundExecutor

BackgroundExecutor 是一个基于线程池的上下文管理器,用于在后台并行运行多个同步任务(函数),并在退出上下文时安全地清理、等待和处理异常。