真机开工体检的编排：把「不报错但不对」的故障源排成现场能照着做的顺序

前一个提交给了四个单项判据（USB / V4L / RDK / 手眼）。这一批把它们的**判据入口**
补齐并编排成一条命令，产出「结论 + 要人工做的事」：

scripts/lib/real_preflight.py
  run_all() 按**现场该处理的顺序**排出结论，不是按代码顺序：
    现场配置 → 遗留进程 → 网络 → 机械臂 RDK → 腕部相机 → USB/触觉/几何
  顺序是有理由的：遗留的 ros2_control 占着 RDK，会让后面每一项的现象都指向别处，
  所以先查它；机械臂能不能动是唯一的人工硬阻断，排在相机之前。

  新增 check_cameras —— **本臂配置的那只腕部相机在不在 USB 上**。
  这是本次实测暴露的判据缺口：2026-09-28 现场 D405 整个从 lsusb 消失
  （只剩顶置 D435），而旧体检**全绿** —— check_usb 只看协商速率，遍历的是
  「发现到的」设备，对「这只根本不在了」无话可说。缺 D405 之后踩的是完全不同
  的一类错误（点云为空 / 找不到工件），排查方向会被带偏很远。
  按 site.yaml 的配置意图核对：robots[<sn>].camera.name → cameras.<name>.model，
  比**型号**而不是序列号 —— 实测这台 D435 的 sysfs serial 是空串，按序列号判会
  永远判「不在」；序列号只在型号对上时做一次「是不是本臂那只」的提醒（WARN）。

  修掉三处会误导现场的输出：
  · _run() 改成三返回值（分离 stdout/stderr，RDK 的 C++ 日志会混进 stdout）
    之后，_ps_output/网络两处还在按两返回值解包 —— 体检**第一步就崩**，
    什么结论都给不出来。现在有 test_run_all_completes_without_crashing 钉住。
  · 人工动作里「去查网线」出现过两条（check_network 与 rdk_status 各写一句
    意思相同、字面不同的话），去重是按整串文字做的，于是人读到两条、还得判断
    「这是一个问题还是两个」。措辞抽到 manual_actions.py，两边字面一致。
  · 触觉 wedged 的处置与 v4l_health 同步（免密的 reset 是首选，重插是兜底）。

scripts/lib/manual_actions.py（新）
  人工动作措辞的**唯一**来源。判据分在两个文件、还跑在两个不同的 Python 上
  （3.11 / 3.10，见 rdk_status 的模块说明），互相 import 不了，所以抽成只依赖
  标准库的公共模块。

scripts/run_real_preflight.sh（新）
  一条命令的入口：只读、不动臂，不需要机械臂能连上。

实测（2026-09-28，arm 89）：体检跑通并给出 NO-GO + 7 条互不重复的人工动作，
新判据抓到「D405 不在 USB 上」（旧版全绿）。同时确认了本机可用的部分：
夹爪 PGEA 初始化完成 / 全开 20.00mm、右指触觉 SDK 300 帧 @29.9Hz、
手眼矩阵自洽、enp4s0 千兆全双工。

测试：test_real_preflight 28 项通过（含跨模块的措辞去重回归、
编排端到端冒烟、腕部相机四种情形）。

Co-authored-by: Cursor <cursoragent@cursor.com>

src/arm_application/test/test_real_preflight.py

@@ -0,0 +1,456 @@
+"""开工体检的编排判据：**根因必须排在症状前面**，且任何一项都不能把体检拖垮。
+
+这个文件钉三件事，都是这次无人值守实测踩出来的：
+
+  1. **顺序**。现场同时存在多个故障时，日志里全是"别人的症状"：
+     示教器还在 Manual（拦住机械臂）、总线 Hub 在内核里卡死（相机全没了）、
+     上一轮栈没退干净（spawner 永远在等）。人看到的是"相机不见了"，
+     于是去查相机 —— 而真正的硬阻断是示教器模式。所以 fail 必须排在最前，
+     并且"要人工做的事"要按这个顺序给。
+  2. **不挂**。一个卡死的 USB 设备曾让体检自己挂到超时被杀（sysfs 读取永久阻塞）。
+     所以任何外部调用都必须有超时，且超时要变成一条结论。
+  3. **不连坐**。某一项自己抛异常时，其余检查必须照跑 —— 否则一个坏掉的
+     可选检查会让"机械臂能不能动"这个最关键的问题没有答案。
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts" / "lib"))
+
+import real_preflight as pf  # noqa: E402
+from real_preflight import Check, Report, check_leftovers, check_network  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# 汇总与顺序
+# ---------------------------------------------------------------------------
+
+
+def test_level_is_fail_if_anything_fails() -> None:
+    rep = Report([Check("a", "A", pf.LEVEL_OK), Check("b", "B", pf.LEVEL_FAIL)])
+    assert rep.level == pf.LEVEL_FAIL
+    rep2 = Report([Check("a", "A", pf.LEVEL_OK), Check("b", "B", pf.LEVEL_WARN)])
+    assert rep2.level == pf.LEVEL_WARN
+    assert Report([Check("a", "A", pf.LEVEL_OK)]).level == pf.LEVEL_OK
+
+
+def test_manual_actions_are_deduped_and_ordered() -> None:
+    """同一句建议被两项都提到时只出现一次（现场按清单做，重复会让人以为漏了）。"""
+    rep = Report(
+        [
+            Check("a", "A", pf.LEVEL_FAIL, actions=["插网线", "开关"]),
+            Check("b", "B", pf.LEVEL_FAIL, actions=["插网线"]),
+        ]
+    )
+    assert rep.manual_actions() == ["插网线", "开关"]
+
+
+def test_root_cause_is_reported_before_the_symptom() -> None:
+    """一条 fail 的"根因"与一条 fail 的"症状"同时存在时，两者都要在，
+    但建议清单里根因的处置必须排在前面。"""
+    root = Check(
+        "arm",
+        "机械臂 RDK",
+        pf.LEVEL_FAIL,
+        ["示教器在 Manual 模式"],
+        actions=["示教器切到 Auto (Remote)"],
+    )
+    symptom = Check(
+        "usb", "USB", pf.LEVEL_FAIL, ["一台相机都没枚举到"], actions=["拔插相机"]
+    )
+    rep = Report([symptom, root])
+    rep.checks.sort(key=lambda c: pf._LEVEL_RANK.get(c.level, 3))  # run_all 的做法
+    assert [c.key for c in rep.checks] == ["usb", "arm"]  # 同级保持稳定顺序
+    # 顺序由 run_all 决定（根因检查项先加入），这里只钉住"根因那条在前"的编排意图：
+    assert pf.Report([root, symptom]).manual_actions()[0] == "示教器切到 Auto (Remote)"
+
+
+def test_run_all_puts_arm_before_symptom_checks() -> None:
+    """真编排里的顺序：机械臂（硬阻断）在 USB/触觉/几何之前。"""
+    src = (Path(pf.__file__)).read_text(encoding="utf-8")
+    body = src[src.index("def run_all") : src.index("def _tactile_pairs")]
+    assert body.index("check_arm_rdk") < body.index("check_usb"), (
+        "机械臂（能不能动）必须排在 USB 之前 —— 它是唯一的人工硬阻断"
+    )
+    assert body.index("check_leftovers") < body.index("check_network"), (
+        "遗留进程要先查：它会让后面每一项的现象都指向别处"
+    )
+
+
+def test_render_says_go_only_when_everything_is_ok() -> None:
+    go = pf.render(Report([Check("a", "A", pf.LEVEL_OK, ["一切正常"])]))
+    assert "GO" in go and "NO-GO" not in go
+    nogo = pf.render(
+        Report([Check("a", "A", pf.LEVEL_FAIL, ["坏了"], actions=["去修"])])
+    )
+    assert "NO-GO" in nogo and "要人工做的事" in nogo and "1. 去修" in nogo
+
+
+# ---------------------------------------------------------------------------
+# 单项判据
+# ---------------------------------------------------------------------------
+
+
+def test_leftovers_reports_orphan_spawner_and_ros2_control() -> None:
+    ps = (
+        "  100 /x/lib/controller_manager/spawner joint_state_broadcaster\n"
+        "  200 /y/lib/controller_manager/ros2_control_node\n"
+        "  300 /usr/bin/htop\n"
+    )
+    c = check_leftovers(ps)
+    assert c.level == pf.LEVEL_WARN
+    assert any("ros2_control" in ln for ln in c.lines)
+    assert c.actions, "要说清怎么清，不能只说'有残留'"
+
+
+def test_leftovers_does_not_match_its_own_command_line() -> None:
+    """曾经的坑：pkill -f 把自己的 shell 也匹配掉了（命令里含同样的字符串）。"""
+    ps = "  42 /bin/bash -c pkill -f '[a]rm_bringup.*stack.launch'\n"
+    c = check_leftovers(ps, running_cmds=("/bin/bash -c pkill",))
+    assert c.level == pf.LEVEL_OK
+
+
+def test_network_ok_when_ping_works() -> None:
+    c = check_network(0, "3 received", [], "192.168.2.101")
+    assert c.level == pf.LEVEL_OK and not c.actions
+
+
+def test_network_flags_same_mac_for_two_robot_ips() -> None:
+    """2026-09-28 实测：.100/.101 是同一个 MAC，ping .101 丢 50~90%。"""
+    arp = [
+        "192.168.2.100 dev enp4s0 lladdr 88:88:88:88:87:88 REACHABLE",
+        "192.168.2.101 dev enp4s0 lladdr 88:88:88:88:87:88 REACHABLE",
+    ]
+    c = check_network(0, "ok", arp, "192.168.2.101")
+    assert c.level == pf.LEVEL_WARN, "ping 通但同 MAC 抢转发，必须提一句"
+    assert any("88:88:88:88:87:88" in ln for ln in c.lines)
+    assert c.actions and "拔掉" in c.actions[0]
+    # 与目标无关的同 MAC 不打扰（正常双机配置不该被误报）
+    c2 = check_network(0, "ok", arp, "192.168.2.55")
+    assert c2.level == pf.LEVEL_OK
+
+
+def test_network_ping_fail_is_fail_and_shows_only_the_summary_line() -> None:
+    out = "PING 192.168.2.101 (192.168.2.101) 56(84) bytes of data.\n\n--- statistics ---\n2 packets transmitted, 0 received, 100% packet loss\n"
+    c = check_network(1, out, [], "192.168.2.101")
+    assert c.level == pf.LEVEL_FAIL
+    assert not any("bytes of data" in ln for ln in c.lines), "不要把 ping 原文刷进来"
+
+
+# ---------------------------------------------------------------------------
+# RDK 结论的翻译
+# ---------------------------------------------------------------------------
+
+
+def test_arm_manual_mode_is_a_hard_block_with_a_manual_action() -> None:
+    payload = {
+        "connected": True,
+        "level": "fail",
+        "summary": "示教器在 Manual 模式（operational_status=OperationalStatus.IN_MANUAL_MODE）",
+        "actions": ["示教器切到 Auto (Remote)"],
+        "mode": "IDLE",
+        "operational": False,
+        "op_status": "OperationalStatus.IN_MANUAL_MODE",
+        "fault": False,
+        "estop_released": True,
+    }
+    c = pf.check_arm_rdk(payload, 0)
+    assert c.level == pf.LEVEL_FAIL
+    assert c.actions == ["示教器切到 Auto (Remote)"]
+    assert any("IN_MANUAL_MODE" in ln for ln in c.lines)
+
+
+def test_arm_timeout_points_at_the_running_stack() -> None:
+    c = pf.check_arm_rdk({}, 124)
+    assert c.level == pf.LEVEL_FAIL
+    assert any("超时" in ln for ln in c.lines)
+    assert c.actions, "超时也要给下一步（RDK 被占用是最常见原因）"
+
+
+def test_unknown_level_from_payload_is_treated_as_fail() -> None:
+    """级别词表外的值不能当成功 —— 脚本按 ok/warn/fail 判定。"""
+    c = pf.check_arm_rdk({"level": "weird"}, 0)
+    assert c.level == pf.LEVEL_FAIL
+
+
+# ---------------------------------------------------------------------------
+# JSON 提取（RDK 的日志会混进 stdout/stderr）
+# ---------------------------------------------------------------------------
+
+
+def test_json_tail_survives_surrounding_logs() -> None:
+    text = (
+        '[info] Flexiv RDK v1.9.1 | {not json}\n'
+        '{"connected": true, "level": "ok"}\n'
+        "[info] >>>>> Flexiv RDK exited <<<<<\n"
+    )
+    assert pf._json_tail(text) == {"connected": True, "level": "ok"}
+
+
+def test_json_tail_picks_the_last_object() -> None:
+    text = '{"a": 1}\nnoise\n{"b": 2}'
+    assert pf._json_tail(text) == {"b": 2}
+
+
+def test_json_tail_returns_empty_when_there_is_nothing() -> None:
+    assert pf._json_tail("no json here") == {}
+    assert pf._json_tail("") == {}
+
+
+def test_json_tail_is_not_confused_by_braces_in_logs() -> None:
+    text = "warning: {unknown token} in config\n" '{"level": "fail", "error": "x"}'
+    assert pf._json_tail(text)["level"] == "fail"
+
+
+# ---------------------------------------------------------------------------
+# 触觉配置的两种写法
+# ---------------------------------------------------------------------------
+
+
+def test_tactile_pairs_accepts_bool_flag(tmp_path: Path) -> None:
+    """robots[].tactile 是布尔开关时不能抛 TypeError（曾经整个触觉检查项没跑）。"""
+    yml = tmp_path / "site.yaml"
+    yml.write_text(
+        "robot:\n"
+        "  sn: Rizon4-000089\n"
+        "  robots:\n"
+        "    - sn: Rizon4-000089\n"
+        "      arm: left\n"
+        "      ip: 192.168.2.101\n"
+        "      tactile: true\n"
+        "tactile:\n"
+        "  left_video: /dev/v4l/by-path/x-3.4.1-y\n"
+        "  right_video: /dev/v4l/by-path/x-3.4.2-y\n",
+        encoding="utf-8",
+    )
+    pairs = pf._tactile_pairs(yml, "Rizon4-000089")
+    assert [n for n, _ in pairs] == ["左指", "右指"]
+
+
+def test_tactile_pairs_empty_when_disabled(tmp_path: Path) -> None:
+    """tactile: false 的臂不该被拉起来查触觉（右臂就没装）。"""
+    yml = tmp_path / "site.yaml"
+    yml.write_text(
+        "robot:\n"
+        "  sn: Rizon4-000089\n"
+        "  robots:\n"
+        "    - sn: Rizon4-000089\n"
+        "      arm: left\n"
+        "      ip: 192.168.2.101\n"
+        "      tactile: false\n"
+        "tactile:\n"
+        "  left_video: /dev/v4l/by-path/x-3.4.1-y\n",
+        encoding="utf-8",
+    )
+    assert pf._tactile_pairs(yml, "Rizon4-000089") == []
+
+
+# ---------------------------------------------------------------------------
+# 腕部相机（D405）在不在 —— "速率对不对"和"在不在"是两件事
+# ---------------------------------------------------------------------------
+
+
+def _site_with_wrist(tmp_path: Path, *, model: str, serial: str) -> Path:
+    yml = tmp_path / "site.yaml"
+    yml.write_text(
+        "robot:\n"
+        "  sn: Rizon4-000089\n"
+        "  robots:\n"
+        "    - sn: Rizon4-000089\n"
+        "      arm: left\n"
+        "      ip: 192.168.2.101\n"
+        "      camera:\n"
+        "        name: d405_left\n"
+        "cameras:\n"
+        "  d405_left:\n"
+        f"    model: {model}\n"
+        f"    serial: \"{serial}\"\n"
+        "    role: left_wrist\n",
+        encoding="utf-8",
+    )
+    return yml
+
+
+def _realsense(path: str, product: str, serial: str = "") -> object:
+    from usb_health import UsbDevice
+
+    return UsbDevice(
+        path=path, vid="8086", pid="0b07", product=product, speed_mbps=5000, serial=serial
+    )
+
+
+def test_cameras_fails_when_the_wrist_camera_is_gone(tmp_path: Path) -> None:
+    """只剩顶置 D435 时必须报 FAIL。
+
+    现场那次（2026-09-28）D405 整个从 lsusb 消失，而旧体检**全绿** ——
+    它只看协商速率，对"这只相机根本不在了"没话说。
+    """
+    yml = _site_with_wrist(tmp_path, model="d405", serial="352122274408")
+    devices = [_realsense("1-3.2", "Intel(R) RealSense(TM) Depth Camera 435")]
+    c = pf.check_cameras(yml, "Rizon4-000089", devices=devices)
+    assert c.level == pf.LEVEL_FAIL
+    assert "d405" in " ".join(c.lines)
+    assert "1-3.2" in " ".join(c.lines), "要把现在有的列出来，人才知道差哪只"
+    assert c.actions, "光报错不够，要说去查什么（腕部活动线最先断）"
+
+
+def test_cameras_ok_when_wrist_camera_present(tmp_path: Path) -> None:
+    yml = _site_with_wrist(tmp_path, model="d405", serial="352122274408")
+    devices = [_realsense("1-3.2", "Intel(R) RealSense(TM) Depth Camera 435")]
+    devices.append(
+        _realsense("1-3.5", "Intel(R) RealSense(TM) Depth Camera 405", "352122274408")
+    )
+    c = pf.check_cameras(yml, "Rizon4-000089", devices=devices)
+    assert c.level == pf.LEVEL_OK
+
+
+def test_cameras_warns_when_another_arms_wrist_camera_is_plugged_in(
+    tmp_path: Path,
+) -> None:
+    """两台臂各有一只同型号：插反了要在开工前看出来（不阻断，但要报）。"""
+    yml = _site_with_wrist(tmp_path, model="d405", serial="352122274408")
+    devices = [
+        _realsense("1-3.5", "Intel(R) RealSense(TM) Depth Camera 405", "260522274442")
+    ]
+    c = pf.check_cameras(yml, "Rizon4-000089", devices=devices)
+    assert c.level == pf.LEVEL_WARN
+    assert "260522274442" in " ".join(c.lines)
+
+
+def test_cameras_does_not_use_serial_alone(tmp_path: Path) -> None:
+    """sysfs 不透出序列号（实测 D435 就是空串）时不能判成"不在"。"""
+    yml = _site_with_wrist(tmp_path, model="d405", serial="352122274408")
+    devices = [_realsense("1-3.5", "Intel(R) RealSense(TM) Depth Camera 405", "")]
+    c = pf.check_cameras(yml, "Rizon4-000089", devices=devices)
+    assert c.level == pf.LEVEL_OK
+
+
+def test_cameras_skips_sim_arm(tmp_path: Path) -> None:
+    """仿真臂不该去 USB 上找相机。"""
+    yml = tmp_path / "site.yaml"
+    yml.write_text(
+        "robot:\n"
+        "  sn: sim\n"
+        "  robots:\n"
+        "    - sn: sim\n"
+        "      arm: sim\n"
+        "      camera:\n"
+        "        name: sim_d405\n"
+        "cameras:\n"
+        "  sim_d405:\n"
+        "    model: d405\n"
+        "    role: sim_wrist\n",
+        encoding="utf-8",
+    )
+    c = pf.check_cameras(yml, "sim", devices=[])
+    assert c.level == pf.LEVEL_OK
+    assert "仿真" in " ".join(c.lines)
+
+
+# ---------------------------------------------------------------------------
+# 同一件事在报告里只能说一次（跨模块的措辞回归）
+# ---------------------------------------------------------------------------
+
+
+def test_network_action_appears_once_in_the_manual_list() -> None:
+    """check_network 与 rdk_status 给的"去查网线"必须是**同一串字**。
+
+    去重是按整串文字做的。两处各写一句意思相同、字面不同的话，现场就会
+    读到两条（实测就是：第 1 条和第 3 条是同一件事），人还得判断
+    "这是两个问题还是一个"。措辞已抽到 manual_actions，这里钉住它。
+    """
+    import rdk_status
+
+    _level, _summary, rdk_actions = rdk_status.classify_state(
+        connected=False, error="连不上"
+    )
+    net = check_network(1, "100% packet loss", [], "192.168.2.101")
+    rep = Report(
+        [
+            net,
+            Check("arm", "机械臂 RDK", pf.LEVEL_FAIL, ["连不上"], actions=list(rdk_actions)),
+        ]
+    )
+    acts = rep.manual_actions()
+    net_acts = [a for a in acts if "check_robot_network.sh" in a]
+    assert len(net_acts) == 1, f"网线那条重复了：{net_acts}"
+
+
+# ---------------------------------------------------------------------------
+# 超时
+# ---------------------------------------------------------------------------
+
+
+def test_run_times_out_instead_of_hanging() -> None:
+    """体检里有子进程时必须有超时：卡死的设备会让普通调用永不返回。"""
+    import time
+
+    t0 = time.monotonic()
+    code, out, err = pf._run([sys.executable, "-c", "import time; time.sleep(30)"], 0.5)
+    assert code == 124
+    assert time.monotonic() - t0 < 10.0
+    assert "超时" in err
+
+
+def test_run_keeps_stdout_and_stderr_separate() -> None:
+    """RDK 的日志走 stderr、JSON 走 stdout；合并会让 JSON 解析必然失败。"""
+    code, out, err = pf._run(
+        [sys.executable, "-c", "import sys; print('{\"a\":1}'); print('log', file=sys.stderr)"],
+        10.0,
+    )
+    assert code == 0
+    assert out.strip() == '{"a":1}'
+    assert "log" in err
+
+
+# ---------------------------------------------------------------------------
+# 编排能真的跑完（回归：_run 改成三返回值后，有一处还在按两返回值解包，
+# 于是体检**第一步就崩**，什么结论都没给出来）
+# ---------------------------------------------------------------------------
+
+
+def test_run_all_completes_without_crashing(tmp_path: Path, monkeypatch) -> None:
+    """整套编排必须能跑到底并返回 Report —— 崩在任何一行都等于"没体检"。"""
+    calls: list = []
+
+    def fake_run(cmd, timeout_s):  # noqa: ANN001
+        calls.append(list(cmd))
+        exe = Path(str(cmd[0])).name
+        if exe == "ps":
+            return 0, "  1 /usr/bin/ros2_control_node --ros-args\n", ""
+        if exe == "ping":
+            return 0, "2 packets transmitted, 2 received, 0% packet loss\n", ""
+        if exe == "ip":
+            return 0, "192.168.2.101 dev enp4s0 lladdr aa:bb:cc:dd:ee:ff REACHABLE\n", ""
+        return (
+            0,
+            '{"level": "ok", "mode": "AUTO", "enabled": true}',
+            "",
+        )
+
+    monkeypatch.setattr(pf, "_run", fake_run)
+    # USB 检查读真 sysfs（真机上可能卡死的正是它）→ 这里注入假 reader
+    monkeypatch.setattr(pf, "check_usb", lambda *a, **k: Check("usb", "USB", pf.LEVEL_OK, ["正常"]))
+    monkeypatch.setattr(
+        pf, "check_v4l", lambda *a, **k: Check("v4l", "触觉", pf.LEVEL_OK, ["正常"])
+    )
+
+    site = tmp_path / "site.yaml"
+    site.write_text(
+        "robot:\n"
+        "  sn: Rizon4-000089\n"
+        "  robots:\n"
+        "    - sn: Rizon4-000089\n"
+        "      arm: left\n"
+        "      ip: 192.168.2.101\n"
+        "      iface: enp4s0\n",
+        encoding="utf-8",
+    )
+    rep = pf.run_all(site, "Rizon4-000089", rdk_python=sys.executable)
+    assert isinstance(rep, Report)
+    assert rep.checks, "至少要有几项结论"
+    assert any(c.key == "network" for c in rep.checks), "网络项没跑到说明中途崩了"
+    assert any(c.key == "arm" for c in rep.checks), "机械臂项没跑到说明中途崩了"