用 PoseBusters V1 抽样 10 个复合物,把 Boltz-2.1(Compute API)、Boltz-2.0(HPC 本地)、Chai-1(HPC 本地)三条路径并排跑一遍,看谁的姿势更准、物理更合理,以及在同样的 14.57 GiB GPU 上谁更先 OOM。

编辑用途说明:这份是 BioTender 试跑记录,不是统计严谨的 benchmark。n=10 太小,结论只能当作"工程踩坑笔记 + 趋势观察",不能直接当作对工具优劣的盖棺定论。

指标	Boltz API (2.1)	Boltz HPC (2.0)	Chai-1 (HPC)
完成数 / 尝试	10 / 10	6 / 10 (OOM 2 + 跳过 2)	6 / 10 (OOM 4)
完成案例中位 RMSD	1.98 Å	2.20 Å	0.93 Å
完成案例 < 2 Å 比例	5/10 (50%)	3/6 (50%)	5/6 (83%)
完成案例 PB-Valid 比例	6/10 (60%)	2/6 (33%)	1/6 (17%)
严格通过 (<2 Å & PB-Valid)	4/10 (40%)	2/6 (33%)	1/6 (17%)
OOM 风险 (HPC 14.57 GiB)	无 (云端)	≥2 链就翻车	1 链 >500 res 或 2 链 >600 res 翻车

两条最值得写的发现:

1. Chai-1 几何最准,但物理合理性最差——中位 RMSD 0.93 Å 明显低于 Boltz API/HPC 的 1.98 / 2.20 Å,但 PoseBusters 22 项物理检查只有 1/6 案例全过。常见失败原因是配体与蛋白/辅因子的最小距离过近。Chai 把配体放对位置,但放得"太近"了。
2. Boltz 和 Chai 在 14.57 GiB 上 OOM 形状完全不同:Boltz HPC 可以跑 1 链 544 残基,但 2 链 465 残基就 OOM;Chai 反过来——2 链 465 残基能跑出 0.90 Å,但 1 链 544 残基 OOM。工程选型时两家是互补的。

Three lanes ran in parallel on 10 randomly sampled PoseBusters V1 complexes: Boltz-2.1 (Compute API), Boltz-2.0 (HPC local), and Chai-1 (HPC local). The goal: see which is more accurate (geometry), which is more physically valid, and which OOMs first on the same 14.57 GiB HPC GPU.

Editorial note: This is a BioTender pilot run, not a statistically rigorous benchmark. n=10 is too small for strong conclusions — read this as "engineering field notes + qualitative trends".

Metric	Boltz API (2.1)	Boltz HPC (2.0)	Chai-1 (HPC)
Completed / attempted	10 / 10	6 / 10 (2 OOM + 2 pre-skipped)	6 / 10 (4 OOM)
Median RMSD (completed)	1.98 Å	2.20 Å	0.93 Å
< 2 Å (completed)	5/10 (50%)	3/6 (50%)	5/6 (83%)
PB-Valid (completed)	6/10 (60%)	2/6 (33%)	1/6 (17%)
Strict pass (<2 Å & PB-Valid)	4/10 (40%)	2/6 (33%)	1/6 (17%)
OOM risk (HPC 14.57 GiB)	None (cloud)	≥2 chains fails	1 chain >500 res or 2 chains >600 res fails

Two most write-worthy findings:

1. Chai-1 has the best raw geometry but the worst physical validity: median RMSD 0.93 Å, much lower than Boltz API/HPC (1.98 / 2.20 Å), but only 1/6 cases pass all 22 PoseBusters physical checks. The common failure mode is ligand atoms too close to protein/cofactors. Chai puts the ligand in the right place, but too close.
2. Boltz and Chai have completely different OOM profiles at 14.57 GiB: Boltz HPC runs 1-chain 544 res but OOMs on 2-chain 465 res; Chai is the opposite — 2-chain 465 res yields 0.90 Å, but 1-chain 544 res OOMs. The two are complementary in deployment.

1.1 三条赛道

赛道	模型	入口	备注
A. Boltz API	boltz-2.1	Boltz Compute API (云端 GPU)	live mode,$0.05/call × 10 = ~$0.50 上限
B. Boltz HPC	boltz-2.0	hpc_run_tool("Boltz-2", ...)	14.57 GiB GPU 限制
C. Chai-1	latest container	hpc_run_tool("Chai-1", ...)	同 GPU 限制

故意保留两路 Boltz:A 跑的是 2.1(已升级),B 跑的是 2.0(HPC 还没更新到 2.1)。这给了一个"同模型不同版本/不同部署环境"的对照点。

1.2 数据集

来源:PoseBusters V1(2023-12 版),PDB ID 划分到 2021-01 之后避免与 Boltz/Chai 训练数据严重重叠。抽样:seed=42 随机抽 10 个,覆盖单链/多链、不同大小、含/不含辅因子。

case_id	n_chains	总残基	备注
7W05_GMP	1	100	小分子
7JUD_MMA	2	254	glucose 类糖
7BTT_F8R	1	304
8AP0_PRP	1	332
8AIJ_M9I	4	456	四聚体
7AFX_R9K	2	465	含 Cl
6W59_SZD	1	544
7KQU_YOF	2	602
7AVI_S2Q	2	944
7LOU_IFM	2	1085	最大

1.3 指标

• 配体 RMSD:对称纠正(spyrmsd),相对配体口袋链做单链 CA 叠合。PoseBusters 论文推荐的 pocket-aware single-chain superposition。
• PB-Valid:PoseBusters 22 项物理合理性检查全过即为 valid。
• 严格成功:RMSD < 2 Å 且 PB-Valid。
• 自报置信度:Boltz 取 structure_confidence (0–1),Chai 取 aggregate_score (0–1)。

1.4 多样本处理

Chai-1 默认输出 5 个 diffusion sample,统一取 aggregate_score 最高的那个作为 top-1。Boltz API/HPC 默认输出 1 个 sample。

工程小坑:本次实验中部分早期 Chai job 只返回了 1 个 sample(容器版本差异?),后期 job 都返回 5 个。当前数据保留实际返回数量并按 top-1 处理,未做重跑。

1.1 Three Lanes

Lane	Model	Entry point	Notes
A. Boltz API	boltz-2.1	Boltz Compute API (cloud GPU)	live mode, $0.05/call × 10 = ~$0.50 cap
B. Boltz HPC	boltz-2.0	hpc_run_tool("Boltz-2", ...)	14.57 GiB GPU limit
C. Chai-1	latest container	hpc_run_tool("Chai-1", ...)	same GPU limit

We deliberately kept two Boltz lanes: A runs 2.1 (upgraded), B runs 2.0 (HPC not yet updated). This gives us a "same model, different version / different deployment" control.

1.2 Dataset

Source: PoseBusters V1 (2023-12), PDB IDs after 2021-01 to minimize training-data overlap with Boltz/Chai. Sampling: seed=42, random 10, covering single/multi-chain, varying size, with/without cofactors.

case_id	n_chains	total residues	notes
7W05_GMP	1	100	small molecule
7JUD_MMA	2	254	glucose-like sugar
7BTT_F8R	1	304
8AP0_PRP	1	332
8AIJ_M9I	4	456	tetramer
7AFX_R9K	2	465	contains Cl
6W59_SZD	1	544
7KQU_YOF	2	602
7AVI_S2Q	2	944
7LOU_IFM	2	1085	largest

1.3 Metrics

• Ligand RMSD: symmetry-corrected (spyrmsd), relative to single-chain CA superposition on the pocket chain. This is the pocket-aware single-chain superposition recommended in the PoseBusters paper.
• PB-Valid: passing all 22 PoseBusters physical-validity checks.
• Strict success: RMSD < 2 Å AND PB-Valid.
• Self-reported confidence: Boltz uses structure_confidence (0–1), Chai uses aggregate_score (0–1).

1.4 Multi-sample handling

Chai-1 outputs 5 diffusion samples by default; we uniformly take the sample with the highest aggregate_score as top-1. Boltz API/HPC outputs 1 sample by default.

Engineering note: in this run, some early Chai jobs only returned 1 sample (container version difference?), while later jobs returned 5. We kept the actual returned count and used top-1 in each case rather than re-running.

完成案例中位 RMSD:Chai 0.93 Å < Boltz API 1.98 Å < Boltz HPC 2.20 Å

但 PB-Valid 完全反过来——Boltz API 6/10 (60%) > Boltz HPC 2/6 (33%) > Chai 1/6 (17%)。

这一组数字看下来,『准』和『合理』是两件事:

• Chai 知道配体应该在哪儿(位置正确),但常常让原子和蛋白/辅因子贴得过近;
• Boltz API 的 RMSD 中位虽然差一档,但更多模型符合物理常识;
• 做 docking 后筛选时,单靠 RMSD 评估会高估 Chai;单靠 PB-Valid 会低估它。

Median RMSD on completed cases: Chai 0.93 Å < Boltz API 1.98 Å < Boltz HPC 2.20 Å

But PB-Valid flips the order: Boltz API 6/10 (60%) > Boltz HPC 2/6 (33%) > Chai 1/6 (17%).

The takeaway: "accurate" and "physically valid" are two different things.

• Chai knows where the ligand goes (right pocket), but tends to place atoms too close to protein/cofactors;
• Boltz API's median RMSD lags by ~1 Å, but more outputs are physically reasonable;
• For post-docking filtering: RMSD alone overrates Chai; PB-Valid alone underrates it.

抽几个 Chai 完成但 PB 不过的案例看失败检查项:

• 7BTT_F8R:PB-Valid ✓(唯一一个通过的)
• 7JUD_MMA、7W05_GMP、8AP0_PRP:常见失败项是 minimum_distance_to_protein 和 minimum_distance_to_organic_cofactors
• 8AIJ_M9I(四聚体):多个 cofactor distance 项失败

典型 Chai 失败模式:模型把配体放到了正确的口袋里、几乎对的姿势,但原子距离蛋白/辅因子太近,违反 van der Waals 限制。这是 diffusion 模型常见的『局部能量未优化』问题。

实用建议:Chai 输出后接一个 GPU-friendly 的能量最小化(比如 Amber 短跑 5 ps),应能大幅提高 PB-Valid 比例,且不破坏 0.93 Å 的几何质量。

Looking at which checks fail for the Chai cases that pass RMSD but fail PB-Valid:

• 7BTT_F8R: PB-Valid ✓ (the only Chai pass)
• 7JUD_MMA, 7W05_GMP, 8AP0_PRP: common failures are minimum_distance_to_protein and minimum_distance_to_organic_cofactors
• 8AIJ_M9I (tetramer): multiple cofactor distance failures

Typical Chai failure mode: the model places the ligand in the right pocket at almost the right pose, but with atoms too close to the protein/cofactors, violating van der Waals limits. This is a known "local energy not minimized" issue with diffusion models.

Practical recommendation: piping Chai output through a quick GPU-friendly energy minimization (e.g. a 5 ps Amber run) should dramatically improve PB-Valid while preserving the 0.93 Å geometry.

在 5 个两路都跑通的 case 上:

case	Boltz API (2.1)	Boltz HPC (2.0)	差异
7BTT_F8R	0.56 Å	0.48 Å	0.08 Å
7JUD_MMA	3.46 Å	3.48 Å	0.02 Å
7W05_GMP	0.63 Å	0.62 Å	0.01 Å
8AIJ_M9I	3.10 Å	3.14 Å	0.04 Å
8AP0_PRP	1.24 Å	1.27 Å	0.03 Å

5 个 case 的 max|diff| = 0.08 Å,median|diff| = 0.03 Å,说明 2.0 → 2.1 的模型权重在这批样本上几乎没动。

但单独有一个 6W59_SZD 完全不同:

• Boltz API (2.1):RMSD = 2.71 Å,confidence = 0.916
• Boltz HPC (2.0):RMSD = 8.57 Å,confidence = 0.918

HPC 版本 confidence 几乎一样高,但 RMSD 高出 5.86 Å——这是一次实打实的『高置信度灾难』。同一个分子、同样的 seed 输入,2.0 → 2.1 之间的某个改动彻底修好了这个 case。详见 §4.2。

On 5 cases that succeeded on both lanes:

case	Boltz API (2.1)	Boltz HPC (2.0)	diff
7BTT_F8R	0.56 Å	0.48 Å	0.08 Å
7JUD_MMA	3.46 Å	3.48 Å	0.02 Å
7W05_GMP	0.63 Å	0.62 Å	0.01 Å
8AIJ_M9I	3.10 Å	3.14 Å	0.04 Å
8AP0_PRP	1.24 Å	1.27 Å	0.03 Å

Across the 5 cases: max|diff| = 0.08 Å, median|diff| = 0.03 Å — the 2.0 → 2.1 weights barely moved on this sample.

But 6W59_SZD is a completely different story:

• Boltz API (2.1): RMSD = 2.71 Å, confidence = 0.916
• Boltz HPC (2.0): RMSD = 8.57 Å, confidence = 0.918

Nearly identical confidence, but RMSD differs by 5.86 Å — a genuine "high-confidence catastrophe". Same molecule, same seed, but something between 2.0 and 2.1 completely fixed this case. See §4.2.

在 14.57 GiB 的 GPU 上,本批 10 个 case 的 OOM 经验上界:

赛道	OOM 起点	OOM 案例
Boltz HPC (2.0)	2-chain ≥ 465 res	7AFX_R9K (2/465), 7AVI_S2Q (2/944),还有 7KQU_YOF / 7LOU_IFM 已主动跳过
Chai-1 HPC	1-chain ≥ 544 res, 2-chain ≥ 602 res	6W59_SZD (1/544), 7KQU_YOF (2/602), 7AVI_S2Q (2/944), 7LOU_IFM (2/1085)

反直觉的点:多链不一定更吃显存——8AIJ_M9I 是 4 链 456 残基,两路都跑得动;但同样 ~450 残基的 7AFX_R9K(2 链)却让 Boltz HPC OOM。

原因可能是 Chai 和 Boltz HPC 对多链做的 padding/MSA 处理不同,导致中等长度 + 2 链时容易超界。用 HPC 跑实际工作前一定要先做 OOM 边界扫描。

On a 14.57 GiB GPU, the empirical OOM upper bounds across this 10-case batch:

Lane	OOM threshold	OOM cases
Boltz HPC (2.0)	2-chain ≥ 465 res	7AFX_R9K (2/465), 7AVI_S2Q (2/944), and 7KQU_YOF / 7LOU_IFM proactively skipped
Chai-1 HPC	1-chain ≥ 544 res, 2-chain ≥ 602 res	6W59_SZD (1/544), 7KQU_YOF (2/602), 7AVI_S2Q (2/944), 7LOU_IFM (2/1085)

Counter-intuitive observation: more chains don't always mean more memory — 8AIJ_M9I has 4 chains × 456 residues and runs fine on both lanes; but 7AFX_R9K with ~450 residues and 2 chains OOMs on Boltz HPC.

The likely cause is different padding/MSA handling for multi-chain inputs between Chai and Boltz HPC, which can overflow on medium-length 2-chain systems. You must scan OOM boundaries before running real work on HPC.

把所有完成案例的(模型自报置信度, 真实 RMSD)做散点图(fig3):

• Boltz API:置信度大多 0.78–0.98,但 RMSD 从 0.56 Å 跨到 4.03 Å,几乎看不出趋势;
• Boltz HPC:6W59_SZD 这个点 confidence=0.918、RMSD=8.57 Å,严重打破任何相关性;
• Chai:agg_score 范围更窄(0.43–0.96),低分案例数量不够看趋势。

结论:这三个模型的自报置信度都不能直接当 ranking 用。在 docking 流水线里,如果你打算用 confidence 过滤 hits,需要先在自己 target 上做校准。

Scatter plot of (self-reported confidence, true RMSD) for all completed cases (fig3):

• Boltz API: confidence mostly 0.78–0.98, but RMSD spans 0.56–4.03 Å — almost no visible trend;
• Boltz HPC: 6W59_SZD sits at confidence=0.918, RMSD=8.57 Å, single-handedly breaking any correlation;
• Chai: agg_score has a narrower range (0.43–0.96), too few low-score cases to see a trend.

Conclusion: none of the three confidences are directly usable as ranking signals. If you plan to use confidence to filter hits in a docking pipeline, you must calibrate on your own targets first.

把『RMSD < 2 Å 占完成案例比例』和『PB-Valid 占完成案例比例』分开看:

模型	RMSD < 2 Å	PB-Valid	严格成功 (<2 Å AND PB-Valid)
Boltz API	5/10 (50%)	6/10 (60%)	4/10 (40%)
Boltz HPC	3/6 (50%)	2/6 (33%)	1/6 (17%)
Chai	5/6 (83%)	1/6 (17%)	1/6 (17%)

这是 PoseBusters 论文里反复强调的一点:评估方式不同,模型排名会颠倒。从 BioTender 选型角度,要根据下游用途选指标——做 hit ranking 重几何(选 Chai),做 lead optimization 进 MM/PBSA 重物理(选 Boltz API)。

Looking at "RMSD < 2 Å on completed cases" and "PB-Valid on completed cases" separately:

Model	RMSD < 2 Å	PB-Valid	Strict success (<2 Å AND PB-Valid)
Boltz API	5/10 (50%)	6/10 (60%)	4/10 (40%)
Boltz HPC	3/6 (50%)	2/6 (33%)	1/6 (17%)
Chai	5/6 (83%)	1/6 (17%)	1/6 (17%)

This echoes the PoseBusters paper: different evaluation criteria reorder the models. From a BioTender selection standpoint, pick the metric by downstream use — hit ranking → prefer geometry (Chai), lead optimization with MM/PBSA → prefer physics (Boltz API).

3 个返回 5 个 sample 的 Chai job(8AP0_PRP / 8AIJ_M9I / 7AFX_R9K)逐个 sample 算 RMSD,得到一个惊人的发现:

case	sample_0 RMSD	sample_0 PB	score-top (sample_4) RMSD	score-top PB	RMSD 损失
8AP0_PRP	0.852 Å	PB ✓	0.973 Å	PB ✗	+0.121 Å
8AIJ_M9I	3.085 Å	PB ✗	3.242 Å	PB ✗	+0.158 Å
7AFX_R9K	0.452 Å	PB ✓	0.896 Å	PB ✗	+0.444 Å

3/3 都是 sample_0 RMSD 最小,但 aggregate_score 偏偏每次都选 sample_4。最严重的 7AFX_R9K:sample_0 是亚埃级 PB-Valid 通过,sample_4 是 0.9 Å PB-Valid 失败,而 aggregate_score 差距 < 0.005。

这 15 个数据点上 aggregate_score 和 RMSD 相关性接近 0。如果当时直接用 sample_0,Chai 在 3 个 case 上的指标会从『1/6 PB-Valid』提升到『3/6 PB-Valid』,严格成功率从 17% 跳到 50%。

实操结论:用 Chai 跑 docking 时,不要只依赖 aggregate_score 选 top-1。可以考虑:(a) 输出多个 sample 给下游 scoring 重排;(b) 把 sample_0 也独立保存做 sanity check。

For the 3 Chai jobs that returned 5 samples (8AP0_PRP / 8AIJ_M9I / 7AFX_R9K), we computed RMSD per-sample and found a striking pattern:

case	sample_0 RMSD	sample_0 PB	score-top (sample_4) RMSD	score-top PB	RMSD penalty
8AP0_PRP	0.852 Å	PB ✓	0.973 Å	PB ✗	+0.121 Å
8AIJ_M9I	3.085 Å	PB ✗	3.242 Å	PB ✗	+0.158 Å
7AFX_R9K	0.452 Å	PB ✓	0.896 Å	PB ✗	+0.444 Å

In 3/3 cases sample_0 had the lowest RMSD, yet aggregate_score picked sample_4 every single time. The worst is 7AFX_R9K: sample_0 is sub-Å and PB-Valid, sample_4 is 0.9 Å and fails PB-Valid — and the aggregate_score gap is < 0.005.

Across these 15 datapoints aggregate_score vs RMSD has near-zero correlation. Had we taken sample_0 directly, Chai's PB-Valid rate on these 3 cases would have jumped from 1/6 to 3/6, and strict-success from 17% to 50%.

Practical takeaway: don't rely on aggregate_score alone to pick top-1 from Chai. Consider: (a) emitting multiple samples to a downstream re-scorer, or (b) always saving sample_0 separately as a sanity check.

Boltz Compute API 在本次实验前 estimate 单价 $0.05/call,10 个 case 上限 $0.50。

• 实际 API 提交 10 次,全部 succeeded;
• retrieve 接口没返回成本字段,auth.me() 也不返回。无法直接读账户余额;
• 但 estimate 阶段 $0.05/call 是合同价,$0.50 上限严格守住。

HPC 路径(Boltz HPC + Chai HPC)由用户的 HPC 配额承担,没有额外现金成本。

Boltz Compute API gave a pre-run estimate of $0.05/call, cap of $0.50 for 10 cases.

• 10 API calls submitted, all succeeded;
• Neither the retrieve endpoint nor auth.me() returns a cost field — we couldn't pull live balance;
• But $0.05/call was the contracted estimate, so the $0.50 cap was honored.

HPC lanes (Boltz HPC + Chai HPC) run on the user's HPC quota and carry no extra cash cost.

30 行 = 10 case × 3 lane。点击表头可排序;OOM 与 skipped 行无 RMSD/置信度。

30 rows = 10 cases × 3 lanes. Click any header to sort; OOM and skipped rows have no RMSD/confidence.

4.1 7AFX_R9K — sample 选错把准度浪费掉

背景:2 链 465 残基。Boltz HPC OOM,Boltz API 给出 3.03 Å(失败),但 Chai 给出 0.90 Å(score-top)。

看 Chai 5 个 sample:

• sample_0:0.452 Å,PB-Valid ✓(几乎完美)
• sample_1:0.45 Å 左右
• sample_2/3:中等
• sample_4:0.896 Å,PB-Valid ✗(score 最高)

aggregate_score 把 sample_4 排第一,差距 < 0.005。这是整个实验里 Chai 损失最多的一个 case——如果用 sample_0,这个 case 就是一个亚埃级 + PB-Valid 通过的完美 docking。

4.2 6W59_SZD — Boltz 2.0 → 2.1 的『单点修复』

背景:1 链 544 残基。Chai OOM。Boltz API (2.1) 给 2.71 Å,Boltz HPC (2.0) 给 8.57 Å。

• HPC 版本 confidence 0.918,API 版本 confidence 0.916,几乎一样。
• 但 RMSD 差距 5.86 Å。

意味着 2.0 → 2.1 的某次更新对这个 case 定向修复了 RMSD,但置信度估计模块完全没跟上。这是非常危险的失败模式:用户拿到 HPC 2.0 的高 confidence 结果,会以为 docking 是成功的,但实际偏离了 8.57 Å。用户向 Boltz 团队提交『2.0 高 conf 错位』bug 时,可以引用这个 case 作为对照。

4.3 8AIJ_M9I — 四聚体上三个模型『谁都不准、但 API 唯一合理』

背景:4 链 456 残基,三路全部跑通。RMSD 都在 3.1 Å 上下:

• Boltz API:3.10 Å,confidence 0.978,PB-Valid ✓
• Boltz HPC:3.14 Å,confidence 0.978,PB-Valid ✗
• Chai:3.24 Å,score 0.959,PB-Valid ✗

三个模型的 RMSD 在 3 Å 这个范围『集体偏离』,可能配体在这个口袋里有未训练的不寻常构象。但 Boltz API 是唯一一个物理合理的输出——其他两个被 22 项 PoseBusters 检查中的某些距离/化学项 reject。

这暴露了一个 BioTender 选型逻辑:当所有模型都『差不多准』时,谁的物理校验更稳更重要。

4.1 7AFX_R9K — sample selection wastes accuracy

Background: 2 chains, 465 residues. Boltz HPC OOMs. Boltz API gives 3.03 Å (fail), Chai gives 0.90 Å (score-top).

Looking at all 5 Chai samples:

• sample_0: 0.452 Å, PB-Valid ✓ (nearly perfect)
• sample_1: ~0.45 Å
• sample_2/3: middling
• sample_4: 0.896 Å, PB-Valid ✗ (highest score)

aggregate_score puts sample_4 on top with a gap < 0.005. This is the single biggest "accuracy left on the table" case in the experiment — sample_0 would have been a sub-Å + PB-Valid perfect docking.

4.2 6W59_SZD — a "spot fix" between Boltz 2.0 and 2.1

Background: 1 chain, 544 residues. Chai OOMs. Boltz API (2.1) gives 2.71 Å, Boltz HPC (2.0) gives 8.57 Å.

• HPC version confidence 0.918, API version confidence 0.916 — virtually identical.
• But RMSD differs by 5.86 Å.

Implication: some 2.0 → 2.1 update fixed the RMSD on this case but the confidence head wasn't retrained alongside. This is a dangerous failure mode — a 2.0 user would see a confident-looking result and trust it, while the structure is 8.57 Å off. This case is a clean reference if you file a "2.0 high-conf misplacement" bug with the Boltz team.

4.3 8AIJ_M9I — a tetramer where all three "miss" but only API is reasonable

Background: 4 chains, 456 residues, all three lanes complete. RMSDs cluster near 3.1 Å:

• Boltz API: 3.10 Å, confidence 0.978, PB-Valid ✓
• Boltz HPC: 3.14 Å, confidence 0.978, PB-Valid ✗
• Chai: 3.24 Å, score 0.959, PB-Valid ✗

All three models miss by ~3 Å — possibly because the ligand sits in an unusual conformation poorly represented in training. But Boltz API is the only physically valid output; the other two get rejected by some of the 22 PoseBusters distance/chemistry checks.

This exposes a BioTender selection principle: when all models are "roughly equally inaccurate", physical validity becomes the deciding factor.

5.1 方法学

• 三路并行提交:为公平比较,三个赛道用同一批 case + 同一份 ground truth(PoseBusters V1)+ 同一个种子。Boltz API 走云端,Boltz HPC 和 Chai HPC 走 hpc_run_tool。
• RMSD 计算:配体口袋单链 CA 叠合 → spyrmsd 对称纠正 → 与 ground truth 配体相比。所有数字均由 boltz_chai_pipeline.process_case() 一致地算出。
• PoseBusters 检查:整套 22 项(键长/键角/平面性/手性/原子距离/分子内 clash 等)。任意一项失败就 PB-Valid = False。
• 多 sample 处理:Chai-1 默认 5 sample,取 aggregate_score 最高者作为 top-1。Boltz API/HPC 默认 1 sample。

5.2 局限

1. n = 10 太小:统计置信区间很宽,某些"差异"可能只是随机波动。这是 BioTender 选型探针,不是 benchmark 级评估。
2. 未做 MSA 控制:三个模型用各自默认 MSA。Boltz API 的『2.1 升级』里可能包含 MSA pipeline 变化,我们无法解耦"权重升级 vs MSA 升级"。
3. Chai sample 数量不均:本批早期 job 只返回 1 个 sample,后期返回 5 个。我们用了实际返回的 top-1,没重跑早期 job。
4. 未做能量最小化对照:Chai 的 PB-Valid 失败大部分应该可以靠后 minimization 修。我们没跑这步,所以低估了"Chai + minimization"的综合表现。
5. 仅做 single-chain pocket superposition:对多链 case(7AVI_S2Q, 7LOU_IFM 等)可能不是最优的 RMSD 估计方式。
6. Boltz HPC OOM 不等于"模型不行":14.57 GiB GPU 是平台限制。如果换 80 GiB A100,大部分 OOM 应该都能跑通。
7. 无法访问 Boltz 账户实际消费数据:我们用估价 $0.05/call 推断 $0.50 上限,无法 cross-check。

5.1 Methods

• Three lanes submitted in parallel: same 10 cases, same ground truth (PoseBusters V1), same seed. Boltz API uses the cloud, Boltz HPC and Chai HPC use hpc_run_tool.
• RMSD calculation: pocket-aware single-chain CA superposition → spyrmsd symmetry-corrected → vs. ground truth ligand. All numbers come from a single boltz_chai_pipeline.process_case() implementation.
• PoseBusters checks: full 22-check battery (bond lengths/angles/planarity/chirality/atom distances/intramolecular clashes etc.). Any failure → PB-Valid = False.
• Multi-sample handling: Chai-1 defaults to 5 samples; we take aggregate_score top-1. Boltz API/HPC return 1 sample.

5.2 Limitations

1. n = 10 is too small: statistical confidence intervals are wide; some "differences" may be random. This is a selection probe for BioTender, not a benchmark.
2. No MSA control: each model used its own default MSA. The Boltz 2.1 "upgrade" may have changed the MSA pipeline; we can't disentangle "weight upgrade vs. MSA upgrade".
3. Chai sample counts uneven: early jobs returned 1 sample, later jobs 5. We used whatever was returned and took top-1; didn't re-run early jobs.
4. No energy minimization control: most Chai PB-Valid failures should be fixable by post-minimization. We didn't run it, so we are under-estimating "Chai + minimization".
5. Single-chain pocket superposition only: may not be optimal for multi-chain cases like 7AVI_S2Q, 7LOU_IFM.
6. Boltz HPC OOM ≠ "model can't handle it": the 14.57 GiB GPU is a platform limit. On an 80 GiB A100 most OOMs would likely run.
7. No direct visibility into Boltz account spend: we relied on the $0.05/call estimate to derive the $0.50 cap; couldn't cross-check.

所有产物都在 boltz_vs_chai_pilot/ 下:

boltz_vs_chai_pilot/
├── REPORT.md                              # 中文 markdown 原稿
├── REPORT.html                            # 本文件(双语 HTML)
├── data/
│   ├── results.csv                        # 30 行单一数据源(case × lane)
│   ├── selected_cases.csv                 # seed=42 抽样的 10 个 case
│   ├── sampling_meta.json                 # 抽样配置
│   ├── posebusters_api.json               # Boltz API 输入 payload
│   ├── posebusters_boltz_hpc.json         # Boltz HPC 输入 payload
│   ├── api_jobs.json                      # API job ID 映射
│   ├── boltz_hpc_jobs.json                # Boltz HPC job ID 映射
│   ├── chai_hpc_jobs.json                 # Chai HPC job ID 映射
│   ├── api_rmsds.json                     # API 每 case RMSD/PB/conf
│   ├── boltz_hpc_rmsds.json               # Boltz HPC 每 case RMSD/PB/conf
│   ├── chai_hpc_rmsds.json                # Chai HPC 每 case RMSD/PB/conf
│   ├── chai_per_sample_rmsd.json          # Chai 5-sample 全 RMSD(§2.7)
│   ├── chai_per_sample_derived.json       # Chai sample 选择诊断
│   ├── {case}_meta.json                   # 每 case 元数据(链/残基/SMILES)
│   ├── ground_truth_ligands/              # PoseBusters 提供的真实配体 SDF
│   └── ground_truth_proteins/             # PoseBusters 提供的真实蛋白 PDB
├── figures/
│   ├── fig1_rmsd_comparison.{png,svg}     # 30 列分组柱状(主图)
│   ├── fig2_success_rates.{png,svg}       # 3-panel 成功率
│   ├── fig3_confidence_vs_rmsd.{png,svg}  # 置信度 vs 真实 RMSD
│   ├── fig4_api_hpc_consistency.{png,svg} # API vs HPC + 三路共有 5 case
│   ├── fig5_oom_landscape.{png,svg}       # (n_chains, residues) OOM 地图
│   └── fig6_chai_sample_selection.{png,svg} # Chai 5-sample 选择诊断
└── predictions/
    ├── boltz_api/                         # API 返回的 CIF + scores.json
    ├── boltz_hpc/                         # HPC 返回的 model_0.cif + scores
    └── chai_hpc/                          # Chai 返回的 5-sample CIF + npz

主流水线模块在 /mnt/shared-workspace/boltz_chai_pipeline.py(跨机器持久化)。

All artifacts live under boltz_vs_chai_pilot/:

boltz_vs_chai_pilot/
├── REPORT.md                              # Chinese markdown source
├── REPORT.html                            # this file (bilingual HTML)
├── data/
│   ├── results.csv                        # 30-row single source of truth (case × lane)
│   ├── selected_cases.csv                 # 10 cases sampled with seed=42
│   ├── sampling_meta.json                 # sampling config
│   ├── posebusters_api.json               # Boltz API input payload
│   ├── posebusters_boltz_hpc.json         # Boltz HPC input payload
│   ├── api_jobs.json                      # API job ID mapping
│   ├── boltz_hpc_jobs.json                # Boltz HPC job ID mapping
│   ├── chai_hpc_jobs.json                 # Chai HPC job ID mapping
│   ├── api_rmsds.json                     # API per-case RMSD/PB/conf
│   ├── boltz_hpc_rmsds.json               # Boltz HPC per-case RMSD/PB/conf
│   ├── chai_hpc_rmsds.json                # Chai HPC per-case RMSD/PB/conf
│   ├── chai_per_sample_rmsd.json          # Chai 5-sample full RMSD (§2.7)
│   ├── chai_per_sample_derived.json       # Chai sample selection diagnostic
│   ├── {case}_meta.json                   # per-case metadata (chains/residues/SMILES)
│   ├── ground_truth_ligands/              # PoseBusters ground-truth ligand SDFs
│   └── ground_truth_proteins/             # PoseBusters ground-truth protein PDBs
├── figures/
│   ├── fig1_rmsd_comparison.{png,svg}     # 30-column grouped bar (main)
│   ├── fig2_success_rates.{png,svg}       # 3-panel success rates
│   ├── fig3_confidence_vs_rmsd.{png,svg}  # confidence vs true RMSD
│   ├── fig4_api_hpc_consistency.{png,svg} # API vs HPC + 5 cases common to all 3 lanes
│   ├── fig5_oom_landscape.{png,svg}       # (n_chains, residues) OOM map
│   └── fig6_chai_sample_selection.{png,svg} # Chai 5-sample selection diagnostic
└── predictions/
    ├── boltz_api/                         # API-returned CIF + scores.json
    ├── boltz_hpc/                         # HPC-returned model_0.cif + scores
    └── chai_hpc/                          # Chai-returned 5-sample CIF + npz

The main pipeline module lives at /mnt/shared-workspace/boltz_chai_pipeline.py (persisted across machines).

实验完成后,有几条值得拎出来直接做选题:

1. 『高置信度也能 8 Å 错位』:Boltz 2.0 → 2.1 的隐藏修复(6W59_SZD 案例)。两版本 confidence 几乎一样,RMSD 差 5.86 Å——给『不要盲信 confidence』的论点提供一个干净对照。
2. 『准的姿势 vs 合理的姿势』:Chai vs Boltz API 的指标颠倒(§2.1 + §2.6)。同一批 case 上,RMSD 排名和 PB-Valid 排名完全相反——这是给『选指标 = 选模型』观点的最直接论据。
3. 『为什么 4 链能跑而 2 链 OOM?』:HPC 部署的反直觉边界(§2.4 + fig5)。8AIJ_M9I (4 链 456 残基) 两路都跑,7AFX_R9K (2 链 465 残基) 让 Boltz HPC OOM——给『跑 docking 前先扫边界』提供具体经验。
4. 『5 个 sample 都跑了, 结果选了最差的那个』(§2.7,新发现)。Chai aggregate_score 在 3/3 多 sample case 上都没选中 sample_0 的最优解,差距小但代价大(最严重 +0.44 Å + PB-Valid 翻转)——给『不要无脑信 ensemble score』提供干净案例。

Several findings are ready to lift as standalone editorial topics:

1. "High confidence can be 8 Å off": the hidden Boltz 2.0 → 2.1 fix (6W59_SZD case). Identical confidence, 5.86 Å RMSD difference between versions — a clean counterexample for the "don't blindly trust confidence" argument.
2. "Accurate pose vs valid pose": the Chai vs Boltz API metric inversion (§2.1 + §2.6). On the same cases, RMSD ranking and PB-Valid ranking flip completely — the cleanest evidence for "choosing the metric = choosing the model".
3. "Why does 4 chains run but 2 chains OOM?": counter-intuitive HPC deployment boundaries (§2.4 + fig5). 8AIJ_M9I (4 chains, 456 res) runs everywhere; 7AFX_R9K (2 chains, 465 res) OOMs Boltz HPC — concrete evidence for "scan the OOM boundary first".
4. "You ran 5 samples and picked the worst one" (§2.7, new finding). Chai's aggregate_score missed sample_0 in 3/3 multi-sample cases, by a tiny score margin but with large quality cost (worst case: +0.44 Å + PB-Valid flip) — a clean case for "don't blindly trust ensemble scores".