Investigate performance of proving for padded height $2^{22}$ and 2^{23}$

marco · May 12, 2025, 3:44am

Investigate performance of proving for padded height $2^{22}$ and 2^{23}$

已打开 09:53PM - 02 Mar 25 UTC

🔴 prio: high ⏩ speedup

Running both with and without `TVM_LDE_TRACE="no_cache"`, I've seen some quite t…roubling performance numbers for proving when the padded height is $2^{23}$. We should get performance numbers and RAM consumption for these three padded heights and investigate: $2^{21}$ (for reference), $2^{22}$, and $2^{23}$. Specifically, a user with 2TB RAM and 256 cores reports that proving for a padded height of $2^{23}$ takes two hours where it, by my calculations, should take less than 600 seconds. And I ran out of RAM when attempting to create these proofs, even with `TVM_LDE_TRACE="no_cache"` and having 768GB RAM, which should be plenty for this configuration.

marco · May 14, 2025, 3:48pm

any update? I meet this issue many time even paded height 512K

danda · May 15, 2025, 3:27am

This is not my area. Hopefully one of the other devs can comment.