BRAVE: Block-wise Structural Regularization via Controlled Evidence Feedback for Reliable Label Aggregation under Sparse Crowdsourcing
With only a few annotations, a model can become more confident without gaining new evidence. BRAVE controls how earlier estimates feed into later updates to reduce this effect.
Zerun Niu — first author; led algorithm design, literature review, experimental design, code implementation, and experimental deployment.
Naive aggregation
correlated agreement treated as independent evidenceBRAVE — controlled feedback
decoupled worker and reliability updates
Dashed line = perfectly calibrated (confidence equals accuracy). Solid curve = this run's calibration curve; the dot marks the average operating point. These illustrative curves show overconfidence and calibration. They do not reproduce BRAVE's algorithm or reported per-benchmark results; see the write-up below for the actual evaluation.
The reliability failure
Sparse crowdsourcing creates a subtle failure mode: workers can share the same error pattern, so their agreement does not always provide independent evidence. Globally coupled inference can repeatedly reuse that correlated agreement, making a label posterior more confident than the annotations justify. We call this illusory evidence accumulation.
What I designed
BRAVE partitions workers into disjoint blocks and separates the posterior used inside each evidence block from the synchronized global posterior for each item. It controls evidence feedback through dual decoupled parameter updates: worker mixture weights use the global posterior, while shared reliability matrices use block-local posterior statistics. The class prior is anchored in observed annotation frequencies. This keeps sharpened cross-block consensus from being recycled into the shared reliability components.
My contribution
As first author, I led the algorithm design, literature review, experimental design, code implementation, experimental deployment, and analysis. I built the study around both predictive performance and uncertainty quality, because reliable aggregation requires more than accuracy alone.
Evidence
Across 14 crowdsourcing benchmarks, BRAVE achieved the lowest negative log-likelihood on 5 datasets and the best or tied-best expected calibration error on 9. Its accuracy was within 0.03 of the strongest external baseline on 11 datasets. We also completed a downstream reward-model calibration transfer experiment.
Status: published in TMLR (2026). Zerun Niu is first author. See the public OpenReview record and official code linked above.