Evaluate LLM over-refusal rates with OR-Bench
Browse and filter machine learning models by category and modality
Leaderboard of information retrieval models in French
Convert Stable Diffusion checkpoint to Diffusers and open a PR
View RL Benchmark Reports
Measure BERT model performance using WASM and WebGPU
Display genomic embedding leaderboard
Search for model performance across languages and benchmarks
Rank machines based on LLaMA 7B v2 benchmark results
Predict customer churn based on input details
Compare audio representation models using benchmark results
Upload ML model to Hugging Face Hub
View LLM Performance Leaderboard
OR-Bench Leaderboard is a benchmarking tool designed to evaluate the performance of large language models (LLMs) with a specific focus on their over-refusal rates. It provides a comprehensive platform to assess how often LLMs refuse to provide answers, even when they should be capable of doing so. This metric is crucial for understanding model reliability and effectiveness in real-world applications.
• Over-refusal rate tracking: Measures how frequently LLMs decline to answer questions they should know. • Comparison across models: Allows users to compare multiple models based on refusal rates. • Real-time leaderboards: Provides up-to-date rankings of LLMs in a competitive format. • Interactive data exploration: Enables users to filter results by specific criteria like model size or dataset. • Transparency and reproducibility: Offers detailed methodologies and datasets for independent verification.
1. Why is OR-Bench Leaderboard important for evaluating LLMs?
OR-Bench Leaderboard is important because it helps identify models that are overly cautious, ensuring they provide meaningful answers rather than refusing when they have the capability to respond.
2. Can anyone submit their model to OR-Bench Leaderboard?
Yes, researchers and developers can submit their models for evaluation by following the submission guidelines provided on the platform.
3. How is the over-refusal rate calculated?
The over-refusal rate is calculated by evaluating how often a model refuses to answer questions it should reasonably be expected to answer, based on its training data and capabilities.
4. Does OR-Bench Leaderboard provide insights into model reliability?
Yes, the leaderboard offers insights into model reliability by highlighting how often models refuse to answer questions, helping users assess their practical effectiveness.
5. Are the datasets used for evaluation publicly accessible?
Yes, the datasets and evaluation methodologies used by OR-Bench Leaderboard are transparent and publicly accessible to ensure reproducibility and fairness.