How to start and use the Dashboard to visualize and analyze benchmark results
Dashboard records all of your precious experiments in one place so that you don’t need spreadsheet anymore to record your experiments.
One key concept in Dashboard is Subrun
and Run
. Subrun
is a single benchmark run of a specific engine/benchmark/hardware/model combination. Run
is a group of Subrun
s.
Starting the dashboard is simple with a single command:
Expected output:
Once the server is running, open your browser and navigate to the local URL or using public URL. You can share the public URL to allow others to access it remotely.
The dashboard is organized into several key areas to help you analyze and compare benchmark results.
Overview Panel
The Overview Panel provides a high-level summary of your benchmark runs:
Key information includes:
This panel helps you quickly gauge the scope of your benchmarking activities.
The chart above shows the results of testing the --max-num-seq
parameter in the vLLM engine. In this run, the parameter was set to 1, 2, 4, 8, 16, 32, and 256, while the benchmark QPS
was fixed at 10. From the chart, we can clearly observe the impact of increasing --max-num-seq
on performance.
Filter of Benchmark runs and subruns
You can select a specific run to view the performance comparison of all its sub-runs. Or a specific sub-run to view all its detailed information.
Features include:
Sub-runs Metrics Comparison
In this section, you can view the metrics comparison of benchmark results for each sub-run:
Information displayed includes:
Use this view when you need to thoroughly analyze a specific benchmark run.
Detailed Information of Sub-run
If you select a specific sub-run, you can access all the information related to that sub-run:
Subrun Detail Information include:
You can upload your sub-run results to the community for others to view and compare using the button of Share Subrun to Gloabal Leaderboard
.
Once you click this button, you may need to log in with GitHub to record the uploader.
If the upload is successful, you will see the following Information, and you can find your uploaded data in the global leaderboard: