criterion performance measurements
overview
want to understand this report?
HashMap/10/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.350511637805802e-6 | 5.454684223823665e-6 | 5.596293689291488e-6 |
| Standard deviation | 3.0180592906119355e-7 | 3.9477831531913154e-7 | 6.372477964052162e-7 |
Outlying measurements have severe (0.780416851345301%) effect on estimated standard deviation.
HashMap/10/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.147708358690615e-6 | 6.241420910321328e-6 | 6.325690688668163e-6 |
| Standard deviation | 2.563806774671315e-7 | 3.035643568209868e-7 | 3.704888805078739e-7 |
Outlying measurements have severe (0.6060650152901881%) effect on estimated standard deviation.
HashMap/10/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.302766796959031e-6 | 5.373163029669289e-6 | 5.4577201287301725e-6 |
| Standard deviation | 2.129768738669435e-7 | 2.6683410811121054e-7 | 3.450388362533162e-7 |
Outlying measurements have severe (0.6184361364010005%) effect on estimated standard deviation.
HashMap/10/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.604135170943669e-6 | 5.657951527082449e-6 | 5.716282817957925e-6 |
| Standard deviation | 1.5382074808312143e-7 | 1.860633399292875e-7 | 2.2741663276126347e-7 |
Outlying measurements have moderate (0.41122898201745983%) effect on estimated standard deviation.
HashMap/100/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.173941969615379e-5 | 6.255514511853873e-5 | 6.364430857400309e-5 |
| Standard deviation | 2.274653686233865e-6 | 3.077953972989967e-6 | 4.306527956644264e-6 |
Outlying measurements have severe (0.5322584397816242%) effect on estimated standard deviation.
HashMap/100/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.48094872552745e-5 | 7.579136045761407e-5 | 7.700396205926436e-5 |
| Standard deviation | 3.0354578236068394e-6 | 3.6198838375456535e-6 | 4.358701881912123e-6 |
Outlying measurements have severe (0.5129034530559006%) effect on estimated standard deviation.
HashMap/100/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.331651052679434e-5 | 6.420342901736961e-5 | 6.527623560499962e-5 |
| Standard deviation | 2.5788119645446477e-6 | 3.2424379747652133e-6 | 4.488457965813061e-6 |
Outlying measurements have severe (0.5480538095125769%) effect on estimated standard deviation.
HashMap/100/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.719109253669906e-5 | 6.801267947121804e-5 | 6.908146895828873e-5 |
| Standard deviation | 2.668047485593825e-6 | 3.1999661927475e-6 | 4.037579211883209e-6 |
Outlying measurements have severe (0.5047311724539719%) effect on estimated standard deviation.
HashMap/1000/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1366073931922398e-3 | 1.1489332564937488e-3 | 1.162970003355041e-3 |
| Standard deviation | 3.300346364784333e-5 | 4.354853501607262e-5 | 5.9742494310732535e-5 |
Outlying measurements have moderate (0.26692519944923976%) effect on estimated standard deviation.
HashMap/1000/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4049321083472266e-3 | 1.4233245240985145e-3 | 1.4475562337495354e-3 |
| Standard deviation | 4.9745377934880994e-5 | 6.670512431535928e-5 | 1.0782800103923173e-4 |
Outlying measurements have moderate (0.34501927120674575%) effect on estimated standard deviation.
HashMap/1000/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.145480435247264e-3 | 1.1579653633479892e-3 | 1.1715911831805745e-3 |
| Standard deviation | 3.4900754104909195e-5 | 4.465097497820876e-5 | 6.072182157538516e-5 |
Outlying measurements have moderate (0.26734669778101217%) effect on estimated standard deviation.
HashMap/1000/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2122509749086625e-3 | 1.2263970277503781e-3 | 1.246356707721818e-3 |
| Standard deviation | 4.515221147991096e-5 | 5.631820247363469e-5 | 7.117986560850641e-5 |
Outlying measurements have moderate (0.34401894493752516%) effect on estimated standard deviation.
HashMap/10000/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4548167924480439e-2 | 1.4747847755554669e-2 | 1.5235545557764981e-2 |
| Standard deviation | 3.270343159539198e-4 | 7.797206547433664e-4 | 1.6228827659457057e-3 |
Outlying measurements have moderate (0.2271210127133228%) effect on estimated standard deviation.
HashMap/10000/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8278270042200336e-2 | 1.8449047439908443e-2 | 1.8712145471327418e-2 |
| Standard deviation | 2.889307034870869e-4 | 4.999297501818292e-4 | 8.193141386742515e-4 |
Outlying measurements have slight (7.325208120541815e-2%) effect on estimated standard deviation.
HashMap/10000/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4464957367747234e-2 | 1.4603947158343551e-2 | 1.4740204432353325e-2 |
| Standard deviation | 2.577575464867356e-4 | 3.5829566101470495e-4 | 5.22549341409506e-4 |
Outlying measurements have slight (3.840000000000025e-2%) effect on estimated standard deviation.
HashMap/10000/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2323201046265846e-2 | 2.250540058226572e-2 | 2.2805930997889537e-2 |
| Standard deviation | 3.451975208873324e-4 | 5.353711532052398e-4 | 8.270924318499385e-4 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
Map/10/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.088458570130062e-6 | 6.1561325261823135e-6 | 6.220781479341807e-6 |
| Standard deviation | 1.7416246276466561e-7 | 2.151141033984622e-7 | 2.6611489458793193e-7 |
Outlying measurements have moderate (0.43886279951346174%) effect on estimated standard deviation.
Map/10/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.342293318417685e-6 | 6.424972518705575e-6 | 6.51469401574787e-6 |
| Standard deviation | 2.413825200179033e-7 | 2.8236077580784476e-7 | 3.4785558557049714e-7 |
Outlying measurements have severe (0.5514678937038551%) effect on estimated standard deviation.
Map/10/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.274694113324057e-6 | 6.364874205169261e-6 | 6.462498459122206e-6 |
| Standard deviation | 2.7388256464700855e-7 | 3.2392800473586864e-7 | 4.1049089773205954e-7 |
Outlying measurements have severe (0.6272632612734041%) effect on estimated standard deviation.
Map/10/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.77292337258571e-6 | 5.8506399655052666e-6 | 5.942043868744923e-6 |
| Standard deviation | 2.5697120213381794e-7 | 2.9652367222002135e-7 | 3.4447875901149844e-7 |
Outlying measurements have severe (0.6257626247260558%) effect on estimated standard deviation.
Map/100/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.026253314317013e-5 | 8.140134027880888e-5 | 8.278765725605201e-5 |
| Standard deviation | 3.3755393446996968e-6 | 4.103466034357068e-6 | 5.152053023857825e-6 |
Outlying measurements have severe (0.530335888550899%) effect on estimated standard deviation.
Map/100/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.780222401865599e-5 | 8.887308232692714e-5 | 8.99388931421662e-5 |
| Standard deviation | 3.072481867534384e-6 | 3.6793466864040608e-6 | 4.8221510951685785e-6 |
Outlying measurements have moderate (0.43018212818753165%) effect on estimated standard deviation.
Map/100/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.161928251511352e-5 | 9.738461429843429e-5 | 1.1422877274769494e-4 |
| Standard deviation | 1.3887000860018575e-5 | 2.843301140929913e-5 | 5.7518120352426003e-5 |
Outlying measurements have severe (0.9813924457048272%) effect on estimated standard deviation.
Map/100/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.89479121958436e-5 | 7.985839746624346e-5 | 8.086725918951054e-5 |
| Standard deviation | 2.8348115424037176e-6 | 3.414609216899798e-6 | 4.174200588190069e-6 |
Outlying measurements have moderate (0.4519216386441377%) effect on estimated standard deviation.
Map/1000/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6350852037593099e-3 | 1.6531410106746859e-3 | 1.673289430114421e-3 |
| Standard deviation | 5.354222495146126e-5 | 6.531532290587842e-5 | 8.25678185132783e-5 |
Outlying measurements have moderate (0.2659441652897075%) effect on estimated standard deviation.
Map/1000/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6441740438060714e-3 | 1.6629440335105793e-3 | 1.6873062281566595e-3 |
| Standard deviation | 5.354154195396869e-5 | 7.086881031731726e-5 | 1.1589430520077978e-4 |
Outlying measurements have moderate (0.29779236642360063%) effect on estimated standard deviation.
Map/1000/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6715745239157588e-3 | 1.6919634487603799e-3 | 1.7183840583928785e-3 |
| Standard deviation | 6.4774157796359e-5 | 7.935958185622123e-5 | 9.807212978369188e-5 |
Outlying measurements have moderate (0.33046314550216516%) effect on estimated standard deviation.
Map/1000/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4650354663869828e-3 | 1.4820990687262434e-3 | 1.501281095210793e-3 |
| Standard deviation | 5.240072061216293e-5 | 6.610574359303498e-5 | 8.776455037334513e-5 |
Outlying measurements have moderate (0.31948131536095664%) effect on estimated standard deviation.
Map/10000/Text
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4607443206879465e-2 | 2.4887222085121875e-2 | 2.5671762633751653e-2 |
| Standard deviation | 3.164685418581758e-4 | 9.125467159240947e-4 | 1.8341760410135544e-3 |
Outlying measurements have slight (9.848577771701898e-2%) effect on estimated standard deviation.
Map/10000/Identity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.430353215768331e-2 | 2.451876550672411e-2 | 2.483093746389403e-2 |
| Standard deviation | 4.277992123943386e-4 | 5.810854802750066e-4 | 8.379066829175414e-4 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
Map/10000/Coerce
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.43434354112207e-2 | 2.457059514939565e-2 | 2.4809929498957117e-2 |
| Standard deviation | 3.4316924114403746e-4 | 5.306609336189058e-4 | 8.331627931127385e-4 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
Map/10000/Parser
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9930190457663895e-2 | 3.026664856479706e-2 | 3.077432740256804e-2 |
| Standard deviation | 5.654034674525462e-4 | 8.299722535655993e-4 | 1.1509800196507988e-3 |
Outlying measurements have slight (5.536332179930796e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.