criterion performance measurements
overview
want to understand this report?
rdf4h/query/TList SPO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8566577344202345e-3 | 1.881220599645832e-3 | 1.973823120207713e-3 |
| Standard deviation | 1.1316747643854409e-5 | 1.5373462656655172e-4 | 3.26374146186185e-4 |
Outlying measurements have severe (0.5954054570889901%) effect on estimated standard deviation.
rdf4h/query/TList SP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8560289369371312e-3 | 1.8805537397983347e-3 | 1.9502732333577996e-3 |
| Standard deviation | 1.4915133390489111e-5 | 1.5233001608519925e-4 | 2.92710018991143e-4 |
Outlying measurements have severe (0.5951425874561408%) effect on estimated standard deviation.
rdf4h/query/TList S
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8581193968387167e-3 | 1.8816555410616084e-3 | 1.9490393640726564e-3 |
| Standard deviation | 9.955745235347027e-6 | 1.4825340486513463e-4 | 2.8528645367643495e-4 |
Outlying measurements have severe (0.5782072183432652%) effect on estimated standard deviation.
rdf4h/query/TList PO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0518252541414234e-3 | 2.0665766116303216e-3 | 2.121701829101684e-3 |
| Standard deviation | 5.738971796135128e-6 | 8.909625745739015e-5 | 1.887640428952701e-4 |
Outlying measurements have moderate (0.28512731886814663%) effect on estimated standard deviation.
rdf4h/query/TList SO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8712194250357459e-3 | 1.8934406566030104e-3 | 1.9646650551096013e-3 |
| Standard deviation | 2.0897661474287074e-5 | 1.420103549067063e-4 | 2.706928655352814e-4 |
Outlying measurements have severe (0.5606366432712175%) effect on estimated standard deviation.
rdf4h/query/TList P
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.387593399343807e-3 | 2.406890899439356e-3 | 2.45105354353868e-3 |
| Standard deviation | 5.939124716243012e-5 | 1.0800546829191704e-4 | 1.766772383007473e-4 |
Outlying measurements have moderate (0.28368092512287374%) effect on estimated standard deviation.
rdf4h/query/TList O
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8420430065911194e-3 | 1.8687350271768264e-3 | 1.9600121396808477e-3 |
| Standard deviation | 8.916813789310034e-5 | 1.7689462202016194e-4 | 3.066076261913321e-4 |
Outlying measurements have severe (0.6643813981147408%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap SPO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.367846149296525e-7 | 9.405896046313825e-7 | 9.490262221681977e-7 |
| Standard deviation | 8.282582920534722e-9 | 2.153248500569954e-8 | 3.7894176437351226e-8 |
Outlying measurements have moderate (0.2890698500249667%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap SP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.1752623685455506e-6 | 4.227466569218847e-6 | 4.342363732695747e-6 |
| Standard deviation | 1.0394437145242637e-7 | 2.4856489353773025e-7 | 4.78369990811878e-7 |
Outlying measurements have severe (0.7016158095421492%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap S
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.968848463955457e-6 | 3.996480707984559e-6 | 4.043496238375734e-6 |
| Standard deviation | 9.734744084484738e-8 | 1.1529666520023121e-7 | 1.2890449894571592e-7 |
Outlying measurements have moderate (0.3574200271467804%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap PO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2233762481392321e-6 | 1.2268546611987533e-6 | 1.2316079587784208e-6 |
| Standard deviation | 1.1570379269256417e-8 | 1.4019316564517396e-8 | 1.8442677495866188e-8 |
Outlying measurements have slight (9.122198983273214e-2%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap SO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.884160716273954e-7 | 8.909984373401903e-7 | 8.9856138204679e-7 |
| Standard deviation | 4.1210317754671105e-9 | 1.5630267625090113e-8 | 2.8707635159985708e-8 |
Outlying measurements have moderate (0.19135521549546416%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap P
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.094108056277097e-2 | 2.148583767404199e-2 | 2.2294680451563915e-2 |
| Standard deviation | 1.4818256304572697e-3 | 1.9079744140440545e-3 | 2.611469284071095e-3 |
Outlying measurements have moderate (0.41124110592388236%) effect on estimated standard deviation.
rdf4h/query/AdjHashMap O
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1709952501787922e-6 | 1.174680920187156e-6 | 1.1792706006452135e-6 |
| Standard deviation | 1.0417290810963409e-8 | 1.4978415118381305e-8 | 2.2473350092959654e-8 |
Outlying measurements have moderate (0.10923340460769347%) effect on estimated standard deviation.
rdf4h/select/TList SPO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.767058181175943e-3 | 2.7827179511510764e-3 | 2.83921873613671e-3 |
| Standard deviation | 6.835009453214368e-5 | 1.0523940075857891e-4 | 1.8406711726139935e-4 |
Outlying measurements have moderate (0.2234349656315285%) effect on estimated standard deviation.
rdf4h/select/TList SP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8420596123784198e-3 | 2.857462921887168e-3 | 2.889379980082532e-3 |
| Standard deviation | 7.026554649888403e-5 | 9.965006483946229e-5 | 1.359851735474789e-4 |
Outlying measurements have moderate (0.18639307658850135%) effect on estimated standard deviation.
rdf4h/select/TList S
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.578207369416664e-3 | 2.5986811200179078e-3 | 2.6581885233089597e-3 |
| Standard deviation | 3.090393256395757e-5 | 1.229905533712502e-4 | 2.2915511901486945e-4 |
Outlying measurements have moderate (0.30721365483448826%) effect on estimated standard deviation.
rdf4h/select/TList PO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.263316557864624e-3 | 3.277608298349415e-3 | 3.3119672404828184e-3 |
| Standard deviation | 2.5279689763985664e-5 | 6.243285975576027e-5 | 1.043566863837092e-4 |
Outlying measurements have slight (5.899549674583188e-2%) effect on estimated standard deviation.
rdf4h/select/TList SO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.47530805256395e-3 | 2.498079688068901e-3 | 2.5508395418558593e-3 |
| Standard deviation | 3.219924687714523e-5 | 1.0399988342799321e-4 | 1.665733790418201e-4 |
Outlying measurements have moderate (0.2522824563578634%) effect on estimated standard deviation.
rdf4h/select/TList P
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8828458990430893e-3 | 2.896308919785602e-3 | 2.9250111842846795e-3 |
| Standard deviation | 5.885626888018041e-6 | 6.096205654635355e-5 | 1.0153182216504443e-4 |
Outlying measurements have slight (9.139230181971615e-2%) effect on estimated standard deviation.
rdf4h/select/TList O
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.920874225382207e-3 | 2.9354128247651803e-3 | 2.9644672395770432e-3 |
| Standard deviation | 4.387986260453897e-5 | 7.54058360821957e-5 | 1.1491701312182197e-4 |
Outlying measurements have moderate (0.1131806815908574%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap SPO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8022642710049724e-3 | 1.8113568757330966e-3 | 1.8339725325412278e-3 |
| Standard deviation | 2.7250572350546167e-5 | 4.984587298445146e-5 | 8.400451152198265e-5 |
Outlying measurements have moderate (0.1444047296645249%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap SP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2686519139667245e-2 | 2.3302860231911634e-2 | 2.3991473580756068e-2 |
| Standard deviation | 1.1565750706331902e-3 | 1.536110967358293e-3 | 2.1375632678286103e-3 |
Outlying measurements have moderate (0.23876318247091013%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap S
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.273964159307047e-2 | 2.3320737092595432e-2 | 2.4414878513918624e-2 |
| Standard deviation | 1.2407708646070522e-3 | 1.534094342104278e-3 | 2.0058179872665207e-3 |
Outlying measurements have moderate (0.23871632790155858%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap PO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.180530829092016e-3 | 6.206196552881583e-3 | 6.263286305220516e-3 |
| Standard deviation | 6.334266486457943e-5 | 1.0573965528321496e-4 | 1.5402920786870264e-4 |
Outlying measurements have slight (2.5623268698060662e-2%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap SO
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7093719871339927e-3 | 1.7286675213993396e-3 | 1.774211608882948e-3 |
| Standard deviation | 4.87594163782474e-5 | 9.48407165710199e-5 | 1.5723011630376966e-4 |
Outlying measurements have moderate (0.40164633287926%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap P
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.3695842499451786e-2 | 5.528449194907525e-2 | 5.6976846094458744e-2 |
| Standard deviation | 1.8659458860257779e-3 | 2.9341764502805656e-3 | 3.8367667196990553e-3 |
Outlying measurements have moderate (0.15382650562403954%) effect on estimated standard deviation.
rdf4h/select/AdjHashMap O
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9095608173787987e-2 | 1.9407658022393786e-2 | 1.9978256531714117e-2 |
| Standard deviation | 9.162535065828509e-4 | 1.1448335498607078e-3 | 1.4971694999696772e-3 |
Outlying measurements have moderate (0.2167299857760461%) effect on estimated standard deviation.
rdf4h/add-remove-triples/TList
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.446198019219572e-3 | 6.795092125195438e-3 | 7.192453099811926e-3 |
| Standard deviation | 7.500875208592397e-4 | 1.065772034744726e-3 | 1.5089402653698503e-3 |
Outlying measurements have severe (0.76710467617356%) effect on estimated standard deviation.
rdf4h/add-remove-triples/AdjHashMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3419606861433644 | 0.3543024817386557 | 0.36252822659589024 |
| Standard deviation | 0.0 | 1.2338618513904862e-2 | 1.4247408022828721e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
rdf4h/count_triples/TList
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0547950904806853e-3 | 2.185984986771324e-3 | 2.3068648762252816e-3 |
| Standard deviation | 3.3138157302153685e-4 | 4.285036440115011e-4 | 5.439996284686539e-4 |
Outlying measurements have severe (0.9046917191607147%) effect on estimated standard deviation.
rdf4h/count_triples/AdjHashMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.805118610714418e-3 | 4.82879394656909e-3 | 4.9021844657503345e-3 |
| Standard deviation | 6.528831526624253e-5 | 1.4538059058190317e-4 | 2.5710487684296404e-4 |
Outlying measurements have moderate (0.13448109509014758%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.