Relative time microbenchmark

Microbenchmark is a very useful function for timing the execution time of one or more programs. I often use it to compare different programs. So I created a function that produces the output relative to the time of one of the programs being tested.

Snippet available at GitHub
Download zip

For example, it is known that the singular value decomposition of a matrix is slower than the eigen-decomposition of its covariance matrix, but what  about the time needed to compute the covariance? Of course it depends on the size of the matrix. The example below shows the use of mbm_rel to compare these times.


M = matrix(runif(4000),200)
S = crossprod(M)
dim(M)# [1] 200  20

eigenS computes the covariance

eigenS <- function(X){
    eigen(var(X))
}
mbm_out = microbenchmark(svd = svd(M), eigen = eigen(S), eigenS = eigenS(X))
mbm_out

# Unit: microseconds
# expr         min       lq               mean      median       uq        max       neval
# svd        704.470 722.0075  911.6752  812.900  878.556  4163.942   100
# eigen     365.281 374.0500  477.9450  425.163  515.628   978.217   100
# eigenS    916.197 941.0050 1746.5251 1085.363 1245.119 50295.797   100

As expected, eigen is faster than svd but eigenS is slower. How much slower? Let's call mbm_rel

mbm_rel(mbm_out)
# Unit: microseconds
#              min      lq   mean median   uq        max          neval
# svd time         705.754 716.233 806.809 735.267 753.873 3459.045   100
# svd/eigen  1.943  1.936  2.009  1.965  1.919      3.694    100
# svd/eigenS  0.637  0.636  0.414  0.640  0.642     0.054     100

The time units can be changed, just like in the original function. For milliseconds, set unit = "ms"

mbm_rel(mbm_out, unit = "ms")   

# Unit: milliseconds

#                        min          lq     mean      median    uq      max    neval

# svd time         0.706    0.716    0.807     0.735    0.754    3.459  100

# svd/eigen       1.943    1.936    2.009    1.965    1.919    3.694   100

# svd/eigenS     0.637    0.636    0.414    0.640    0.642    0.054   100

The first row gives the time for svd and the following the svd time relative to the other functions. So svd takes almost twice the time of eigen but 60% of the time of eigenS. A more informative output would be the time of the slowest. easily done, set relpos= 3 (the third function)

mbm_rel(mbm_out, relpos = 3, unit = "ms")
# Unit: milliseconds
#                   min              lq mean median     uq max       neval
# eigenS time      1.109 1.126 1.948 1.149 1.174 64.490 100
# eigenS/svd       1.571 1.572 2.415 1.563 1.557 18.644 100
# eigenS/eigen     3.053 3.044 4.852 3.070 2.987 68.877 100

Easier to interpret.

It's also possible to reverse the relativity, that is have the proportion of time of the methods with respect to the target method by setting inverse = TRUE

mbm_rel(mbm_out, relpos = 3, inverse = TRUE, unit = "ms")   
# Unit: milliseconds
#                    min     lq         mean median   uq           max neval
# eigenS time     1.109 1.126 1.948 1.149 1.174 64.490 100
# svd/eigenS      0.637 0.636 0.414 0.640 0.642 0.054 100
# eigen/eigenS    0.328 0.329 0.206 0.326 0.335 0.015 100

One can retrieve the time units with

attributes(mt)$unit

# [1] "microseconds"

The argument rel_name assigns a different name to the target method.

mbm_rel returns the relative times table, setting reltime = FALSE, returns ones instead of the target method time summaries.

mbm_r <- mbm_rel(mbm_out, relpos = 3, reltime = FALSE,  unit = "ms")

mbm_r

#                           min        lq        mean    median    uq      max    neval

# eigenS             1.000    1.000    1.000    1.000    1.000     1.000    100

# eigenS/svd      1.571    1.572    2.415    1.563    1.557    18.644    100

# eigenS/eigen   3.053    3.044    4.852    3.070    2.987    68.877    100

A possible plot

barplot(mbm_r$median, names = rownames(mbm_r), main = "relative execution times")