Benchmarking doesn't hurt
While working on a problem you probably choose between several ways of implementing a solution. How do you decide which implementation to go for? Do you base this on style, lines of code or something else?
As you're considering your options please also consider performance. Maintainability and readability are very important, but when your project is serving many visitors a minute you'll want to speed things up.
So what can we do? One of the things we can do is benchmark our options.
Benchmarking sounds like a lot of work and headaches, but it doesn't have to be. If done right it doesn't take too much of your time and you can quickly run the next benchmark.
Ruby benchmark
Ruby ships with its own benchmarking library which is quite useful if you quickly want to compare two implementations. You specify two or more pieces of code, an iterations count and run it.
# benchmark.rb
require 'benchmark'
n = 10_000_000
Benchmark.bmbm do |x|
x.report %(string = '') do
n.times do
foo = 'bar'
foo = ''
end
end
x.report 'string.clear' do
n.times do
foo = 'bar'
foo.clear
end
end
end
The benchmark will give you a report with how long everything took to complete. In this case the lower number is better.
$ ruby ./benchmark.rb
Rehearsal ------------------------------------------------
string = '' 3.150000 0.570000 3.720000 ( 3.790554)
string.clear 2.460000 0.060000 2.520000 ( 2.534461)
--------------------------------------- total: 6.240000sec
user system total real
string = '' 2.910000 1.130000 4.040000 ( 4.084821)
string.clear 2.510000 0.070000 2.580000 ( 2.609390)
Using the iterations count you can quickly tests how well your code performs over time and when dealing with more objects. This is important because how many times code is called and how many objects you give it can greatly affect the outcome of your benchmarks.
Some tweaking of variables and Garbage Collection is involved here. Reading the documentation is a must!
benchmark-ips gem
There's also benchmark-ips, a gem which allows you to benchmark iterations per second. This is probably the easiest method for most tests. It's not the perfect benchmarking solution but it does take away a lot of the benchmarking headaches.
require 'benchmark/ips'
ARRAY = (1..100).to_a
Benchmark.ips do |x|
x.report('Array#each + push') do
array = []
ARRAY.each { |i| array.push i }
end
x.report('Array#map') do
ARRAY.map { |i| i }
end
x.compare! # Print the comparison
end
source: JuanitoFatas/fast-ruby
In this report the conclusion is already printed out for you making it even easier to see what performs beter.
Calculating -------------------------------------
Array#each + push 9.025k i/100ms
Array#map 13.947k i/100ms
-------------------------------------------------
Array#each + push 99.634k (± 3.2%) i/s - 505.400k
Array#map 158.091k (± 4.2%) i/s - 794.979k
Comparison:
Array#map: 158090.9 i/s
Array#each + push: 99634.2 i/s - 1.59x slower
What should we benchmark?
The standard benchmarking library in Ruby and the benchmark-ips gem make it really easy to set up a benchmark in minutes. There really is no excuse not to benchmark your code when you have doubts about its speed.
There is also a lot of benchmarking already done for Ruby. There is a very good repository fast-ruby by Juanito Fatas based on Erik Michaels-Ober's Baruco 2014 presentation. This repository contains a lists of comparisons between standard Ruby implementations and the benchmark code.
What should we benchmark then?
Unfortunately most implementation decisions aren't a matter of Array#each + push vs Array#map. The fast-ruby repository is set up to test code that perform the same kind of logic, not solve a specific problem.
There's also code involved from other people by using frameworks and libraries. We don't call one method to solve the entire problem, but the program performs multiple operations on one or more objects. All those operations combined determine the actual speed of the program.
Go back to one of your recent projects, find some complex logic and consider alternative implementations. You might be surprised how much faster another solution might be.
Just an example
A problem in a recent project at work was to wrap each not empty line in a string with span elements. In this case we needed all the speed we could get, because this operation was performed for a lot of translatable records from our database per request.
There are quite a lot of implementations to accomplish this, but I'll limit the solutions in the example below.
require "benchmark/ips"
STRING = "\nLorem ipsum\ntempor invidunt\n\naliquyam erat\nvero eos\nno sea. Lorem\n\nipsum\n"
# Desired end result:
# <span>Lorem ipsum</span>
# <span>tempor invidunt</span>
# <span>aliquyam erat</span>
# <span>vero eos</span>
# <span>no sea. Lorem</span>
# <span>ipsum</span>
Benchmark.ips do |x|
x.report("lines + map") do
STRING.squeeze("\n").strip.lines.map { |line| "<span>#{line}</span>" }.join
end
x.report("each_line + inject") do
STRING.squeeze("\n").strip.each_line.inject("") { |s, line| s << "<span>#{line}</span>" }
end
x.report("map with if") do
STRING.lines.map { |line| "<span>#{line}</span>" unless line.strip.empty? }.join
end
x.report("reject + map") do
STRING.lines.reject { |line| line.strip.empty? }.map { |line| "<span>#{line}</span>" }.join
end
x.report("gsub") do
STRING.gsub(/^(.+)$/) { |line| "<span>#{line}</span>" }
end
x.compare! # Output the comparison
end
Warming up --------------------------------------
lines + map 58.261k i/100ms
each_line + inject 49.970k i/100ms
map with if 38.717k i/100ms
reject + map 43.452k i/100ms
gsub 31.151k i/100ms
Calculating -------------------------------------
lines + map 582.281k (± 1.1%) i/s - 2.913M in 5.003431s
each_line + inject 500.948k (± 0.8%) i/s - 2.548M in 5.087657s
map with if 388.120k (± 0.3%) i/s - 1.975M in 5.087582s
reject + map 436.801k (± 1.2%) i/s - 2.216M in 5.074160s
gsub 309.976k (± 1.0%) i/s - 1.558M in 5.025262s
Comparison:
lines + map: 582281.2 i/s
each_line + inject: 500948.4 i/s - 1.16x (± 0.00) slower
reject + map: 436800.7 i/s - 1.33x (± 0.00) slower
map with if: 388119.7 i/s - 1.50x (± 0.00) slower
gsub: 309975.7 i/s - 1.88x (± 0.00) slower
You see that a combination of calls on different methods and objects can make quite the bit of difference.
I had actually hoped the gsub would have been the fastest, because it performs the least amount of operations on the surface. It wasn't really really an option as it becomes even slower with larger strings. (Some of the other solutions however, create more objects and take up more memory.)
Conclusion
The next time you're considering your options, set up a quick benchmark and see what's faster. Only add the code you need to solve the problem. Avoid database queries and large libraries, unless that's what you're benchmarking.
However, benchmarking does not always give you the complete picture about why your code might be slow, but it does give you a good indication about how your code is performing.
For more details on how to setup benchmarks in Ruby, take a good look at the Ruby docs and the benchmark-ips README.
So benchmark your code and hopefully your applications will respond faster next time.