One Weird Trick That Will Speed Up Your Bundle Install

By Brian John

Ok ok, I apologize for the clickbait title. But this time it’s for real a weird trick and it for real (well, maybe) will improve your bundle performance. And no, I’m not referring to the --jobs argument, though while we’re on the subject let’s talk about that too.

As a note I’m going to run benchmarks with the following dependency versions on a Macbook Pro with an 8-core i9 and 32G of RAM:

1. Rails 6.1.3.1
2. Ruby 3.0.1
3. Bundler 2.2.15

--jobs

This is a commonly used argument with bundle install to improve performance. According to the documentation:

--jobs=[<number>], -j[<number>]          
    The maximum number of parallel download and install jobs. The default is 1.

Well now that seems pretty neat. Let’s try it on a new Rails project.

First the setup:

$ rails new bundle-install-performance
<SNIP>
$ gem uninstall --all --force --silent # clear out the gems so we can benchmark install

Ok now let’s see what performance is like by default:

$ time bundle install
<SNIP>
bundle install  
90.16s user 29.66s system 139% cpu 1:26.08 total

Ok, now we have our baseline of roughly a minute and a half. Let’s try the --jobs argument to speed things up. Since I’m using an 8-core machine, let’s try 1 job per core:

$ time bundle install --jobs 8
<SNIP>
bundle install --jobs 8  
86.79s user 28.04s system 138% cpu 1:22.91 total

Hmmm…well that didn’t seem to help hardly at all. I guess we might have picked up a few seconds. Just to rule out context switching, let’s try --jobs 4:

$ time bundle install --jobs v
<SNIP>
bundle install --jobs 4  
87.18s user 25.16s system 134% cpu 1:23.41 total

Ok, seems about the same.

But why? We have 8 cores and plenty of memory, why doesn’t more jobs make this faster? Well, as far as I can tell it looks like Bundler uses threads for parallelism, which means our old friend the GIL/GVL is in play, at least when using MRI.

At a high level, this means that we can get concurrency in our I/O operations, but not in CPU operations (this isn’t 100% true, but we could spend a whole series of blog posts on that). Rails includes a bunch of native gem extensions by default, which means they have to be compiled, which means lots of CPU operations.

It looks like Bundler switched from using processes to threads for MRI back in 2014, for what seem to be really solid reasons. So I don’t think that is going to change any time soon.

The MAKE environment variable

So we know that more threads don’t seem to help us much, at least not with our default Rails/MRI setup, and that we’re probably going to need to have a way to get multiple CPU cores involved in order to use parallelism to speed things up. Bundler doesn’t seem to have anything out of the box that will help us here, so what will?

Well, it just so happens that most native extensions use the rake-compiler library, which uses make under the hood, which also happens to support a --jobs argument:

-j [jobs], --jobs[=jobs]

Specifies the number of jobs (commands) to run simultaneously. If there is more than one -j option, the last one is effective. If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.

The nice thing about this --jobs argument is that it does in fact parallelize across CPU cores. So if we can find a way to pass that argument when running bundle install, we should be able to speed things up.

It turns out that there is a way to do this, and indeed we can see it being used in the rake-compiler-dev-box library to improve compile performance:

$ echo 'export MAKE="make -j$(nproc)"' >> $home/.bash_profile

Neat! So it looks like we can use the MAKE environment variable to pass a custom make command that includes the --jobs (or -j) argument! Let’s give it a try:

$ time MAKE="make --jobs 8" bundle install
<SNIP>
bundle install
133.25s user 35.47s system 468% cpu 36.004 total

Whoa! 36 seconds?! That sped our bundle install up by almost 3X, a pretty nice improvement.

Wrapping things up

If your Gemfile has a bunch of native extensions in it, and you’re using MRI, then you can use the MAKE environment variable to pass the --jobs argument to make, and it will likely improve bundle install performance dramatically (assuming you have multiple CPU cores available, caveat emptor, etc. etc.).

About the Author

Brian joined BetterUp in 2017 as a Senior Full Stack Engineer. He lives in the Minneapolis area with his spouse and two sons. Brian is a lover of family time, hiking, biking, video games, and serial optimization.