This is Fundbase Nerds, written by the team behind Fundbase.

Go to Fundbase  

Rails & Mongoid Data Loading Optimizations

Posted by Marek Stanczyk on

In this post, I’ll write about how the need for optimizing a backend response slowly sneaks into our application as it grows, and how we can tackle this situation from the point of view of loading data from the database. It’s based on Rails with Mongoid, but some ideas are valid for ActiveRecord too.

Happy beginnings

Usually we start with an application that has a few models, controllers and pages that operate on those models (CRUD, more or less). The controllers have simple actions which look like they were copied from a Rails guide (or Rails generated), and tidy templates that render our nicely structured data (where a single each loop is the most complicated piece of code). And life is good.

Getting bigger

Our application is growing. We have more models which in turn have more relations. Some of the relations are becoming more complicated, e.g. polymorphic. The pages are displaying more aggregated data (e.g. dashboards and such) and quietly become slower. Initially, we prefer not to notice it, while the request takes at most a second or two, it’s kind of ok. At 5 seconds, we’re getting slightly nervous, and 10 seconds or more is no longer acceptable. Even worse if that’s just one of the requests from a frontend application, such as Fundbase. And thus comes the time to optimize. I won’t touch approaches such as splitting to more requests, caching, database indexing, etc., but will take a look at how to make it faster without changing the request or response.

Diving into the data load

There are many possible reasons why our request is slow, and I’m going to focus specifically on how the data is loaded from the database, processed in the controller and rendered in the template. Some issues are simple and rather easy to fix. If a page shows all records and there are many, we can add pagination, infinite scroll, limit the data with date conditions, etc.

When we’re displaying relations, we might have the n+1 queries problem, which can result in dozens or even hundreds of database queries, and can be fixed by eager loading. But there are some specifics with Mongoid, namely that it can’t eager-load polymorphic relations, nor more than one level of relations. Example (include Mongoid::Document omitted):

class Organization

class User
  belongs_to :organization

class Activity
  belongs_to :author, class_name: 'User'
  belongs_to :source, polymorphic: true

Let’s say we’re loading a list of activities

@activities = Activity.all

and rendering them with the user, its organization and the source name, e.g. (HAML):

- @activities.each do |activity|

This will result in one query for the activities + 3 queries (author, organization, source) for each activity, so for 10 activities it would be

1 (activities) + 10 (users) + 10 (organizations) + 10 (sources) = 31 queries.

We can reduce the User queries by using simple eager loading:

@activities = Activity.all.includes(:author)

which will result in only one query for users, i.e.

1 (activities) + 1 (users) + 10 (organizations) + 10 (sources) = 22 queries.

In ActiveRecord, we could enhance it to includes(author: [:organization]), but Mongoid doesn’t support this, nor eager-loading of polymorphic relations, so we’ll manually preload related organizations to a Hash (id => object):

@activities = Activity.all.includes(:author)
authors =
organization_ids =
organizations = Organization.any_in(id: organization_ids)
@organizations = { |o| [, o] }.to_h

We’ll modify the organization line in the template to:

= @organizations[]

and we’re down to

1 (activities) + 1 (users) + 1 (organizations) + 10 (sources) = 13 queries.

But what about the sources? They’re polymorphic, so we’ll have to do a little more work, i.e. split the sources by class name (stored in source_type) and getting their source_ids. Then we’ll preload the sources for each type, mapping them to a Hash again (we can use a single Hash for all types, because the object ids are unique in Mongoid).

sources_by_type = @activities.group_by(&:source_type)
@sources = sources_by_type.each_with_object({}) do
    |(type, activities), sources|
  source_ids =
  items = type.constantize.any_in(id: source_ids)
  sources.merge( { |item| [, item] }.to_h)

And adjust the template respectively:

= @sources[activity.source_id]

Now the query count depends on actual number of different source models, e.g. 3:

1 (activities) + 1 (users) + 1 (organizations) + 3 (sources) = 5 queries.

84% queries less, not bad. My actual optimization on the Fundbase User Dashboard page made the activities request more than 10x faster in production (from about 7 seconds to 0.5..1 second).

Polymorphic preloading gets more interesting if the models use inheritance, in which case you’d want to query only the root of the model hierarchy, not every subclass you encounter, but I’ll leave this as an exercise.

Additional notes

While reducing the number of queries, consider how many collection iterations you’re adding because your Ruby loops could take long too.

When you don’t need many attributes of the models, it can make a surprising difference not to fetch the entire documents from the database. There are a few options:

A bonus for the patient readers

Those of you who made it this far without falling asleep, you might have thought that it would be helpful to measure the changes you’re making, which in this case is the query count. I’m using a small script (a shell alias), which looks like this (single line):

alias count_queries='perl -nle "print $& if m{(?<=collection=)(\w+)|(?<=:count=>\")(\w+)}" log/development.log | sort | uniq -c; cat log/development.log | grep MOPED | wc -l'

It used grep -E originally, but later I needed to run it on a Mac, so switched to Perl. The usage is

count_queries; : > log/development.log

And the output looks like this (the last line being the total):

1 activities
2 channels
1 connection_requests
1 master_options
10 organizations
2 users

I hope you can put some of these ideas to good use. Thanks for reading!

Marek Stanczyk

Marek is a Fullstack developer at Fundbase.
Loves beautiful code and enjoys developing with Ruby and Rails the most.