How to Make a Data Grid Scale

At Smartly.io, our users have diverse, sometimes very demanding needs when it comes to tables for displaying and manipulating data. We wanted to cover all of the possible use cases with our new campaign management data grid implementation, where expandable rows, sticky columns and good performance are just some of the most important requirements.

In this blog post, we go through how we found the right tool for the job, and overcame the most critical challenges to build a truly well-performing and scalable data grid.

Our old data grid implementation was written in Angular and worked decently once the entire DOM had been created. However, creating it could take several seconds in the best case scenario, and as the creation time increased when the amount of data grew, it wasn’t really a scalable solution. Even though the top-level rows were paginated, pivoting (i.e. expanding rows) would increase the row count dramatically, which would slow things down.

With the new implementation, we wanted to take advantage of virtualization, which is a technique used for creating only those DOM nodes that are visible in the viewport. Using virtualization would result in a performance of O(1) instead of ~O(n), so the time to render the table would be constant regardless of the amount of data.

Find the right tools

The react-virtualized library felt like a good technology to go with, as we’ve successfully implemented it elsewhere in our tool before. We experimented with the Table component, which seemed to be quite flexible, and provided several useful features out of the box. It had one critical shortcoming, though: it didn’t have horizontal (↔) scrolling.

To bypass this limitation, we placed the table in a container, and allowed the table to grow horizontally (↔) as much as was needed to display all columns. We set the overflow-x CSS property on the container to auto, to allow scrolling of horizontally (↔) overflowing content. The workaround seemed perfect—until we noticed that the vertical (↕︎) scrollbar that was still in the table container, was hidden unless the horizontal (↔) scroll bar was scrolled to the far right.

We could have worked around this hiccup by building our own scrollbar with HTML. But, as we knew there were other alternatives, and our time and resources were limited, we decided that making our own solution from scratch wasn’t worth the effort yet. So, we went back to the drawing board.

Next, we tried a more low-level component from the react-virtualized library : the Grid component, which supports horizontal (↔) scrolling. We decided to try it out after seeing this demo involving Grid that was very close to what we were looking for with fixed columns and headers.

However, there was one major problem: window scrolling couldn’t be used with Grid. In addition, Grid is a very low-level component, which means we would’ve had to implement a lot of basic functionality ourselves before getting it to work how we wanted. So again, we put the Grid-based implementation on hold, and moved to explore the next alternative.

Fixed-data-table is a library built by Facebook who, unsurprisingly, has very similar needs when it comes to displaying tabular data. It’s the most feature-rich library we came across, covering most of our use-cases (even resizeable columns) out of the box. The only problem was that the library wasn’t actively maintained anymore. Luckily, a community-maintained fork of the library called fixed-data-table-2 had emerged, which, as a bonus, provided some useful features that the original library didn’t have.

Make it scroll

Even after finding a suitable library, we faced one big problem: vertical (↕︎) scrolling. In our tool, we have content above the table, some of which we want to scroll away, but some of which should stick above the table while the table content is scrolling. Our first approach involved overflowing the table by the height of the static content above it.

The problem here is that we want to enable scrolling the table contents with a wheel and touch pad regardless of where the cursor is on the page. Otherwise the user experience is suboptimal, as the user has to position the cursor on top of the table contents in order to scroll them. An effective user experience is one of our competitive advantages, so we don’t sacrifice it light-heartedly.

We worked around this obstacle by taking control of and manually calculating the scroll position of the table contents by measuring the height of the headers, and where the window was scrolled to. Unfortunately, disabling the table’s own wheel events was not possible at the time, so we had to make a contribution to fixed-data-table-2 to allow it.

But still, we had two scrollbars: one for the entire viewport and one for the contents of the table. Although wheel-powered scrolling worked nicely, hiding either scrollbar was not an option, as dragging the scrollbar would then become impossible. Luckily, we came up with another solution that, although being quite hacky, turned out to work pretty well.

The table row height is static due to virtualization, so once we know how many items are in it, it’s possible to calculate the height of the table as if it wasn’t virtualized. We added the height of the content above the table to the calculated height of the table contents, and set the resulting value to the height of window.body:

window.body.style.height = headerHeight + stickyToolbarHeight + rowCount * rowHeight;

We wrapped the sticky header content and the table in a container, and once reaching the top edge of this container, we set it’s position attribute to fixed and started delegating window scroll events to the table component that now has its own scrollbar hidden. This solution still resorts to some hacks like initially overflowing the table, but it provided us with very well-performing native window scrolling. Finally, we got what we were looking for.

The aftermath

With React, it is relatively easy to extract a reusable table component that, although not providing virtualization, shares a lot of their styles and functionality.

This was the first major refactoring effort in our Angular-to-React migration, and the benefits are obvious: the code is better structured, it’s more performant, functionality can be shared across our app easily, and the flow of data is more expectable. For our customers, this means an effective and coherent user experience with less bugs.