NZIER and Statistics NZ

Prototyping ways to consume and explore complex statistical data with NZIER and Statistics NZ

In 2016 Springload worked on a couple of projects making complex statistical information available for exploration by the public. Both the projects were ‘alphas’ – public prototypes intended to uncover obstacles and opportunities plus test market appetite early. In the spirit of alpha release software there were plenty of rough edges – but we learned lots of interesting lessons along the way.

A dashboard for key figures

We created the backend and interface for a prototype dashboard conceived by Statistics New Zealand and their partner Empathy.

The dashboard was designed to allow key business consumers of Statistics NZ data to monitor the economical indicators that matter to their organisations. Users can pick particular measures or relevant subsets of those measures to a dashboard, then dig into historical trends as required.

Re-orderable metric cards on the Statistics NZ Dashboard

Comparing longterm trends

The New Zealand Institute of Economic Research is an independent consultancy with the longest running set of economic data in New Zealand. This longterm dataset provides invaluable information about the country’s past economic performance, and helps people consider the future. Springload was engaged to make this data accessible.

Data1850 lets you create and share graphs that answer questions like, how does spending on health compare to other government spending? Or how does New Zealand’s GDP growth compare to the growth in wages and salaries?

It’s an invaluable tool for economists, journalists, students and the general public who want to improve their understanding of changes to economics in New Zealand. As well as being able to make comparisons between two linked datasets, the entire longterm dataset of New Zealand economic history can be downloaded and manipulated using your own tools.

Comparing long term datasets for Data1850

Technology

Prototyping with data calls for technical choices that enable fast iterations, which can be taken to production when the time is right. Django, our go-to web framework, is well suited as a data delivery platform for the front-end application, which is in charge of manipulating and displaying the data. The data and metadata are provided via a simple REST API and the interface is built with React, providing great performance at scale out of the box. For the engagement with interactive charts to be high, it is paramount that the tool reacts quickly to user interaction regardless of the amount of data being manipulated.

We also wanted our charts to be as visually appealing as possible while being statistically accurate and highly customisable – hence the need for a good data visualisation tool like D3. It is now the de facto standard for building public-facing visualisations, requiring a high amount of expertise but providing the highest level of control over the final rendering of the charts.

The data exploration experience is more enriching when context is provided with additional content and related resources. This need is best served by a CMS like Wagtail. It is a drop-in addition to Django, providing a comprehensive back-end solution without compromising on the data API nor the administration experience that project stakeholders now expect.

Lessons learned

Both the organisations we worked with are used to having their staff act as a filter for the consumption of their data. That might mean creating reports from lots of different inputs or finding the right dataset to meet a client’s needs. Some of the biggest challenges we faced were about having a computer do these jobs instead of a human.

Normalising the data is hard

Computers aren’t nearly as good as humans at adapting to variation – they need everything to be in a predictable format. The effort involved in getting the data into the right shape can be significant. To speed the process up for NZIER, we built a suite of browser-based data processing tools to help their team quickly and painlessly diagnose errors or omissions. 

Web-based tools for NZIER to examine the API, trouble-shoot the metadata and proof rendered data for anomalies

Labelling and organising data in easily understood ways is hard

There's a strong tension between being both statistically correct and clear to nonexperts. A description that makes sense to a statistician for a single ‘atom’ of data may not make sense as a label or title. This makes it important to get the metadata (data about the data) right so we can use aliases for nice titles and specific terms where required. Understanding those gaps and how to fill them is an important step before design gets too detailed or entrenched.

The data defines the functionality and its interface

Both of these projects started with an existing set of ingredients – the underlying data. The shape and consistency of that data has an enormous influence over what can be done with it and how tricky the doing will be. Thorough analysis of that data is a mandatory first step – either with internal experts before considering vendors, or as initial discovery work so the build can then be costed with greater accuracy.

It's impossible with highly variable information to uncover all the gotchas upfront in mockups, which rely on example data. We kept flow and architecture explorations as lo-fi as possible to get a working prototype up early. This let us make decisions based on real datasets and real interactions between them.