The Danger of Letting Data Define the “Thing”
by Nick Redding, Hawaii Data Collaborative
I am a research psychologist by training. That means I have spent a good amount of time bringing people (mostly college freshman) into a research lab to participate in experiments. These experiments were carefully designed to collect data about people – their thoughts, behaviors, emotions, etc. The what of measurement was always deeply informed by theory and prior research that guided how we defined – or operationalized – what we wanted to observe. In doing this type of research I had the ability to collect the data most relevant for measuring what I was interested in.
Shortly after arriving in Hawaii, I became involved in a project that was working to address a series of questions: What do we know about the well-being of communities across Hawaii? Which communities are doing well and which are struggling? How is this changing over time? These are very different questions than I was used to addressing, but I was up for the challenge and found inspiration in other projects working to address these questions globally. One example is the Legatum Prosperity Index, which measures and compares the well-being of 149 countries across the world.
The Legatum methodology was straightforward: they started by reviewing academic research on well-being, identifying over 200 indicators across eight domains including health, education, safety & security, economic quality, etc. From there, they looked to see which indicators were available across at least 80 percent of countries, compiled that data, and used regression analysis to identify those indicators most predictive of country well-being. In the end, 89 indicators were used to calculate an overall well-being score for each country (this is an index score – the topic of my next post).[1]
What struck me from reviewing their methodology was the step that was taken from identifying the indicators that research found to be most representative of well-being, to settling on the subset of indicators based on data that is actually available. Pragmatically, this makes a lot of sense: If the data doesn’t exist, it cannot be included in the analysis. However, it raises what I think is a really important question: Is the availability of data limiting how well-being is defined?
We are what we measure…or are we?
Don’t get me wrong, the Legatum Prosperity Index is a solid effort – in my view one of the best there is. I’m raising this limitation in their approach to highlight a fundamental challenge: the data that is available to us is not the data we need to holistically understand the well-being of a place. The Legatum Prosperity Index, and other similar efforts, rely completely on data that was collected for other purposes – albeit for many data points the purpose was to address some sub-component of well-being. However, without coordination across collection efforts, there will always and inevitably be critical data gaps.
There is growing interest from a broad range of local stakeholders in better understanding our local challenges through data. This cadre of stakeholders believes that making community indicators more accessible will be incredibly useful for building a better understanding of well-being – and that understanding can shape the actions we take to improve our communities. If decision makers can get a high-level sense of conditions and outcomes related to health, education, economic opportunity, housing, etc., they can make more informed decisions regarding how to allocate resources to improve disparities.
The Hawaii Data Collaborative was launched in recognition of this need – to elevate data to improve the quality of decision made around community well-being – and we responded by inventorying existing data and making it readily accessible to a broader audience. We have organized 75 data points into a 10-domain framework, modified to be more reflective local considerations, to make it easy to find available data based on the well-being topic of interest. However, our approach has represented a bit of a dilemma for us: How can ensure that the data we have now does not define our efforts to improve well-being going forward?
Here are a few ideas:
We can recognize that the data we currently have may reflect different priorities than those for assessing local community well-being. For example, our most prolific data source is the US Census which reflects the broadest data needs covering all U.S. communities. The strength of this approach is the consistency of the data, which readily allows for regional comparisons, flexible aggregation, and examination of trends over time. The tradeoff, however, is that the content of the survey is informed by national considerations, rather than local community contexts. This very is similar to many of our existing data sources. Given this, it is important to consider existing data more as a starting point than a ready-made dashboard covering key indicators of well-being.
Let’s avoid relying only on existing data to guide action. Instead we can draw from a more holistic set of resources to build a working theory of the system, and then use existing data to test hypotheses that naturally follow from our theories. These resources include local historical data, academic research – both broad and local, data derived from existing programs and services, and the wisdom and experiences of change leaders across sectors. The available data will almost certainly not be sufficient to test the theories that would result from a holistic examination of the system, which is fine – inevitable in fact. We must simply make a plan to test those aspects of our theories using other methods.
Where data is really lacking, we must work to fill the gaps. If the data exists, but is currently not publicly available, let’s explore what it would take to make it available. If the data does not exist, we can collaborate with others to collect it. For example, social capital has been shown be to fundamental for individual, family and community thriving.[2] In Hawaii, we have very little data regarding the support individuals and families receive from others, and we know equally little about levels of social capital activity across our communities – even though this a fundamental component of well-being.
Other examples of data gaps include measures of subjective well-being (how people feel about their situation), culture (the ability to practice one’s own culture safely, and experience other cultures), and civic engagement (the extent to which individuals participate in our political processes) beyond voter turnout. For matters of critical importance, we must elevate the data need, engage with our public agencies to reevaluate data collection priorities, and lobby legislators to secure funds to expand data collection capacities.
Elevating data to guide action is our mission, and it is an aspiration shared by many across sectors and interests. Let’s work together to elevate the data we need, not just the data we have, for a thriving Hawaii.
Nick Redding, PhD, is the Director of Social Research & Strategy at Ulupono Initiative, and Project Lead for the Hawaii Data Collaborative.
