Before this summer, I had little experience with investing, much less sustainable investing. Like most people my age, creating a better world is important to me, so I jumped at the opportunity to learn how firms like Just Invest help investors seek both financial returns and positive change. This is a quickly growing field. US SIF reports sustainable investments now total over $8 trillion dollars, which represents more than 20% of assets professionally managed in the US. With this growth of socially-responsible investing, fund managers and investors are demanding robust, transparent, and accurate data for making sustainable choices. Just as funds have been constructed by evaluating companies along important financial metrics such as EPS, book value, and profitability, investors increasingly evaluate companies on the impacts and risks of Environmental, Social, and Governance (ESG) issues. Being an economics major, I was very curious to better understand this data, and in particular, how it is used to evaluate the ESG quality of an investment portfolio. By the way, if you are wondering “What is ESG anyway?,” give that link a quick read.

Measuring Impact…

Today, a number of firms collect and report extensive ESG data about companies. In turn, fund managers use this data both to construct portfolios for investors and to report on the ESG profile of their funds. What I found, though, may surprise most investors. Unlike financial data, which is largely standardized, ESG metrics vary across vendors, even for seemingly identical issues. Understanding that such differences exist is important to ensure investors get the exposures and impacts they intend.

A number of ESG data providers have emerged.

To highlight this issue, I selected 28 publicly-traded, ESG-themed Exchange Traded Funds (ETFs) and compared the ESG characteristics as reported by two leading ESG-rating services. Both firms provide an overall score as well as separate E, S, and G scores. Given the similar labels and descriptions of these categories, I expected the scores for each fund to largely match between ratings. To get the data, I relied on Just Invest’s subscriptions to investment tools and datasets to which most personal investors don’t have access, though ESG ratings are becoming more mainstream via services like Morningstar. Finally, it is worth noting that given the short history of most of the funds, I only compared current scores as of July 2017.

Let’s take a look at some of the data.The following chart shows the overall scores for each ETF (note one rating system uses a 0–10 scale while the other uses a 0–100 scale; I adjusted all scores to a 0–10 scale for ease of comparison). One vendor reports the Overall ESG score for each fund as a weighted aggregation of the underlying E, S and G component scores. The second vendor reports a Fund Sustainability score, where they too perform a weighted sum of the underlying E,S and G scores but also add a penalty for any controversies (for example, the account opening scandal at Wells Fargo Ticker:WFG). If the two ratings systems are consistent, we should see the scores neatly aligned along the 45 degree line. Instead we see a strong bias of higher scores under one rating system versus the other.

For the statistically inclined, the correlation between these scores is just 0.51. While it is positive, it suggests only a moderate relationship between the scores. That’s surprising given both systems are scoring on E,S and G factors. Given this bias in scores, I wondered if I could pinpoint the source of difference. To do so, I next looked at the separate E, S, and G scores for each of the funds. These scores roll-up into the overall ESG score for a fund, and it stands that a large discrepancy in the rating methodology of one might explain the low correlation of the total score. The following are scatterplots for the pairwise scores of each fund in the three (E, S, and G) categories.

28 ESG ETFs scored according to their overall Environmental rating (0–10 scale, 10 is best)

Environmental scores from both ratings providers are built on assessments of items such as pollution/emissions, water usage, and carbon footprint. While the sub-categories are not identical, they are quite similar. These are also arguably the most objective items that can be measured for companies, and we do see a higher correlation (0.76) than with the Social and Governance ratings, though again there is a systematic bias of one rating system to score funds higher than the other.

28 ESG ETFs scored according to their overall Social rating (0–10 scale, 10 is best)

The Social score evaluates a company along factors such as employee development programs, labor relations, employee satisfaction, health & safety, product quality and safety, and supply chain management. Again the categories are largely consistent between the two rating services, but the assessments appear more qualitative than the Environment scores. This becomes apparent in the lower correlation of only 0.43 (mildly positive) between the two rating providers. Also similar to the E score case, we see that one of the vendors consistently scores higher than the other, though here the bias is flipped.

Lastly, we show results in the Governance category, in which both ratings services evaluate a company based on aspects such as their Board independence, diversity and oversight of the company as well as executive composition and compensation. The correlation for the Governance scores is 0.54, quite similar to the overall ESG correlation figure we saw earlier.

28 ESG ETFs scored according to their overall Governance rating (0–10 scale, 10 is best)

So far I have shown a comparison across a broad set of ESG themed funds where some funds address overall ESG performance whereas others target a specific theme such as Low-Carbon or Gender Diversity. I next focus on a sub-set of Environmental themed funds. The main themes of these funds are low carbon emissions and fossil free/clean energy. Given the more narrow focus of these funds, one might expect much tighter clustering of scores. The following table summarizes each fund’s ESG and Environmental pillar score under the two rating systems. Not surprisingly, the scores are less dispersed than in the general population, but there are still notable differences. Take for example, the First Trust Global Wind Energy ETF (FAN) which received overall scores of 4.96 vs 7.17, and Environmental ratings of 5.41 vs 6.53. Those big gaps for arguably the same evaluation criteria!

Finally, I wanted to see how these ESG-themed funds compare to the broader market. In the final chart, I plot the overall ESG scores for those ESG ETFs which invest only in US stocks. The red vertical and horizontal lines are the overall ESG score of the S&P 500 (as represented by IVV, the iShares Core S&P 500 ETF) without any ESG selections applied, as scored under the two rating systems. The upper right quadrant of the chart shows those funds that score higher than the general market. While 7 of the 10 funds do land in the upper quadrant, surprisingly the remaining 3 all fall below the S&P 500 under both rating systems! Wow — if you thought you were buying a better environmental fund than the S&P500, you may be surprised.

Specialized US ESG funds compared to the S&P500 under the two ESG ratings systems.

Admittedly this was not an exhaustive comparison of ESG ratings, as I was limited to the overall and E, S, and G scores for the evaluated funds. Also, most of these funds are new, so little or no history of scores is available to assess how their scores are evolving over time. Nonetheless, even these top-level rating differences highlight that a given fund can look quite different depending on which rating system one uses. Further, as we started to decompose the overall score into the individual category scores, we see that the factors, weightings, and interpretations that go into each category score introduce more opportunity for differences.

As you seek a fund that matches your sustainable goals, it is clear that you may not get what you expect. To me this is a classic case of buyer beware. Not that these are bad funds, but rather that a simple score does not provide enough insight to ensure your goals are being achieved. Understanding the factors driving a given rating and focusing on the factors that are most important to you are crucial to achieving your intended impact.

To learn more about ESG factors in investing and see how Just Invest is deploying advanced data visualization to improve sustainable investing, visit www.justinvest.com/models.