Deconstructing Surveys

As one can tell from the deluge of data, surveys are easy to do. In the financial world it seems not a week goes by without the release of some survey results. Most of these contribute nothing unexpected, because they simply parrot the questions of other surveys—which is not always bad—but sometimes the results may not be what they seem.

Representativeness refers to how well the survey sample represents the larger audience. For example, a survey stating “98 percent of Americans say the St. Louis Cardinals are the best baseball team ever” may not represent America if the survey sample was taken in a two mile circle around Busch Stadium. A good sample can be difficult to obtain in financial surveys for several reasons. One is that surveys usually rely on self-reporting and people tend to embellish their income and assets. Another problem is people reporting what they think they should report or intentionally misstating their answers. All of these issues are why multiple surveys covering similar ground can be a good thing—if different groups provide similar answers the results are more likely to be accurate.

Intentionally biased questions and responses are more often found in non-financial areas. These are the ones where the headline is “People surveyed agree that…” and the only answer choices were: Completely agree, agree, and partially agree. More often are cases of poorly defined questions that can be interpreted in different ways. For example, the question might ask whether a person feels there will be enough income in retirement to cover their needs. The problem is that one might respond thinking of needs being food and shelter and another may feel they need tropical vacations and five star dining. Although the survey answer may be helpful from a psychological perspective, it isn’t very good if you’re trying to pin down a dollar answer for a group.

Even when you have representative, well-written surveys another issue is understanding what the results really mean. Say that a survey finds that 30 percent of people say they feel they won’t have enough money to cover food and shelter in retirement. Let’s say the statement is correct and the consensus is that this is an unacceptably high percentage. Before solutions can be proposed we need to understand the why behind the number and this requires more questions. If further questioning reveals those surveyed feel this way because they don’t know how to save, then financial education might be the solution. If the answer is they won’t have enough money because every time they get a few dollars in their 401(k) plan they withdraw and spend it, then all of the “nudging” ideas—where people have to opt out to avoid contributions—will prove useless. For this group mandatory, untouchable retirement accounts may be the only answer. All too often survey results are used as proof of a problem or solution that the survey never proved.

There are certain red flags to look out for. The confidence interval should be at least 95 percent and the margin of error around three percent—this means the sample reflects the population and the actual result will be plus or minus three percent from the percentage given. Another is the sample itself; a sample consisting of millennials will generally produce different results than one of retirees; the sample group should mirror the population you’re looking at. Open internet surveys—the kind where anyone can answer—generally attract those that feel strongly and this tends to distort the results. However, targeted internet surveys sent to a known population are okay.

Better data is obtained through observation—watching what people really do and what they say. The problem is that these studies are far more expensive than doing a survey. In the meantime, we’ll keep seeing survey results, both the good and the bad.

Jack Marrion provides research and consulting services to insurance companies and financial firms in a variety of annuity areas. He also serves as director of research for the National Association for Fixed Annuities and as a research fellow for Webster University.

In 1994 he wrote a book to help banks market investment and insurance solutions to their small business clients. In 1996 he produced the first independent hypothetical return monthly publication comparing all index annuities on the market, and in 1997 created the first comprehensive report of index annuity sales, products and trends, “Advantage Index Product Sales & Market Report” (quarterly).

His insights on the annuity and retirement income world have appeared in hundreds of publications. In 2006 the National Association of Insurance Commissioners asked him to address their annual meeting and teach regulators the realities of index annuities. He was invited back in 2009 to talk to the NAIC about the effects of aging on senior decision-making. He is a frequent speaker at industry functions.

Prior to forming Advantage Com­pen­dium, Marrion was president and owner of an NASD broker/dealer with offices in nine states. Previous to that he was vice president of a life insurance company and vice president of an NYSE investment banking firm. He has a BBA from the University of Iowa, an MBA from the University of Missouri, and a doctorate from Webster University.

Marrion can be reached at Ad­van­­tage Compendium. Telephone: 314-255-6531. Email: ­marrion@advantagecompendium.com.