Copyright © 1997- 2007, Ethan A. Winning, All Rights Reserved
Preface: Although the following has its base in salary surveys for financial institutions, the problems with analysis and statistical validation (i.e., getting at reality) that we've encountered do not vary much with all other compensation surveys regardless of industry. We ask the reader to do some extrapolation to their own experiences and industries. The truth will out; salary surveys and their results as published are based upon some fairly weak methodology and information. The original article was written in 1996 and revised in 1998. I recently received another request to conduct a salary survey, and found that what I said in the 90s still holds true in the 00s.
For more than 20 years, we've been conducting annual compensation and benefits surveys in and for financial and lending institutions including banks, savings and loan associations, and credit unions. Usually, after completion, we've compared the results with as many other surveys as possible, and our analysis has been within a few (percentage) points of the others. Not so in 1996 and 1997 or 2002.
In response to a request from a client, our last survey began in March 2002. This time we had four other executive compensation surveys for comparison, and we were bothered by what we found:
1. Our surveys have almost always been conducted for institutions with less than $500 million in assets. At one time, 225 such institutions responded to survey questionnaires. Last year, there were fewer than 30. In fact, we had to combine S&L's and banks in order to obtain a sample which could yield statistically significant results. We also had to rely on proxy and annual statements for reliable data...and annual reports are notoriously suspect in their analysis of the past and their rosy view of the future. Many of them should be called "Disclaimer Reports," i.e., "Last year was a difficult one for Last National Piggy Bank, but that was because of the one-time (they're always "one-time") write-off when we bought Next-To-Last National whose assets size was five times ours, and we pay 10 times book. Next year, we fully expect to show phenomenal profits {and the unstated, 'God willing'}."
Note, too, that credit unions which often use bank and S&L data were included although we failed miserably in matching CU's data with that of the more statistically "sophisticated" bank surveys. The best way to put this in perspective -- for those of you who are with credit unions -- is that banks and S&L's have had 50 years more experience diddling with (survey) figures than CU's.
2. Although asset size is still used by all others, that is no longer a viable criterion. Three of the other surveys combined institutions with over $1 billion in assets with those with under $150 million. Talk about apples and tangelos...
3. Peer groups were difficult to find. Since asset size limited the number of institutions sampled, we chose two other criteria: ROAA of at least 1% and ROE of 10%. Certainly, there are still problems with the use of such benchmarks, but at least those yardsticks showed the relative health of the organization.
4. More apples and oranges: It was bad enough that institutions with wide-ranging asset sizes were mixed. To compound that problem, two of the other survey companies used organizations from Kentucky to Kansas as "peer groups" for individual institutions in California (our primary focus in '96). One national survey broke data down by "Pacific Region" and "Metropolitan Areas." The range of executive compensation between the two diverged as much as $100,000 from the midpoint!
5. Still another listed a single-incumbent position (only one reported) as having a low-mid-max range as $5,000-5,000-5,000. Simply, the position should not have been reported. Another company, whose report we returned, reported nationwide data with only two California institutions reporting. This is significant when one considers that there are more financial institutions in California than any other state. (Much as one may hate to hear about this state ad infinitum, remember that one-eighth of the country lives in California.)
6. The use of titles doesn't work. For example, there can be a world of difference between "Chief Loan Officers," "Chief Administrative Officers," and even "Chief Executive Officers." How different are the CLO's or CFO's of a $500 million operation from one that is at $20 million? That's a rhetorical question. Yet two of the surveys didn't even list job summaries in order for an analysis to be conducted. (The same was true for "lower-level" positions where functional titles can be even more confusing.) Those responsible for filling out the questionnaires were often haphazard in the way in which they matched titles to responsibilities.
If this doesn't strike a chord, think of the term and title, "Secretary." A secretary in one company may well be at an executive level; in another, it may be a data-input clerk; and in one company with which we deal the title is only used to mean Secretary to the Board of Directors. "Receptionists" in some financial institution greet customers; in others, they may open new accounts, a minimum of a three-grade spread no matter how you look at it.
This was very much compounded by a tremendous variation in job descriptions or, more appropriately, job synopses provided either by the reporting institution or the survey takers.
7. What does the mean mean? Without getting too statistically heavy-handed, the reader should note that the median should be used only when one expects the results of an analysis to be skewed. Quartiles and percentiles are not statistically significant when there is an assumption that the results will indeed be "out of whack." For these and other reasons we use means and standard deviations, with some assumption that there will be a "normal" range for a position's data. Typical of assumptions, this one most often is flawed.
Solutions: To accept the figures presented in other surveys would have been an exercise in proving the adage, "garbage in, garbage out;" some were about as useful as hubcaps on a tractor. The steps taken to resolve some of these problems helped in obtaining meaningful data and results, "meaningful" equating to statistical reliability of at least the .85 level. (It is realized that a truly significant level is .95 but the deck is really stacked against such a level in executive compensation analysis.)
We first determined (or assumed) that some banks and S&L's were sufficiently alike so that they could be used as part of the same survey. (Credit unions had to be taken separately although they are getting much "closer" to other financial institutions at least in terms of services offered. We often found ourselves thinking of credit unions as the 90's version of the 70's S&L -- very much customer oriented with specialized lending, credit, and other financial services to be offered.) Had we not done so, we could not have come up with a sufficient number. (Most statistical analyses demand an N=20.We were truly lucky to find twenty-one in each of three peer groups.)
We decided to forego asset size in favor of the ROAA and ROE criteria. It should be noted that the institution we were conducting these analyses for more than met these criteria.
We temporarily discarded all other surveys and conducted telephone interviews with the newly established peer group. The peer group was also examined to meet the additional factors of being as much alike in terms of the "philosophy" and "structure" as possible, e.g., "community-based and oriented" organizations were not compared with those having a large number of branches in other areas.
A job summary or full job description was used in gathering data. For example, in one survey we did not combine Chief Lending Officers who supervised a number of Loan Agents with Chief Lending Officers who were the primary "originators" of loans.
A note to those who pass on the survey questionnaires to be filled out by others: To complete a good questionnaire for 30 positions takes three to four hours if one attempts to match existing position descriptions with the (usually) attached job summaries. If the survey is important to you, then you have to explain the importance to the person completing the forms.
Statistically, we used the mean and standard deviations rather than the median to determine the wage-range spread. By using standard deviations, it is mush easier to know which top- and bottom-end salaries have to be dropped from the data. (Some may say that the sample does not follow a normal curve distribution, but we found that it did.)
Some findings: Our survey showed compensation ranges for many positions lower than the other surveys. But, we also found that increase in base compensation were above the national average as reported in the Wall Street Journal and the most recent BLS findings. And as should have been expected, although bonuses dropped in 1995, they were still considerable, i.e., up to 60% of base even when the bank's "profitability" dropped by 8-10% over 1994. Of course, one must consider that a company that made $4 million in 1995 and only $3 million in 1994...
Journal and Barron's watchers, note that our client base and those interested in our surveys are not the Wells Fargo's, BoA's, Shawmut's, Chemical's, etc. of the world. They are the smaller, community-oriented institutions still very much customer or member-oriented. Some still give out dog biscuits to customers (usually with dogs), some still have drive-up windows, and some even demand that tellers (yes, tellers) know customers by name. The philosophy lives even though as institutions, they may be a dying breed.
[Footnote: Most of the organizations we spoke with were expecting 1997 profits to be either lower than 1996 or continued "flat." Of the (total) 37 institutions in our combined surveys, six (14%) anticipated layoffs while another four (11%) were looking at "reorganization" to increase productivity. Reorganization usually means downsizing, but those four refused to use the term. Eight companies, by the way, said that they were looking to expand operations which would negatively affect short-term profits. In other words, it's a mixed bag, and I am not certain that there will be "old-time" stability by the end of 1997.]
For a perspective of the author's leanings and personal relationship with his Big Bank, see the article, "See a Teller, Go to Jail.")
All Rights Reserved. Copyright 1997-2007. E. A. Winning Associates.