Few topics have raised more intense feelings and debate over the last 15 years in the human resource (HR) community than this one: “What is the proper use of multisource (360 degree) feedback?” Central to this question is the disagreement over whether this feedback should be used for decision making or development. In addition to countless articles that have touched on this issue, a complete book
(Bracken, Dalton, Jako, McCauley and Pollman, 1997) and book chapter (London, 2001) have been devoted to it. The debate is fueled largely by personal opinion, anecdotal experiences, and sometimes flawed research to a point where unfounded fear often drives the final decision.
We will propose in this article that development versus decision making is not an either/or decision and, in fact, the use of “development only” is often misleading and misused. In this article, we hope to present an overview of the issues, relevant research, and what will hopefully be a new way of looking at this question. We believe that this debate creates an opportunity to examine best practices, and ultimately is a means to examine the many decisions that are made in the design, implementation, and use of 360° feedback processes.
What does “development only” mean?
If we are to take “development only” literally, then it indicates that no one other than the ratee (the label we will use throughout this article, versus “rater,” the feedback provider) will have access to the feedback results, either directly or indirectly.
For the developmental purists, such as alumni of the Center for Creative Leadership (Tornow, 1999; London, 1998), keeping the data in the hands of the ratee has at least three benefits: this practice creates a “safe” environment for the ratee who can then use the feedback in a way that he/she
is most comfortable with, it prevents the organization from using it in some way that may be inappropriate, such as making decisions regarding pay, and it keeps the 360º process safe from distortions caused by raters who do not give fully honest input when they know that the data will have some tangible impact on the ratee. These three benefits will be examined later in this paper.
In order to satisfy this “development-only” requirement, the distribution of the data must be strictly controlled so that the ratee is the only person who has access to it. Here, when we say “data,” we mean the actual feedback report and the database that creates it. The typical scenario this creates is that the data is collected and generated by an outside entity (vendor), and only one copy of the report is generated and placed in the hands of the ratee.
Another way of looking at this issue of sharing the feedback data is the question of who also “co-owns” the data (Bracken, 1996). In its simplest form, the distinction holds that in a development-only process, the ratee owns the data. If the organization (and its representatives, including the boss) has access to the data, then at that point the organization “owns” it and has the right to use it as it sees fit.
One of the major points here is that there are probably few 360º processes where results are not shared in some form or fashion with the boss and/or a coach, and for good reason. This basic decision about who will have access to the data, directly or indirectly, is one of the most crucial ones to be made, yet is all too often left to the discretion of the ratee with statements such as, “is encouraged but not required to share,” or “is encouraged or required to reference their feedback in their development plan.” This kind of ambiguity at the very least creates inconsistencies across the organization.
This issue of sharing results with others, and whether it is a good or bad idea, is critical on many levels. The reality of organizational life is that, more often than not, the boss is a critical resource for the ratee. One tangible sign of that fact is that almost all 360º processes require that the boss be one of the raters and, in some systems, the boss rating is given extra weight. Another indication of that fact is that the boss often holds the keys to developmental resources, whether they be informal (e.g., on-the-job experiences) or formal (e.g., training).
What is “decision making”?
Tornow (1993) offered a taxonomy of uses, beginning with development and the focus on increased self awareness. His next use is appraisal, in effect putting information into the hands of the ratee’s boss to help round out the information used to give performance reviews and presumably affect associated decisions (e.g., pay, bonus). The third use is for assignment or selection purposes; that is, for the prediction of future performance and the likelihood of success. Finally, he sees 360º feedback as a potential instrument of organizational change that aligns leadership behavior with the organization’s strategy.
It may be more productive to look at “development versus decision making” not as a dichotomy but more as a continuum. Figure 1 proposes that a pure “development only” or a pure “decision making” use of 360º feedback is actually a rare phenomenon, and, to some extent, almost every use has an emphasis on both. Figure 1 offers as examples the most common uses for 360º feedback, though we hope that downsizing is the least frequent and least encouraged.
Figure 1: The Development vs. Decision Making Continuum
The placement of uses along this continuum can include a consideration of such factors as:
Clearly, with the presentation of this concept of a continuum, we are calling into question almost any defense of the use of 360º feedback as being “for development only. If some use has no consideration for any of the three factors listed above, then we must question the utility of the feedback.
What is the problem with using 360º for decision making?
The various objections to using 360º feedback for decision making purposes fall into categories that run along a continuum from purely philosophical reservations to fears regarding the quality of the data under such circumstances. For example, perhaps the most “pure” objection comes from the previously cited philosophy supported by the CCL community and its proponents. As noted earlier, CCL is committed to the development-only approach and, as best it can, prohibits the use of its assessments for purposes other than development. This stems from a belief that a positive development experience is best accomplished when the participant feels safe and can make decisions free of external encumbrances.
Another philosophical argument originates in the long-standing history of separating pay from development discussions. Since compensation is often the single decision that raises the most emotion in this debate, it is more than germane to examine the veracity of this position. Fortunately, Rynes, Gerhart and Parks (2005) have done the work for us in the Annual Review of Psychology. Their insightful review of literature and research leads them to the following summary conclusion:
“The use of PFP (Pay for Performance) can be considerably more effective than it is portrayed in the psychological literature, and that research has failed to validate Meyer et al.’s (Meyer, Kay, and French, 1965) claims about the dangers of integrating feedback and reward in more traditional appraisal settings…” (p. 594)
At the most tangible end of the continuum of reasons for resistance, researchers such as Antonioni (1994) have reported actual shifts in the distribution of ratings when raters know that their feedback may have a direct effect on the ratee’s “well being,” such as pay or promotion decisions. The effect is typically in the upward direction, that is, an inflation of ratings or leniency error.
Some of the resistance comes from emotional responses that are often hard to document and classify. These might include feelings that putting this kind of “power” in the hands of subordinates violates the boss-subordinate power relationship. This is often accompanied by a fear that subordinates might use this as a means of retaliating against a boss by giving very low ratings.
Since compensation is often the single decision that raises the most emotion in this debate, it is more than germane to examine the veracity of the position.
Finally, there are real logistical challenges that are created with large-scale 360º implementations, especially when they are required annually and fall under deservedly high scrutiny. The hard costs are very real, though they are being reduced as technological advances take place. An equally valid concern is the soft cost, in particular the time required for raters to complete their task, often on multiple leaders. Each of these problems has at least a partial solution (e.g., keeping instruments to reasonable length, administering online), but in the end the organization must make its own assessment of costs versus benefits.
Part of the problem in combating these objections is that some, such as the philosophical type, cannot be addressed by research and others are more anecdotal than systematic. Another problem is that every 360º process is unique; each process occurs in a unique context (e.g., culture) and history that are organizationally specific, as well as each having its own design characteristics. One of those design characteristics (that we will address more completely later) is the use of rater training as a means to combat some of the rater behaviors that can produce less than reliable results. These multiple variations across 360º processes make broad generalizations impossible, and we encourage parties on both sides of this debate to acknowledge that there are processes that run the gamut from very effective to very bad.
As a tangible example of how 360º processes can have design features that can drastically alter the nature of the feedback, we offer the case study from a large organization that began administering a “developmental” feedback process as part of a training intervention to leaders from multiple levels (i.e., pay grades) across the company. After reviewing the effectiveness of the feedback after the first year, they came to the conclusion that the data were not providing useful feedback, largely because of the skewed results toward the positive direction. Based on their review of the characteristics of effective 360º processes, they implemented two simultaneous changes before the second round of participation:
1) instituted rater training consisting of a set of slides that raters were required to at least scan through before completing their first rating, and 2) changed the rating scale. The primary objectives of the rater training were to reduce rater error and calibrate ratings through improved understanding of how to use the rating scale. The rating scale was changed from the standard Likert-type Agree/Disagree format to one where raters are encouraged to gauge effectiveness by comparing the ratee’s performance against peers.
The effect of these two design changes was dramatic and (for your information), replicated across all pay levels. Most strikingly, the percent of “favorable” (4 or 5 on the 5-point scale) decreased from 83% to 61%, and the choice of the midpoint, redefined from “Sometimes Agree/Sometimes Disagree” to “Average Effectiveness,” went from 13% to 32%. Unfortunately, we cannot separate out the relative effects of the two interventions (i.e., rating scale change and rater training), but at the same time we propose that the two are closely related and, in fact, mostly likely have a multiplicative relationship. We are quite sure, for example, that the rating scale change would not by itself be as effective without the rater training that accompanied it.
These interventions represent just two of the many design features that vary across (and, in this case, within) 360º processes that make comparing systems almost impossible. The variety of rating scales used across 360º processes alone makes comparisons difficult, if not impossible. We and others (e.g., Bracken and Timmreck, 2001; Antonioni and Woehr, 2001) also believe that rater training is a potentially powerful intervention that is ignored more often than not, at the peril of the system’s health and validity.
The key points we would like you to consider related to the uniqueness of your 360º process include:
The variety of rating scales used across 360º processes alone makes comparisons difficult, if not impossible.
1. A 360º process is, intentionally or unintentionally, a product of the organization’s history, culture, strategy, and values that will uniquely determine its ultimate success.
2. A 360º process is a complex interaction of many variables. In that way, no two 360º processes are the same, and no 360º process can be assumed to be valid in two different contexts.
3. A 360º process is always evolving, potentially in a good way or bad way.
4. A valid, reliable instrument (questionnaire) is a necessary, but not sufficient, determinant of the validity of the system.
5. An interface with HR systems creates a symbiotic relationship—the 360º process gains credibility and alignment, and the HR systems gain reliable data for personnel decisions.
Is there a risk in NOT using 360º assessments for decision making?
As we noted earlier in this paper, the primary risk in not using the 360º data for decision making is in making the wrong decisions. As one extreme example, one of the major themes of a recently published book called Snakes in Suits (Babiak and Hare, 2006), that examines the problem of psychopaths in organizations, is that dysfunctional and even destructive employees can amazingly escape detection when multisource methods are bypassed at the direction of senior leaders who are enamored with these charismatic, charming individuals.
Reading between the lines of some earlier comments regarding “development-only” processes that do directly or indirectly influence decisions, there is a substantial risk in the use of 360º assessments for purposes that are not intended. These risks are multifaceted:
1. Claims of inconsistent use of 360º information for decision making raise questions of discrimination and unfairness.
2. Use of an unvalidated 360º process for decisions will not only be a legal risk, but will certainly draw into question the quality of the decision.
3. Using 360º data for a purpose other than that for which it was designed and communicated will affect the quality of the data in future generations of the process, including impacts on honesty and response rates.
Conclusion
Is 360º feedback used for decision making always a good idea? Of course not. It is hard to do well enough to defend its usage in many organizations. We find many cases where an organization is unwilling to either create or sustain the resources necessary to have a successful 360º process, and in those cases the process hopefully dies a timely death. On the other side of the equation, some “development-only” processes live on indefinitely, not drawing away too many organizational resources and existing benignly under the radar. These 360º processes do not cause much damage, but often do little good.
The best solution may indeed be to acknowledge that both sides of this “debate” are right, and each can benefit from the other. We frankly find the “development only” a little too dogmatic at times and unwilling to acknowledge that using 360º data for decision making can and does work in many organizations (Edwards and Ewen, 1996; Church and Waclawski, 2001). It also has the risk of being misused or evolving into a process that can be manipulated. Therefore, a 360º process requires constant nurturing and review to ensure that the health of the system is being maintained and being administered in ways consistent with its purpose.
The primary risk in not using the 360º data for decision making is in making the wrong decisions.
References
6 Ways to Defend Against Compensation Discrimination Claims
Since the release of the latest iteration of the OFCCP's stated procedures for investigating compensation discrimination (i.e., Directive 307 - effective February 28, 2013), there has been no shortage of webinars, blogs, and warnings from … Read more...
6 Ways to Defend Against Compensation Discrimination Claims
Since the release of the latest iteration of the OFCCP's stated procedures for investigating compensation discrimination (i.e., Directive 307 - effective February 28, 2013), there has been no shortage of webinars, blogs, and warnings from … Read more...
