Assess Fairly: Interviewing, Assessing, and Reference Checking for Essential Leadership Qualities

The following article is a part of our Reimagining the Academic Leader Search Process series published in May 2019:

How can we ensure we are measuring the right leadership competencies at each stage of the hiring process, while also being open to candidates from diverse backgrounds? A reimagined approach to candidate assessment is essential.

Candidate assessments in the academic leadership search process have become more consistent than ever. Collective bargaining, precedents set through grievance processes, and more professional human resource and faculty relations processes have all served to bring more uniformity to the approaches to assessment. But has that focus on consistent process resulted in better hires? Some would argue that process has trumped outcome and especially in a context where we desire to be more open to diverse backgrounds and perspectives, we need to revisit our traditional approaches to assessing candidates.

Our experience with more than 260 leadership searches for universities and colleges nationwide in the last five years tells us that clarity, consistency, reliability, reciprocity, and equity must all be considered in reimagining the approach to candidate assessment.

Clarity: What do we want to assess? 
Many academic institutions perform well in the early stages of the search process. They take the time to discuss and gather diverse insight from across the organization to inform the ideal candidate profile. They translate this carefully into the advertisement and often use it methodically to assess applications on paper. But assessment against defined candidate characteristics can falter at the interview stage. Search committees often stray from the competency profile for the role in assessing interview performance. This can affect community perceptions of transparency of process, when an agreed-upon set of criteria appear to have been disregarded as the process moves forward. And in open search processes where candidates interact with a number of the organization’s community members, we often see that the target competencies for success in the role play little or no part in the assessment feedback.

How can your organization ensure that the clarity you worked to define at the outset of your search is carried through in your assessment process? We recommend that the assessment approach be aligned  with the target competencies. This could include ensuring that interview questions, tests, reference questions, and presentation topics are developed with the target competencies in plain sight. Strategies such as rating rubrics, ‘consider points’ to guide assessment of interview responses, and structured surveys that ask community members to specifically comment on candidates’ alignment with the ideal candidate profile can also focus assessment on the factors you have determined will be essential for success.

Consistency: Improving interviews
To minimize the impact of personal biases and create greater uniformity in the process, panel-style interviews have been widely adopted in academic institutions. When well run, this type of interview generally produces valid information for assessment and is judged to be fair by candidates, but it may also dampen a candidates’ feeling of rapport and attraction to the job or organization.

Structured interviews, where the same questions are posed in the same way to all candidates, are similarly common (often overlaid with a panel approach). While numerous studies show that situational (what would you do in this situation) and behavioural (what have you done in similar situations) interview questions both have merit in effectively predicting performance, behavioural questions have been shown to be slightly more effective for assessing performance for more complex management positions. 

There are, however, potential shortcomings in both panel-style and structured interviews. Panelists often do not have experience in assessing complex competencies through interviews so they may be persuaded by interview performance or content that is not aligned with the actual competencies being assessed. And when candidates bring more diverse backgrounds and experience it is more difficult for panelists to assess competencies using a one-size-fits-all approach to questioning. In the name of consistency, many panels restrict follow-up questions which further limits the panel’s ability to seek greater understanding.

One refinement that can help is allowing the panel to ask questions from a list of options, all of which are targeted to assess the same competency but vary in how that competency might be described or revealed by candidates with different backgrounds. This approach maintains the benefits of consistency but recognizes the different backgrounds of candidates.

Alternative question formats can also help when used in concert with more traditional behavioural and situational questions. These may be targeted at assessing the more complex competencies of academic leadership: influence and persuasion, conflict resolution, solutions- focus, and/or openness to diverse perspectives. For example, realistic (not necessarily real) situations that embody an inherent conflict that a leader might encounter can be an effective approach to learning about judgment and decision-making. Psychometric assessments (administered by certified professionals, adhering to relevant ethical standards) where results are provided to the candidate to use as a way of describing their leadership can be effective, introducing objective language and constructs that the candidate and committee can use as the basis for deeper discussion of a candidate’s tendencies and preferences.

Reliability: Checking references
References continue to be a critical component in candidate assessment. Some institutions invite letters of reference but most have moved to reference calls for leadership appointments. While it is tempting for committee members to reach out to their networks to gather insights on candidates, this approach is not necessarily reliable or valid as an assessment approach.

Reference calls should be conducted by experienced individuals and guided by consistent questions that explore each candidate in context to the ideal candidate profile. When properly executed, they provide an opportunity to validate assessment and to explore issues that may have arisen through early assessment of the candidate. In our experience, the reliability of reference information in assessing leadership candidates is improved by having the committee identify the referees from whom they would like information. This mitigates the biases a candidate might introduce by choosing only favourable referees. Conducting reference calls with a number of referees and continuing to gather reference feedback until the committee is satisfied that they have a reliable perspective on the candidate also improves the value of this element of candidate assessment.

Reciprocity: The view from the other side
Panel-style interviews are generally judged to be fair by candidates but they may also result in candidates feeling less rapport or attraction to the job or organization. In a context where leadership candidates have options, it is important to ensure that the candidates' interview experience is at best attracting them and at a minimum neutral in its effect on their assessment of the opportunity and the institution.

How can you mitigate the negative perceptions that candidates might develop through the formal interview process without reducing the reliability and validity of structured interviews? Leaning on human resource professionals or search consultants to host and support candidates through the search process, allowing time for candidate questions during interviews, and making candidates comfortable in the interview setting are effective strategies. Search consultants can also serve as an intermediary, providing candidates with highly sought feedback on their cover letter, CV, and interview, ensuring candidates are prepared and engaged throughout the process.

It is also important to consider the perceived risk that candidates often feel when they progress to the open phase of the search process, making their candidacy broadly public. The prospect is daunting for most and a deterrent for many. Our experience tells us that setting clear expectations for all parties in terms of how presentations and informal discussions will be used in decision-making is critical, as is clarity around how the information about the candidate will be shared to the community. Candidates that engage in the open phase of a search process accept the inherent risks of the visit and they will use the occasion to assess the opportunity and the community. With this in mind, the organization should adopt an ambassadorial or host mindset in preparing for and conducting all aspects of the open process.

Equity: Reflecting on biases and barriers
As we encourage candidates with more diverse backgrounds and abilities to consider leadership opportunities, it is essential to reflect on the biases and barriers inherent to the traditional, tested approaches to assessment. This reflection extends not only to the biases that exist in the language, words, and norms used in assessment tools, but also to the ways in which these approaches to assessment are applied. A few examples can be found drawing on my own experience as a 5-foot tall woman with hearing impairment. Most boardroom chairs are designed for people 8 to 10 inches taller so I have often found myself seated around a table where my visible height difference can invite the well-documented bias that correlates height with leadership ability. Height adjustable chairs and a few words on how to operate them are a simple way to mitigate this bias. In large meeting rooms with high ceilings and challenging acoustics or in large group settings where I need to quickly adjust to different voices amidst ambient noise, I am often anxious about whether I will be able to hear what I have been asked or engage effectively in conversation. Asking candidates early in the process about the accommodations that might allow them to fully participate in interviews and assessments, and offering alternatives where possible helps to make search processes more inclusive and to mitigate disadvantages and biases.

KBRS is committed to academic leadership success.

KBRS has been a strategic talent advisor to universities and colleges across Canada for more than 45 years. Over the past five years, our firm has had the privilege of leading more than 260 leadership searches for universities and colleges nationwide. One of Canada’s largest independently owned executive search and consulting firms, we pair the flexibility and focus of a boutique firm with the research and rigour expected of a multinational organization.

This experience has informed our approach discussed in the articles below.

We invite you to connect with our Academic Search Partners to further discuss any of the ideas presented.