Predictive Impact: The Science Behind 15Five’s AI-powered Pr…

Last Updated: November 2024

Summary

HR leaders can often sense when employee engagement is suffering across their organization or even in small pockets of departments, demographics, or teams. This disengagement can often lead to performance and retention issues, among other problems. It is important to connect anecdotal evidence with specific, systematic insights about engagement. 15Five’s Predictive Impact Model is a machine learning model that helps HR teams predict the impact that taking certain strategic actions for groups of employees will have on their engagement. It predicts how changes in specific engagement survey statement answers can influence overall engagement for specific employee groups.  The statements that a group has the lowest scores on numerically are not always the statements that are most negatively impacting their engagement. The Predictive Impact Scores provide a method of uncovering which survey statements and groups you should focus on, accurately predicting where to take action to make the biggest impact on engagement specifically. 

Our definition of engagement

Unlike employee turnover, where rigid numeric measurement is inherent, employee engagement is difficult to quantify because it is strongly related to human emotion and thought. 15Five defines employee engagement as the emotional commitment and passion that an employee has to their work, their organization, its goals, and its values. Engaged employees are motivated to contribute to the organization’s success, feel a sense of purpose and satisfaction in their work, and often go above and beyond in their roles. It is characterized by enthusiasm, dedication, and a willingness to invest effort in work activities. High levels of engagement typically result in increased productivity, better performance, and lower turnover rates. Encompassing both meaning and purpose, workplace engagement stretches beyond weaker measures such as employee satisfaction.

Why 15Five’s engagement score matters

Engagement vs. satisfaction

Workplace satisfaction is the most popular and most dangerous concept that is used as a measure of engagement. In reality, it does not incorporate the conceptual reality of engagement.  Whereas satisfaction measures an employee’s contentment with an employer, engagement corresponds to their passion and desire to help the employer. According to industrial and organizational (I-O) psychologists William Macey and Benjamin Schneider, “Engagement is above and beyond simple satisfaction with the employment arrangement or basic loyalty to the employer— characteristics that most companies have measured for many years. Engagement, in contrast, is about passion and commitment — the willingness to invest oneself and expand one’s discretionary effort to help the employer succeed.”^[3]

Engagement is a psychological state

Workplace engagement is a persistent, cognitive state rather than a personality trait or behavior. Psychological states tend to fluctuate in response to external inputs more rapidly than behaviors and personality traits. In accordance to leading I-O research, 15Five considers employee engagement to be a leading indicator of workplace behavior. It is a state of mind that can be influenced by an employee’s personality, but ultimately is most impacted by various organizational drivers. Examples of such drivers are knowing “why” a business exists, being able to connect daily work to this purpose, having clarity about what that work is, and having trust and respect in the working environment among team members. 

The engagement score

The Three F’s: the dimensions of engagement

An individual’s state of engagement consists of three dimensions – focus, feeling, and force. Focus is an expression of the individual’s ability to work intensely without distraction; it is a manner in which engagement is expressed and experienced. Feeling is an expression of what it feels like to work. Engagement is not painful. Thus, if an individual experiences a good feeling while working, it may be evidence of engagement. Lastly, force is an expression of the internal drive to work for the organization and contribute in positive ways. When a person is engaged, they feel an internal force to get work done and help their organization. This dimension builds on the concepts of motivation or inspiration.

Scoring engagement

To score an employee’s state of engagement, agreement with seven statements across each of these three dimensions is established on a 5-point agreement scale (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree). In academic literature, work force is described as dedication, work feeling as vigor, and work focus as absorption.

These statements are:

1. [Work Force] The work that I do gives me a sense of pride. 
2. [Work Feeling] I feel a sense of happiness when I am working very hard.
3. [Work Focus] I find it very easy to stay focused on what is important for me to accomplish at work. 
4. [Work Force] I find my work to be full of meaning and purpose. 
5. [Work Feeling] When I wake up, I feel like going to work. 
6. [Work Focus] I am able to get into a state of complete focus while working. 
7. [Work Feeling] I love the feeling of working. 

Next, each response to these statements is converted into a value of 0-4 for the purpose of calculating the overall engagement score. All employee responses are weighted equally. Finally, the responses are summed per employee and scaled up to be between 0 and 100 to produce the Engagement Score.

Below, Table 1 summarizes all 15Five’s 17 engagement drivers. 15Five’s standard engagement survey contains 48 statements that encompass these drivers of engagement. Employees’ work experiences across these drivers, reflected in their level of agreement with the 48 statements, correlate with their engagement level.

Table 1: 17 drivers of engagement

NEW: Predictive impact scores

The Predictive Impact Model is a machine learning model that predicts how changes in specific engagement survey statement answers (across the 17 drivers shown in Table 1) can impact overall engagement for specific employee groups.

Benefits compared to our previous scoring

Previously, 15Five’s engagement reporting consisted of two explanatory scores –  the Driver Score and the Driver Influence Score. Three advantages of the new Predictive Impact Score are: 

1. It is dynamic with different groups of employees. 
2. It connects the drivers (and their statements!) directly to a group’s engagement level.
3. It enables specific, targeted actions based on statement-level insights.

The Predictive Impact Score is calculated at an individual employee level and thus can be aggregated up across any group of employees. There is one score per employee survey statement. Thus, the scores can also be obtained per engagement driver.

*Note – 15Five always ensures engagement survey participation is confidential, no individual survey responses are identified, written responses are always anonymous, and groupings of data are only available in groupings of at least 3-5 to ensure confidentiality. 

Driver Score

For a given group, a Driver Score measures how well it scored in a driver relative to other groups at other companies. A Driver Score answers the question: Which drivers are other, external groups doing better than us at? This is valuable but different than the Predictive Impact Score. Conversely, the impact score explains: Which drivers or statements are most negatively and positively impacting engagement for this group?. The drivers or statements that a group does the worst on are not always the ones most negatively impacting their engagement. The Impact Scores are targeted at engagement specifically. 

How does it work?

The Predictive Impact Model is a machine learning model trained on over 600K complete engagement surveys in the 15Five dataset. It is a correlational model that detects thousands of patterns in survey statement answers that tend to correlate with distinct engagement scores. It uses these well-established patterns to predict which specific statements are pushing engagement away from a base level of engagement. 

This base level of engagement is the average engagement of a person in the 15Five dataset: an engagement score of 75. It is essential that the model considers impact “from a base level of engagement” so that the impact scores are realistically possible to improve. Improving a group’s answers to a statement typically only requires improving it to the average answer across all companies rather than a perfect “Strongly Agree” answer. 

Moreover, the Predictive Impact Model outputs impact scores for a single employee per survey statement. These individual scores are then aggregated across all groupings within an organization. Examples of such groups include: hierarchies, teams, divisions, departments, pay levels, location, and etc. The statement Impact Scores can also be combined across drivers to produce driver level scores as well. 

The Predictive Impact Model contains two parts – a predictive, machine learning model and an explanatory layer. The predictive model consists of a decision tree-based regression model^[1] and the explanatory layer utilizes explainable AI to generate the Impact Scores. Unlike causation-based methods, this system can run in seconds rather than hours and scale from a per-employee-level to any group of employees or even an entire organization. 

Identifying pockets of disengagement

Suppose an organization is highly engaged with no large, negative Impact Scores at the company level. Even in this example, the organization could still contain pockets of disengagement with larger negative Impact Scores for specific groups of employees. Despite the high overall engagement for an organization, these pockets of disengagement could still benefit from strategic action focused around increasing the engagement of only certain employees.

Below, Table 2 illustrates such an example for departments at a fictional company across a selection of survey statements. For simplicity, only two statements and their corresponding Impact Scores are shown. In the product, Impact Scores will be shown for all statements. 

Table 2: Example of Impact Scores by department

In this example, despite the company’s high average engagement score of 79, two of the five departments are actually very disengaged on average. Here, a typical employee in the Engineering and Human Resources has a low engagement score. If the Impact Scores for the two statements are combined together they sum to -7 for the Engineering department and -18 for the Human Resources department. These impact scores can be interpreted as the following for both departments:

• Engineering: The Impact Model predicts that My job activities are personally meaningful to me and My job challenges me in a positive way have a combined -7 point impact on the engagement score. 
• Human Resources: The Impact Model predicts that My job activities are personally meaningful to me and My job challenges me in a positive way have a combined -18 point impact on the engagement score. 

Seeing this, an HR leader or department head may create an action plan targeted towards improving scores across these departments on those two specific statements. Similarly, for the other departments that have large positive Impact Scores, the leader may focus on maintaining the high statement responses for those particular employee groups.

This was a very simple example. Most companies have many more groupings, with a much greater potential for pockets of disengagement, in need of focused action across a greater variety of engagement drivers and statements. 

In this way, 15Five works as a strategic command center for HR – enabling our customers to diagnose problem areas with precise accuracy, create targeted action plans to address the pockets of disengagement, and then take action through managers.

Diagnosing, Planning, and Acting  to Improve Engagement

Measuring employee engagement is only the first step in driving meaningful change. Without a structured approach to acting on survey results, the insights gathered lose their potential to improve the employee experience. Action planning bridges the gap between data and outcomes, transforming insights into tangible steps that elevate engagement, performance, and retention.

The Predictive Impact Model empowers HR leaders and department heads by identifying the areas that have the greatest potential for improving engagement, and quantifying that potential improvement! With this data, organizations can prioritize actions that will have the greatest impact on engagement, avoiding guesswork and focusing on what matters most.

Creating targeted action plans ensures that survey findings are not just informative but actionable. Some of the key benefits of actions plans include: 

• A clear structure for ownership and timelines: Documented action plans ensure that efforts to improve engagement are intentional and measurable. Regular check-ins keep teams aligned and accountable.
• Alignment with organizational goals: Well-designed action plans align engagement initiatives with broader business objectives, helping to improve both employee well-being and business outcomes. Engaged employees are more productive, innovative, and committed to the organization’s success.
• Continuous improvement loop: Action plans create a feedback loop by measuring the effectiveness of each intervention. Leaders can adjust strategies over time based on what drives the most meaningful change, ensuring continuous progress.

By integrating action planning directly into the post-survey process, organizations move beyond passive reporting toward a culture of action and accountability. This approach, empowered by the data from the predictive impact model, ensures that engagement data leads to meaningful improvements, empowering HR leaders to make strategic decisions that foster long-term employee satisfaction and retention. 

Conclusion

Employee engagement is a critical driver of productivity, retention, and overall business success. However, engagement surveys alone are not enough—meaningful change requires action. 

The Predictive Impact Model leverages AI to provide leaders with precise insights, helping them identify the areas with the greatest potential for improving engagement.

By focusing efforts where they will have the most impact, targeted action plans ensure survey results translate into tangible outcomes. These plans create clear accountability, align with organizational goals, and foster continuous improvement through measurable progress.

15Five’s Predictive Impact Model has made this level of diagnosing, planning, and acting possible for every HR leader. 

See a demo of the Predictive Impact Model

References

1. Anghel, A., Papandreou, N., Parnell, T., De Palma, A., & Pozidis, H. (2018). Benchmarking and optimization of gradient boosting decision tree algorithms. arXiv preprint arXiv:1809.04559.
2. Bakker, A.B. & Demerouti, E. (2007). The Job Demands-Resources Model: State of the art. Journal of Managerial Psychology, 22(3), 309-328.
3. Macy, W.H. & Schneider, B.S. (2008). The Meaning of Employee Engagement. Industrial and Organizational Psychology. 1, 3-30.
4. Schaufeli, W.B. & Bakker, A.B. (2004). Utrecht Work Engagement Scale (UWES) Preliminary Manual.
5. Schaufeli, W.B., Bakker, A.B., & Salanova, M. (2006). The Measurement of Work Engagement with a Short Questionnaire: A Cross-National Study. Educational and Psychological Measurement, 66(4), 701-716.