You are the head of analytics for an online retailer focused on athleisure and sports attire for mature women and men. The executive team of your company mirrors the customer demographic they target, and they believe that having a workforce that reflects their targeted market segment is a critical driver of their historical success in athleisure and sports attire. Your company has seen a spike in searches for novel sports equipment inspired by the summer Olympics in Tokyo, e.g., high-end longboard skateboards and related safety equipment and attire, and they want to expand to capitalize on this trend and prepare for another wave of demand driven by the winter Olympics in Beijing. The executive team is excited by the potential opportunity and want to move the company rapidly in the new direction with a new line of business.
Given the tight labor market, your company is concerned about missing the current trend by taking too much time to fill positions. The Head of HR gives you 5 years of data on the company’s employees, including those who left the company over the past 2 years. The data includes employee personal details such as address, date of birth, job title, length of service, salary, annual performance review results, promotion history, manager assessment of the employee’s long-term potential, recruitment details, qualifications held, technical certificates and training courses. The Head of HR asks you to develop a model that is based on job success of the company’s employees in order to screen and predict candidates who will become successful new employees for their new line of business.
Assume that you developed your model based on the 5 years of historical data. You are now screening 200 resumes of candidates, all of whom have come through LinkedIn and Glass Door. The candidates have tailored their resumes to reflect what they have learned from these websites about the qualifications of people hired by your company. You have calculated the following confusion matrix and associated parameters:
Predicted Qualified Predicted Not Qualified
Actual Qualified 100 5
Actual Not Qualified 15 80
Sensitivity = 95%
Sensitivity is the true positive rate. It is the number of positive predictions as percent of the total number of actual qualified candidates.
Specificity = 84%
Specificity is the true negative rate. It is the number of negative predictions as percent of the total number of actual unqualified candidates.
Precision = 87%
Precision is the positive predictive value. It is the number of correct/actual positive predictions as a percent of the total number of positive predictions.
Accuracy = 90%
Accuracy is the number of correctly predicted candidates (positive & negative) as a percent of the total number of candidates.
What to Do
Answer the following questions:
What are the false positive rate and the false negative rate? Describe what false positive and false negative mean. (10 points)
If your goal is to develop a model to screen resumes and identify candidates to be invited for an interview, which type of error is worse – false positive or false negative? Explain the rationale for your answer. (10 points)
If you want to improve the performance of your model to identify candidates to be invited for an interview, and minimize the error that you defined in question 2, which parameter (sensitivity, specificity, precision, accuracy) would you use to guide your work? (10 points)
If your goal is to develop a model to identify candidates who will receive a job offer, which type of error is worse – false positive or false negative? Explain the rationale for your answer. (10 points)
If you want to improve the performance of your model to identify candidates to receive a job offer, and minimize the error that you defined in question 4, which parameter (sensitivity, specificity, precision, accuracy) would you use to guide your work? (10 points)
What attributes in the HR data would you use to define success for the company’s current employees so you can train your model? (10 points)
The Head of HR is impressed by your work, and she wants to use your model to identify candidates who will receive a job offer. Would you recommend using your model to make the decision to offer jobs to candidates? Discuss the rationale for your answer. (10 points)
Fast forward one year. The company deployed the model that you developed with the 5 years of data on current employees, and people were hired based on your predictions. The Head of HR has now come back to you with a concern that not all of the new hires were “good”. Fifteen of the 100 people hired were not qualified, and did not work out. What parameter in the confusion matrix would you use to understand if your model worked better than you expected, as well as you expected or worse than you expected? How well did the model work? Explain the rationale for your answers. (10 points)
What aspects of the data would you explore if you want to improve the performance of your model to offer jobs to candidates? (10 points)
Describe any limitations and/or concerns associated with your approach. (10 points)
