This article describes how Artificial Intelligence (AI) is used in the Plum Platform and how Plum ensures that AI use is governed at Plum. A separate article, Protecting Your Privacy, discusses how we ensure your personal information is protected.
Overview
The Plum Platform automates the process of performing and interpreting psychometric assessments of candidates and employees using advanced analytics and artificial intelligence.
Plum is committed to ensuring our use of AI is ethical, transparent, and trusted. At Plum, great care is taken to ensure all data used to build our models is ethically sourced with full consent from contributors and is free from biases.
Plum strives for transparency. We publish details of how we use and build our algorithms. But we go further to ensure that all outputs are presented in a manner that humans can understand and validate the results.
Plum ensures that our AI systems are trusted. Plum maintains a robust risk management program and tests and monitors all algorithms for accuracy, effectiveness, and safety
Artificial intelligence powers the features illustrated below.
Artificial Intelligence in the Plum Platform
- The Plum Talent Profile is the heart of the Plum Platform. Based on a deployment of the Plum Measurement Model, it is an automated psychometric assessment of an individual’s workplace behaviors. It provides individuals with insights into their durable skills and enables them to explore their career options. For employers, it powers Plum’s Talent Match which is used in hiring to assess candidate durable skills. It also powers a growing suite of talent management capabilities including Culture Assessment, Talent Map, and Leadership Potential.
- Ask Violet provides a Large Language Model (LLM) driven natural language interface through which an individual can explore their talent profile.
- Ask Violet for Groups provides a Large Language Model (LLM) driven natural language interface through which employers can, as part of their talent management activities, explore the talent profiles of employees.
- The Plum Role Model is an AI-powered tool that analyzes a job description to create a "behavioral blueprint" of the skills and traits needed for success in that role. It enables employers to rapidly scale their use of Plum in hiring while still allowing human experts to contribute to the behavioral requirements for a role. It uses a Large Language Model to process job descriptions and find the best match to a standardized set of job descriptions and retrieve Job Match Criteria. The Job Match Criteria is used as input to the Plum Talent Match feature.
Plum Measurement Model
What is the Plum Measurement?
The Plum Measurement Model powers the Plum Talent Profile, Plum Talent Match, Plum Talent Map, Plum Culture Assessment, and Plum Leadership Potential features.
The Plum Measurement Model is a custom AI built and deployed by Plum
The Plum Measurement Model takes a user’s responses to the Plum Discovery Survey and produces the Plum Talent Profile. The algorithm encodes a psychometric assessment based on the science of I/O Psychology.
The Plum Discovery Survey is a structured survey designed to produce stable measurements of durable skills and to counter attempts to intentionally influence the Plum Talent Profile.
How is the Plum Measurement Model Used?
Recruiting - Plum Talent Match
The Plum Talent Match provides employers with the ability to assess applicant’s durable skills relative to specified job criteria.
The employer creates a job in the Plum Platform and assigns Match Criteria to that job. The match criteria represent the durable skills (Plum Talents) important for the job.
When a candidate applies for a job, the candidate’s talent scores and match criteria are inputs to an algorithm that produces the Talent Match score. The score and a summary of the candidate’s talents contributing to the score are presented to the employer to support the selection process.
Talent Management
Employers have employees complete the Plum Discovery Survey and create a Plum Profile. Using Plum, the employer can conduct a number of analyses based on their employees’ Plum Talent Profiles. These include:
- Talent Maps assess employees’ durable skills across opportunities within the company. The employer assigns Match Criteria to opportunities and Plum assesses employees using the Talent Match score. Talent Maps support several use cases such as strategic internal mobility, reorg/redeployment, and workforce planning, employee mentoring opportunities and stretch assignments, and building project teams.
- Culture Fit Assessment: is an aggregate analysis of employees’ talents to assess alignment between the workforce and organization values.
- Leadership Potential: using the Plum Talent Profile for elect employees, compute a Leadership Potential score to identify the employees who have the potential to succeed in leadership positions.
How the Plum Measurement Model Works
The Plum Talent Profile employs established science in psychometrics and I/O psychology to convert responses to the discovery survey into a network of measurements which are then used to provide candidates, employees, and employers with assessments and insights discussed above.
The Plum Measurement Model is a feed-forward network as illustrated below. The first four layers (Discovery Survey Responses, Traits, Competencies, and Talents) are used to create the Plum Talent Profile. Additional layers are added to the network to deliver Talent Match, Culture Assessment, and Leadership Potential.
The network was “trained” using established methods in the science of psychometrics and I/O psychology. Responses to the Discovery Survey from a carefully curated set approximately 80,000 individuals were used to calibrate the network. During development each layer of the network was tested independently for stability and predictive validity. Unlike many AI models, each layer of the Plum Measurement Model is meaningful and the internal results can and are explained to users in the Platform.
You can read more about The Science Behind Plum and details about the development and testing of the Plum Measurement Model in our Technical Manual.
Plum Measurement Model
Development and Testing of the Plum Measurement Model
The Plum Measurement Model is developed and deployed by Plum. It is a static model; that is, the algorithm does not use other inputs to learn and does not change over time. In fact, the stability of the Plum Measurement Model is a key feature. It is designed to measure innate workplace behavioral characteristics and to produce consistent results at the individual and the population level.
The development of the Plum Measurement Model used data from carefully curated set of approximately 80,000 anonymized responses to the Discovery Survey collected directly from respondents with their consent to train the algorithm.
Prior to deployment, the Plum Measurement Model was subjected to rigorous testing to establish its validity (i.e., its ability to predict) and safety (adverse impacts). When new features based on the Plum Measurement Model (e.g., Culture Assessment), these features are subject to the same level of rigorous testing prior to deployment.
Although the algorithm is static, Plum continues to test after deployment.
- Plum is audited annually for compliance with NYC Law 144 concerning bias in Automated Employment Decision Tools.
- Plum conducts ongoing studies related to validity and effectiveness.
- Plum conducts regular internal analyses to monitor for potential bias that may emerge.
- Customers have chosen to conduct independent studies and in many cases have made these available to Plum.
Plum Role Model
What is Plum Role Model™
The Plum Role Model™ is a science-based feature within Plum’s talent assessment platform that defines the behavioral traits, problem-solving abilities, and personality strengths required for success in a specific role. By instantly transforming job descriptions into validated behavioral blueprints, Role Model™ enables organizations to benchmark candidates and employees against what great looks like, beyond résumés, gut feel, or outdated credentials.
How is Plum Role Model™ Used?
The Plum Role Model™ is used to create Job Match Criteria required to produce a Talent Match for jobs. See Recruiting - Plum Talent Match. The Plum Role Model ™ can be used on its own or as one expert contributor alongside human job experts.
The Plum Role Model:
- Takes job descriptions as input. This is low sensitive customer data that will generally be made public.
- Does not use customer data to train the system.
- Does not process personal data.
- Is used only indirectly in assessing individuals for employment and only after direct human intervention.
How does Plum Role Model™ Work?
The Plum Role Model™ takes the text of a job description as input and uses a Large Language Model (LLM) to find the closest match from a standardized set of job descriptions. Using the standard job description, Plum looks up the validated match criteria for that job description.
Using standard job descriptions from O*NET, Plum developed match criteria by having multiple LLM agents complete the Plum Job Analysis Survey (also referred to as Match Criteria Survey) for each standard job description. The derived match criteria were subjected to validation studies to ensure that the durable skills (Plum Talents) specified by the match criteria were predictive of performance in these jobs. These match criteria were also analyzed to ensure that they were unlikely to result in adverse effects on protected groups.
The mapping function that takes arbitrary job descriptions to standard O*NTE job descriptions was tested to ensure:
- Stability: similar job descriptions map to the same standard job description, and
- Accuracy: the standard job description is a good match to the input.
Development and Testing of Plum Role Model
Plum Role Model is a tailored and constrained deployment of a large language model (LLM) service (currently by Open AI). The system operates over a fixed, curated set of predefined occupations, preventing open-ended or speculative generation. Each predefined occupation is associated with Job Match Criteria, determined using multiple agents completing the Plum Job Analysis Survey on the occupation. When an employer submits a job description, a deterministic bi-encoder retrieval step selects a limited top-k candidates from the predefined jobs set. The LLM is then used to re-rank these candidates and generate explanatory text within a tightly structured prompt.
Plum Role Model is used only to recommend match criteria for a job and the recommendation is subject to review by the employer before being used to rank applicants.
Customer inputs are processed transiently for inference only and are not used to train or fine-tune models.
Outputs are human-interpretable and reviewable, enabling oversight and validation.
The Match Criteria associated with the pre-defined occupations were subjected to extensive validity and adverse effects (bias) analyses to assess predictive capacity and to assess risk of bias from the use of Plum Role Model.
Operational monitoring of customer use and feedback is ongoing.
Ask Violet and Ask Violet for Groups
What are Ask Violet and Ask Violet for Groups?
Ask Violet provides individuals with an intelligent natural language interface to their profile. Ask Violet for Groups provides the same intelligent natural language interface to a group of profiles and intended to be used by employers for understanding the durable skills of their team.
How is Ask Violet and Ask Violet Used?
For individuals, ask Violet is available by default in their profile. Employers can choose to make ask Violet available to their employees. When enabled, a user accesses Ask Violet via a chat icon in the web app. The user asks Ask Violet about their profile and Ask Violet provides answers based on their profile and the feature’s general language knowledge.
Employers who have enabled Ask Violet for their employees, have the option of enabling Ask Violet for Groups at the level of a Talent Collections.
How does Ask Violet Work?
For Ask Violet, an LLM is provided with the content that makes up the user profile and responds to questions from the user about their profile. Information Is provided to the LLM only when the user
For Ask Violet for Groups, an LLM is provided with the content of profiles of group members enabling an authorized manager to use the LLM to use natural language queries regarding the group.
Ask Violet for Groups is only active when the feature is enabled for the customer (it is disabled by default) and the customer enables it for the group.
Development and Testing of Ask Violet
Both Ask Violet and Ask Violet for Groups have been tested to ensure that responses from the AI are stable and accurate.
Plum has established a monitoring program to ensure that responses continue to be stable and accurate.