Questions & Answers from the Webinar:
- What are your validity and reliability coefficients?
Whenever you use a test or assessment in the workplace, you need to make sure it actually measures what it is supposed to measure. In the assessment world, we show this by assessing the strengths of the relationship between the assessment and the outcome of interest (e.g., performance ratings, turnover, or in this case, injuries). This is done by conducting validation studies where we measure these correlations and refer to them as “Validity Co-efficients” because they indicate the level of criterion-related validity (i.e., predictive accuracy) of the test.
As we stated in the webinar, Select International’s validity co-efficients are high by industry standards. Our assessments have overall co-efficients that typically range between 0.45 and 0.55, and in some cases as high as 0.60. This indicates that the assessments are very accurate in predicting outcomes. In the case of our safety-specific outcomes, our assessments show correlations between -0.48 and -0.56 with safety incidents and injuries, respectively. They are also significantly correlated with supervisor ratings of safety behavior with co-efficients ranging from 0.35 to 0.45. It is also important to mention that we have been able to achieve these high validity co-efficients while having very low sub-group differences, which result in no adverse impact, particularly with our safety-based assessments. This table below can help put the magnitude of these co-efficients into perspective.
And in terms of our reliability co-efficients, our assessments consistently meet recommended guidelines, with reliability co-efficients above 0.70 and mostly in the 0.80's
- What is the adverse impact of your selection tool?
We routinely conduct analyses on all of our assessments to continuously minimize any potential adverse impact on protected minority groups in the U.S. The safety assessments we discussed in our webinar, including SecureFit, which is used for selection purposes, have never been found to have any adverse impact on any minority group. Furthermore, it shows near-zero (minority vs. majority) group differences in average scores and pass/fail rates, even in datasets of thousands of applicants across the variety of industries. In other words, the likelihood of adverse impact occurring is negligible because minority and majority group members tend to have very similar test results. We accomplish this by using test content that is always highly job-relevant, incorporating assessment methods that are shown by research to minimize adverse impact, and continuously improving our test content to increase accuracy and fairness.
- We saw the statistics for Wabtec using your assessments and programs. Are your statistics with other companies similar?
On the selection (pre-hire) side, yes we have similar results from other companies where we have helped reduce injuries and workers compensation claims over time through our safety pre-hire assessments. Our safety programs for development are newer than our selection tools, therefore we have somewhat more limited client results for those tools as we are still collecting data on clients currently using the programs. However, of the companies who have used our full programs for existing employees for at least 6 months, over half of them have seen significant reduction in injury metrics such as TRIR, number of recordables, or lost days due to injury. In contrast, none of them have seen any significant increases in injury rates since implementing the programs. These companies are from various types of manufacturing sectors, as well as the petrochemical, transportation and heavy equipment maintenance.
- What is the implementation process like for the safety training programs for front-line leaders?
Our implementation process can vary depending on the needs, situation, and structure of the company we are working with. All solutions are client-focused based on the situation, but critical aspects that we utilized in most projects are: measuring Safety Leadership profiles through validated assessments, providing training and skills development to leaders, and coaching them on their unique strengths and blind spots in order to reduce the risk to their team members.
- Does Select International offer ebooks or whitepapers on the topic of SafetyDNA?
Absolutely. We have multiple whitepapers on the topic of SafetyDNA that cover the theory, research and results behind our safety solutions. We have an eBook available and are currently working on a new one that will be available very soon. If you’re interested in downloading these to learn more about SafetyDNA, visit our resource library, and browse through our eBooks, business outcomes, videos and whitepapers.
- Todd, in your experience, what accident or event could have been averted had you had the Select tools in place?
As I shared in the webinar, the event that I believe could have been averted with these tools in place was probably the unfortunate and significant injury we had last summer, where an employee was struck by a forklift moving in reverse (driver not looking in direction of travel) and struck an employee standing in front of load positioned by another stationary forklift carrying a cage type tote bin full of steel parts. This resulted in major injuries and the individual being out of work for nearly a year. It was a devastating and tragic event that had tremendous impact on the site and the company as a whole. Surprisingly, the individual who was responsible for the incident was actually a Supervisor and the long time forklift trainer at the site. The entire event was captured on a digital security camera, where it was very clear that he never looked behind him or to the side when backing up the forklift multiple times while unloading a delivery truck. When questioned about his role in the event, the Supervisor displayed little personal ownership for what happened, indicating that it was only "33%" his fault, as two others were involved and present (including the injured employee). He went on to share how, after 40 years of driving a forklift, he did not really need to look back more than once when backing up because with just one quick glance he knew exactly where an object was going to be and could stop within one inch of it - without taking a second look. His risk tolerance was unacceptably high and eventually it led to this injury. While the company was weighing its options in the midst of the investigation, he decided to retire.
The point of the story is that as safety professional we need "sleep assurance"...when life altering events occur we need to take comfort in the fact we've taken ALL measures to prevent their occurrence. After this event, we implemented SafetyMirror for Leaders through Select International. My thinking is while we checked all the regulatory compliance boxes, had the SM4L program been available and implemented at that site, the driver's behavior - SafetyDNA been identified - the process may have alerted him and others to his unsafe ways and perhaps, just maybe, the incident averted.
- Todd, do you use the Red Flag Condition cards in place of near miss reporting or in addition to? Do you input the cards into any database?
That is generally site specific. If the site used a near miss report they may elect to keep and supplement with RFC cards. Yes, many sites set up spreadsheets, prioritize by risk and established completion dates and assigned "owners" of abatement.
- Todd, are you using a specific web-based software system to track and manage safety incidents? If so, which one?
We don't utilize an off the shelf product. We house worldwide safety metrics on our intranet. As part of our operational monthly FLASH calls with the executive office - including CEO, each site discusses where they are in terms of the business plan and any Recordable accidents. Any lost times accidents are provided to me on a specific form which is sent to Senior HR VP and CEO as soon as possible.
- Todd, do you use a behavior based program as well?
In many respects the "2 Minute Warning" puts a "name badge" on BBS. It was created as a method to modify behavior both at work and at home with a roadmap on how to do. Repetition and reminders are key. Like I said on the webinar, some people can sit down and play the piano at the first lesson, others may need to read music and never play by ear, still others will never get it. Put in SafetyDNA terms, we want the first example, can develop the second, and steer clear of the third. Select International enables us to be able to do that.
