Blog
Improving Clustering on Occupational Text Data through Dimensionality Reduction
In this study, we focused on proposing an optimal clustering mechanism for the occupations defined in the well-known US-based occupational database, O*NET. Even though all occupations are defined according to well-conducted surveys in the US, their definitions can vary for different firms and countries. Hence, if one wants to expand the data that is already collected in O*NET for the occupations defined with different tasks, a map between the definitions will be a vital requirement. We proposed a pipeline using several BERT-based techniques with various clustering approaches to obtain such a map. We also examined the effect of dimensionality reduction approaches on several metrics used in measuring performance of clustering algorithms. Finally, we improved our results by using a specialized silhouette approach. This new clustering-based mapping approach with dimensionality reduction may help distinguish the occupations automatically, creating new paths for people wanting to change their careers.
Preparing the Workforce for an Automated Future
Over the past few years, artificial intelligence has experienced a remarkable development. The surge of deep neural networks, with the subsequent development of computer vision and foundational models, has gradually allowed machines to perform increasingly better at tasks that were previously reserved for humans.
Unveiling the nexus of AI and gender at work with our recent working paper!
We are delighted to share this pivotal research under our project, providing multi-layered insights into the evolving roles and challenges. You can reach our study from PDF attached to this page.