LinkedIn open-sources LiFT to enable the measurement of fairness in large-scale machine learning workflows

Although there are many approaches and libraries presently in practice to analyze fairness in machine learning models, they all are still limited to some or other factors. These tools are either specifically not addressing large-scale problems (and the inherent challenges that come with such scale) or tied to a specific cloud environment.

To solve this issue, LinkedIn introduces an open-source tool called LiFT (LinkedIn Fairness Toolkit). LiFT is a Scala/Spark library that enables the measurement of fairness, according to a multitude of fairness definitions, in large-scale machine learning workflows.

The main features of LiFT include flexibility towards deployment in ML workflows and integration in different stages of ML training systems. It is designed as a reusable library at its core and can also be used for ad-hoc fairness analysis.. Apart from flexibility, LiFT is also scalable and can be distributed on several nodes to analyze and scale bias measurements in various machine learning models.

https://engineering.linkedin.com/blog/2020/lift-addressing-bias-in-large-scale-ai-applications

Github: https://github.com/linkedin/LiFT

Linkedin: https://engineering.linkedin.com/blog/2020/lift-addressing-bias-in-large-scale-ai-applications

Asif Razzaq is an AI Journalist and Cofounder of Marktechpost, LLC. He is a visionary, entrepreneur and engineer who aspires to use the power of Artificial Intelligence for good.

Asif's latest venture is the development of an Artificial Intelligence Media Platform (Marktechpost) that will revolutionize how people can find relevant news related to Artificial Intelligence, Data Science and Machine Learning.

Asif was featured by Onalytica in it’s ‘Who’s Who in AI? (Influential Voices & Brands)’ as one of the 'Influential Journalists in AI' (https://onalytica.com/wp-content/uploads/2021/09/Whos-Who-In-AI.pdf). His interview was also featured by Onalytica (https://onalytica.com/blog/posts/interview-with-asif-razzaq/).