Uber Open-Sources ‘Fiber’, A Python Distributed Computing Library For Modern Computer Clusters

Uber Introduces Fiber, a Python-based distributed computing library for modern computer clusters. Now you can code/program your computer cluster instead of programming your desktop or laptop. It was originally developed to power large scale parallel scientific computation projects like POET, Go-ExploreGTN.

Here are some of the key features for ‘Fiber’ (a Python distributed computing library for modern computer clusters).

  • Fiber is easy to use. It allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.
  • Fiber is easy to learn. It provides the same API as Python’s standard multiprocessing library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.
  • Fiber is fast. It’s communication backbone is built on top of Nanomsg which is a high-performance asynchronous messaging library to allow fast and reliable communication.
  • Fiber doesn’t need deployment.
  • Fiber is reliable. It has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.
  • Dynamic Scaling: Fiber can scale up and down according to the computation needed by the user.

Originally, it was developed to power large scale parallel scientific computation projects like POET and it has been used to power similar projects within Uber.

Installation (Copied from https://uber.github.io/fiber/)

pip install fiber

Code example: Estimating Pi with Fiber (Copied from https://uber.github.io/fiber/)

from fiber import Pool
import random

NUM_SAMPLES = int(1e6)

def is_inside(p):
    x, y = random.random(), random.random()
    return x * x + y * y < 1

def main():
    pool = Pool(processes=4)
    pi = 4.0 * sum(pool.map(is_inside, range(0, NUM_SAMPLES))) / NUM_SAMPLES
    print("Pi is roughly {}".format(pi))

if __name__ == '__main__':

Paper: https://arxiv.org/pdf/2003.11164.pdf

Github: https://github.com/uber/fiber

Tutorial: https://uber.github.io/fiber/getting-started/

Asif Razzaq is an AI Journalist and Cofounder of Marktechpost, LLC. He is a visionary, entrepreneur and engineer who aspires to use the power of Artificial Intelligence for good.

Asif's latest venture is the development of an Artificial Intelligence Media Platform (Marktechpost) that will revolutionize how people can find relevant news related to Artificial Intelligence, Data Science and Machine Learning.

Asif was featured by Onalytica in it’s ‘Who’s Who in AI? (Influential Voices & Brands)’ as one of the 'Influential Journalists in AI' (https://onalytica.com/wp-content/uploads/2021/09/Whos-Who-In-AI.pdf). His interview was also featured by Onalytica (https://onalytica.com/blog/posts/interview-with-asif-razzaq/).