Uber Open-Sources ‘Fiber’, A Python Distributed Computing Library For Modern Computer Clusters

Image Source: https://uber.github.io/fiber/introduction/

Uber Introduces Fiber, a Python-based distributed computing library for modern computer clusters. Now you can code/program your computer cluster instead of programming your desktop or laptop. It was originally developed to power large scale parallel scientific computation projects like POET, Go-ExploreGTN.

Here are some of the key features for ‘Fiber’ (a Python distributed computing library for modern computer clusters).

  • Fiber is easy to use. It allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.
  • Fiber is easy to learn. It provides the same API as Python’s standard multiprocessing library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.
  • Fiber is fast. It’s communication backbone is built on top of Nanomsg which is a high-performance asynchronous messaging library to allow fast and reliable communication.
  • Fiber doesn’t need deployment.
  • Fiber is reliable. It has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.
  • Dynamic Scaling: Fiber can scale up and down according to the computation needed by the user.

Originally, it was developed to power large scale parallel scientific computation projects like POET and it has been used to power similar projects within Uber.

AdvertisementCoursera Plus banner featuring Johns Hopkins University, Google, and University of Michigan courses highlighting data science career-advancing content

Installation (Copied from https://uber.github.io/fiber/)

pip install fiber

Code example: Estimating Pi with Fiber (Copied from https://uber.github.io/fiber/)

from fiber import Pool
import random

NUM_SAMPLES = int(1e6)

def is_inside(p):
    x, y = random.random(), random.random()
    return x * x + y * y < 1

def main():
    pool = Pool(processes=4)
    pi = 4.0 * sum(pool.map(is_inside, range(0, NUM_SAMPLES))) / NUM_SAMPLES
    print("Pi is roughly {}".format(pi))

if __name__ == '__main__':

Paper: https://arxiv.org/pdf/2003.11164.pdf

Github: https://github.com/uber/fiber

Tutorial: https://uber.github.io/fiber/getting-started/



Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.