Friday, June 20, 2008

Parallel Python - an awesome distributed computing library for Python

http://www.parallelpython.com/

Parallel Python helps you execute your Python code on multiple cores on the same machine, or on a whole cluster of network machines, all transparently, load balanced and with fault tolerance.

"Features:
  • Parallel execution of python code on SMP and clusters
  • Easy to understand and implement job-based parallelization technique (easy to convert serial application in parallel)
  • Automatic detection of the optimal configuration (by default the number of worker processes is set to the number of effective processors)
  • Dynamic processors allocation (number of worker processes can be changed at runtime)
  • Low overhead for subsequent jobs with the same function (transparent caching is implemented to decrease the overhead)
  • Dynamic load balancing (jobs are distributed between processors at runtime)
  • Fault-tolerance (if one of the nodes fails tasks are rescheduled on others)
  • Auto-discovery of computational resources
  • Dynamic allocation of computational resources (consequence of auto-discovery and fault-tolerance)
  • SHA based authentication for network connections
  • Cross-platform portability and interoperability (Windows, Linux, Unix, Mac OS X)
  • Cross-architecture portability and interoperability (x86, x86-64, etc.)
  • Open source"
I tried it, it's very easy to set up.

A review: http://www.devchix.com/2007/08/03/parallel-python-review-in-a-nutshell-wow/

No comments: