Understanding Multi-Processing in Python: A Simiplified Guide

Posted on September 11th, 2023

Multiprocessing is a simple yet very useful generalization for us to solve complex or, we can say, vast challenges by running the processes concurrently. The multiprocessing concept has been a foundational aspect of computer architecture for many decades by utilizing the capabilities of multi-core CPUs.

Let’s understand it in a real-world scenario of using multiprocessing with small python code.

Imagine you are cooking 4 dishes sequentially; it takes more time. But you can make multiple dishes simultaneously if you are a group of 4 cooks. 

from multiprocessing import Process, current_process

def cook_dish(dish):
    print(f”person {current_process().name} is cooking {dish}…”)

if __name__ == “__main__”:
    dishes = [“salad”, “soup”, “main course”, “dessert”]
    for dish in dishes:
        Process(target=cook_dish, args=(dish,)).start()

You just need to understand the last line in the above code clearly. Let’s understand it by breaking down the line.

  1. Process: Process is the class from the multiprocessing module. In the above case, the process helps create separate kitchens(processes) for your friends to cook in. Here is an understandable image of it.
  1. target=cook_dish: This specifies the target to execute. In this case, the cook_dish function tells each friend how to cook a dish.
  2. args=(dish, ): this is giving each friend their recipe to cook. It passes the argument, which is a dish, in the cook_dish function. The comma in the (dish, ) looks strange. This is because when passing a single argument to a function using the args parameter, you need to mention a comma to indicate that it’s a tuple containing a single element.
  3. .start(): After setting up the processes and their arguments, the .start() method executes each separate process concurrently. This means each individual can cook their own dish at the same time. However, the order in which processes start might differ from the order of dishes in the list. It is due to various factors, including operating system scheduling algorithm, CPU cores, CPU utilization, I/O operations, etc.

The output of the above example code gives something like the below where process-X is each individual person that cooks:

person Process-1 is cooking salad…
person Process-4 is cooking dessert…
person Process-3 is cooking main course…
person Process-2 is cooking soup…

Let’s deep dive into multiprocessing intricacies, from managing parallel tasks to optimizing performance. 

Multiprocessing vs Multithreading

Many people have questions like, can multithreading achieve true parallelism in Python? The answer is No. The threads cannot run in parallel because of Python’s Global Interpreter Lock(GIL). Need help to understand what GIL is? Be at ease; we will comprehend this topic later in this article.      

The key difference in both is multiprocessing involves CPU-bound tasks(processing images in parallel), and in other cases, multithreading involves I/O-bound tasks like reading files or downloading files.

Let’s break down multiprocessing and multithreading step by step.

Multiprocessing

In multiprocessing, multiple processes run on specific CPUs to complete the respective tasks, and because each process runs on an individual CPU core, it uses its own Python interpreter and memory space.

In other terms, each process can fully utilize one CPU core and is not subjected to the GIL restrictions.

Here is the simple Python code that uses multiprocessing.

import multiprocessing
import requests

def download_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        print(f”Downloaded {url} ({len(response.content)} bytes)”)

if __name__ == “__main__”:
    urls = [
        “https://www.serveravatar.com”,
        “https://www.openai.com”,
        “https://www.python.org”,
    ]

    num_processes = len(urls)
    pool = multiprocessing.Pool(processes=num_processes)
    pool.map(download_page, urls)
    pool.close()
    pool.join()

In the above code,

  1. First, we define a function called download_page using a requests library to download the content of the page and print the URL and size in bytes.
  2. Then, in the if __name__ == “__main__” block, we define the list of URLs we download concurrently
  3. Then we specify the number of processes using the num_process variable that is equal to the number of URLs in the list. 
  4. After that, we create a multiprocessing pool and use the pool.map to achieve parallelism.
  5. At last, we close the pool and wait for all processes to complete using pool.join()

Multithreading

Multithreading works especially in I/O bound tasks, which means tasks involving waiting for I/O operations to complete, such as reading/writing to file, database query, etc. During this waiting time, the CPU is often idle.

All threads within the single process share the same memory space.

Although the multithreading name says running multiple threads concurrently, the GIL in CPython allows only one thread to execute Python bytecode within a single process.  

Here is the simple Python code that uses multiprocessing.

import threading
import requests

def download_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        print(f”Downloaded {url} ({len(response.content)} bytes)”)

if __name__ == “__main__”:
    urls = [
        “https://www.example.com”,
        “https://www.openai.com”,
        “https://www.python.org”,
    ]

    threads = []

    for url in urls:
        thread = threading.Thread(target=download_page, args=(url,))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

In the above piece of code:

  1. We create a function called download_page, similar to the multiprocessing example. 
  2. In the if __name__ = “__main__” block, we define the list of URLs to download concurrently.
  3. After that, we create an empty list called threads to store the thread objects.
  4. Then, we create separate threads for each URL by passing the url as an argument to the download_page function. 
  5. After that, we wait for all threads to complete using thread.join() method.

Core Concepts of Python Multiprocessing

Understanding the core concepts of Python multiprocessing makes sense to use the Python multiprocessing module. It can help you write effective, scalable, high-performance applications, mainly when dealing with concurrent tasks. It lets you use the power of modern computer hardware.

Let’s look at the core concepts of Python multiprocessing with example code.

1. Process creation

In Python, you can create a process using multiprocessing.Process class. Each process runs independently and with its own memory space, program counter, and resources.

The below code demonstrates creating and running two separate processes using multiprocessing.Process class.

import multiprocessing

def worker(num):
    print(f”Worker {num} is running.”)

if __name__ == “__main__”:
    process1 = multiprocessing.Process(target=worker, args=(1,))
    process2 = multiprocessing.Process(target=worker, args=(2,))
   
    process1.start()
    process2.start()
   
    process1.join()
    process2.join()
   
    print(“All processes have finished.”)

2. Parallelism

Parallelism is a concept about maximizing the use of hardware resources. In the below code, we create a list of numbers and use parallel processes to calculate the square of numbers concurrently.

import multiprocessing

def square_number(number):
    result = number * number
    print(f”The square of {number} is {result})

if __name__ == “__main__”:
    numbers = [1, 2, 3, 4, 5]
    processes = []

    for num in numbers:
        process = multiprocessing.Process(target=square_number, args=(num,))
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

    print(“All calculations completed.”)

In the above code, we wait for each process to complete before printing the completion message using the process.join() method. 

3. Shared Memory

Using Shared Memory, multiple concurrent processes can communicate and share the data to each other. The multiprocessing module provides mechanisms like Array and Value to transfer data between processes concurrently. 

Let’s understand with a real-life scenario example. 

Assume that you have a team of workers in a warehouse. All the workers have to count as many products as they have packed. But with that, you must also maintain a shared total count of all the workers.

import multiprocessing

def worker(worker_id, shared_total):
    # Simulate each worker packing products
    products_packed = 0
    for _ in range(10):
        products_packed += 1
        shared_total.value += 1
        print(f”Worker {worker_id}: Packed 1 product (Total: {shared_total.value} products)”)

if __name__ == “__main__”:
    shared_total = multiprocessing.Value(“i”, 0# Shared integer value
   
    # Create worker processes
    processes = []
    for i in range(5):  # 5 workers
        process = multiprocessing.Process(target=worker, args=(i, shared_total))
        processes.append(process)
        process.start()

    # Wait for all workers to finish
    for process in processes:
        process.join()

    print(f”Total products packed: {shared_total.value})

The fundamental of the above code is it can share the data among multiple processes using multiprocessing.Value

In the above code, 

  1. We create a worker function that simulates each worker packing product. 
  2. product_packed += 1 variable to keep track of how many products the individual worker has packed. 
  3. Shared_total.value += 1 also increments the value by 1, and it’s a shared variable(of type multiprocessing.value) that all worker processes can access. So it keeps track of the total number of products packed by all workers. 
  4. In the main program script, a shared integer value is created using multiprocessing.Value. It is responsible for sharing data among multiple processes. 
  5. Then, we create a list with variable processes to store the references to the worker processes.
  6. We create 5 workers using a for loop, and inside the loop, each process is initialized with the worker function and in the argument parameter, we pass worker’s ID(i) and the total of each worker.
  7. Append the processes in the processes list and start the process with process.start()
  8. The last loop iterates through the list of processes and waits for each worker process to complete using the process.join() method.

4. Inter-Process Communication(IPC)

Compared to shared memory, IPC allows processes to communicate in a more abstract and message-based manner, which is often safe and more flexible. 

Python multiprocessing offers several IPC mechanisms, including queues, pipes, and shared memory, to facilitate communication between processes.  

Imagine you have a small bakery with a few bakers and a cashier. The baker bakes the fresh bread, and the cashier serves customers and collects payments. For better communication, they use order slips to take and fulfill customer orders.

import multiprocessing

def baker(order_queue):
    # Simulate the baker taking and fulfilling orders
    while True:
        order = order_queue.get()
        if order == “exit”:
            break
        print(f”Baker: Preparing {order} bread”)

def cashier(order_queue):
    # Simulate the cashier taking customer orders and sending them to the baker
    orders = [“Baguette”, “Ciabatta”, “Sourdough”, “exit”]
    for order in orders:
        order_queue.put(order)

if __name__ == “__main__”:
    order_queue = multiprocessing.Queue()  # Queue for orders
   
    # Create processes for the baker and cashier
    baker_process = multiprocessing.Process(target=baker, args=(order_queue,))
    cashier_process = multiprocessing.Process(target=cashier, args=(order_queue,))
   
    baker_process.start()
    cashier_process.start()

    # Wait for both processes to finish
    baker_process.join()
    cashier_process.join()

    print(“Bakery is closed for the day.”)

The fundamentals of the above code are backers, and the cashier communicates using a multiprocessing.queue class.

In the above code: 

  1. First, create a baker function to simulate the baker taking and fulfilling orders. It continuously checks in the order_queue for the recent orders(using order_queue.get(order)), which is a shared queue between processes. If it sees an exit, it terminates the order and closes the loop. Otherwise, print the message indicating the baker is preparing a xyz bread.
  2. The cashier function represents the cashier’s behavior. All the customers’ orders, including an “exit” message to signal the end of the day, are on the list. The for loop iterates all the orders and puts them in the order_queue using order_queue.put(order) method.
  3. In the main program, we create a variable order_queue to communicate between baker and cashier using multiprocessing.Queue().
  4. Then, we create processes for baker and cashier to work in parallelism. 
  5. We start the processes and wait for all the processes to finish using the .join() method. 
  6. The Exit message is executed after the baker and cashier processes finish their work.

5. Global Interpreter Lock(GIL)

Global Interpreter Lock is a limitation in the CPython interpreter that allows the execution of only one thread to execute Python bytecode at a time within a process.

GIL is still in the Python feature list because CPython’s memory management systems are not thread-safe. So, the GIL  is used to serialize access to objects and memory to prevent race conditions.

Let’s understand the code example demonstrating the GIL. 

import threading

def worker(worker_id, iterations):
    for _ in range(iterations):
        print(f”Worker {worker_id} is working.”)

if __name__ == “__main__”:
    num_workers = 4
    iterations_per_worker = 10000

    threads = []
    for i in range(num_workers):
        thread = threading.Thread(target=worker, args=(i, iterations_per_worker))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

    print(“All workers have finished.”)

In the above code:

  1. The worker function represents the task that each worker thread will perform. It takes two parameters: worker_id, which identifies the worker, and iterations, which determines how often the worker performs the tasks. 
  2. In the main program, we define the number of workers and iteration per worker using the num_workers and  iterations_per_worker variables. 
  3. Then, we create a list to hold thread objects and use a for loop to create and start the worker threads. For each worker, a new threading.Thread object is created specifying the worker function as the target and passes the argument as the worker’s ID(‘i’) and number of iterations.
  4. The thread appends to the threads list, and then it is started using the thread.start() method.
  5. The last for loop is used to wait for each thread to finish its execution. The .join() method blocks all the programs until each thread has completed its tasks. After that, we print the message indicating that all workers have finished their tasks.

Using the Multiprocessing Pool

The pool concept in Python multiprocessing is useful when you have a set of independent tasks that can be executed concurrently. It refers to a mechanism for managing and distributing tasks among a fixed number of worker processes.

Let’s look at key features of pool class in Python multiprocessing.

  • Worker Processes: A pool consisting of predefined numbers of worker processes. And it can be set when creating the pool. 
  • Task Distribution: The pool distributes all the tasks among the available worker processes for execution. Each worker process picks up a task from the pool’s task queue and executes it. 
  • Concurrency: All the predefined workers can execute tasks concurrently in the pool. This allows you to take advantage of multi-core CPUs to perform multiple tasks concurrently. 
  • Task Queuing: The pool maintains the task queue, where tasks are placed when you submit them. It continuously checks the queue for new tasks to execute. When a worker process completes a task, it retrieves the next task from the queue. 
  • Synchronization: Pools have capabilities to coordinate between worker processes and also handle synchronization to ensure that tasks are executed safely and without any conflict. This includes managing access to shared resources and avoiding race conditions.
  • Result Retrieval: The pool returns the results to the main program in the order they were completed. So it makes it easy to collect and process the results. 
  • Terminate: When you decide to close the pool or all tasks are completed, the worker process is terminated, and the pool is cleaned up. 

Here are the real-life scenario codes that help you understand the pool process easily.

Assume that you run a photography studio and you have to resize, watermark and save thousands of high-resolution images; this is time-consuming, especially when dealing with large numbers of images, right?

And that’s how the pool concept came to parallelize this task:

import os
import multiprocessing
from PIL import Image

# Function to process an image
def process_image(image_file):
    # Load the image
    img = Image.open(image_file)

    # Resize the image
    img = img.resize((800, 600))

    # Add a watermark
    watermark = Image.open(‘watermark.png’)
    img.paste(watermark, (10, 10), watermark)

    # Save the processed image
    output_path = os.path.join(‘output’, os.path.basename(image_file))
    img.save(output_path)

if __name__ == “__main__”:
    input_folder = ‘input_images’
    image_files = [os.path.join(input_folder, file) for file in os.listdir(input_folder)]

    # Create a multiprocessing Pool with 4 worker processes
    pool = multiprocessing.Pool(processes=4)

    # Distribute the image processing tasks across the Pool
    pool.map(process_image, image_files)

    # Close the Pool and wait for all tasks to complete
    pool.close()
    pool.join()

    print(“All images processed and saved.”)

In the above code: 

  1. We define a process_image function that performs image processing tasks, like resizing, and adding watermarks in it. The function takes image_file as an argument which is a path to an image file. 
  2. Then, we load the image, resize it and add a watermark to it through the function.
  3. In the main program, the input_folder variable stores the folder name containing the input images.
  4. The image_files variable is a list comprehension that creates a list of image paths. For example, the input folder name is input_iamges, and it contains an image file like iamge1.jpg, then the image_files variable contains an element like [‘input_image/image1.jpg’]
  5. After that, multiprocessing.pool is created using 4 worker processes to perform parallel image processing. 
  6. The pool.map method is used to distribute the process_image function across the multiple worker processes in the pool. Each worker processes a different image from the image_files list concurrently.
  7. In the end, we closed the pool using the pool.close() method and wait for the other tasks to be completed using the pool.join() method. Once all the tasks are completed, we print the completion message. 

So that’s all for the Python multiprocessing concept. You can check how to debug multiprocessing in Python to avoid deadlocks and ensure data consistency. Click on the link to study more about Python process-based parallelism

Leave a Reply