Completion in Linux Device Driver – Linux Device Driver Tutorial Part 28

This article is a continuation of the  Series on Linux Device Drivers and carries the discussion on Linux device drivers and their implementation. The aim of this series is to provide easy and practical examples that anyone can understand. This is the Completion in Linux Device Driver – Linux Device Driver Tutorial Part 28.

Prerequisites

In the example section, I had used kthread to explain this completion. If you don’t know what is kthread and how to use it, then I would recommend you to explore that by using the below link.

  1. Kthread Tutorial in Linux Kernel
  2. Waitqueue Tutorial in Linux Kernel

Completion

Completion, the name itself says. When we want to notify or wake up some thread or something when we finished some work, then we can use completion. We’ll take one situation. We want to wait for one thread for something to run. Until that time that thread has to sleep. Once that process finished then we need to wake up that thread that is sleeping. We can do this by using completion without race conditions.

These completions are a synchronization mechanism which is a good method in the above situation mentioned rather than using improper locks/semaphores and busy-loops.

Completion in Linux Device Driver

In the Linux kernel, Completions are developed by using waitqueue.

The advantage of using completion in Linux is that they have a well defined, focused purpose which makes it very easy to see the intent of the code, but they also result in more efficient code as all threads can continue execution until the result is actually needed, and both the waiting and the signaling is highly efficient using low-level scheduler sleep/wakeup facilities.

There are 5 important steps in Completions.

  1. Initializing Completion in Linux
  2. Re-Initializing Completion in Linux
  3. Waiting for completion (The code is waiting and sleeping for something to finish)
  4. Waking Up Task (Sending a signal to sleeping part)
  5. Check the status

Initialize Completion

We have to include <linux/completion.h> and creating a variable of type struct completion,which has only two fields:

struct completion { 
    unsigned int done; 
    wait_queue_head_t wait; 
};

Where, wait is the waitqueue to place tasks on for waiting (if any). done is the completion flag for indicating whether it’s completed or not.

We can create the struct variable in two ways.

  1. Static Method
  2. Dynamic Method

You can use any one of the methods.

Static Method

DECLARE_COMPLETION(data_read_done);

Where the “data_read_done” is the name of the struct which is going to create statically.

Dynamic Method

init_completion (struct completion * x);

Where, x – completion structure that is to be initialized

Example:

struct completion data_read_done;

init_completion(&data_read_done);

In this init_completion call we initialize the waitqueue and set done to 0, i.e. “not completed” or “not done”.

Re-Initializing Completion

reinit_completion (struct completion * x);

Where, x – completion structure that is to be reinitialized

Example:

reinit_completion(&data_read_done);

This function should be used to reinitialize a completion structure so it can be reused. This is especially important after complete_all is used. This simply resets the ->done field to 0 (“not done”), without touching the waitqueue. Callers of this function must make sure that there are no racy wait_for_completion() calls going on in parallel.

Waiting for completion

For a thread to wait for some concurrent activity to finish, it calls anyone of the function based on the use case.

wait_for_completion

This is used to make the function waits for the completion of a task.

void wait_for_completion (struct completion * x);

Where, x – holds the state of this particular completion

This waits to be signaled for completion of a specific task. It is NOT interruptible and there is no timeout.

Example:

wait_for_completion (&data_read_done);

Note that wait_for_completion() is calling spin_lock_irq()/spin_unlock_irq(), so it can only be called safely when you know that interrupts are enabled. Calling it from IRQs-off atomic contexts will result in hard-to-detect spurious enabling of interrupts.

wait_for_completion_timeout

This is used to make the function waits for the completion of a task with a timeout. Timeouts are preferably calculated with msecs_to_jiffies() or usecs_to_jiffies(), to make the code largely HZ-invariant.

unsigned long wait_for_completion_timeout (struct completion * x, unsigned long timeout);

where,x – holds the state of this particular completion

timeout – timeout value in jiffies

This waits for either completion of a specific task to be signaled or for a specified timeout to expire. The timeout is in jiffies. It is not interruptible.

It returns if timed out, and positive (at least 1, or the number of jiffies left till timeout) if completed.

Example:

wait_for_completion_timeout (&data_read_done);

wait_for_completion_interruptible

This waits for the completion of a specific task to be signaled. It is interruptible.

int wait_for_completion_interruptible (struct completion * x);

where, x – holds the state of this particular completion

It return -ERESTARTSYS if interrupted, 0 if completed.

wait_for_completion_interruptible_timeout

This waits for either completion of a specific task to be signaled or for a specified timeout to expire. It is interruptible. The timeout is in jiffies. Timeouts are preferably calculated with msecs_to_jiffies() or usecs_to_jiffies(), to make the code largely HZ-invariant.

long wait_for_completion_interruptible_timeout (struct completion * x, unsigned long timeout);

where, x – holds the state of this particular completion

timeout – timeout value in jiffies

It returns -ERESTARTSYS if interrupted, 0 if timed out, positive (at least 1, or a number of jiffies left till timeout) if completed.

wait_for_completion_killable

This waits to be signaled for completion of a specific task. It can be interrupted by a kill signal.

int wait_for_completion_killable (struct completion * x);

where, x – holds the state of this particular completion

It returns -ERESTARTSYS if interrupted, 0 if completed.

wait_for_completion_killable_timeout

This waits for either completion of a specific task to be signaled or for a specified timeout to expire. It can be interrupted by a kill signal. The timeout is in jiffies. Timeouts are preferably calculated with msecs_to_jiffies() or usecs_to_jiffies(), to make the code largely HZ-invariant.

long wait_for_completion_killable_timeout (struct completion * x, unsigned long timeout);

where, x – holds the state of this particular completion

timeout – timeout value in jiffies

It returns -ERESTARTSYS if interrupted, 0 if timed out, positive (at least 1, or a number of jiffies left till timeout) if completed.

try_wait_for_completion

This function will not put the thread on the wait queue but rather returns false if it would need to en queue (block) the thread, else it consumes one posted completion and returns true.

bool try_wait_for_completion (struct completion * x);

where, x – holds the state of this particular completion

It returns 0 if completion is not available 1 if a got it succeeded.

This try_wait_for_completion() is safe to be called in IRQ or atomic context.

Waking Up Task

complete

This will wake up a single thread waiting on this completion. Threads will be awakened in the same order in which they were queued.

void complete (struct completion * x);

where, x – holds the state of this particular completion

Example:

complete(&data_read_done);

complete_all

This will wake up all threads waiting on this particular completion event.

void complete_all (struct completion * x);

where, x – holds the state of this particular completion

Check the status

completion_done

This is the test to see if completion has any waiters.

bool completion_done (struct completion * x);

where, x – holds the state of this particular completion

It returns 0 if there are waiters (wait_for_completion in progress) 1 if there are no waiters.

This completion_done() is safe to be called in IRQ or atomic context.

Driver Source Code – Completion in Linux

First, I will explain to you the concept of driver code.

In this source code, two places we are sending the complete call. One from the read function and another one from the driver exit function.

I’ve created one thread (wait_function) which has while(1). That thread will always wait for the event to complete. It will be sleeping until it gets a complete call. When it gets the complete call, it will check the condition. If the condition is 1 then the complete came from the read function. It is 2, then the complete came from the exit function. If complete came from the read function, it will print the read count and it will again wait. If it is coming from the exit function, it will exit from the thread.

Here I’ve added two versions of the code.

  1. Completion created by static method
  2. Completion created by dynamic method

But operation-wise, both are the same.

[Get  the source code from GitHub]

Completion created by static method

/***************************************************************************//**
*  \file       driver.c
*
*  \details    Simple linux driver (Completion Static method)
*
*  \author     EmbeTronicX
*
*  \Tested with Linux raspberrypi 5.10.27-v7l-embetronicx-custom+
*
*******************************************************************************/
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kdev_t.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/slab.h>                 //kmalloc()
#include <linux/uaccess.h>              //copy_to/from_user()
 
#include <linux/kthread.h>
#include <linux/completion.h>           // Required for the completion
#include <linux/err.h>
 
uint32_t read_count = 0;
static struct task_struct *wait_thread;
 
DECLARE_COMPLETION(data_read_done);
 
dev_t dev = 0;
static struct class *dev_class;
static struct cdev etx_cdev;
int completion_flag = 0;
 
static int __init etx_driver_init(void);
static void __exit etx_driver_exit(void);
 
/*************** Driver Functions **********************/
static int etx_open(struct inode *inode, struct file *file);
static int etx_release(struct inode *inode, struct file *file);
static ssize_t etx_read(struct file *filp, 
                                char __user *buf, size_t len,loff_t * off);
static ssize_t etx_write(struct file *filp, 
                                const char *buf, size_t len, loff_t * off);
/******************************************************/

//File operation structure
static struct file_operations fops =
{
        .owner          = THIS_MODULE,
        .read           = etx_read,
        .write          = etx_write,
        .open           = etx_open,
        .release        = etx_release,
};

/*
** Waitqueue thread
*/ 
static int wait_function(void *unused)
{
        
        while(1) {
                pr_info("Waiting For Event...\n");
                wait_for_completion (&data_read_done);
                if(completion_flag == 2) {
                        pr_info("Event Came From Exit Function\n");
                        return 0;
                }
                pr_info("Event Came From Read Function - %d\n", ++read_count);
                completion_flag = 0;
        }
        do_exit(0);
        return 0;
}

/*
** This function will be called when we open the Device file
*/ 
static int etx_open(struct inode *inode, struct file *file)
{
        pr_info("Device File Opened...!!!\n");
        return 0;
}

/*
** This function will be called when we close the Device file
*/ 
static int etx_release(struct inode *inode, struct file *file)
{
        pr_info("Device File Closed...!!!\n");
        return 0;
}

/*
** This function will be called when we read the Device file
*/
static ssize_t etx_read(struct file *filp, char __user *buf, size_t len, loff_t *off)
{
        pr_info("Read Function\n");
        completion_flag = 1;
        if(!completion_done (&data_read_done)) {
            complete (&data_read_done);
        }
        return 0;
}

/*
** This function will be called when we write the Device file
*/
static ssize_t etx_write(struct file *filp, const char __user *buf, size_t len, loff_t *off)
{
        pr_info("Write function\n");
        return len;
}

/*
** Module Init function
*/
static int __init etx_driver_init(void)
{
        /*Allocating Major number*/
        if((alloc_chrdev_region(&dev, 0, 1, "etx_Dev")) <0){
                pr_err("Cannot allocate major number\n");
                return -1;
        }
        pr_info("Major = %d Minor = %d \n",MAJOR(dev), MINOR(dev));
 
        /*Creating cdev structure*/
        cdev_init(&etx_cdev,&fops);
        etx_cdev.owner = THIS_MODULE;
        etx_cdev.ops = &fops;
 
        /*Adding character device to the system*/
        if((cdev_add(&etx_cdev,dev,1)) < 0){
            pr_err("Cannot add the device to the system\n");
            goto r_class;
        }
 
        /*Creating struct class*/
        if(IS_ERR(dev_class = class_create(THIS_MODULE,"etx_class"))){
            pr_err("Cannot create the struct class\n");
            goto r_class;
        }
 
        /*Creating device*/
        if(IS_ERR(device_create(dev_class,NULL,dev,NULL,"etx_device"))){
            pr_err("Cannot create the Device 1\n");
            goto r_device;
        }
 
        //Create the kernel thread with name 'mythread'
        wait_thread = kthread_create(wait_function, NULL, "WaitThread");
        if (wait_thread) {
                pr_info("Thread Created successfully\n");
                wake_up_process(wait_thread);
        } else
                pr_err("Thread creation failed\n");
 
        pr_info("Device Driver Insert...Done!!!\n");
        return 0;
 
r_device:
        class_destroy(dev_class);
r_class:
        unregister_chrdev_region(dev,1);
        return -1;
}

/*
** Module exit function
*/ 
static void __exit etx_driver_exit(void)
{
        completion_flag = 2;
        if(!completion_done (&data_read_done)) {
            complete (&data_read_done);
        }
        device_destroy(dev_class,dev);
        class_destroy(dev_class);
        cdev_del(&etx_cdev);
        unregister_chrdev_region(dev, 1);
        pr_info("Device Driver Remove...Done!!!\n");
}
 
module_init(etx_driver_init);
module_exit(etx_driver_exit);
 
MODULE_LICENSE("GPL");
MODULE_AUTHOR("EmbeTronicX <[email protected]>");
MODULE_DESCRIPTION("A simple device driver - Completion (Static Method)");
MODULE_VERSION("1.23");

Completion created by dynamic method

/****************************************************************************//**
*  \file       driver.c
*
*  \details    Simple linux driver (Completion Dynamic method)
*
*  \author     EmbeTronicX
*
*  \Tested with Linux raspberrypi 5.10.27-v7l-embetronicx-custom+
*
*******************************************************************************/
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kdev_t.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/slab.h>                 //kmalloc()
#include <linux/uaccess.h>              //copy_to/from_user()
 
#include <linux/kthread.h>
#include <linux/completion.h>           // Required for the completion
#include <linux/err.h>
 
uint32_t read_count = 0;
static struct task_struct *wait_thread;
 
struct completion data_read_done;
 
dev_t dev = 0;
static struct class *dev_class;
static struct cdev etx_cdev;
int completion_flag = 0;
 
static int __init etx_driver_init(void);
static void __exit etx_driver_exit(void);
 
/*************** Driver Functions **********************/
static int etx_open(struct inode *inode, struct file *file);
static int etx_release(struct inode *inode, struct file *file);
static ssize_t etx_read(struct file *filp, char __user *buf, size_t len,loff_t * off);
static ssize_t etx_write(struct file *filp, const char *buf, size_t len, loff_t * off);
/******************************************************/

//File operation structure
static struct file_operations fops =
{
        .owner          = THIS_MODULE,
        .read           = etx_read,
        .write          = etx_write,
        .open           = etx_open,
        .release        = etx_release,
};

/*
** Waitqueue thread
*/ 
static int wait_function(void *unused)
{
        
        while(1) {
                pr_info("Waiting For Event...\n");
                wait_for_completion (&data_read_done);
                if(completion_flag == 2) {
                        pr_info("Event Came From Exit Function\n");
                        return 0;
                }
                pr_info("Event Came From Read Function - %d\n", ++read_count);
                completion_flag = 0;
        }
        do_exit(0);
        return 0;
}

/*
** This function will be called when we open the Device file
*/ 
static int etx_open(struct inode *inode, struct file *file)
{
        pr_info("Device File Opened...!!!\n");
        return 0;
}

/*
** This function will be called when we close the Device file
*/ 
static int etx_release(struct inode *inode, struct file *file)
{
        pr_info("Device File Closed...!!!\n");
        return 0;
}

/*
** This function will be called when we read the Device file
*/
static ssize_t etx_read(struct file *filp, char __user *buf, size_t len, loff_t *off)
{
        pr_info("Read Function\n");
        completion_flag = 1;
        if(!completion_done (&data_read_done)) {
            complete (&data_read_done);
        }
        return 0;
}

/*
** This function will be called when we write the Device file
*/
static ssize_t etx_write(struct file *filp, const char __user *buf, size_t len, loff_t *off)
{
        pr_info("Write function\n");
        return len;
}

/*
** Module Init function
*/
static int __init etx_driver_init(void)
{
        /*Allocating Major number*/
        if((alloc_chrdev_region(&dev, 0, 1, "etx_Dev")) <0){
                pr_err("Cannot allocate major number\n");
                return -1;
        }
        pr_info("Major = %d Minor = %d \n",MAJOR(dev), MINOR(dev));
 
        /*Creating cdev structure*/
        cdev_init(&etx_cdev,&fops);
        etx_cdev.owner = THIS_MODULE;
        etx_cdev.ops = &fops;
 
        /*Adding character device to the system*/
        if((cdev_add(&etx_cdev,dev,1)) < 0){
            pr_err("Cannot add the device to the system\n");
            goto r_class;
        }
 
        /*Creating struct class*/
        if(IS_ERR(dev_class = class_create(THIS_MODULE,"etx_class"))){
            pr_err("Cannot create the struct class\n");
            goto r_class;
        }
 
        /*Creating device*/
        if(IS_ERR(device_create(dev_class,NULL,dev,NULL,"etx_device"))){
            pr_err("Cannot create the Device 1\n");
            goto r_device;
        }
 
        //Create the kernel thread with name 'mythread'
        wait_thread = kthread_create(wait_function, NULL, "WaitThread");
        if (wait_thread) {
                pr_info("Thread Created successfully\n");
                wake_up_process(wait_thread);
        } else {
                pr_err("Thread creation failed\n");
        }
 
        //Initializing Completion
        init_completion(&data_read_done);
 
        pr_info("Device Driver Insert...Done!!!\n");
        return 0;
 
r_device:
        class_destroy(dev_class);
r_class:
        unregister_chrdev_region(dev,1);
        return -1;
}

/*
** Module exit function
*/ 
static void __exit etx_driver_exit(void)
{
        completion_flag = 2;
        if(!completion_done (&data_read_done)) {
            complete (&data_read_done);
        }
        device_destroy(dev_class,dev);
        class_destroy(dev_class);
        cdev_del(&etx_cdev);
        unregister_chrdev_region(dev, 1);
        pr_info("Device Driver Remove...Done!!!\n");
}
 
module_init(etx_driver_init);
module_exit(etx_driver_exit);
 
MODULE_LICENSE("GPL");
MODULE_AUTHOR("EmbeTronicX <[email protected]>");
MODULE_DESCRIPTION("A simple device driver - Completion (Dynamic Method)");
MODULE_VERSION("1.24");

MakeFile

obj-m += driver.o
 
KDIR = /lib/modules/$(shell uname -r)/build
 
 
all:
  make -C $(KDIR)  M=$(shell pwd) modules
 
clean:
  make -C $(KDIR)  M=$(shell pwd) clean

Building and Testing Driver

  • Build the driver by using Makefile (sudo make)
  • Load the driver using sudo insmod driver.ko
  • Then Check the Dmesg
Major = 246 Minor = 0
Thread Created successfully
Device Driver Insert…Done!!!
Waiting For Event…
  • So that thread is waiting for the event. Now we will send the event by reading the driver using sudo cat /dev/etx_device
  • Now check the dmesg
Device File Opened…!!!
Read Function
Event Came From Read Function – 1
Waiting For Event…
Device File Closed…!!!
  • We send the complete from the read function, So it will print the read count, and then again it will sleep. Now send the event from the exit function by sudo rmmod driver
Event Came From Exit Function
Device Driver Remove…Done!!!
  • Now the condition was 2. So it will return from the thread and remove the driver.

Please find the other Linux device driver tutorials here.

You can also read the below tutorials.

Linux Device Driver TutorialsC Programming Tutorials
FreeRTOS TutorialsNuttX RTOS Tutorials
RTX RTOS TutorialsInterrupts Basics
I2C Protocol – Part 1 (Basics)I2C Protocol – Part 2 (Advanced Topics)
STM32 TutorialsLPC2148 (ARM7) Tutorials
PIC16F877A Tutorials8051 Tutorials
Unit Testing in C TutorialsESP32-IDF Tutorials
Raspberry Pi TutorialsEmbedded Interview Topics
Reset Sequence in ARM Cortex-M4BLE Basics
VIC and NVIC in ARMSPI – Serial Peripheral Interface Protocol
STM32F7 Bootloader TutorialsRaspberry PI Pico Tutorials
STM32F103 Bootloader TutorialsRT-Thread RTOS Tutorials
Zephyr RTOS Tutorials - STM32Zephyr RTOS Tutorials - ESP32
AUTOSAR TutorialsUDS Protocol Tutorials
Product ReviewsSTM32 MikroC Bootloader Tutorial
VHDL Tutorials
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Table of Contents