Topics

FIFO is not empty after getting all data items; reference pointer still points to another item


Weber, Dominik <dominik.weber@...>
 

Hey everyone,

 

I’m currently having problems when putting items to a FIFO. As long as I put data items to my FIFO queue, everything is ok and works as expected. If I stop putting data to the queue, the reading thread continues reading the last data item out of the queue over and over again, although the queue should be empty but is not (k_fifo_is_empty() says it’s not empty).

 

There’s a writing and a reading thread in my software project.

 

I have the following data item, defined in the header file of the reading thread:

 

typedef struct

{

    void *fifoReserved;     ///< 1st word reserved for use by fifo

    int data1;               

    int data2;              

 

s_data_item_fifo_ecg_t;

 

 

 

 

The writing thread is like:

 

(This is a static variable in the software module)

static s_data_item_fifo_ecg_t ecgDataStruct;  ///< This data item contains the ecg information to put to the data fifo.

 

(Inside the thread)

while (1)

    {

        /* Thread will be unready until signal was raised, signal will be raised when data ready interrupt comes */

        k_poll(daEvents1, K_FOREVER);

 

        /* Reset signal */

        daEvents[0].signal->signaled = 0;

        daEvents[0].state = K_POLL_STATE_NOT_READY;

 

        /* …get data from hardware chip here… */

 

        ecgDataStruct.data1 = ;

        ecgDataStruct.data2 = ;

 

        dataProcessingPutFifoECG(&ecgDataStruct);

    }

 

The last function is part of the reading thread software module and just puts the item to the queue with k_fifo_put().

 

 

 

The reading thread is like:

 

(This is a static variable in the software module)

static struct k_fifo dpFifoECG;      ///< FIFO structure for handling incoming ECG data

 

There is an init function, where this is called:

k_fifo_init(&dpFifoECG);

 

 

s_data_item_fifo_ecg_t *fifoDataECG;

 

    while (1

    {

        /* Get data from queue. */

        fifoDataECG = k_fifo_get(&dpFifoECG, K_FOREVER);

 

        printk("%d,%d\n"fifoDataECG->data1fifoDataECG->data2);

 

    }

 

I checked the print outputs with a terminal program and realized that the data is still printed, even if I stop the measurement and stop putting data to the queue, which means that the program is not waiting at k_fifo_get() as expected. It just continues reading the last item which should be removed from the queue.

 

When I have a look at the fifoReserved pointer for these unwanted items, I see that the pointer is always the same and not NULL. Shouldn’t the pointer be NULL if there are no more items in the list?

 

This is really driving me crazy.

 

Both threads do have a maximum memory thread stack size of 8kB.

 

The priorities are -2 for the writing thread (implements SPI communication) and 1 for the reading thread. (There are three threads running on the system, excluding main thread and idle thread).

 

A general question:

 

For me, it is not really clear, if a data item is passed by copy or by reference to the fifo queue. I assume, the items are passed by reference, since I faced some serious problems when passing data items from an ISR, which are lying on the ISR stack. The content of the data item was sometimes different after reading the item compared to when the item was putted to the queue. The solution to this problem was to define a static data item on the memory heap of the software module and to only adjust the data of the same item and put it again to the queue. This seems very unintuitive to me, since I put the same item to the queue over and over again, only with different content. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

My setup:

·         Zephyr 2.3 Build 99

·         Nordic nRF52840 DK

·         Segger J-Link OB-SAM3U128-V2-NordicSemi

·         Eclipse 2019-09

·         Python 3.8.3

·         West 0.7.2

·         C compiler GNU 9.2.1

 

Thanks for your help!

 


Carles Cufi
 

Hi Dominik,

 

>. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

FIFOs do not copy the contents of the message, they only store the pointer in a linked list. Maybe this is not clear enough in the documentation, please consider sending a Pull Request enhancing the doc.

 

If you need the kernel to copy the contents of your message for you, you can use Message Queues:

https://docs.zephyrproject.org/latest/reference/kernel/data_passing/message_queues.html

 

Thanks,

 

Carles

 

From: users@... <users@...> On Behalf Of Weber, Dominik via lists.zephyrproject.org
Sent: 28 August 2020 13:22
To: users@...
Subject: [Zephyr-users] FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hey everyone,

 

I’m currently having problems when putting items to a FIFO. As long as I put data items to my FIFO queue, everything is ok and works as expected. If I stop putting data to the queue, the reading thread continues reading the last data item out of the queue over and over again, although the queue should be empty but is not (k_fifo_is_empty() says it’s not empty).

 

There’s a writing and a reading thread in my software project.

 

I have the following data item, defined in the header file of the reading thread:

 

typedef struct

{

    void *fifoReserved;     ///< 1st word reserved for use by fifo

    int data1;               

    int data2;              

 

s_data_item_fifo_ecg_t;

 

 

 

 

The writing thread is like:

 

(This is a static variable in the software module)

static s_data_item_fifo_ecg_t ecgDataStruct;  ///< This data item contains the ecg information to put to the data fifo.

 

(Inside the thread)

while (1)

    {

        /* Thread will be unready until signal was raised, signal will be raised when data ready interrupt comes */

        k_poll(daEvents1, K_FOREVER);

 

        /* Reset signal */

        daEvents[0].signal->signaled = 0;

        daEvents[0].state = K_POLL_STATE_NOT_READY;

 

        /* …get data from hardware chip here… */

 

        ecgDataStruct.data1 = ;

        ecgDataStruct.data2 = ;

 

        dataProcessingPutFifoECG(&ecgDataStruct);

    }

 

The last function is part of the reading thread software module and just puts the item to the queue with k_fifo_put().

 

 

 

The reading thread is like:

 

(This is a static variable in the software module)

static struct k_fifo dpFifoECG;      ///< FIFO structure for handling incoming ECG data

 

There is an init function, where this is called:

k_fifo_init(&dpFifoECG);

 

 

s_data_item_fifo_ecg_t *fifoDataECG;

 

    while (1

    {

        /* Get data from queue. */

        fifoDataECG = k_fifo_get(&dpFifoECG, K_FOREVER);

 

        printk("%d,%d\n"fifoDataECG->data1fifoDataECG->data2);

 

    }

 

I checked the print outputs with a terminal program and realized that the data is still printed, even if I stop the measurement and stop putting data to the queue, which means that the program is not waiting at k_fifo_get() as expected. It just continues reading the last item which should be removed from the queue.

 

When I have a look at the fifoReserved pointer for these unwanted items, I see that the pointer is always the same and not NULL. Shouldn’t the pointer be NULL if there are no more items in the list?

 

This is really driving me crazy.

 

Both threads do have a maximum memory thread stack size of 8kB.

 

The priorities are -2 for the writing thread (implements SPI communication) and 1 for the reading thread. (There are three threads running on the system, excluding main thread and idle thread).

 

A general question:

 

For me, it is not really clear, if a data item is passed by copy or by reference to the fifo queue. I assume, the items are passed by reference, since I faced some serious problems when passing data items from an ISR, which are lying on the ISR stack. The content of the data item was sometimes different after reading the item compared to when the item was putted to the queue. The solution to this problem was to define a static data item on the memory heap of the software module and to only adjust the data of the same item and put it again to the queue. This seems very unintuitive to me, since I put the same item to the queue over and over again, only with different content. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

My setup:

  • Zephyr 2.3 Build 99
  • Nordic nRF52840 DK
  • Segger J-Link OB-SAM3U128-V2-NordicSemi
  • Eclipse 2019-09
  • Python 3.8.3
  • West 0.7.2
  • C compiler GNU 9.2.1

 

Thanks for your help!

 


Weber, Dominik <dominik.weber@...>
 

Hi Carles,

 

thanks for your reply. This solved my problem.

 

But why does Zephyr not copy the items to the list? For me, there’s absolutely no reason to NOT copy data items to a queue . It runs the risk to…  

·         … put data items to the queue which are on the stack of a function or an ISR. Thus, these elements would be not valid anymore when the function returns. This is a major security issue, since you won’t be able to recognize the error until it will be overwritten randomly in memory…

·         … put the same data item several times to the queue which has the same memory address but different content (which was actually my problem)

Furthermore I would need a separate buffer to store my queue items, if I only put pointers to the queue. So one of the main features of a queue would be obsolete.

 

(The way FreeRTOS does it, is way more intuitive for me: https://www.freertos.org/Embedded-RTOS-Queues.html)

 

Best Regards,

Dominik

 

Von: Cufi, Carles <Carles.Cufi@...>
Gesendet: Freitag, 28. August 2020 16:29
An: Weber, Dominik <dominik.weber@iis
.fraunhofer.de>; users@...
Betreff: RE: FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hi Dominik,

 

>. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

FIFOs do not copy the contents of the message, they only store the pointer in a linked list. Maybe this is not clear enough in the documentation, please consider sending a Pull Request enhancing the doc.

 

If you need the kernel to copy the contents of your message for you, you can use Message Queues:

https://docs.zephyrproject.org/latest/reference/kernel/data_passing/message_queues.html

 

Thanks,

 

Carles

 

From: users@... <users@...> On Behalf Of Weber, Dominik via lists.zephyrproject.org
Sent: 28 August 2020 13:22
To: users@...
Subject: [Zephyr-users] FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hey everyone,

 

I’m currently having problems when putting items to a FIFO. As long as I put data items to my FIFO queue, everything is ok and works as expected. If I stop putting data to the queue, the reading thread continues reading the last data item out of the queue over and over again, although the queue should be empty but is not (k_fifo_is_empty() says it’s not empty).

 

There’s a writing and a reading thread in my software project.

 

I have the following data item, defined in the header file of the reading thread:

 

typedef struct

{

    void *fifoReserved;     ///< 1st word reserved for use by fifo

    int data1;               

    int data2;              

 

s_data_item_fifo_ecg_t;

 

 

 

 

The writing thread is like:

 

(This is a static variable in the software module)

static s_data_item_fifo_ecg_t ecgDataStruct;  ///< This data item contains the ecg information to put to the data fifo.

 

(Inside the thread)

while (1)

    {

        /* Thread will be unready until signal was raised, signal will be raised when data ready interrupt comes */

        k_poll(daEvents1, K_FOREVER);

 

        /* Reset signal */

        daEvents[0].signal->signaled = 0;

        daEvents[0].state = K_POLL_STATE_NOT_READY;

 

        /* …get data from hardware chip here… */

 

        ecgDataStruct.data1 = ;

        ecgDataStruct.data2 = ;

 

        dataProcessingPutFifoECG(&ecgDataStruct);

    }

 

The last function is part of the reading thread software module and just puts the item to the queue with k_fifo_put().

 

 

 

The reading thread is like:

 

(This is a static variable in the software module)

static struct k_fifo dpFifoECG;      ///< FIFO structure for handling incoming ECG data

 

There is an init function, where this is called:

k_fifo_init(&dpFifoECG);

 

 

s_data_item_fifo_ecg_t *fifoDataECG;

 

    while (1

    {

        /* Get data from queue. */

        fifoDataECG = k_fifo_get(&dpFifoECG, K_FOREVER);

 

        printk("%d,%d\n"fifoDataECG->data1fifoDataECG->data2);

 

    }

 

I checked the print outputs with a terminal program and realized that the data is still printed, even if I stop the measurement and stop putting data to the queue, which means that the program is not waiting at k_fifo_get() as expected. It just continues reading the last item which should be removed from the queue.

 

When I have a look at the fifoReserved pointer for these unwanted items, I see that the pointer is always the same and not NULL. Shouldn’t the pointer be NULL if there are no more items in the list?

 

This is really driving me crazy.

 

Both threads do have a maximum memory thread stack size of 8kB.

 

The priorities are -2 for the writing thread (implements SPI communication) and 1 for the reading thread. (There are three threads running on the system, excluding main thread and idle thread).

 

A general question:

 

For me, it is not really clear, if a data item is passed by copy or by reference to the fifo queue. I assume, the items are passed by reference, since I faced some serious problems when passing data items from an ISR, which are lying on the ISR stack. The content of the data item was sometimes different after reading the item compared to when the item was putted to the queue. The solution to this problem was to define a static data item on the memory heap of the software module and to only adjust the data of the same item and put it again to the queue. This seems very unintuitive to me, since I put the same item to the queue over and over again, only with different content. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

My setup:

  • Zephyr 2.3 Build 99
  • Nordic nRF52840 DK
  • Segger J-Link OB-SAM3U128-V2-NordicSemi
  • Eclipse 2019-09
  • Python 3.8.3
  • West 0.7.2
  • C compiler GNU 9.2.1

 

Thanks for your help!

 


Carles Cufi
 

The reason that Zephyr does not copy the items to the list is that it offers primitives that copy them (message queues) and primitives that do not (FIFOs). Then you can use FIFOs if you have your own allocation scheme that is optimized for your usecase, and message queues if you want the kernel to handle that for you.

 

Carles

 

From: users@... <users@...> On Behalf Of Weber, Dominik via lists.zephyrproject.org
Sent: 17 September 2020 16:57
To: Cufi, Carles <Carles.Cufi@...>; users@...
Subject: Re: [Zephyr-users] FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hi Carles,

 

thanks for your reply. This solved my problem.

 

But why does Zephyr not copy the items to the list? For me, there’s absolutely no reason to NOT copy data items to a queue . It runs the risk to…  

  • … put data items to the queue which are on the stack of a function or an ISR. Thus, these elements would be not valid anymore when the function returns. This is a major security issue, since you won’t be able to recognize the error until it will be overwritten randomly in memory…
  • … put the same data item several times to the queue which has the same memory address but different content (which was actually my problem)

Furthermore I would need a separate buffer to store my queue items, if I only put pointers to the queue. So one of the main features of a queue would be obsolete.

 

(The way FreeRTOS does it, is way more intuitive for me: https://www.freertos.org/Embedded-RTOS-Queues.html)

 

Best Regards,

Dominik

 

Von: Cufi, Carles <Carles.Cufi@...>
Gesendet: Freitag, 28. August 2020 16:29
An: Weber, Dominik <dominik.weber@iis.fraunhofer.de
>; users@...
Betreff: RE: FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hi Dominik,

 

>. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

FIFOs do not copy the contents of the message, they only store the pointer in a linked list. Maybe this is not clear enough in the documentation, please consider sending a Pull Request enhancing the doc.

 

If you need the kernel to copy the contents of your message for you, you can use Message Queues:

https://docs.zephyrproject.org/latest/reference/kernel/data_passing/message_queues.html

 

Thanks,

 

Carles

 

From: users@... <users@...> On Behalf Of Weber, Dominik via lists.zephyrproject.org
Sent: 28 August 2020 13:22
To: users@...
Subject: [Zephyr-users] FIFO is not empty after getting all data items; reference pointer still points to another item

 

Hey everyone,

 

I’m currently having problems when putting items to a FIFO. As long as I put data items to my FIFO queue, everything is ok and works as expected. If I stop putting data to the queue, the reading thread continues reading the last data item out of the queue over and over again, although the queue should be empty but is not (k_fifo_is_empty() says it’s not empty).

 

There’s a writing and a reading thread in my software project.

 

I have the following data item, defined in the header file of the reading thread:

 

typedef struct

{

    void *fifoReserved;     ///< 1st word reserved for use by fifo

    int data1;               

    int data2;              

 

s_data_item_fifo_ecg_t;

 

 

 

 

The writing thread is like:

 

(This is a static variable in the software module)

static s_data_item_fifo_ecg_t ecgDataStruct;  ///< This data item contains the ecg information to put to the data fifo.

 

(Inside the thread)

while (1)

    {

        /* Thread will be unready until signal was raised, signal will be raised when data ready interrupt comes */

        k_poll(daEvents1, K_FOREVER);

 

        /* Reset signal */

        daEvents[0].signal->signaled = 0;

        daEvents[0].state = K_POLL_STATE_NOT_READY;

 

        /* …get data from hardware chip here… */

 

        ecgDataStruct.data1 = ;

        ecgDataStruct.data2 = ;

 

        dataProcessingPutFifoECG(&ecgDataStruct);

    }

 

The last function is part of the reading thread software module and just puts the item to the queue with k_fifo_put().

 

 

 

The reading thread is like:

 

(This is a static variable in the software module)

static struct k_fifo dpFifoECG;      ///< FIFO structure for handling incoming ECG data

 

There is an init function, where this is called:

k_fifo_init(&dpFifoECG);

 

 

s_data_item_fifo_ecg_t *fifoDataECG;

 

    while (1

    {

        /* Get data from queue. */

        fifoDataECG = k_fifo_get(&dpFifoECG, K_FOREVER);

 

        printk("%d,%d\n"fifoDataECG->data1fifoDataECG->data2);

 

    }

 

I checked the print outputs with a terminal program and realized that the data is still printed, even if I stop the measurement and stop putting data to the queue, which means that the program is not waiting at k_fifo_get() as expected. It just continues reading the last item which should be removed from the queue.

 

When I have a look at the fifoReserved pointer for these unwanted items, I see that the pointer is always the same and not NULL. Shouldn’t the pointer be NULL if there are no more items in the list?

 

This is really driving me crazy.

 

Both threads do have a maximum memory thread stack size of 8kB.

 

The priorities are -2 for the writing thread (implements SPI communication) and 1 for the reading thread. (There are three threads running on the system, excluding main thread and idle thread).

 

A general question:

 

For me, it is not really clear, if a data item is passed by copy or by reference to the fifo queue. I assume, the items are passed by reference, since I faced some serious problems when passing data items from an ISR, which are lying on the ISR stack. The content of the data item was sometimes different after reading the item compared to when the item was putted to the queue. The solution to this problem was to define a static data item on the memory heap of the software module and to only adjust the data of the same item and put it again to the queue. This seems very unintuitive to me, since I put the same item to the queue over and over again, only with different content. Is there a reason why Zephyr does not copy the items to the queue like FreeRTOS does it? The Zephyr documentation lacks this kind of information!

 

My setup:

  • Zephyr 2.3 Build 99
  • Nordic nRF52840 DK
  • Segger J-Link OB-SAM3U128-V2-NordicSemi
  • Eclipse 2019-09
  • Python 3.8.3
  • West 0.7.2
  • C compiler GNU 9.2.1

 

Thanks for your help!