re-assembling large notification data #ble #bluetooth


alok.sethi1+zephyr@...
 

Hello,

I am fairly new to bluetooth n BLE and would need advice about how to transfer large amount of data (data larger than the MTU) from a sensor to a central device.
So the sensor is using notifications to send the data in multiple chunks and on the central device, my notification callback is called multiple times with a data pointer and length.

I believe that the notifications are acknowledged at the link layer so the application does not get any marker that there is more data pending. is that true?
Further, are the notifications guaranteed to be delivered in sequence?

Currently the sensor is putting a header on only the first chunk and not on the rest of chunks. I can run a re-assembly timer and just assemble the data from different notification calls, provided that the fragments are guaranteed to be delivered in sequence.
Below is just a snapshot of logs taken from the notification callback handler, showing the length of the fragment and the MTU of the connection.

[00:57:40.765,045] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.765,716] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,357] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,815] <dbg> ble_hdlr: notify_func: notify len:2, mtu:65


BR


Jeremy Herbert
 

Hi,

I have pondered this same question myself, and there appears to be a lot of conflicting information on the internet. Here is what I have determined.

From the BLE5.3 spec (vol 3, part G):

Notifications: This sub-procedure is used when a server is configured to notify a Characteristic Value to a client without expecting any Attribute Protocol layer acknowledgment that the notification was successfully received.
Indications: This sub-procedure is used when a server is configured to indicate a Characteristic Value to a client and expects an Attribute Protocol layer acknowledgment that the indication was successfully received.

As far as I know, notifications are not guaranteed to be delivered *to the application layer*, they are kind of like UDP in that they are a "send and hope it gets delivered" kind of thing. To use the UDP example, even if you could guarantee that an ethernet link between two devices was good and it would always deliver packets uncorrupted from one end to the other, the UDP packet could also be dropped by the networking stack on the server or the client (perhaps due to out of memory, etc). So even if the notification doesn't get dropped while in the air by the link layer, it could be dropped before it was transmitted or after it was received.

The equivalent to a TCP level of reliability (though still packetised in BLE, not a stream) is through indications, not notifications. However, the downside of indications is that they require more round-trips back and forth between the two peers, sort of like the SYN-ACK handshake in TCP. If you are transferring lots of data, this can result in a significantly lower data rate overall; almost all vendors run their throughput test examples with notifications blasting out, not indications. The indication must either be delivered and acknowledged by the receiver (assuming one has subscribed), or an error is returned to the application layer according to the core spec.

Note that what is required is not necessarily the same as what is actually implemented. For example, looking through subsys/bluetooth/host/att.c and subsys/bluetooth/host/gatt.c in Zephyr, it seems that notifications are much more reliable than they are required to be by the standard in terms of retries and such. There is nothing preventing a BLE stack from implementing notifications as indications, but just not reporting the ACK to the application layer. It wouldn't make sense to do this really, but it is allowed by the spec. So particularly for devices which have proprietary, closed source BLE stacks (which is basically all of them), you can expect that there will be different behaviours here.

An approach I have used before was to separate a large payload into chunks which are larger than the MTU, and then send those larger chunks as notifications of the MTU. Each chunk is wrapped with a header that contains the length of the chunk and a checksum, and some special control characters to indicate a new transmission, etc. This way, the receiver can determine if each chunk is valid and depending on the circumstances can either request a retransmission via the application layer, or error out. You then tune the chunk size depending on how reliable your connection is expected to be in the deployed environment.

The packet encoding I have used is fairly simple and available as an open source library here: https://github.com/jeremyherbert/simplehdlc if you are curious.

I am not sure about the reordering of packets (I am not sure this is even defined for notifications), but I would imagine if a notification fails (ie no buffers or something) it just gets dropped so there is no reordering anyway, and if an indication fails, it will either report an error up through the stack, or the connection itself may need to be renegotiated depending on the circumstances. But there are some hints which make me think reordering might be possible, for example this comment: https://github.com/zephyrproject-rtos/zephyr/blob/580f370867ff3e5309ad1c92fd064c01bed93510/subsys/bluetooth/host/att.c#L193

/* In case of success the ownership of the buffer is transferred to the stack
 * which takes care of releasing it when it completes transmitting to the
 * controller.
 *
 * In case bt_l2cap_send_cb fails the buffer state and ownership are retained
 * so the buffer can be safely pushed back to the queue to be processed later.

Thanks,
Jeremy


On Thu, 28 Jul 2022 at 10:40, <alok.sethi1+zephyr@...> wrote:
Hello,

I am fairly new to bluetooth n BLE and would need advice about how to transfer large amount of data (data larger than the MTU) from a sensor to a central device.
So the sensor is using notifications to send the data in multiple chunks and on the central device, my notification callback is called multiple times with a data pointer and length.

I believe that the notifications are acknowledged at the link layer so the application does not get any marker that there is more data pending. is that true?
Further, are the notifications guaranteed to be delivered in sequence?

Currently the sensor is putting a header on only the first chunk and not on the rest of chunks. I can run a re-assembly timer and just assemble the data from different notification calls, provided that the fragments are guaranteed to be delivered in sequence.
Below is just a snapshot of logs taken from the notification callback handler, showing the length of the fragment and the MTU of the connection.

[00:57:40.765,045] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.765,716] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,357] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,815] <dbg> ble_hdlr: notify_func: notify len:2, mtu:65


BR


Alok Sethi <alok.sethi1@...>
 

Hello Jeremy,

Thanks for the detailed information. I will have a look at the packet encoding library.
Regarding reordering of notifications, for the time being i think just ignore the scenario n maybe drop the data in case that happens.

BR


On Thu, Jul 28, 2022 at 5:48 AM Jeremy Herbert <jeremy.006@...> wrote:
Hi,

I have pondered this same question myself, and there appears to be a lot of conflicting information on the internet. Here is what I have determined.

From the BLE5.3 spec (vol 3, part G):

Notifications: This sub-procedure is used when a server is configured to notify a Characteristic Value to a client without expecting any Attribute Protocol layer acknowledgment that the notification was successfully received.
Indications: This sub-procedure is used when a server is configured to indicate a Characteristic Value to a client and expects an Attribute Protocol layer acknowledgment that the indication was successfully received.

As far as I know, notifications are not guaranteed to be delivered *to the application layer*, they are kind of like UDP in that they are a "send and hope it gets delivered" kind of thing. To use the UDP example, even if you could guarantee that an ethernet link between two devices was good and it would always deliver packets uncorrupted from one end to the other, the UDP packet could also be dropped by the networking stack on the server or the client (perhaps due to out of memory, etc). So even if the notification doesn't get dropped while in the air by the link layer, it could be dropped before it was transmitted or after it was received.

The equivalent to a TCP level of reliability (though still packetised in BLE, not a stream) is through indications, not notifications. However, the downside of indications is that they require more round-trips back and forth between the two peers, sort of like the SYN-ACK handshake in TCP. If you are transferring lots of data, this can result in a significantly lower data rate overall; almost all vendors run their throughput test examples with notifications blasting out, not indications. The indication must either be delivered and acknowledged by the receiver (assuming one has subscribed), or an error is returned to the application layer according to the core spec.

Note that what is required is not necessarily the same as what is actually implemented. For example, looking through subsys/bluetooth/host/att.c and subsys/bluetooth/host/gatt.c in Zephyr, it seems that notifications are much more reliable than they are required to be by the standard in terms of retries and such. There is nothing preventing a BLE stack from implementing notifications as indications, but just not reporting the ACK to the application layer. It wouldn't make sense to do this really, but it is allowed by the spec. So particularly for devices which have proprietary, closed source BLE stacks (which is basically all of them), you can expect that there will be different behaviours here.

An approach I have used before was to separate a large payload into chunks which are larger than the MTU, and then send those larger chunks as notifications of the MTU. Each chunk is wrapped with a header that contains the length of the chunk and a checksum, and some special control characters to indicate a new transmission, etc. This way, the receiver can determine if each chunk is valid and depending on the circumstances can either request a retransmission via the application layer, or error out. You then tune the chunk size depending on how reliable your connection is expected to be in the deployed environment.

The packet encoding I have used is fairly simple and available as an open source library here: https://github.com/jeremyherbert/simplehdlc if you are curious.

I am not sure about the reordering of packets (I am not sure this is even defined for notifications), but I would imagine if a notification fails (ie no buffers or something) it just gets dropped so there is no reordering anyway, and if an indication fails, it will either report an error up through the stack, or the connection itself may need to be renegotiated depending on the circumstances. But there are some hints which make me think reordering might be possible, for example this comment: https://github.com/zephyrproject-rtos/zephyr/blob/580f370867ff3e5309ad1c92fd064c01bed93510/subsys/bluetooth/host/att.c#L193

/* In case of success the ownership of the buffer is transferred to the stack
 * which takes care of releasing it when it completes transmitting to the
 * controller.
 *
 * In case bt_l2cap_send_cb fails the buffer state and ownership are retained
 * so the buffer can be safely pushed back to the queue to be processed later.

Thanks,
Jeremy

On Thu, 28 Jul 2022 at 10:40, <alok.sethi1+zephyr@...> wrote:
Hello,

I am fairly new to bluetooth n BLE and would need advice about how to transfer large amount of data (data larger than the MTU) from a sensor to a central device.
So the sensor is using notifications to send the data in multiple chunks and on the central device, my notification callback is called multiple times with a data pointer and length.

I believe that the notifications are acknowledged at the link layer so the application does not get any marker that there is more data pending. is that true?
Further, are the notifications guaranteed to be delivered in sequence?

Currently the sensor is putting a header on only the first chunk and not on the rest of chunks. I can run a re-assembly timer and just assemble the data from different notification calls, provided that the fragments are guaranteed to be delivered in sequence.
Below is just a snapshot of logs taken from the notification callback handler, showing the length of the fragment and the MTU of the connection.

[00:57:40.765,045] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.765,716] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,357] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,815] <dbg> ble_hdlr: notify_func: notify len:2, mtu:65


BR



--
BR
alok


Carles Cufi
 

Hi there,

 

BLE is fully reliable for GATT, and notifications are guaranteed to be delivered in sequence.

But what you really want is to configure your GATT MTU to the size of the data you want to send. Then the stack will do all the heavy lifting for you and it should be transparent. Note that the GATT MTU is independent of the LL MTU and thus of DLE.

 

Carles

 

From: <users@...> on behalf of "alok.sethi1+zephyr via lists.zephyrproject.org" <alok.sethi1+zephyr=gmail.com@...>
Reply-To: "alok.sethi1+zephyr@..." <alok.sethi1+zephyr@...>
Date: Thursday, 28 July 2022 at 02:40
To: "users@..." <users@...>
Subject: [Zephyr-users] re-assembling large notification data #ble #bluetooth

 

Hello,

I am fairly new to bluetooth n BLE and would need advice about how to transfer large amount of data (data larger than the MTU) from a sensor to a central device.
So the sensor is using notifications to send the data in multiple chunks and on the central device, my notification callback is called multiple times with a data pointer and length.

I believe that the notifications are acknowledged at the link layer so the application does not get any marker that there is more data pending. is that true?
Further, are the notifications guaranteed to be delivered in sequence?

Currently the sensor is putting a header on only the first chunk and not on the rest of chunks. I can run a re-assembly timer and just assemble the data from different notification calls, provided that the fragments are guaranteed to be delivered in sequence.
Below is just a snapshot of logs taken from the notification callback handler, showing the length of the fragment and the MTU of the connection.

[00:57:40.765,045] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.765,716] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,357] <dbg> ble_hdlr: notify_func: notify len:62, mtu:65
[00:57:40.766,815] <dbg> ble_hdlr: notify_func: notify len:2, mtu:65


BR