vikrant8051 <vikrant8051@...>
Hi to all,
Suddenly get this non-stoppable FAULT
***** BUS FAULT ***** Executing thread ID (thread): 0x20000220 Faulting instruction address: 0x1c9a4 Precise data bus error Address: 0xe5f7ffbe Fatal fault in thread 0x20000220! Aborting.
Input -->
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
Output -->
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Thank You !!
toggle quoted message
Show quoted text
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote: Hi Carles,
Now get these FAULT/OOPS,
***** MPU FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Instruction Access Violation Fatal fault in thread 0x20001eb0! Aborting. ***** USAGE FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Illegal load of EXC_RETURN into PC Fatal fault in thread 0x20001eb0! Aborting. ***** USAGE FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Illegal load of EXC_RETURN into PC Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! ***** Current thread ID = 0x20001eb0 Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
Input -->
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
Output -->
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
Thanks !!
|
|
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
toggle quoted message
Show quoted text
From: Vikrant More <vikrant8051@...>
Sent: 30 May 2018 13:05
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
What board is this on? Check for stack overflow, on devices with NXP MPUs these get reported as bus faults.
Andrew
toggle quoted message
Show quoted text
From: users@... [mailto:users@...]
On Behalf Of Cufi, Carles
Sent: Wednesday, May 30, 2018 7:00 AM
To: Vikrant More <vikrant8051@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Hedberg, Johan <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
From: Vikrant More <vikrant8051@...>
Sent: 30 May 2018 13:05
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
vikrant8051 <vikrant8051@...>
Hi,
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4 ==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550 ==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46 ==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x4ea8 ==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
Thank You !!
toggle quoted message
Show quoted text
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
toggle quoted message
Show quoted text
From: Vikrant More <vikrant8051@...>
Sent: 31 May 2018 16:13
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
vikrant8051 <vikrant8051@...>
Hi Carles,
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS. But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code. 🤔
Thank You !!
toggle quoted message
Show quoted text
On Thu, May 31, 2018 at 7:53 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
Hi Vikrant,
OK in that case I will try to run the onoff mesh app to see if I can reproduce at all.
Thanks for the help in debugging this.
Carles
toggle quoted message
Show quoted text
From: users@... <users@...>
On Behalf Of vikrant8051
Sent: 31 May 2018 17:06
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS.
But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code.
🤔
On Thu, May 31, 2018 at 7:53 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
vikrant8051 <vikrant8051@...>
Hi,
I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. // BT_ASSERT(buf);
And after that everything is working perfectly normal.
Thanks !!
toggle quoted message
Show quoted text
On Thu, May 31, 2018 at 8:36 PM, vikrant8051 <vikrant8051@...> wrote: Hi Carles,
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS. But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code. 🤔
Thank You !!
|
|
vikrant8051 <vikrant8051@...>
Hi,
It is basically modified version of samples/bluetooth/mesh as well as inspired from samples/boards/nrf52/mesh/onoff-app.
Thank You !!
toggle quoted message
Show quoted text
On Fri, Jun 1, 2018 at 2:52 PM, vikrant8051 <vikrant8051@...> wrote: Hi,
I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. // BT_ASSERT(buf);
And after that everything is working perfectly normal.
Thanks !!
|
|