|
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From: users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...; users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi Carles,
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From: users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...; users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
From: Vikrant More <vikrant8051@...>
Sent: 30 May 2018 12:11
To: Cufi, Carles <carles.cufi@...>
Cc: devel@...; users@...
Subject: Re: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
I will but don't know, how to do that ?
toggle quoted messageShow quoted text
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles < Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi Carles,
Input ->
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x5222
Output -> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
Thank You !!
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
From: users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 12:31
To: Cufi, Carles <carles.cufi@...>
Cc: devel@...; users@...
Subject: Re: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
toggle quoted messageShow quoted text
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles < Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi Carles,
Now get these FAULT/OOPS,
***** MPU FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Instruction Access Violation Fatal fault in thread 0x20001eb0! Aborting. ***** USAGE FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Illegal load of EXC_RETURN into PC Fatal fault in thread 0x20001eb0! Aborting. ***** USAGE FAULT ***** Executing thread ID (thread): 0x20001eb0 Faulting instruction address: 0x200006b8 Illegal load of EXC_RETURN into PC Fatal fault in thread 0x20001eb0! Aborting. ***** Kernel OOPS! ***** Current thread ID = 0x20001eb0 Faulting instruction address = 0x14f10Fatal fault in ISR! Spinning...
Input -->
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
Output -->
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
Thanks !!
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi to all,
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
Input -->
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
Output -->
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Thank You !!
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote: Hi Carles,
Now get these FAULT/OOPS,
***** MPU FAULT ***** Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
Input -->
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
Output -->
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
Thanks !!
|
|
|
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
From: Vikrant More <vikrant8051@...>
Sent: 30 May 2018 13:05
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
toggle quoted messageShow quoted text
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 < vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
What board is this on? Check for stack overflow, on devices with NXP MPUs these get reported as bus faults.
Andrew
From: users@... [mailto:users@...]
On Behalf Of Cufi, Carles
Sent: Wednesday, May 30, 2018 7:00 AM
To: Vikrant More <vikrant8051@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Hedberg, Johan <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
From: Vikrant More <vikrant8051@...>
Sent: 30 May 2018 13:05
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
toggle quoted messageShow quoted text
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi,
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4 ==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550 ==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh#
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46 ==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line
-e
/home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x4ea8 ==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
Thank You !!
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
From: Vikrant More <vikrant8051@...>
Sent: 31 May 2018 16:13
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
toggle quoted messageShow quoted text
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles < Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi Carles,
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS. But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code. 🤔
Thank You !!
On Thu, May 31, 2018 at 7:53 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf 0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
Hi Vikrant,
OK in that case I will try to run the onoff mesh app to see if I can reproduce at all.
Thanks for the help in debugging this.
Carles
From: users@... <users@...>
On Behalf Of vikrant8051
Sent: 31 May 2018 17:06
To: Cufi, Carles <carles.cufi@...>
Cc: Von Dentz, Luiz <luiz.von.dentz@...>; Johan Hedberg <johan.hedberg@...>; devel@...; users@...
Subject: Re: [Zephyr-devel] [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS.
But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code.
🤔
toggle quoted messageShow quoted text
On Thu, May 31, 2018 at 7:53 PM, Cufi, Carles < Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please try to reproduce this with standard samples present in the Zephyr tree?
Thanks,
Carles
These are today's OOPS & Faults
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a7c4
==> /home/vikrant/projects/zephyr/zephyr/include/net/buf.h:370
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1a550
==> /home/vikrant/projects/zephyr/zephyr/subsys/net/buf.c:60
root@vikrant-iMac:/home/vikrant/projects/bluez/mesh# /opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x6e46
==> /home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/conn.c:1354
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x4ea8
==> /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/include/generated/syscalls/kernel.h:99
On Wed, May 30, 2018 at 7:30 PM, Cufi, Carles <Carles.Cufi@...> wrote:
This is starting to look more and more like a memory corruption or kernel issue at this point.
I cannot reproduce this easily by running our standard Bluetooth samples, so I am a bit at a loss here. I will try to run a mesh sample to see if it fails in
the same way.
Suddenly get this non-stoppable FAULT
***** BUS FAULT *****
Executing thread ID (thread): 0x20000220
Faulting instruction address: 0x1c9a4
Precise data bus error
Address: 0xe5f7ffbe
Fatal fault in thread 0x20000220! Aborting.
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x1c9a4
/home/vikrant/projects/zephyr/zephyr/kernel/include/ksched.h:153
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
On Wed, May 30, 2018 at 4:20 PM, vikrant8051 <vikrant8051@...> wrote:
Now get these FAULT/OOPS,
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Instruction Access Violation
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** USAGE FAULT *****
Executing thread ID (thread): 0x20001eb0
Faulting instruction address: 0x200006b8
Illegal load of EXC_RETURN into PC
Fatal fault in thread 0x20001eb0! Aborting.
***** Kernel OOPS! *****
Current thread ID = 0x20001eb0
Faulting instruction address = 0x14f10
Fatal fault in ISR! Spinning...
#/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x14f10
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/controller/ll_sw/ctrl.c:4702
On Wed, May 30, 2018 at 4:11 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Thanks for looking into this. It seems like we do not get a buffer back from the command tx queue:
https://github.com/zephyrproject-rtos/zephyr/blob/master/subsys/bluetooth/host/hci_core.c#L3318
I have copied Luiz and Johan from Intel in case they know what the cause might be.
Regards,
Carles
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e /home/vikrant/projects/zephyr/zephyr/samples/bluetooth/mesh2/build/zephyr/zephyr.elf
0x5222
/home/vikrant/projects/zephyr/zephyr/subsys/bluetooth/host/hci_core.c:3318
On Wed, May 30, 2018 at 3:51 PM, Cufi, Carles <Carles.Cufi@...> wrote:
If you are using the Zephyr SDK:
path-to-zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-addr2line -e zephyr.elf <addr>
Carles
I will but don't know, how to do that ?
On Wed, May 30, 2018 at 3:19 PM, Cufi, Carles <Carles.Cufi@...> wrote:
Hi Vikrant,
Can you please run addr2line with this address present in the error log:
0x5222?
Thanks,
Carles
From:
users@... <users@...>
On Behalf Of vikrant8051
Sent: 30 May 2018 11:29
To: devel@...;
users@...
Subject: [Zephyr-users] #BluetoothMesh ...about latest kernel OOPS & exception
Today after upgrading to v1.12.0-rc2, I'm getting following error while playing with sample code which is inspired from sample/bluetooth/mesh
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [WRN] process_events: Unexpected k_poll event state 7
[bt] [WRN] process_events: Unexpected k_poll event state 8
[bt] [ERR] hci_tx_thread: assert: 'err == 0' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x5222
Fatal fault in thread 0x20000484! Aborting.
[bt] [ERR] event_scan_prepare: assert: '!_radio.ticker_id_prepare' failed
***** Kernel OOPS! *****
Current thread ID = 0x20000484
Faulting instruction address = 0x1b77e
Fatal fault in ISR! Spinning...
-------------------------------------------------------------------------------------------------------------------------
Sometime it completely hangs & nothing get displayed on debug terminal
-------------------------------------------------------------------------------------------------------------------------
|
|
|
vikrant8051 <vikrant8051@...>
Hi,
I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. // BT_ASSERT(buf);
And after that everything is working perfectly normal.
Thanks !!
On Thu, May 31, 2018 at 8:36 PM, vikrant8051 <vikrant8051@...> wrote: Hi Carles,
I tried it with samples/boards/nrf52/mesh/onoff-app ....Here too, I faced kernel OOPS. But in that case, occurrence frequency was very low.
I'm facing all these issues after 1.12.0-rc2 update. Previously, everything was working perfectly.
Let me again thoroughly go through with my code. 🤔
Thank You !!
|
|
|
vikrant8051 <vikrant8051@...>
Hi,
It is basically modified version of samples/bluetooth/mesh as well as inspired from samples/boards/nrf52/mesh/onoff-app.
Thank You !!
On Fri, Jun 1, 2018 at 2:52 PM, vikrant8051 <vikrant8051@...> wrote: Hi,
I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. // BT_ASSERT(buf);
And after that everything is working perfectly normal.
Thanks !!
|
|
|
Vikrant8051 <vikrant8051@...> wrote: I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. //
BT_ASSERT(buf);
And after that everything is working perfectly normal. I was pointed to this on IRC. The symptom you've got there sounds a lot like a scheduler mistake that got fixed yesterday, where ostensibly-pended threads could be incorrectly swapped back in. They would then wake up (generally with an -EAGAIN return value from _Swap()) and take whatever default action is approparite (like returning a NULL value from the empty list), so it's a little subtle to recognize as a bug and often "recoverable" by existing handling. But it's a real bug. Can you verify that you tree contains commit 43553da9b2bb9e5 and see if that was the same root cause? There's at least one other report of instability after that merged, but this definitely sounds like it might be your local problem. I'm trying to track this down locally, but alas our test suite passes, so I'm hoping to get more information. Andy
|
|
|
vikrant8051 <vikrant8051@...>
Hi Andy,
commit 43553da9b2bb9e537185695899fb5184c3b17ebe Author: Andy Ross < andrew.j.ross@...> Date: Thu May 31 11:13:49 2018 -0700 kernel/sched: Fix preemption logic The should_preempt() code was catching some of the "unrunnable" cases but not all of them, opening the possibility of failing to preempt a just-pended thread and thus waking it up synchronously. There are reports of this causing spin loops over k_poll() in the network stack work queues (see #8049). Note that the previous _is_dummy() call is folded into (the somewhat verbosely named) _is_thread_prevented_from_running(), and that the order of tests has been changed/optimized to hopefully catch common cases earlier. Suggested-by: Michael Scott < michael@...> Signed-off-by: Andy Ross < andrew.j.ross@...>
Is this what you are pointing?
It is there in my local repository.
May I now comment out line no. line no. 3318 of subsys/bluetooth/host/hci_core.c?
--------------------------------------------------------------------------------------------------------------------------------------
Could anybody help me to find out serious bugs in my PR:8101?
It is very simple & will help for testing. I am not interested in merging it, but need a complete bug free/easy to test App which will help in such scenarios.
Thank You !!
On Sat, Jun 2, 2018 at 4:10 AM, Andy Ross <andrew.j.ross@...> wrote: Vikrant8051 <vikrant8051@...> wrote:
> I commented line no. 3318 of subsys/bluetooth/host/hci_core.c i.e. //
> BT_ASSERT(buf);
>
> And after that everything is working perfectly normal.
I was pointed to this on IRC. The symptom you've got there sounds a
lot like a scheduler mistake that got fixed yesterday, where
ostensibly-pended threads could be incorrectly swapped back in. They
would then wake up (generally with an -EAGAIN return value from
_Swap()) and take whatever default action is approparite (like
returning a NULL value from the empty list), so it's a little subtle
to recognize as a bug and often "recoverable" by existing handling.
But it's a real bug. Can you verify that you tree contains commit
43553da9b2bb9e5 and see if that was the same root cause?
There's at least one other report of instability after that merged,
but this definitely sounds like it might be your local problem. I'm
trying to track this down locally, but alas our test suite passes, so
I'm hoping to get more information.
Andy
|
|
|
