At least some versions of the SiLabs app had a bug with OOB handling that was causing similar results as above. A simple solution was to disable OOB provisioning support from the Zephyr side, i.e. set all output* members to zero in struct bt_mesh_prov in main.c. That said, I'm not sure if the issue you're seeing is the same.
[bt] [ERR] hci_acl_handle: Invalid Tx Enqueue [bt] [ERR] send_frag: Unable to send to driver (err -22)
This is something coming from the controller code that I haven't seen before. Maybe the controller maintainers from Nordic have some idea.