Whenever the USB DFU driver and the USB BLE HCI driver are both enabled, neither works as expected . BLE HCI over USB works when the DFU USB driver is disabled via prj.conf (CONFIG_USB_DFU_CLASS=n). DFU works when the bluetooth kernel module is removed in Linux (via modprobe btusb).

Are there any known bugs/limitations with the USB stack which would prevent DFU and BLE from operating simultaneously? Alternatively, is it possible that I am missing some settings in dfu-util?

Environment:

    Ubuntu 16.04 (host)
    nRF52840 PCA10056 hw v. 1.1.0
    Zephyr 1.14
    Zephyr SDK 0.10.0
    dfu-util 0.9 (modified for increased timeout during flash erase)

prj.conf:

    CONFIG_STDOUT_CONSOLE=y
    CONFIG_GPIO=y
    CONFIG_SERIAL=y
    CONFIG_UART_INTERRUPT_DRIVEN=y

    CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
    CONFIG_CLOCK_CONTROL_NRF_K32SRC_20PPM=y

    CONFIG_BT=y
    CONFIG_BT_HCI_RAW=y
    CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
    CONFIG_BT_RX_BUF_LEN=258
    CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

    CONFIG_USB=y
    CONFIG_USB_DEVICE_STACK=y
   
    # Bluetooth works when the below setting is set to no
    CONFIG_USB_DFU_CLASS=y
    CONFIG_USB_DEVICE_BLUETOOTH=y

    CONFIG_FLASH=y
    CONFIG_SOC_FLASH_NRF=y
    CONFIG_BOOTLOADER_MCUBOOT=y
    CONFIG_IMG_MANAGER=y
    CONFIG_MCUBOOT_IMG_MANAGER=y

    CONFIG_USB_DEVICE_VID=0x167f
    CONFIG_USB_DEVICE_PID=0x3001

    CONFIG_USB_DEVICE_MANUFACTURER="Company"
    CONFIG_USB_DEVICE_PRODUCT="Product"
    CONFIG_USB_DEVICE_SN="123abc"

Flashing method:

    source zephyr/zephyr/zephyr-env.sh
    mkdir -p build/nrf52840_pca10056 && cd build/nrf52840_pca10056
    cmake -GNinja -DBOARD=nrf52840_pca10056 -Dapp_VERSION_BUILD=9999 ../..
    ninja
    ~/repos/mcuboot/scripts/imgtool.py sign --key ~/repos/mcuboot/root-ec-p256.pem --header-size 0x200 --align 8 --version 1.2 -S 0x69000 ./zephyr/zephyr.hex signed-hello.hex
    nrfjprog --family NRF52 --program signed-hello.hex --sectorerase
    ~/repos/mcuboot/scripts/imgtool.py sign --key ~/repos/mcuboot/root-ec-p256.pem --header-size 0x200 --align 8 --version 1.2 -S 0x69000 ./zephyr/zephyr.bin signed-hello.bin
    sudo dfu-util --alt 1 --download signed-hello.bin

Output of failed run of dfu-util:

    dfu-util 0.9

    Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
    Copyright 2010-2019 Tormod Volden and Stefan Schmidt
    This program is Free Software and has ABSOLUTELY NO WARRANTY
    Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

    dfu-util: Invalid DFU suffix signature
    dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
    Opening DFU capable USB device...
    ID 167f:3001
    Run-time device DFU version 0110
    Claiming USB DFU Runtime Interface...
    dfu-util: Cannot claim interface 1: LIBUSB_ERROR_BUSY

Output of successful run of dfu-util:

    dfu-util 0.9

    Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
    Copyright 2010-2019 Tormod Volden and Stefan Schmidt
    This program is Free Software and has ABSOLUTELY NO WARRANTY
    Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

    dfu-util: Invalid DFU suffix signature
    dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
    Opening DFU capable USB device...
    ID 167f:3001
    Run-time device DFU version 0110
    Claiming USB DFU Interface...
    Setting Alternate Setting #1 ...
    Determining device status: state = dfuIDLE, status = 0
    dfuIDLE, continuing
    DFU mode device DFU version 0110
    Device returned transfer size 64
    Copying data from PC to DFU device
    Download    [=========================] 100%        85970 bytes
    Download done.
    state(2) = dfuIDLE, status(0) = No error condition is present
    Done!

I created a Github issue for the above incompatibility:

https://github.com/zephyrproject-rtos/zephyr/issues/16240

It looks like DFU cannot be used at the same time as Bluetooth HCI in v1.14 or master but there is active development on composite USB.