I Set CMSIS_DSP_TABLES_ALL_FAST and CMSIS_DSP_TABLES_ALL_FFT to n and the file size went down and I went on to add tables that I needed:

CONFIG_CMSIS_DSP_TABLES_RFFT_FAST_F32_32=y
CONFIG_CMSIS_DSP_TABLES_ARM_SIN_F32=y
CONFIG_CMSIS_DSP_TABLES_ARM_COS_F32=y
CONFIG_CMSIS_DSP_TABLES_ARM_CMPLX_MAG_Q31=y
CONFIG_CMSIS_DSP_TABLES_RFFT_F32_512=y

But when running the code I get a hard fault after arm_rfft_fast_init_f32() with the fftLen=512

#if !defined(ARM_DSP_CONFIG_TABLES) || defined(ARM_ALL_FFT_TABLES) || (defined(ARM_TABLE_TWIDDLECOEF_F32_256) && defined(ARM_TABLE_BITREVIDX_FLT_256) && defined(ARM_TABLE_TWIDDLECOEF_F32_256) && defined(ARM_TABLE_TWIDDLECOEF_RFFT_F32_512))
  case 512U:
    fptr = arm_rfft_512_fast_init_f32;
    break;
#endif

I cannot figure out which table CONFIGs are
  ARM_TABLE_TWIDDLECOEF_F32_256
  ARM_TABLE_BITREVIDX_FLT_256
  ARM_TABLE_TWIDDLECOEF_F32_256
  ARM_TABLE_TWIDDLECOEF_RFFT_F32_512

Thanks David