I added the rfcomm directory in the file component/bt/component.mk and fixed the compile errors.
Have a look here: https://github.com/ursfassler/esp-idf, branch rfcomm

Thank you for the example. I found 2 problems in it, unfortunately it does not prevent the cyclic rebooting. It does not reboot all the time when the callbacks are not registered. The problems are: the hci_cmd_buf is too small. If I got it correctly, it should be HCIC_PREAMBLE_SIZE+HCI_COMMAND_SIZE ...

I am also looking for an rfcomm example.

Meanwhile I try it on my own. I am able to call RFCOMM_CreateConnection on the esp. Unfortunately I can not connect from another device to the ESP32, not sure where the problem is.