Just a follow-up to this thread, I am successfully loading the ulp binary blob into the final elf and running a test sketch using the ulp to increment a global variable. Looks like I can use .s extension for ulp assembly and .S for xtensa assembly and have a python script do the heavy lifting of creating an ulp binary blob to embed into larger elf. I'm trying my hardest to keep the functionality of how you write code for the ulp the same as you would do with esp-idf environment, but there are few snags I need to figure out.
First being on how Arduino compiles code makes it difficult for the sketch to see any global variables defined in the assembly because the Arduino ide doesn't have a platform.txt hook before Arduino detects libraries. The way esp-idf does it is it creates a header file that gives your c, c++ code an externed variables that links to the assembly global variables. What happens is Arduino detects all the libraries and sketch ino files before I can compile the ulp code and create that header file. What works is to have the user create that header file manually in the ide and define the externed global variables that they want to access in their sketch. This is the only way I can think of so far, not to bad though I think?
I am in the process of moving but once I get chance I'll post a how-to for MacOS and possibly linux and upload the files to my github account.