VS1103 triggering synchronized wave clips using RT-MIDI meta

Writing software that inputs and/or outputs audio and performs DSP algorithms such as filters, new codecs or audio effects.
Post Reply
mart
User
Posts: 3
Joined: Sun 2022-04-17 1:58

VS1103 triggering synchronized wave clips using RT-MIDI meta

Post by mart »

Hello!

I'm totally new to DSPs in general and VSDSP in particular, and am hoping to determine the feasibility of a design for a VS1103 application involving real-time MIDI synchronized with the playback of small wave clips streamed from an attached external memory.

My goal is that software running on the system's main microcontroller can first upload small PCM/ADPCM clips into an SPI RAM that is also connected to the VS1103 chip during startup, and then afterwards would ideally use the RT-MIDI interface (over UART) both to play normal MIDI notes using the built-in functionality and interleave commands to trigger synchronized playback of the PCM/ADPCM audio clips from the RAM to produce music with effects that cannot be generated using MIDI alone.

From studying the VS1103 datasheet, the VS-DSP4 user's manual, and some discussions elsewhere in this forum I have learned the following which suggest to me that what I'm looking to do is plausible:
  • The VS1103 natively supports playback of both MIDI and PCM/ADPCM data at the same time, although typically as two separate input channels controlled separately by the host controller, rather than using MIDI messages to trigger Wave playback.
  • The VS1103 module illustrates connecting an SPI Flash to the four GPIO pins and, with the spimodule firmware, playing back wave content streamed from that SPI Flash. I'm assuming that I could connect an SPI SRAM instead and achieve a similar result with some custom firmware.
  • In some commentary about the supported MIDI commands I see a passing mention of the MIDI decoder reacting to most MIDI meta messages by calling something called "MidiMeta()". I see this both in the VS1053 datasheet and in a forum post with similar content, though I've not found details on exactly what "MidiMeta" refers to here.
My current idea then is to write a custom plugin that reacts to this "MidiMeta" call for a custom meta message which specifies an address into the SPI RAM where playback should begin and the length of the clip to play in bytes. It would then trigger some behavior similar to the "spimodule" firmware, reading the designated slice of the SPI RAM and passing it through the built-in PCM/ADPCM playback functionality to be mixed in with the normal RT-MIDI playback running concurrently.

I'm primarily curious to hear about whether this application seems plausible to implement with a VS1103 at all. I understand that VS1103 has a relatively small instruction RAM but am hoping to have relatively little system-control-oriented code in the instruction RAM and use built-in features in the instruction ROM for the "real work". I'd be willing to consider a more general/complex VSDSP chip like VS1053 too, but I've focused on VS1103 so far because I have no need for MP3/Vorbis decoding.

If this does seem like a realistic goal, I'd also love to learn more about this "MidiMeta()" call. I've not found any further mention of it in any other documentation, so I feel unsure as to what kind of thing "MidiMeta" is: since specific function names would be erased during compilation I assume this is referring to something more significant than a firmware function. In the datasheets I do see the ideas of "system vector functions" which are for user application code to call, and "system vector tags" which are for user application code to react to various events. Is "MidiMeta" a "system vector tag" that just isn't included in the documentation, or something else? Is there some further documentation about it somewhere that I've not found yet?

Some "nice to have" (but not required) features would be:
  • The meta message can optionally specify a pitch shift to apply during wave playback, so that these clips can potentially be used as custom instruments to augment those provided in the MIDI synth. My main need is percussion-like sound effects and vocal recordings rather than tuned instruments, but being able to use clips also as basic "tuned" instruments would allow for some nice additional flexibility for composers (in similar vein to a "soundtracker" module, albeit with only one wave channel). Is this something achievable as a DSP filter placed after the PCM/ADPCM in the audio datapath?
  • I have no hard requirement for the application code running in the main MCU to be able to modify the content SRAM while playback is ongoing, so am intending to just connect the SPI SRAM to both the VS1103 GPIO pins and a main MCU SPI peripheral at the same time and, in the worst case, keep the VS1103 held in reset until the main MCU has finished writing to the SRAM. However, if possible I would like the main MCU code to be able to request to be given back control of the SRAM to write new audio content, which I imagine as being another MIDI meta message to immediately halt PCM/ADPCM playback and switch the VS1103 GPIOs all into digital input mode (HiZ) so that the main MCU can temporarily use the SRAM without totally resetting the VS1103, and then later command the custom VS1103 plugin to retake control of that bus and enable wave playback again.
I'd appreciate any other relevant information that a reader might offer related to this use-case too. I'm still early in my learning process so I'm sure there are questions I'm not even thinking to ask yet.

Thanks!
User avatar
pasi
VLSI Staff
Posts: 2019
Joined: Thu 2010-07-15 16:04

Re: VS1103 triggering synchronized wave clips using RT-MIDI meta

Post by pasi »

The vs10xx chips are not intended to be user-programmable. The amount of RAM is historically very small, and as you found out, user documentation is lacking. Internally we can always go back to the source code to see things, which is not possible for the general public.

The VS1103 Module skips the MIDI decoder + WAV mixer of the ROM firmware, and implements the Standalone player with either the MIDI (file) decoder or the WAV decoder. The real-time MIDI mode doesn't allow mixing of WAV decoder output. However, the SPI data read routines are not as big as the SD and FAT filesystem code, so it should not be completely impossible.

Some random comments:
- Mixing audio at the MIDI samplerate (44100Hz) is fairly easy through the ApplAddr hook. If the rates do not match, a lot of CPU gets spent.
- Pitch shifting well is impossible. Pitch shifting adequately is hard. Wavetable playback is not that hard. The MIDI decoder has a passable wavtable interpolation routine, and you can improve by using a different source table (pre-filtered) for larger changes in pitch.
- MidiMeta() is a callback function with a vector in RAM. All non-tempo MIDI Meta messages are sent to this function one byte at a time (0x8000|type as a start marker, 0xc000 as an end marker).

With all that said, rereading your requirements, vs1053b would probably be a better IC for your application.
1.PCM mixer can play WAV data at the same time (See vs1053b PCM Mixer https://www.vlsi.fi/en/support/software ... ugins.html ).
You can thus get straight to prototyping.
2.More available instruction (and data) RAM, more GPIO, better audio path (makes PCM Mixer possible).
3.More examples and some documentation if you want to add wavtable playing at 44.1kHz instead of the PCM mixer.
4.VS1053b Real-Time MIDI Input Application ( https://www.vlsi.fi/en/support/software ... tions.html ) fixes a few things in the real-time MIDI code. If the PCM Mixer doesn't natively work with the RT-MIDI mode, the mixer plugin and rtmidi app could be merged.

Please don't hesitate to ask more.
Visit https://www.facebook.com/VLSISolution VLSI Solution on Facebook
mart
User
Posts: 3
Joined: Sun 2022-04-17 1:58

Re: VS1103 triggering synchronized wave clips using RT-MIDI meta

Post by mart »

Thank you very much for this advice and background information.

I will take some time to study again with the VS1053b in mind so I can hopefully come back with some better/deeper questions once I've understood the basics.

Thankfully I seem to have Adafruit's VS1053b Breakout Board in stock from a previous project, so hopefully I can quickly begin prototyping by using it instead of the VS1103 module.

Thanks again!
mart
User
Posts: 3
Joined: Sun 2022-04-17 1:58

Re: VS1103 triggering synchronized wave clips using RT-MIDI meta

Post by mart »

Hello again!

Unfortunately I got distracted by some other projects and so I wasn't able to look into this immediately, but I began further investigation today. So far I'm just trying to understand how things fit together with the toolchain and IDE.

What I've noticed so far is that the VSIDE package includes a pre-built "stub" object file for each target processor (I'm focused on lib1053b here, so rom1053b.o) which contains the addresses of the various built-in ROM functions, so the linker can resolve calls to those. This object file also includes the expected addresses for the IRAM hooks like the MidiMeta hook we were discussing earlier.

Where I started to get a bit lost is in navigating the various header files in the lib1053b directory. In the provided documentation I see lots of functions and other symbols that apparently belong to header files midi.h and midilib.h, but there don't appear to be any such files in the VSIDE distribution. I can of course write out my own declarations based on the information in the documentation, and I'm prepared to do that if that's the expected way to use that functionality, but I'm wondering if there are additional header files available somewhere else that I've missed.

The rest of this is more speculative since I've so far only read some documentation and not tried to put this into practice...

I see the RealTimeMidi function in common.h which I understand as a way to just give full control to the real-time MIDI application in ROM. With the original strategy I was considering in earlier comments I suppose I could place some code at the MidiMeta address, then jump into the real-time MIDI application and let it call back into my meta function when a MIDI meta command appears.

However, now that I can see (slightly) a better view of the shape of things I'm wondering if I should instead try to integrate with the MIDI decoder at a lower level, where my own application can still "control the loop". Specifically, at a high level I'm imagining a design something like the following:
  • Handle the UART setup and data transfer myself, but mostly just pass the received data over to the MIDI codec so it can continue to do most of its work. I see various CodMidi... functions and struct CodecMidi in the docs, and wonder if those can help me.
  • ...but also accept commands over a separate channel (probably SCI?) to provide the address and length of a sample in an SPI RAM attached to the GPIO pins, which would then "cue up" the sample for later playback when triggered.
  • Using either the MidiMeta hook or some other means, trigger playback of the previously-designated sample and mix it in with any MIDI synth notes currently playing. (turns out that direct synchronization with MIDI playback isn't as important as I originally thought, so doing this triggering over SCI/SDI instead could be acceptable if it works out better to keep these two paths separated.)
In the details I'm expecting to write a "producer" which streams ADPCM data from the SPI SRAM into a FIFO ring buffer and then a consumer that (when triggered to start playback) reads data from that FIFO, performs ADPCM decoding, and mixes the result into the audio path. I'm hoping that I can use the MIDI codec and the ADPCM codec from ROM "simultaneously" but I've seen mention elsewhere of the different codecs using conflicting parts of data RAM, so I'm not sure if that's actually possible for this particular pair of decoders.

Does it seem like I'm on a plausible path here, or am I making some incorrect assumptions about what is feasible/practical? Is there some more documentation available about the details of the MIDI synth code in ROM?

Thanks in advance for any advice you are able to share!
User avatar
pasi
VLSI Staff
Posts: 2019
Joined: Thu 2010-07-15 16:04

Re: VS1103 triggering synchronized wave clips using RT-MIDI meta

Post by pasi »

The RealTime MIDI mode is actually a rather small wrapper that parses the real-time MIDI stream and calls the regular MIDI decoding functions.

So, replicating that would allow you to add all kinds of things.

Unfortunately the source code for the VS1003b/VS1033c/VS1053b Real-Time MIDI Input Application ( https://www.vlsi.fi/en/support/software ... tions.html ) isn't a VSIDE solution at the moment. I could still provide that "one of these days" (quite busy with production now) as-is.

The answer to the MidiMeta hook: The MidiMeta callback gets called for every byte of the meta message plus once with a starting marker and once with an end marker as parameter.
Visit https://www.facebook.com/VLSISolution VLSI Solution on Facebook
Post Reply