Optimize memory with SysEx Arrays

[RiNoize]

Hi, in my project I have thousands of SysEx arrays like these:

const byte arpUPup[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf7};
const byte arpDOWN[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xf7};
const byte arpALT1[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xf7};
const byte arpALT2[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0xf7};
const byte arpRNDOM[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0xf7};
const byte arp0000[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0xf7};
const byte arp0001[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0xf7};
const byte arp0002[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x07, 0xf7};

And I read them this way:

case 287: //
MIDI.sendSysEx(18, (memcpy_P(Buffer, arp0001, 18)),true);
break;

Everything works fine, as expected, the arrays remain in the program memory, freeing up the ram, but with so many arrays the flash memory also fills up quickly.
How can I optimize or save more memory space?
Since most of the bits in the arrays are repeated.

I am using serial tx as the MIDI output of an Arduino AVR.
Thanks in advance for your attention.

[franky47]

If there's a clear pattern in your SysEx messages, could you try reconstructing them programmatically from a template that contains the common bytes and placeholder empty bytes where things change ?

Example:

const byte arp0000[18] PROGMEM = {
  // Constant header
  0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,

  0x00, // Placeholder byte here

  // Constant footer
  0xf7
};

[RiNoize]

ok, i think i understand the concept, but i'm new to all this arduino and i don't know how to implement it, could you write the complete example? thanks franky

[franky47]

I'd need more of your code, I see a case statement but I have a feeling there's a lot of duplicated code there. The Arduino runs programs, and programs automate repetition, so long as there's some logic behind it.

[RiNoize]

Do you say something like that?:

const byte arpHeader[15] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00};

const byte arpUPup[3] PROGMEM = {0x00, 0x00, 0xf7};
const byte arpDOWN[3] PROGMEM = {0x00, 0x01, 0xf7};
const byte arpALT1[3] PROGMEM = {0x00, 0x02, 0xf7};
const byte arpALT2[3] PROGMEM = {0x00, 0x03, 0xf7};
const byte arpRDOM[3] PROGMEM = {0x00, 0x04, 0xf7};
const byte arp0000[3] PROGMEM = {0x00, 0x05, 0xf7};
const byte arp0001[3] PROGMEM = {0x00, 0x06, 0xf7};
const byte arp0002[3] PROGMEM = {0x00, 0x07, 0xf7};
const byte arp0003[3] PROGMEM = {0x00, 0x08, 0xf7};

And rebuild the arrays like this?:

case 44: //
MIDI.sendSysEx(15, (memcpy_P(Buffer, arpHeader, 15)),true);
MIDI.sendSysEx(3, (memcpy_P(Buffer, arp0000, 3)),true);
break;

The sketch is very large, the "case" is from a library that I am using: PS2 Keyboard Advanced.
I'm using a qwert keyboard, assigning and firing hundreds of SysEx with the keys controlling a Korg Kross 2.
But if you want the sketch, I'll cut it out and share it with you.

[franky47]

I see. One way I see to optimise this is to use a lookup table, a matrix of "keyboard keys" vs "SysEx variable". Rather than implementing every possible case in a switch/case statement, have the key number be the input index of the lookup table, which gives out a variable in your SysEx message.

Taking back your initial example:

const byte sysExTemplate[18] PROGMEM = {0xf0, 0x42, 0x30, 0x00, 0x01, 0x49, 0x41, 0x02, 0x00, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf7};

// Copied from PROGMEM into a local buffer
byte sysExBuffer[18];
memcpy_P(sysExBuffer, sysExTemplate, 18);

const int keyNumber = 287; // todo: Read from I/O

// Use the keyNumber to map to a specific SysEx value
const byte sysExVariable = readFromLookupTable(keyNumber); // eg: 0x06

// Inject variable value at the correct location in the SysEx template
sysExBuffer[16] = sysExVariable;

// Send the customised SysEx
MIDI.sendSysEx(18, sysExBuffer, true);

It may even not be necessary to store the SysEx template in PROGMEM. The lookup table however could get quite big, and could be stored that way, and accessed efficiently by offsetting the base address with the index:
https://www.nongnu.org/avr-libc/user-manual/group__avr__pgmspace.html

[RiNoize]

Gracias franky, I will see how I implement your explanation in my sketch...
Greetings from Argentina