Two Quick Technical Tidbits

Trump is blathering about about "fire, fury, and - frankly - power," and I'm trying to thinkg about something less distressing, so I'm going to write briefly about two technical issues I've come across, in the hopes that Google places this within reach of someone else looking for a solution.

Creating a Shared Memory Block with Keil µVision and the Atmel SAM4E16E

In my work, I have created firmware for the Atmel SAM4E16E, a nifty chip with an ARM core. My toolchain is Keil µVision version 5.23.

What I'm trying to do is conceptually very simple. We have a bootloader image and an application image. When the bootloader executes, I want it to put some specialized values into a block of memory, at a fixed, known address. When the bootloader executes to run the main application, I want the main application to read some information from that block of memory.

The challenge is not so much in writing the C code to handle this -- that's quite easy. The challenge is configuring your tools to get out of your way. In particular, you want the linker to set aside the memory in the right place, and the startup code to leave it alone (not zero it out). Documentation on this is a bit sparse and confusing. It took me quite a bit of trial and error to get it working. Along the way I discovered that the tools are both more poorly-documented and less robust than I hoped. But it did work, and here's how.

First, I created a common header file for describing the memory structure. It looks something like this:

typedef struct Bootloader_Shared_Memory_s
{
    uint32_t prefix;
    uint32_t version;
    uint32_t unused0;
    uint32_t unused1;
    uint32_t unused2;
    uint32_t unused3;
    uint32_t unused4;
    uint32_t unused5;
    uint32_t unused6;
    uint32_t unused7;
    uint32_t suffix;

} Bootloader_Shared_Memory_t;

extern Bootloader_Shared_Memory_t bootloader_shared_memory;

That just gives us a series of 32-bit words in memory. I want to set a prefix and suffix value to some special values that I will look for, to see if the shared memory block looks like it was configured as I expect. It is very unlikely that the prefix and suffix would have these values unless they were deliberately put there.

#define BOOTLOADER_SHARED_DATA_PREFIX ( 0x00ABACAB )
#define BOOTLOADER_SHARED_DATA_SUFFIX ( 0xDEFEEDED )

Then I just define a function that will configure the shared memory, instead of specifying inital values in the definition. That's because I don't want the compiler to treat this block as part of its initialized memory (the ".data" section of memory). See: https://en.wikipedia.org/wiki/Data_segment

void Configure_Bootloader_Shared_Memory( void );

The bootloader code calls this function, which looks like this:

void Configure_Bootloader_Shared_Memory( void )
{
    bootloader_shared_memory.prefix = BOOTLOADER_SHARED_DATA_PREFIX;
    bootloader_shared_memory.version = ( ( BOOTLOADER_REVISION_HIGH   << 16 ) |
                                         ( BOOTLOADER_REVISION_MIDDLE << 8  ) |
                                         ( BOOTLOADER_REVISION_LOW )            );
    bootloader_shared_memory.suffix = BOOTLOADER_SHARED_DATA_SUFFIX;

}

Where BOOTLOADER_REVISION_HIGH etc. are #defines which specify the current version number.

Now, we've declared the data structure and made a function that operates on it. We need to define the data structure. I do this in a separate C file:

/*
    Highest possible base address (last RAM address is 0x2001FFFF), masked to zero off two low bits of address for 4-byte alignment
*/
#define BOOTLOADER_SHARED_MEMORY_BASE_ADDRESS ( ( 0x20020000 - sizeof( Bootloader_Shared_Memory_t ) ) & 0xFFFFFFFC )

/*
    Note: comes out to 0x2001FFD4, the data structure is 0x2C bytes long; will have to change the scatter file if our data structure changes
*/
__attribute__((at(BOOTLOADER_SHARED_MEMORY_BASE_ADDRESS),zero_init))

Bootloader_Shared_Memory_t bootloader_shared_memory;

Note that non-standard attribute. It creates a separate memory area to be passed to the linker. It specifies the base address. The *zero_init* is confusing; suffice it to say that I found it mentioned in ARM's documentation as a workaround. It doesn't mean, apparently, "initialize this block of memory to zero." It seems to mean "do no initialization of this block of memory."

Now, you might think that we've told the linker enough, but apparently we haven't. We have to use a scatter-gather file. This is a file with the extension .sct. Now, we're getting pretty deep in the weeds. The documentation on this file format is pretty sparse. I found few useful examples. The standard way to build my project seemed to result in the toolchain creating its own scatter-gather file, which looks like this:

LR_IROM1 0x00400000 0x000E0000  {    ; load region size_region
  ER_IROM1 0x00400000 0x000E0000  {  ; load address = execution address
   *.o (RESET, +First)
   *(InRoot$$Sections)
   .ANY (+RO)
  }
  RW_IRAM1 0x20000000 0x00020000  {  ; RW data
  .ANY (+RW +ZI)
  }
}

Apparently this specifies the memory regions which the linker will use. It hunts through them applying some rules to determine if each code or data object can go in a region. The RW_IRAM1 is our RAM.

We can create our own scatter-gather file by starting with this tool-generated file and adding a region:

LR_IROM1 0x00400000 0x000E0000  {    ; load region size_region
  ER_IROM1 0x00400000 0x000E0000  {  ; load address = execution address
   *.o (RESET, +First)
   *(InRoot$$Sections)
   .ANY (+RO)
  }
  bootloader_shared 0x2001FFD4 UNINIT 0x0000002C {
    Bootloader_Shared_Memory.o (+RW +ZI)
  }
  RW_IRAM1 0x20000000 0x00020000  {  ; RW data
   .ANY (+RW +ZI)
  }
}

I've added a special region which will fit any data object in the BootloaderSharedMemory.o object file. I've added the special obscure "UNINIT," which I also found mentioned in some somewhat obscure ARM documentation.

Anyway, to make a long story short, this seems to work for me, while any variation of it does not work for me. If you mess up your scatter-gather file a bit, you can see strange behavior -- for example, my image file became over a gigabyte in size at one point. The toolchain doesn't seem very robust as far as error-checking goes, here. If you don't use both the special "UNINIT" and "zero_init" specifiers, things don't work right. It was quite confusing.

To make the build process use the custom scatter-gather file, I had to turn off the checkbox in the project's linker options that reads "Use Memory Layout from Target Dialog." Then I had to provide a path to the scatter file. The linker options dialog should show a -scatter option with a path to your custom scatter-gather file. It isn't enough just to provide a file.

You should be able to tell if it worked right by looking at the link map. It shows the address of the shared memory object. The attribute created the special memory region named with its address.

.ARM.__AT_0x2001FFD4                     0x2001ffd4   Section       44  bootloader_shared_memory.o(.ARM.__AT_0x2001FFD4)

Then, in my application firmware, which reads the information from the shared memory object, I have to configure the linker to use the same scatter-gather file. Then, elsewhere in my code, I can look for the special prefix and suffix to decide if I want to treat the version field in the structure as legitimate data:

if ( ( BOOTLOADER_SHARED_DATA_PREFIX == bootloader_shared_memory.prefix ) && 
     ( BOOTLOADER_SHARED_DATA_SUFFIX == bootloader_shared_memory.suffix )    )
{
    /*
        Do something with bootloader_shared_memory.version
    */
}

This may not be the absolute best or simplest way to achieve what I set out to achieve -- if I've done something glaringly wrong, please leave a comment. But it seems to work, and so I hope someone else might find this useful.

Tagging MP3 files with LAME (Unsolved Mysteries)

I've been trying to simplify my podcast workflow. My old workflow, in part, looked like this:

1. Bounce a track from Logic Pro at a 24 bit/96KHz. Tell Logic to do the conversion to 16 bit/44.1KHz.
2. Don't ask Logic to make an MP3 because if I do it this way, it creates a 48KHz MP3, and I don't want that.
3. Import the 16 bit/44.1KHz track into iTunes.
4. Have iTunes create the MP3 file (annoying because I have to configure the conversion using CD import settings).
5. Use the iTunes "Get Info" tag editor to tag the MP3 file.
6. Find where iTunes put it using "Show File in Finder" and get it back out, then that becomes the tagged file I upload to my server.

I'd rather do it like this instead:

1. Bounce a track from Logic Pro at a 24 bit/96KHz. Tell Logic to do the conversion to 16/44.1.
2. Use a little shell script to do both the conversion to MP3 conversion and tagging with LAME.

This seems to work fine, in that it creates the MP3 file with the bit rate I expect, and the tags I expect, as seen in the MacOS X Finder. That is, if I do "Get Info" on the MP3 file, it shows all the tag I added, including the comments field.

However, if I then bring that MP3 file into iTunes, either just by importing it or by downloading it via my podcast feed, the tags all work except for the comment field. The comment field appears to be empty.

If I look at the LAME-tagged file with Windows file explorer using "Properties," the comment tag shows up as expected. When I look at the iTunes-tagged file, it shows the "Comments" field as either containing 0, or some hexadecimal garbage. In other words, with the Windows file explorer the iTunes-tagged file appears broken, while the LAME-tagged file looks fine. So clearly there is some kind of compatibility problem going on.

Using a Windows program called "Tag Clinic," I can see a difference -- it looks like the LAME-created file has a comment tag that is ID3v2.3 with the Language field showing nothing. The iTunes-tagged file shows as ID3v2.2 with the Language field showing English.

So the question seems to be: can I make LAME export a tag that iTunes will read? If not, is this enough of a deal-breaker that I have to stop using LAME? I would like to create MP3 files with maximum compatibility, and a lot of folks (including me) use iTunes. I care less about "who is broken/buggy/not honoring the standard" as "how can I make my workflow easier while still creating podcast files that will work well for most users."

I didn't get any suggestions yet on Reddit. I had a comment on the Hydrogen Audio forum, suggesting that I "try forcing 2.2" (use ID3 version 2.2 comment tags). However, I don't see a way to force LAME to do that. There's a --id3v2-only switch, but I think that turns off the use of version 1 tags.

I'm left wondering if there is a different command-line utility that will encode and tag my file (ffmpeg, maybe?) Or if I should skip having LAME add the comment tag and instead use a separate command-line utility, id3tag.

But I can't help feeling that I'm missing something -- can the most popular command-line encoding tool really not create MP3 files that play nicely with iTunes? It seems like there must be a way to make this easier. If you've got a suggestion, please leave a comment. Comments are moderated, so I'll have to approve them.

Ypsilanti, Michigan
August 8, 2017