Saturday, August 12, 2017

A Workaround for the MP3 Tagging Problem

I have found a workaround for the problem I was having. To quickly summarize, I use Logic Pro 9 to export projects to wave files. Then I want to convert them to fully tagged MP3 files.

I've been doing the conversion by hand using iTunes, and tagging in iTunes too. My projects tend to be in higher sample rates and bit depths -- for example, 24-bit, 96KHz. I would use Logic Pro to export the files to MP3 format, and tag them, but I have the following issues:

  1. In Logic Pro 9, if I ask Logic to bounce this project to an MP3 file, it does a long series of conversions and the end result is that I get an MP3 file at 48KHz, which is not what I want. There does not seem to be an option to force it to use 44.1KHz.
  2. In Logic Pro 9, you can also ask Logic to tag your file, but if you supply long fields, it will truncate them. I think this is because it supports a somewhat out-of-date version of the ID3 standard, while iTunes supports a later version which supports longer fields.

So why not just upgrade to Logic Pro X? Well, first, I'm not sure if this would actually fix the problem. At the moment, money is a bit tight. And I'm recording on a 2009 "3,1" Mac Mini. It works great, although it is slow. I don't think installing Logic Pro X on this machine is likely to make my projects easier. Most likely, Pro X eats considerably more memory and CPU and disc space than Logic Pro 9. So the plan is to make do, as much as possible, with what I have, until I can do some major upgrades, including replacing the computer.

Anyhow, to work around these issues with Logic Pro 9, I've been bouncing to a WAVE file, then bringing the 16-bit, 44.1KHz WAVE file into iTunes, creating an MP3 using iTunes, then tagging it by hand.

I wanted to see if I could do some of these steps on the command line, so I could do it in a script, a BBEdit worksheet, or even a Makefile. I'd like to automate that somewhat, not so much because I'm spending that much time producing podcasts, but because all these steps are error-prone. I'd also like to automate, at least partially, the generation of the entries in the podcast feed file. I'm always screwing up the time zone offset in dates, or forgetting to update the size of the file in bytes. It would be nice to have a script to do the grunt work, especially since I am often making versions to test, before I am happy enough with them to add them to the live podcast feed.

So I was trying to use LAME to do the encoding and tagging, but I discovered that iTunes would not import the "comment" field in the MP3 files created and tagged by LAME.

I asked Dan Benjamin on Twitter, and was happy that he replied, but he just wrote "Why not just bounce correctly from Logic?"

Maybe that works for him, but as I explained above, and in the link I sent him, it doesn't work for me, because I wind up with MP3 files encoded at 48KHz. I don't want to get too far into the weeds here, but I believe that making 48KHz MP3 files for podcasts are fairly pointless for most users, since they will need to be resampled on playback and resampling is lossy. For most listeners playing the files back on typical devices, 48KHz is a waste of storage space and will not provide a quality boost over a 44.1KHz file.

I also want to be able to use comment fields like this:

This work by Paul R. Potts is released under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License (http://creativecommons.org/licenses/by-nc-sa/3.0/). See http://generalpurposepodcast.blogspot.com for more information.

And not have them truncated.

In the Hydrogen Audio forum I got a couple of useful replies, but no solutions as such. The root of the problem seems to be that iTunes silently fails to import certain types of comment fields in MP3 tags. I don't want to go too far down the ID3 rabbit hole, but it seems like if the language specified for the comment field is "XXX," iTunes will not import it.

When LAME writes an ID3v2 comment tag, it seems to set the language to either some unicode string, or if the command-line switch --id3v2-latin1 is used, to "XXX." There doesn't seem to be an option to set it to something else. In either case, iTunes will not import this field.

My wife Grace said "your encoder is lame."

I tried to use the id3v2 command-line tool to add the tags to my MP3 file instead. Since I'm making a script, I have no qualms about using two command-line tools instead of one, if that works. But id3v2 seems to have the same problem. It supports the --id3v2-only switch, but there does not seem to be any equivalent of the --id3v2-latin1 switch, so I can't get it to write an iTunes-compatible comment field either.

My workaround was to rebuild LAME, replacing several hard-coded instances of "XXX" with "eng." This is not exactly a bug to report, since I'm not sure that LAME is actually "wrong" per se, according to the standard. But I can't fix the issue in iTunes. And like it or not, if I want maximum compatibility, I have to generate files that work well with iTunes.

A better solution would probably be to give LAME more options. Specifically for this problem an option to set the comment language would be nice. I would hesitate to add a specific switch like --workarounditunes which just forced the comment language to "eng," but maybe the ability to set the comment language would be useful for other folks, while providing a workaround for this compatibility problem.

Ypsilanti, Michigan
August 12, 2017

Tuesday, August 08, 2017

Two Quick Technical Tidbits

Trump is blathering about about "fire, fury, and - frankly - power," and I'm trying to thinkg about something less distressing, so I'm going to write briefly about two technical issues I've come across, in the hopes that Google places this within reach of someone else looking for a solution.

Creating a Shared Memory Block with Keil µVision and the Atmel SAM4E16E

In my work, I have created firmware for the Atmel SAM4E16E, a nifty chip with an ARM core. My toolchain is Keil µVision version 5.23.

What I'm trying to do is conceptually very simple. We have a bootloader image and an application image. When the bootloader executes, I want it to put some specialized values into a block of memory, at a fixed, known address. When the bootloader executes to run the main application, I want the main application to read some information from that block of memory.

The challenge is not so much in writing the C code to handle this -- that's quite easy. The challenge is configuring your tools to get out of your way. In particular, you want the linker to set aside the memory in the right place, and the startup code to leave it alone (not zero it out). Documentation on this is a bit sparse and confusing. It took me quite a bit of trial and error to get it working. Along the way I discovered that the tools are both more poorly-documented and less robust than I hoped. But it did work, and here's how.

First, I created a common header file for describing the memory structure. It looks something like this:

typedef struct Bootloader_Shared_Memory_s
{
    uint32_t prefix;
    uint32_t version;
    uint32_t unused0;
    uint32_t unused1;
    uint32_t unused2;
    uint32_t unused3;
    uint32_t unused4;
    uint32_t unused5;
    uint32_t unused6;
    uint32_t unused7;
    uint32_t suffix;

} Bootloader_Shared_Memory_t;

extern Bootloader_Shared_Memory_t bootloader_shared_memory;

That just gives us a series of 32-bit words in memory. I want to set a prefix and suffix value to some special values that I will look for, to see if the shared memory block looks like it was configured as I expect. It is very unlikely that the prefix and suffix would have these values unless they were deliberately put there.

#define BOOTLOADER_SHARED_DATA_PREFIX ( 0x00ABACAB )
#define BOOTLOADER_SHARED_DATA_SUFFIX ( 0xDEFEEDED )

Then I just define a function that will configure the shared memory, instead of specifying inital values in the definition. That's because I don't want the compiler to treat this block as part of its initialized memory (the ".data" section of memory). See: https://en.wikipedia.org/wiki/Data_segment

void Configure_Bootloader_Shared_Memory( void );

The bootloader code calls this function, which looks like this:

void Configure_Bootloader_Shared_Memory( void )
{
    bootloader_shared_memory.prefix = BOOTLOADER_SHARED_DATA_PREFIX;
    bootloader_shared_memory.version = ( ( BOOTLOADER_REVISION_HIGH   << 16 ) |
                                         ( BOOTLOADER_REVISION_MIDDLE << 8  ) |
                                         ( BOOTLOADER_REVISION_LOW )            );
    bootloader_shared_memory.suffix = BOOTLOADER_SHARED_DATA_SUFFIX;

}

Where BOOTLOADER_REVISION_HIGH etc. are #defines which specify the current version number.

Now, we've declared the data structure and made a function that operates on it. We need to define the data structure. I do this in a separate C file:

/*
    Highest possible base address (last RAM address is 0x2001FFFF), masked to zero off two low bits of address for 4-byte alignment
*/
#define BOOTLOADER_SHARED_MEMORY_BASE_ADDRESS ( ( 0x20020000 - sizeof( Bootloader_Shared_Memory_t ) ) & 0xFFFFFFFC )

/*
    Note: comes out to 0x2001FFD4, the data structure is 0x2C bytes long; will have to change the scatter file if our data structure changes
*/
__attribute__((at(BOOTLOADER_SHARED_MEMORY_BASE_ADDRESS),zero_init))

Bootloader_Shared_Memory_t bootloader_shared_memory;

Note that non-standard attribute. It creates a separate memory area to be passed to the linker. It specifies the base address. The *zero_init* is confusing; suffice it to say that I found it mentioned in ARM's documentation as a workaround. It doesn't mean, apparently, "initialize this block of memory to zero." It seems to mean "do no initialization of this block of memory."

Now, you might think that we've told the linker enough, but apparently we haven't. We have to use a scatter-gather file. This is a file with the extension .sct. Now, we're getting pretty deep in the weeds. The documentation on this file format is pretty sparse. I found few useful examples. The standard way to build my project seemed to result in the toolchain creating its own scatter-gather file, which looks like this:

LR_IROM1 0x00400000 0x000E0000  {    ; load region size_region
  ER_IROM1 0x00400000 0x000E0000  {  ; load address = execution address
   *.o (RESET, +First)
   *(InRoot$$Sections)
   .ANY (+RO)
  }
  RW_IRAM1 0x20000000 0x00020000  {  ; RW data
  .ANY (+RW +ZI)
  }
}

Apparently this specifies the memory regions which the linker will use. It hunts through them applying some rules to determine if each code or data object can go in a region. The RW_IRAM1 is our RAM.

We can create our own scatter-gather file by starting with this tool-generated file and adding a region:

LR_IROM1 0x00400000 0x000E0000  {    ; load region size_region
  ER_IROM1 0x00400000 0x000E0000  {  ; load address = execution address
   *.o (RESET, +First)
   *(InRoot$$Sections)
   .ANY (+RO)
  }
  bootloader_shared 0x2001FFD4 UNINIT 0x0000002C {
    Bootloader_Shared_Memory.o (+RW +ZI)
  }
  RW_IRAM1 0x20000000 0x00020000  {  ; RW data
   .ANY (+RW +ZI)
  }
}

I've added a special region which will fit any data object in the BootloaderSharedMemory.o object file. I've added the special obscure "UNINIT," which I also found mentioned in some somewhat obscure ARM documentation.

Anyway, to make a long story short, this seems to work for me, while any variation of it does not work for me. If you mess up your scatter-gather file a bit, you can see strange behavior -- for example, my image file became over a gigabyte in size at one point. The toolchain doesn't seem very robust as far as error-checking goes, here. If you don't use both the special "UNINIT" and "zero_init" specifiers, things don't work right. It was quite confusing.

To make the build process use the custom scatter-gather file, I had to turn off the checkbox in the project's linker options that reads "Use Memory Layout from Target Dialog." Then I had to provide a path to the scatter file. The linker options dialog should show a -scatter option with a path to your custom scatter-gather file. It isn't enough just to provide a file.

You should be able to tell if it worked right by looking at the link map. It shows the address of the shared memory object. The attribute created the special memory region named with its address.

.ARM.__AT_0x2001FFD4                     0x2001ffd4   Section       44  bootloader_shared_memory.o(.ARM.__AT_0x2001FFD4)

Then, in my application firmware, which reads the information from the shared memory object, I have to configure the linker to use the same scatter-gather file. Then, elsewhere in my code, I can look for the special prefix and suffix to decide if I want to treat the version field in the structure as legitimate data:

if ( ( BOOTLOADER_SHARED_DATA_PREFIX == bootloader_shared_memory.prefix ) && 
     ( BOOTLOADER_SHARED_DATA_SUFFIX == bootloader_shared_memory.suffix )    )
{
    /*
        Do something with bootloader_shared_memory.version
    */
}

This may not be the absolute best or simplest way to achieve what I set out to achieve -- if I've done something glaringly wrong, please leave a comment. But it seems to work, and so I hope someone else might find this useful.

Tagging MP3 files with LAME (Unsolved Mysteries)

I've been trying to simplify my podcast workflow. My old workflow, in part, looked like this:

1. Bounce a track from Logic Pro at a 24 bit/96KHz. Tell Logic to do the conversion to 16 bit/44.1KHz.
2. Don't ask Logic to make an MP3 because if I do it this way, it creates a 48KHz MP3, and I don't want that.
3. Import the 16 bit/44.1KHz track into iTunes.
4. Have iTunes create the MP3 file (annoying because I have to configure the conversion using CD import settings).
5. Use the iTunes "Get Info" tag editor to tag the MP3 file.
6. Find where iTunes put it using "Show File in Finder" and get it back out, then that becomes the tagged file I upload to my server.

I'd rather do it like this instead:

1. Bounce a track from Logic Pro at a 24 bit/96KHz. Tell Logic to do the conversion to 16/44.1.
2. Use a little shell script to do both the conversion to MP3 conversion and tagging with LAME.

This seems to work fine, in that it creates the MP3 file with the bit rate I expect, and the tags I expect, as seen in the MacOS X Finder. That is, if I do "Get Info" on the MP3 file, it shows all the tag I added, including the comments field.

However, if I then bring that MP3 file into iTunes, either just by importing it or by downloading it via my podcast feed, the tags all work except for the comment field. The comment field appears to be empty.

If I look at the LAME-tagged file with Windows file explorer using "Properties," the comment tag shows up as expected. When I look at the iTunes-tagged file, it shows the "Comments" field as either containing 0, or some hexadecimal garbage. In other words, with the Windows file explorer the iTunes-tagged file appears broken, while the LAME-tagged file looks fine. So clearly there is some kind of compatibility problem going on.

Using a Windows program called "Tag Clinic," I can see a difference -- it looks like the LAME-created file has a comment tag that is ID3v2.3 with the Language field showing nothing. The iTunes-tagged file shows as ID3v2.2 with the Language field showing English.

So the question seems to be: can I make LAME export a tag that iTunes will read? If not, is this enough of a deal-breaker that I have to stop using LAME? I would like to create MP3 files with maximum compatibility, and a lot of folks (including me) use iTunes. I care less about "who is broken/buggy/not honoring the standard" as "how can I make my workflow easier while still creating podcast files that will work well for most users."

I didn't get any suggestions yet on Reddit. I had a comment on the Hydrogen Audio forum, suggesting that I "try forcing 2.2" (use ID3 version 2.2 comment tags). However, I don't see a way to force LAME to do that. There's a --id3v2-only switch, but I think that turns off the use of version 1 tags.

I'm left wondering if there is a different command-line utility that will encode and tag my file (ffmpeg, maybe?) Or if I should skip having LAME add the comment tag and instead use a separate command-line utility, id3tag.

But I can't help feeling that I'm missing something -- can the most popular command-line encoding tool really not create MP3 files that play nicely with iTunes? It seems like there must be a way to make this easier. If you've got a suggestion, please leave a comment. Comments are moderated, so I'll have to approve them.

Ypsilanti, Michigan
August 8, 2017