Sunday, April 05, 2009

I Stream, UStream, We All Stream

So, this weekend I figured out how to do a live broadcast via ustream.tv that is a little bit more sophisticated than just turning on my webcam and playing my guitar in front of it.

What I wanted was to avoid relying on the single mono microphone built in to the iSight webcam I'm using, keeping in mind that even though it is sounds pretty decent, the ideal place to put a microphone is not necessarily the same as the ideal place to put a camera. What I wanted was the ability to fully mix live audio sources, and that's mostly what I got.

If you haven't seen UStream, it's a really cool site that allows you to follow live streaming audio and/or video from any of a bazillion live contributors at any given time, and makes it easy to set up you own streaming upload. It provides a very nice Java application which, on my Mac, worked very seamlessly with either my wife's laptop with built-in iSight camera (which it considers a USB camera) or one of the discontinued iSight cameras I picked up recently. The Java application looks like this:


Let me point out a few key features. First, there is the webcam image. You can view either the local image or the one coming back down from the server. The broadcast console has some amazing features: can set set the image quality along a whole range of values, which corresponds to the level of pixelation (how grainy the image is). You can set the frame rate down as low as one frame per second. If you have a slow connection, or if as in my case you want to crank up the audio quality and don't care much about the video quality, you can tweak it just the way you like it. The audio quality can be set from CD quality (44.1 KHz) down to phone quality or less (5 KHz).

Note also that you can choose to stream video, audio, or both. So, if you want to set up your own internet radio station, there you go.

There's only one minor problem: the console application is designed to work with a webcam. You can tell it what audio and video source to use, but if, for example, you want to send audio from iTunes into your stream, you can't, because iTunes, or the system audio, does not show up as a source. And what if you want to play DJ and mix the camera audio over the iTunes audio, or use a separate microphone, or, like me, play guitar into an audio interface and sing on another microphone into the audio interface? You can't.

(Note that the following instructions apply only to Macs; there are presumably similar situations for Windows or Linux on PC hardware, but I have not investigated them).

What you need to do is set up another audio bus -- a virtual bus. The ticket here is a free application from Cycling '74 called Soundflower (Google for it). This will give you two new audio devices, both of which are virtual (they have no hardware associated with them). One is a two-channel bus and one is 16-channel bus. This virtual device can be set up to 96KHz and uses 32-bit samples, so you won't lose audio quality needlessly by passing 16-bit or 24-bit audio through it.

Now, what I really wanted to do was to allow Logic Pro to work with one big audio device. I wanted to use the audio inputs on my Apogee Ensemble for guitar, microphone, and CD player, and the audio outputs for headphones and speakers for monitoring, but I _also_ wanted to use the SoundFlower two-channel device as a way to send exactly what I wanted, all mixed together, to my audio stream. Logic and CoreAudio only work with one device at a time, so this seems impossible.

Apple's solution is to create something called an Aggregate Device. If you run the Audio Midi Setup utility, in the Audio menu you'll see a command called "Open Aggregate Device Editor." You can then create a mash-up of audio devices. In this case, a real hardware device and a virtual device can all be combined. We're using 8 inputs and outputs on the Ensemble, which include four mic preamps, two of which can be used to plug in guitars directly. The two virtual channels from Soundflower are 9 and 10. The aggregate device looks like this in the editor:


So, there's my virtual device. Now Logic can use it; you have to do a little tweaking in Preferences/Audio, but wham, it was pretty simple. And now I can set up my devices.

The way I did it was to configure channel 1 as a guitar input, set up with virtual effects. Channel 4 is my vocal mic, configured with more virtual effects. Channels 7 and 8 are wired to the combo CD/tape player in my rack, so I wanted to be able to use that as an input source as well. Outputs 1 and 2 are the Apogee headphone outputs. Outputs 7 and 8 are the Apogee outputs to some monitor speakers, although to avoid feedback and strange delay issues I generally just listen to headphones.

Now things get a little stranger: inputs 9 and 10 correspond to the 2-channel virtual SoundFlower device, and outputs 9 and 10 represent the outgoing stream.

So, I have lots of options. In Logic, I created a virtual mixer to help me keep this all straight:


Let me try to give you a quick tour of that mixer. From left to right, channel 1 get input 1, which is my guitar, and the inserts are guitar effects. Instead of setting it up with a fixed output, I configured two aux busses, 1 and 2. Bus 1 represents my headphone monitoring. Bus 2 represents the outgoing stream. So the mixer is currently set up to feed the guitar, mic, and CD/tape deck into my headphones.

The fourth channel strip from the left is interesting. Its purpose in life is to allow me to feed the output of iTunes into Logic and mix it in to my headphones as well. For this purpose it has as its input channels 9 and 10, which is the Soundflower virtual bus. It has a send to the headphone bus, but no send to the Soundflower output. The reason for this is that I don't want iTunes audio to be going into the stream twice, with different latencies. I am not sure what might happen there but it looked like it could get ugly, so I thought it best to avoid that situation.

One last thing: iTunes does not have a configurable output; I can't tell it to play on a particular audio device and channel. Instead, it always plays on the default output. So if I want it to feed audio to SoundFlower, that means I have to set the default output to Soundflower, using the Audio MIDI Setup application. I left the system output to 1 and 2 on the Ensemble so that any beeps or what-not produced by my Mac did not wind up on the stream.

If you decide to play audio from iTunes into your UStream show, keep in mind that the 50% mark on the volume control on the Ustream console seems to be digital full scale or 0 dB, when the iTunes volume control is all the way up. This seems to give you the fullest range of volumes. If you go any higher than that you will likely get digital clipping, which sounds very nasty. Experiment with boosting the sources you feed into Soundflower up by adding gain or compression, rather than pushing the final volume past 50%.

Note that you don't have to use Logic; you could use MainStage, or if you want a lighter-weight setup, I think you could probably do most or all of what I'm doing using the wonderful Audio Hijack Pro. And you don't need an Apogee Ensemble; I have one for doing recording with its nice-sounding preamps and low noise, but for live use a much less expensive audio interface would be nearly indistinguishable. You could try the Duet, also by Apogee, or something like an Edirol FA-66, or something like a MOTU traveler. I can't recommend any USB audio interfaces, but this Firewire combination seems to work very well for me.

A word on processing power: I'm doing this all live on a Mac Mini with a G4 processor. However, when I turn on guitar and vocal effects, it starts to bog down. The effects use quite a bit of CPU time. Handling live video also makes the G4 work. If you have anything faster than this -- a G5 or something with an Intel processor -- it should be plenty fast enough to do what I'm doing. You may hear the fan ramp up. One way that I could save some processor power is by moving all the guitar effects to outboard pedals. I don't have an outboard compressor or voice channel strip so the voice processing will have to stay in the computer for now. It does make me want to trade in my G4 Mac Mini for a newer Intel version, which would be able to run MainStage and which would presumably have quite a bit more CPU horsepower to handle audio effects. Oh, and if you are using a laptop, make sure to set it so it doesn't go to sleep or try to cycle down the processor. This might make it get hot, so be sure you have adequate airflow and your studio environment is not too warm.

Here is a picture showing pretty much my entire live setup, minus the guitars and pedals;


Looks impressive, right? Well, except that everything on the rack shelf was very dusty from disuse, and I had not cleaned it all up yet. Also, when you see a shot from the other side, taken while I was setting it all up, and notice the dirty socks on the floor... and the beer... maybe not so much!


And, yes, I did play harmonica on my punk version of Jewel's song "I'm Sensitive." (If this were Twitter, I would then type LOLZ).

You might be saying "wow, this is very complicated," and you wouldn't be wrong. However, I am very impressed, and almost amazed, that it worked. Apple really has their system software together now, and these tools are incredibly flexible. It's particularly nice to be able to do all this mixing and routing right inside the box. And you don't need a setup this elaborate; I know folks using a cheap webcam and microphone and their live streams sound fine for casual listening. We aren't producing a professional-quality video for Blu-Ray here. If I were setting up something from scratch, I'd consider a laptop, an iSight, and an Edirol FA-66, and a mic; the FA-66 can be had for under $300, and a Shure SM-58 microphone for under $100. Add a few cables and spend some time on the setup and you have near-broadcast audio quality (although, of course, not video quality, but with the proper lighting the iSight can produce some impressive results).

Finally, if you would like to listen to one of my shows, my Ustream channel can be found here. I post notes on Twitter and Facebook, or on my chat, indicating when I expect to be on. So far I have done two kinds of shows -- live music with video, and canned playlists from iTunes. But as you can see, just about anything is possible!

13 comments:

Tumbling Dice said...

Very cool. I'd like to do the same thing. I just tried downloading soundflower and setting this up, but I can't get Ustream to actually detect the audio. Even if I select soundflower or the virtual aggregate device as the audio input in Ustream. Not sure if that's the weak link or if Apple Mainstage is not sending the signal to the right place. I can't get Mainstage to detect the virtual aggregate device but it does detect soundflower.
Also, why do you need soundflower, can't you just use the Apogee Ensemble to mix everything together. I can listen to Itunes through it while I'm doing other things with it - I assume, haven't tested it yet.

Paul R. Potts said...

Hi Tumbling Dice,

Thanks for posting a comment. Comments are moderated and with 4 kids I don't get on here to moderate as often as I'd like.

I don't have a lot of time today to think about this -- lots of errands and what-not -- but let me start with your last question: "can't you just use the Apogee Ensemble to mix everything together?"

That works fine if you want to listen to multiple sources, say sending them all to the headphones or monitors. The Ensemble will mix multiple inputs to outputs and CoreAudio will mix multiple applications playing on those inputs, say iTunes with system sounds.

But it breaks down because of this fact: different applications seem to have greater or lesser flexibility as to how they can route audio. This is an area where Apple could really improve, in my opinion, with a standard control panel for application audio input and output, maybe with a basic mode and an advanced mode.

iTunes seems to always send its audio to the default output, as selected in Audio MIDI Setup or the Sound control panel. There doesn't seem to be a setting within iTunes itself to pick something other than the default output. So if I want to send my iTunes output to uStream, the default output has to be set to something that uStream can use as an _input_. Soundflower acts as the "virtual" device for that purpose. I guess I could also route two of the Ensemble outputs back to Ensemble inputs with cables or something, or maybe there is a way to do it in the Maestro application using the mixer, but I mostly have tried to avoid using Maestro's mixer because it seems very buggy -- it crashes a lot, it won't obey my settings, it won't restore my settings, it comes up with random routings.

Logic is another special case with its audio behavior -- it wants to work with a single CoreAudio device for both its inputs and outputs -- it won't let me pick and choose separate devices for input and output. So if I want Logic to be able to take inputs from the Ensemble and send outputs to the Soundflower bus, I have to create an aggregate device that lets me combine Soundflower with the Ensemble.

I have not used MainStage very much -- although I have experimented a little for doing MIDI drum things with a used Yamaha DD-55 pad I just picked up on eBay. I did notice that MainStage seems to have much more flexible audio routing -- it appears to let you use separate devices for input and output, although I have not played with it much.

It becomes even more complicated when you add video. I want to record live video of my playing, using my mics and instruments going into the Ensemble and having Logic do all the effects and mixing. Apple's iMovie will let me record from the iSight camera I have on my desk but it seems to only take the audio stream coming from the camera's built-in mics, which is not what I want. So to record like this the only application I found that would do what I want is actually Photo Booth, which records audio from the default input (in this case, Soundflower).

The whole mess is not working right for me, though, because Soundflower seems to have some serious problems (see my most recent posts). So I may have to find an entirely different solution.

That's my take on it -- it's always possible that I'm missing something obvious that would let me do what I want without Soundflower. I hope it will either get fixed, or that I can find some kind of replacement.

Paul R. Potts said...

Oh, now I remember why I don't use MainStage. It runs great on my Mac Pro, but _will not run_ on my Mac Mini G4, which is installed in my rack so I can set it up for live use. Logic runs just fine (although I can't use very many plug-ins at all). MainStage complains that the video hardware isn't powerful enough. Maybe at some point I will replace the old Mac Mini with a new one. I love the Mini for putting on a rack shelf because they are extremely quiet, even quieter than a laptop.

Tumbling Dice said...

I didn't realize Mainstage caused that problem. Separate and aside from that, I can't even get Ensemble to work with Ustream. When I try to use it, just to add a better microphone than the one on my webcam, it makes the video stream go into slow motion and the sound is in super slow motion. I select "Ensemble" as the audio source and that's when it goes into slow motion. Do you know what might be causing that?

Paul R. Potts said...

Hmmm. About the problem you're having with using the Ensemble as an input into UStream -- I have not tried that exact setup because I wanted from the beginning to use Logic to handle effects. It could be a bottleneck with trying to share your FireWire connection with both the camera and the audio -- are you using a FireWire camera?

One thing you might try is plugging the camera into the FIreWire through on the Ensemble.

I'll see if I can reproduce this problem using a straight mic input into the Ensemble as the audio for UStream.

Paul R. Potts said...

OK, I was able to do an audio/video mix to Ustream using the Ensemble + iSight. Here's a little video clip here (go to this link, and look for "Ensemble Mic Input Test" clip).

http://www.ustream.tv/channel/pottsandpans

This is strictly a little experiment done off the cuff to see if I can make it work as you described -- not intended to be high-quality.

Whether I can get that kind of video quality seems to depend heavily on how my internet access is going at any given time -- it changes depending on network traffic.

I didn't mention it but when I have the camera plugged in to the Ensemble and turned on, the Maestro control panel's "bandwidth used" setting jumps from 13%, which is where it is when I have it set to 8x8 mixer mode, to something like 39%, so the camera uses a lot of bandwidth. But this time at least it didn't seem to break up the audio or the video. In previous experiments involving effects through Logic and Soundflower as the audio input to UStream the audio quality got really poor, and I have not figured out exactly where the problem lies.

Paul R. Potts said...

Well, for some reason my test clip was playing back for me in the UStream console, but will not play back for me from the web page. I don't know what that is all about -- it appears to be another problem on their end. My older clips seem to play OK (they were recorded at lower video quality), but they exhibit some of the audio problems I was talking about.

I can't mess around with it any more this afternoon, but I'll see if I can get a better test clip recorded. I swear it was working...

Tumbling Dice said...

Thanks so much for your help! I really appreciate the time you put into trying to help me solve this. I tried plugging the camera into the ensemble. I didn't even know you could do that. The camera worked fine until I tried it on Ustream that way. Unfortunately, it produced the same result, which was the extremely slow motion video and sound. Hmmm. I'll keep trying to figure out a way. I also sent a support request to Ustream to see if they had any advice about it. It seems like one should be able to use a separate microphone from the web camera on Ustream. I'm not even trying to do anything nearly as fancy as what you are doing.

Tumbling Dice said...

I should have mentioned, the problem produced itself when I tried to play the video back in the test console. I didn't even get as far as you did.

Paul R. Potts said...

When I go to that page today, I am able to play back my test clip. So maybe it had just not been encoded yet. (It would be nice if the UStream site said something to that effect, rather than just giving me the spinning animation when attempting to play the clip).

It is hard to know from here whether the problem you are seeing is a function of bandwidth on your internet connection. Even a pretty low-power Mac should be able to handle the video and audio streams -- it worked on my older G4 Mac Mini, although as I mentioned I could not add very many plug-ins before pegging out the CPU and destroying the audio.

Tumbling Dice said...

Again, thanks so much for your help with this! I really appreciate the effort you have put into this. I watched your video clip testing a setup where you just use the Ensemble for the microphone input and it sounded great. The sound quality was very good and the video seemed to work perfectly. I just tried doing it again and still have the slow motion problem. I thought maybe the problem was that I had a bunch of different microphones plugged into various inputs on the Ensemble and I thought maybe that was taking up too much bandwith or processor power or something, but I still had the problem after I unplugged everything but one microphone.
I recorded a video so you can see what I'm talking about:
http://www.ustream.tv/recorded/1596067

nathan said...

thanks! this was very helpful. I am using it to mic bands and play live sessions. Here is my video page: Always Good Music

Alicia Morgan said...

Yay!!

I have been trying to stream (decent) audio and video via Ustream for quite some time, but I have stymied by the Ustream broadcaster's inability to recognize the audio I was sending from Logic. I have been using my Digi002 to mix my audio (a vocal mic, sometimes a bass, and keyboard using virtual instruments.)

For some bizarre reason, Ustream would recognize only the vocal mic input and not the keyboard - either a direct input or a MIDI trigger for VIs - even though I could hear them both fine, and I can broadcast the audio by itself perfectly.

Your post helpd me figure out how to route my audio using an aggregate device with Soundflower. I don't use the Digi002 because that Digi driver is such a nightmare unless that's all you're using - it makes Soundflower explode - but I'm using an M-Audio Fast Track Pro, which will get the job done.

Now I can hook up a camera and use the Aggregate Device for the audio input! Thank you so much for the great post and explanation (and for the very useful Logic pic!)