The Rock Band game engine has been greatly improved in Rock Band 3. You've got more instruments, "Pro" modes, faster load times, more sorting options for songs, and the list goes on and on.
Where the game is somewhat disappointing is in song selection. You get about 80 songs of middling quality. Rock Band 1 and 2 had fewer songs, but they were better quality overall. Luckily there are plenty of awesome tracks to download from the Rock Band store, but of course you have to pay extra for those.
And, if you don't already have instruments from Rock Band 1 or 2, you'll have to buy those separately, too, and they are not cheap. Except for a keyboard, no bundles are available yet for Rock Band 3.
All-in-all, a worthwhile update to the series. Just be prepared to spend a couple hundred extra for better songs and instruments.