In the egg, there is a cord that attaches the developing embryo to the yolk sac. When the bird hatches, there is a residual scar where the cord used to be. While the bird is a nestling, you can still see what would be the avian equivalent of a belly button. 
(Photo © reddirtpics) However, as the bird develops, that area becomes more compact and in an adult bird there is virtually nothing to be seen of what once was the scar. So technically baby birds have belly buttons, but unlike the belly buttons of humans, these go away as they grow up.