Well, you are, aren't you? Most people are, despite the fact that it is by definition an irrational fear. Put simply, those who believe in an afterlife expect to go to a "better place," while those who believe there is nothing beyond the physical world expect existence to cease--neither of which should be a particularly frightening prospect. (As the ancient philosopher Epicurus wrote, "When we exist, death is not; and when death exists, we are not.)
The fear, of course, comes from the singularly unknown quantity of death. Much as children fear mysterious "monsters" in their closet rather than, say, lions or bears, it is the unknown that makes our skin crawl, keeps us up at night, and causes us to cover our eyes when the innocent girl goes into the dark basement in a horror movie. This is why the scariest horror movies, paradoxically, are typically those that show the least amount of violence. It is much more frightening to see the girl go into the basement and hear her scream--i.e., let our imaginations run wild--than to witness whatever is actually happening. Thus with death, which is unique in that while we will all experience it, there exists not a single person to tell us what to expect.
Interestingly, many people say they are not afraid to die, but simply don't want to die yet. While we have sympathy for such a view, we think most people who say such things are deluding themselves. Put a different way, while the absurd man makes a conscious decision to live rather than die, he also accepts that all is meaningless, and thus does not particularly care when or how he dies. While he does not seek death, neither does he shrink from it. Those who have not made such a choice, on the other hand, see death as something to be avoided at all costs, even if this is consciously expressed simply as a desire to not die yet.
As the incomparable Ernest Becker put it in his classic The Denial of Death: "The idea of death, the fear of it, haunts the human animal like nothing else; it is a mainspring of human activity--activity designed largely to avoid the fatality of death, to overcome it by denying in some way that it is the final destiny for man."
This, of course, cuts right to the heart of the absurd struggle, which is that humans are the only animals capable of contemplating their own demise (and irrelevance). The truth that we will all die, that the physical world is all there is, and that everything we do is of no consequence whatsoever...well, it is easy to see why humans have evolved sophisticated defense mechanisms (such as religion) to keep these thoughts well-buried in the psyche.
As Becker noted: "It is hard for a man to work steadfastly when his work can mean no more than the digestive noises, wind-breakings, and cries of dinosaurs--noises now silenced forever." Indeed, it is one thing to say you do not fear death; quite another to agree that all of human achievement is in fact of no more consequence than dinosaur droppings.
Finally, we came across an interesting piece in the Financial Times recently that addressed this very issue. As the author (Stephen Cave) put it: "The fact of death imbues our life with passion and urgency, but it is that very passion for life that makes death tragic. Our eye on the reaper’s hourglass prompts us to strive for our highest achievements; but that very striving means that when he comes knocking, it is always too soon."
This is the delicate balance for which the absurd man strives--to live with passion while never losing sight of the meaningless of it all. This is the ultimate (and never-ending) struggle for which he chooses life over death--the rebellion against his knowledge that all is futile, and his commitment to live as if things matter...even when he knows they do not.