All the novels are great, maybe except "Pulp" where the opinions are divided. "Post Office" would be a good place to start. It's also his first novel.
As for short stories, you might start with "South Of No North". It's one of the best short story collections.
As for poetry, you could start with "Burning In Water, Drowning In Flame". It contains some of his best early poetry. Then there's "The Last Night Of The Earth Poems", which is one of his great late poetry books.
Then there's "Septuagenarian Stew" and "Betting On The Muse", both of them a mix of poetry and short stories.
At some point you should also watch the Bukowski documentaries and poetry readings. They're all out on DVD. There's also many CD's with poetry readings, and then there's the movies, "Barfly", "Crazy Love", and "Factotum".