I don't find the movie a very big shot either, but think Matt Dillon does a very fine job in it.
First of all: we have to remind, that Bukowski was the VERY BEST BUKOWSKI EVER and we, knowing so many of his appearances, should be aware of that, whenever watching other people doing 'the Buk'. There will NEVER be anybody, doing justice to the real one. We have to relative due to that (or come to the conclusion, that Nobody should ever have the right to do any performance on him).
2nd: I don't agree, there are No horrors in it. There is one very long, slow scene - where Henry leaves Jane - that justifies the whole movie. It's so sad, without being pathetic. I love it.
3rd: talking of 'slowness' - the whole movie is very slow in a European, even Scandinavian way (you ever seen any movie by Aki Kaurismaki?). The whole style is different from Hollywood-style, and in some way, this suits the subject. The whole matter of this movie, and the style it's done in, appears in the song at the end:
"It's just a slow day moving into a slow night
It doesn't matter what you do ..."
( - Buk: 'wind the clock' in: 'What matters most', p.87)
That sums up the thing.
Give it another try and watch it again, with this in mind.
And now the most heretic one:
I don't find the novel to be very great either.
I think a lot of the problems, that occure with this movie come from the choice to adapt that novel. It's a lose episode-film with no dramaturgic plot or continuity, and tho it may look like this is secondary - it isn't. The whole structure (and this is something that comes from the book) is DISSATISFYING. Maybe I'm thinking too much about dramaturgy, like 'Mr. Hitchcock, what've you done' or something. Just my opinion.
I don't know about the status, the novel has in the US, but in Germany, (I know from a survey), for many fans, 'Factotum' is considered to be one of his weakest novels. And personally I go with that. (sure it has many fantastic scenes; and sure it beats lots of novels of other so called writers, but that's not my point.) I even asked Bent Hamer about the weakness of this novel, but, he disagreed with me, finding it a very good one. (and btw. had very good reasons to name for his oppinion!)
Anyway. My contribution so far.