It's from Notes of a Dirty Old Man, page 177 in the newest City Lights edition. The exact quote is imbedded in a dialogue between two men:
"you mean to say that you can LOVE a piece of buttered toast?
only some, sir. on certain mornings. in certain rays of sunlight. love arrives and departs without notice.
is it possible to love a human being?
of course, especially if you don't know them too well."
This is only 12 pages removed from his "scribbling on shirt cardboards during two-day drunks" riff, which contains the famous "an artist is a man who" quote and "If you want to know where God is, ask a drunk."