I know I mentioned this project (Spanish Moss Celtic Cables) at least twice in the last meme (30 Days of Knitting), but I couldn't help using it again. It is the most challenging thing I've knit to date, I believe. Working the cables (and reading the chart for them) required a fair amount of concentration for me, though I've read that some knitters simply memorize the pattern for "easy" cables like these. (I don't see myself ever getting that comfortable. I'm too paranoid. I must refer to the chart.) I love it, though, and I definitely want to knit more cables in the future. For me, cables are the epitome of Real Knitting.
I'd have a much harder time choosing a most challenging project for crochet. It would be doilies, I guess-- just doilies in general. They are a ton of fun to crochet, but they do present something of a challenge. The tiny hook-- the thin thread-- the involved instructions, usually different for every round. Then when you're done, you really must block them to get it to look its best. (And blocking is a challenge in and of itself!) But doily-making is a very satisfying process, I think.