To say that two things, superficially alike, are fundamentally different in their qualities.
English has hundreds of phrases to describe the similarity of one thing with another, such as ‘alike as two peas in a pod.’ There are far fewer expressions that explicitly refer to the difference between things.
‘As different as chalk and cheese’ is an old expression, and the earliest citation is in John Gower’s Middle English text Confessio Amantis, 1390:
Lo, how they feignen chalk for chese.
This example suggests that some shopkeeper was making an illicit profit by adulterating his wares: “And thus ful ofte chalk for cheese he changeth with ful littel cost.” The buyer could not have been paying attention. Though some British cheeses, particularly unaged ones, are rather chalk-like in appearance, substituting more than a tiny proportion of cheese with chalk wouldn’t fool anybody for very long.
By the sixteenth century, the phrase had become a fixed expression. Hugh Latimer wrote rather sarcastically around 1555: “As though I could not discern cheese from chalk!”
Whoever coined the phrase — and it may have been John Gower himself — had a flair for choosing short, snappy words that alliterate. Something like ‘as different as a cormorant and a lamp-post’ just doesn’t have the same punch. Alliteration sounds good, which explains why so many common phrases use it: hocus-pocus, the bee’s knees, riff-raff, and so on.
I believe this expression is more common in England than it is in North America. Here, I think we would be more inclined to say ‘as different as night and day or oil and water.’