Faced with two dangerous alternatives, with neither option offering any clear benefits.
The first print record of ‘the devil and the deep sea’ is in Robert Monro’s His expedition with the worthy Scots regiment called Mac-keyes, 1637:
“I, with my partie, did lie on our poste, as betwixt the devill and the deep sea.”
You’ll note, from that source, that originally the phrase was ‘between the devil and the deep sea.’ I suspect that ‘blue’ was added when the phrase was used as the title of a song, because ‘between the devil and the deep blue sea’ has better rhythm. The song was Between the Devil and the Deep Blue Sea, written by Ted Koehler and Harold Arlen, and recorded by Cab Calloway in 1931.
As to the source, who knows? It might be as simple as the fact that the devil is evil and falling in the sea could mean death, so having to choose one or the other would put one in something of a nasty quandary.
If the answer is that simple, the source may date back to Greek mythology. Homer uses a similar idea when, in his Odyssey, he refers to Odysseus being caught between Scylla (a six-headed monster) and Charybdis (a whirlpool).
There is, however, another explanation. CANOE, the Committee to Ascribe a Nautical Origin to Everything (yes, this actually exists), says the phrase has a nautical origin. To justify this claim, we need to understand some sailing terms.
“Devil” means the seam between the deck planking and the topmost plank of the ship’s side. This seam must be watertight and would need caulking (filling) from time to time. On a ship at sea this would, we assume, require a sailor to be suspended over the side, or at least to work at the very edge of the deck. Either way it’s easy to see how that might be described as ‘between the devil and the deep sea’.
Does the seafaring explanation hold water? That depends on whether the nautical term ‘devil’ was in use when the phrase was coined. There’s no evidence to show the word used in that context until over two hundred years after the first use of the phrase.
So, I suppose that leaves the mystery of the source unsolved, but that shouldn’t stop anybody using the phrase. It’s more poetic than being ‘on the horns of a dilemma.’