A Point in Favor of John Quincy Adams
Of the forty-four Presidents, only Theodore Roosevelt and John Quincy Adams were not sworn in on the Bible. Apparently no Bible was convenient when Roosevelt was sworn in, but for Adams the choice was considered:
John Quincy Adams, according to his own letters, placed his hand on a constitutional law volume rather than a Bible to indicate where his fealty lay.
Apparently, there’s also some question as to whether Lyndon Johnson was sworn in on a Bible, and evidence for the earliest Presidents (Washington excepted) is sketchy.