Browsing Ayscough's An Index to the Remarkable Passages and Words made use of by Shakspeare ; Calculated to Point Out the Different Meanings to Which the Words are Applied (Thomas Tegg, London 1827) I checked out its dozen or so entries under 'books'. It is fairly comprehensive (Samuel Ayscough was known as 'The Prince of Indexers') but at about 500 pages is not a 'concordance' and its intention was somewhat different, as stated in the title. Henry IV (Part 2) seems to be the play with the most bookish references
Books
Burn but his books. Tempest, Act 3, Scene 2.
Drown my books. Ibid, Act 5, Scene 1.
The gentleman is not in your books. Much Ado About Nothing, Act 1, Scene 1.
Tire the hearer with a book of words. Ibid, Act 1, Scene 1.
These trees shall be my books. As You Like It, Act 3, Scene 2.
I have unclasp'd to thee the book of my secret soul. Twelfth Night, Act 1, Scene 4.
By what time shall our book, I think be drawn. 1 Henry IV, Act 3, Scene 1.
By this our book is drawn; we will but seal and then to horse immediately. Ibid, Act 3, Scene 1.
Turning your books to graves, your ink to blood, your pens to lances. 1 Henry IV, Act 4, Scene 1.
Blotting your names from books of history. 2 Henry IV, Act 1, Scene 1.
Our fore-fathers had no other books, but the score and the tally. Ibid, Act 4, Scene 7.
Made him my book, wherein my soul recorded the history of all her secret thoughts. Richard III, Act 3, Scene 5.
I have been the book of his good acts. Coriolanus, Act 5, Scene 2.
A book! O rare one! be not as is our fangled world, nobler than that it covers. Cymbeline, Act 5, Scene 4.
That book in many's eyes doth share the glory, that in gold clasps locks in the golden story. Romeo and Juliet, Act 1, Scene 3.
Was ever book, containing such vile matter, so fairly bound. Ibid, Act 3, Scene 2.
Booked
Let it be booked with the rest of the day's deeds - 2 Henry IV, Act 4, Scene 3.
Bookish
Though I am not bookish, yet I can read waiting-gentlewoman in the scape. Winter's Tale, Act 3, Scene 3.
I'll make him yield the crown, whose bookish rule hath pull'd fair England down. 2 Henry IV, Act 1, Scene 1.
Book-Mates
One that makes sport to the prince, and his book-mates. Love's Labour Lost, Act 4, Scene 1.
Book-Oath
I put thee now to thy book-oath; dry it if thou canst. 2 Henry IV, Act 2, Scene 1.
Book of Sport
Or like a book of sport thou'lt read me o'er. Troilus and Cressida, Act 4, Scene 5.
And from The Tempest these fine lines - 'My library was dukedom large enough'...