Sources and Methods

Extracting features

To begin, each verse in the Bible is turned into tokens, or n-grams. Take, for instance, the first verse of the Bible: “In the beginning, God created the heaven and the earth” (Genesis 1:1 KJV). This verse would be turned into tokens ranging from three to five words long, skipping stop words which convey little meaning (such as an, of, the). So the text of Genesis 1:1 becomes these tokens.

• "god created heaven"
• "beginning god created heaven earth"
• "beginning god created"
• "created heaven earth"
• "beginning god created heaven"
• "god created heaven earth"

These tokens then are used to create a document-term matrix, where the rows are the Bible verses, the columns are the tokens, and the cells indicate how many times that token appears in that verse. For instance, a subset of the Bible matrix looks like this.

                  beginning god created god created heaven without form void
Genesis 1:1 (KJV)                     1                  1                 0
Genesis 1:2 (KJV)                     0                  0                 1
Genesis 1:3 (KJV)                     0                  0                 0

Then each newspaper page is turned into tokens using the exact same function that was used for tokenizing the Bible. This creates a second matrix where the rows are newspaper pages, the columns are three to five word tokens from the Bible, and the cells indicate how many times that string of words from the Bible is found on the newspaper page. Although the newspapers include vastly more possible tokens than are found in just the Bible, this method restricts the number of columns in the newspaper document-term matrix to the same as in the Bible matrix.

       beginning god created god created heaven without form void
page_A                     1                  1                 0
page_B                     0                  0                 1
page_C                     0                  0                 0

Because the two matrices share a dimension, the Bible matrix can be multipled by the transpose of the newspaper matrix. The result is a matrix with Bible verses in the rows and newspaper pages in the columns. The numbers in the cells of the matrix indicate how many tokens from that verse were found on that newspaper page. So in the sample matrix below, page_A shares two tokens with Genesis 1:1 and page_B shares one token with Genesis 1:2, indicating that those verses might appear on those pages. Of course the vast majority of cells in the resulting matrix are zeros.

                  page_A page_B page_C
Genesis 1:1 (KJV)      2      0      0
Genesis 1:2 (KJV)      0      1      0
Genesis 1:3 (KJV)      0      0      0

The multiplication of document-term matrices is the primary means I have used to find matches, but the token count is only one of the features about potential matches that I measure. These are the features (or predictors) that I currently use.

• Token count: the number of tokens from a particular verse that appear on a particular newspaper page.
• TF-IDF: Not every token contains the same amount of information about whether a newspaper page contains a particular biblical verse. For instance, the phrase “went into the city” could be a quotation from a dozen or more Bible verses, but it might just as well be any English sentence. But the phrase “through a glass, darkly” is obviously a reference to 1 Corinthians 13:12, as these search results demonstrate. By weighting the matching tokens according to their term frequency-inverse document frequency, more significant terms count for more in determining a match.
• Proportion: Bible verses can vary in length, from just two words (“Jesus wept” [John 11:35] and “Rejoice evermore” [1 Thessalonians 5:16]) to the longest, Esther 8:9, which has ninety words in the King James Version. (In fact, Esther 8:9 appears in several Chronicling America newspapers as the punchline of a joke about a “boy who boasted of his wonderful memory.”¹) This feature measures what proportion of the entire verse is found on the page.
• Runs test: Where the matching tokens appear on the page is as important as how many matches there are. If the tokens appear widely scattered across the page, then they are likely to be just random matches to unimportant phrases. If the tokens are all clustered right next to each other (perhaps with a few gaps for incorrect OCR), then they are likely to be a quotation from the verse. This feature uses a statistical test to determine whether the sequence of matches (called a “run”) is random or not.

Labeling data and training the model

After measuring the potential matches, we need a means of distinguishing between accurate matches and false positives. This is a difficult problem because of the way that the Bible was quoted in newspapers (or indeed, used more generally). If we were looking for complete quotations, then we would look for candidates where there were many matching tokens, or where a high proportion of the matching verse is present on the page. But often quotations can be highly compressed. A single unusual phrase (“Quench not the Spirit” or “Remember Lot’s wife” or “The Lord called Samuel”) may be enough to identify one quotation, where even a half dozen commonplace matching phrases might not actually be a quotation. Then too, sometimes allusions function by changing the actual words while retaining the syntax or cadence, as in this joke.²

Rather than specify arbitrary thresholds, a more accurate approach is to teach an algorithm to distinguish between quotations and noise by showing it what many genuine matches and false positives look like. After taking a sample of potential matches, I identified some 1,700 possible matches as either genuine or not. (You can see the labeled data here.) This makes it possible to observe patterns in the features that have been measured. In the charts below, for instance, show that genuine matches tend to have a much higher token count, a much higher TF-IDF score, and a very low p-value for the runs test. But it is not possible to draw a single line on either chart which cleanly distinguishes between all genuine matches and all false positives.

I then used that data to train and test a machine learning model. This model takes the predictors mentioned above, and assigns it a class (“quotation” or not) and a probability that that classification is correct. While I evaluated a number of models, including random forests, support vector machines, and ensembles of other models, a neural network classifier had the best performance. I measured accuracy using the area under the receiver operating characteristic curve. The idea there is simple: the best classifier is the one that maximizes the number of genuine matches while minimizing the number of false positives.