TF-IDF Weight From Scratch

Difficulty: Medium · Concept: Word Embeddings — Count Vectorization and TF-IDF

Raw word counts overweight common words that appear everywhere and carry little meaning ("the", "a"). TF-IDF fixes this by multiplying two signals: term frequency (how often the word appears in this document) and inverse document frequency (how rare the word is across the whole corpus):

$\text{tf}(t, d) = \frac{\text{count}(t, d)}{|d|} \qquad \text{idf}(t) = \log\left(\frac{N}{\text{df}(t)}\right) \qquad \text{tf-idf}(t, d) = \text{tf}(t, d) \times \text{idf}(t)$

where $N$ is the number of documents in the corpus and $\text{df}(t)$ is how many of them contain $t$ at least once.

Your task: implement tf_idf(term, document, corpus), where document is a list of tokens (assume it's one of the entries in corpus), and corpus is a list of token-lists.

Implement it yourself

assert abs(tf_idf("cat", ["the", "cat", "sat"], [["the", "cat", "sat"], ["the", "dog", "ran"], ["the", "cat", "ran"]]) - 0.13515503603605478) < 1e-9
assert abs(tf_idf("the", ["the", "cat", "sat"], [["the", "cat", "sat"], ["the", "dog", "ran"], ["the", "cat", "ran"]]) - 0.0) < 1e-9
assert abs(tf_idf("dog", ["the", "dog", "ran"], [["the", "cat", "sat"], ["the", "dog", "ran"], ["the", "cat", "ran"]]) - 0.3662040962227032) < 1e-9
assert abs(tf_idf("cat", ["cat", "cat", "mouse", "cat"], [["cat", "cat", "mouse", "cat"], ["dog", "mouse"], ["cat", "bird"]]) - 0.3040988310811233) < 1e-9

Next: 2D Convolution From Scratch