Clean and Tokenize Texteasy

Clean and Tokenize Text

easy Python for AI 10–15 min

NumPy allowed · plain Python earns 10 bonus points

Implement basic text normalization and tokenization in pure Python -- the very first step of any classical NLP or search pre-processing pipeline.

Related Skills

Classical NLP

Task

Implement clean_and_tokenize(text): convert text to lowercase, replace any character that is not alphanumeric (a-z, 0-9) or whitespace with a space, and split into words by whitespace. Return a list of normalized string tokens.

Function Signature

clean_and_tokenize(text: str) -> list[str]

Examples

Example 1: Standard Sentence with Punctuation
Input: text = 'Hello, World! Welcome to AI.'
Output: ['hello', 'world', 'welcome', 'to', 'ai']

Example 2: Hyphens and Version Digits
Input: text = 'state-of-the-art multi-agent system -- v2.0'
Output: ['state', 'of', 'the', 'art', 'multi', 'agent', 'system', 'v2', '0']

Constraints

Libraries (like standard re) are allowed; A plain-Python solution earns 10 bonus points.
Tokens must contain only lowercase letters and digits.
Consecutive whitespace and punctuation must not yield empty string tokens.
Empty input text returns [].

Python3Saved ✓

def clean_and_tokenize(text: str) -> list[str]:

"""Normalize and tokenize text into lowercase words.

Convert to lowercase, replace non-alphanumeric characters (except

whitespace) with spaces, and split into whitespace-delimited tokens.

Empty string returns an empty list."""

# Your implementation here

pass

Case 1: Standard Sentence with Punctuation

Input: text = 'Hello, World! Welcome to AI.'

Expected: ['hello', 'world', 'welcome', 'to', 'ai']

Case 2: Hyphens and Version Digits

Input: text = 'state-of-the-art multi-agent system -- v2.0'

Expected: ['state', 'of', 'the', 'art', 'multi', 'agent', 'system', 'v2', '0']