Oracle Cloud Infrastructure 2023 Architect Associate — Question 43

Which stage of the indexing pipeline divides text into tokens?

Answer options

• A. TOKENIZER
• B. LEXER
• C. SECTIONER
• D. FILTER

Correct answer: B

Explanation

The correct answer is B, LEXER, as it is specifically responsible for breaking down the input text into manageable tokens. The TOKENIZER is often confused with the LEXER, but it typically refers to the process rather than the component that handles tokenization. SECTIONER and FILTER do not perform tokenization; SECTIONER organizes content into sections, while FILTER processes tokens after they have already been created.