AWS Certified AI Practitioner (AIF-C01) — Question 74

What are tokens in the context of generative AI models?

Answer options

• A. Tokens are the basic units of input and output that a generative AI model operates on, representing words, subwords, or other linguistic units.
• B. Tokens are the mathematical representations of words or concepts used in generative AI models.
• C. Tokens are the pre-trained weights of a generative AI model that are fine-tuned for specific tasks.
• D. Tokens are the specific prompts or instructions given to a generative AI model to generate output.

Correct answer: A

Explanation

The correct answer is A because tokens serve as the fundamental components that generative AI models process, encompassing words and subwords. Options B, C, and D are incorrect as they misrepresent the role of tokens; B describes a different aspect of representation, C refers to model parameters rather than input units, and D pertains to prompts rather than the units of representation.