AWS Certified Data Analytics – Specialty — Question 4

A company that produces network devices has millions of users. Data is collected from the devices on an hourly basis and stored in an Amazon S3 data lake.
The company runs analyses on the last 24 hours of data flow logs for abnormality detection and to troubleshoot and resolve user issues. The company also analyzes historical logs dating back 2 years to discover patterns and look for improvement opportunities.
The data flow logs contain many metrics, such as date, timestamp, source IP, and target IP. There are about 10 billion events every day.
How should this data be stored for optimal performance?

Answer options

• A. In Apache ORC partitioned by date and sorted by source IP
• B. In compressed .csv partitioned by date and sorted by source IP
• C. In Apache Parquet partitioned by source IP and sorted by date
• D. In compressed nested JSON partitioned by source IP and sorted by date

Correct answer: A

Explanation

Storing the data in Apache ORC format, partitioned by date and sorted by source IP, optimizes performance for both analytical queries and data retrieval. ORC files are designed for high compression and efficient read access, which is essential given the volume of data. The other options, while still functional, do not provide the same level of performance or efficiency as ORC for the specified use case.