AI Data Collection: Key Concepts & Best Practices

Executive Summary

AI data collection is the process of gathering data to build AI use cases within an organization. The use cases can involve projects based on statistical machine learning, deep learning, or even LLMs. While use cases involving statistical machine learning and deep learning require data for both training and evaluation, those based on LLMs predominantly only need data for inference. At a high level, AI data collection involves collecting large volumes of data from various sources, including websites, APIs, sensors, social media, and user interactions on the organization’s website. The collected data can be structured or unstructured and of varying quality. Effective AI data collection requires that data quality management and governance processes be integrated with the data collection process.