In a retrospective secondary-use EHR study identifying a cohort of Non-Valvular Atrial Fibrillation (NVAF) patients, chart abstraction was done by two sets of clinicians to create a gold standard for risk measures CHA2DS2-VASc and HAS-BLED. Inter-rater reliability between each set of clinicians for NVAF and the outcomes of interest were variable, ranging from extremely low agreement to high agreement. To assess the chart abstraction process, a focus group and a survey was conducted. Survey findings revealed patterns of difficulty in assessing certain items dealing with temporality and social data. The focus group raised issues on the quality and completeness of EHR data, including missing encounters, truncated notes, and low granularity. It also raised the issue of the usability of the data system, the Clinical Data Viewer, which did not mirror a live EHR and made it difficult to record outcomes. Finally, the focus group found it was difficult to infer certain outcomes, like severity, from the provided data. These factors produced differences in clinician rated outcomes.