Pathology suffers from an agreement problem for cases that are not clear-cut. It is easy for pathologists (and similarly, computational models) to distinguish between normal and cancerous samples, but frequently there are classes in between that pathologists may not agree on. This discordance is hampering automatic visual detection and classification for these difficult cases. Only recently did researchers come across this problem for breast cell biopsy whole slide image classification, which generated much publicity and led experts to recommend that patients seek a second opinion. In this talk, we will explore ideas that may allow us to pinpoint why classification discordance occurs in hard cases, and how we can exploit this knowledge to benefit pathologists, computational vision researchers, and machine learning researchers in the future.