Second Opinion

By Kurt Luther 

In my last column, I presented some initial facts and figures showing that Civil War Photo Sleuth (CWPS), our free website for identifying unknown soldiers in photos, was off to a strong start. In just the first month since its public launch in August 2018, the site’s users had newly identified more than 100 unidentified soldier photos. As of that writing, more than 4,000 users had signed up for the site.

Since then, bolstered by additional media attention, CWPS has hit several new milestones. We now boast over 10,000 registered users who have added over 10,000 photos to the site. Thanks to this amazing community, in less than a year, CWPS has become the world’s largest public database of Civil War portraits, with more than 25,000 total images.

As the site has grown, users have sent us a steady stream of valuable feedback, offering new feature suggestions and helping us find and fix software glitches. Through this feedback and our ongoing interviews with active users, we have also discovered clever and creative ways people are using the site that we had not even considered. In this column, I will discuss one such example and present a new enhancement to CWPS, called “Second Opinion,” that we developed in response to this unexpected need.

The last mile problem

When CWPS users seek to identify an unknown photo, the site returns potential matches of identified photos from the database of reference images. The site’s automated facial recognition software narrows down possibilities by showing only the most similar faces. At this point, the user must carefully inspect the shortlist of the most promising candidates and make a final decision. We call this the “last mile” problem.

This decision can be quite challenging. With a growing database of photos, the odds of finding similar-looking faces increase every day. For a mystery soldier of lower rank and unremarkable facial features, strong candidates are easy to come by. Suppose an identification is promising, but uncertain. What else can a responsible photo sleuth do at this point?

Seeking a Second Opinion

Many users, we discovered, would seek out second opinions. Some users sought the perspectives of particular collectors of specific states or units, hoping to benefit from their specialized expertise. Others worried that collectors had too much skin in the game, and instead asked family or friends for an unbiased point of view. And still others shared potential matches on Facebook groups, seeking consensus from a large, diverse mix of people.

These requests could sometimes yield valuable insights. But users also mentioned various drawbacks. First, users with big piles of photos to identify worried about exhausting the good will of the experts, family, or friends by asking for help too often. Second, the feedback users received about facial features was not very systematic—people tended to focus on certain features and sometimes overlooked important similarities or differences. Third, users felt that existing sites, like Facebook and CWPS, were not designed for detailed discussions of facial nuances.

CWPS does not provide a mechanism for users to weigh in on proposed identifications. So users resorted to posting screenshots on Facebook, and asked others to leave feedback as comments, which were hard to organize, easy to lose, and remained separate from the photos on CWPS.

Based on this feedback, we created a new enhancement for CWPS called “Second Opinion,” specifically aimed at helping users with the last mile of photo sleuthing when they are deciding among one or more strong candidates and would typically seek a second opinion. Second Opinion brings in crowdsourcing and visualization techniques to try to overcome the challenges mentioned above.

Defining and judging the features

When a user finds a possible match for their mystery photo on CWPS, they can use the Second Opinion feature to request a detailed analysis from a group of other people, called “crowd workers.” By default, the software asks each crowd worker to look for differences in facial features across six typical categories: eyes, eyebrows, nose, ears, head hair, and facial hair. For example, the system asks the crowd worker, “Do the soldiers in both photos have similar ears?” If the worker votes no, the software then asks them to elaborate on the difference (e.g., attached versus detached earlobes).

The user can also ask the crowd to look for unique, differentiating features. For instance, if the mystery soldier is missing an arm or has a distinctive scar on his cheek, the user can highlight these unique features for the crowd, since either the presence or absence of them would send an especially strong signal of similarity.

Finally, each crowd worker provides an overall judgment on whether the two photos show the same person or not.

Visualizing the results

Once crowd workers provide their judgments, the results are summarized for the user as visualizations. An overview page ranks the possible matches using the crowd’s overall similarity judgments. The user can click on any photo to see a details page showing how the crowd compared that photo to the mystery photo. These comparisons include both the six typical feature categories (e.g., eyes, ears), plus unique features specified by the user (e.g., eye patch, cleft chin).

Creating a crowd

The current Second Opinion prototype provides feedback from six crowd workers for every photo pair. Every question about the typical features, unique features, and overall similarity is answered, and the level of agreement within the crowd is visualized for the user with color-coded icons. For example, a strong consensus might show all six crowd workers agreeing that the mystery soldier’s nose is similar to the identified soldier’s nose, whereas mixed results might show only half the crowd agreeing that the noses are similar.

Who are these crowd workers? Initially, we hired freelancers from Mechanical Turk (MTurk), a paid crowdsourcing platform run by Amazon.com. Because MTurk’s labor pool has thousands of workers, we could expect results very quickly. And, because the workers are paid, we could post jobs repeatedly without worrying about burnout. The MTurk crowds were also Civil War novices. Alternatively, the user could recruit volunteer crowds, such as friends, family and people with specialized knowledge.

Testing with real experts

We asked 10 experienced Civil War photo sleuths to test the Second Opinion software. We provided each expert with a mystery soldier photo they had not seen before, along with the top five results from CWPS based on facial similarity. We also tested five different image sets: low- and high-ranking white Union and Confederate soldiers, as well as black USCT soldiers.

The expert’s task was to determine which, if any, of the five results was the correct match for the mystery soldier. Since all five were strong candidates, we hoped the crowdsourced feedback from Second Opinion could help the expert make an informed decision.

Our early results suggest that it does. In most of the 10 sessions, the crowd ranked the correct match first among the five possibilities. Experts who used Second Opinion picked the correct match in all 10 sessions. Several pointed out ways that the crowd’s comments helped them notice similarities or differences they had originally missed.

Because we hired crowd workers from MTurk, most experts received crowd feedback for all five pairs in less than 15 minutes. A few experts noted that this speed would be critical in situations with time as a factor, such as deciding whether to bid on an online auction or to buy a photo for sale at a flea market. Many experts liked that the second opinions came from unbiased novices. But most preferred a mix of novices and Civil War experts.

Next steps

Based on these promising initial results, we are exploring ways to integrate the ideas behind Second Opinion into the core Civil War Photo Sleuth website. Our intent is not to replace individual expertise with majority rule; indeed, all proposed identifications on the site will continue to be made by, and attributed, to individual users. Rather, we hope that this tool will provide photo sleuths with a more visual, systematic and integrated way to seek a fresh perspective from the broader community.  

We welcome your feedback on CivilWarPhotoSleuth.com, either via email or by submitting the feedback form on CWPS. We will continue to make regular updates and improvements, announced through our Facebook page and elsewhere.

Kurt Luther is an assistant professor of computer science and, by courtesy, history at Virginia Tech. He writes and speaks about ways that technology can support historical research, education and preservation.

Military Images is America’s only magazine solely dedicated to Civil War portrait photography. Not a Military Images member? Consider our special introductory offer for new members only: Subscribe to Military Images magazine for $24.95 a year and receive four quarterly issues PLUS a free upgrade to our digital edition, which includes PDFs of each issue that are yours to keep and unlimited access to our premium web edition—a $5 savings. Subscribe and become a member now.

© Military Images Magazine. The contents of this page may not be reproduced in whole or part without the written consent of the publisher. Views expressed by the authors do not necessarily represent those of Military Images or Military Images, LLC.