A Guide to Exploring Photo Sleuthing and Generative AI

By Kurt Luther 

Artificial intelligence (AI) seems to be everywhere lately–in the news, on our phones, in schools and workplaces. Civil War photo sleuthing is no exception. AI has been in our field for a long time, and used in diverse ways, thanks in part to the breadth of the term “AI.” Initially, those ways focused on analysis, especially of text documents. Search engines and online databases have long used a flavor of AI to help us quickly sift through thousands of records to find wartime photos or military records based on a keyword, or to make recommendations for new leads based on our recent searches. 

More recently, AI advances have allowed users to “see” photos and directly analyze image content, rather than the text surrounding or describing an image, known as metadata. General-purpose “reverse image search” tools are now built into common search engines like Google and Bing. They allow users to upload a photo of interest, crop a smaller section of particular interest if desired, and the search engine returns a list of visually similar images from around the web. Sometimes a photo sleuth will get lucky and find an exact match to an unknown Civil War soldier portrait, especially cartes de visite due to their production in batches, via a reverse image search. But these tools are usually too general-purpose to find different poses or views of the same Civil War subject.

Generative AI per se is not good or evil. Like any tool, it depends greatly on who is using it, their skills, and their intentions.

Specialized AI image analysis tools have come along to address these limitations. Civil War Photo Sleuth (CWPS) used AI-based facial recognition, in combination with crowdsourced human expertise, to help users identify unknown Civil War portraits. In facial recognition, AI analyzes a face in a photo and identifies key landmarks, such as the corners of the mouth or the tip of the nose, in order to create a unique “faceprint.” This faceprint can then be automatically compared to other, known faceprints in a reference database. On CWPS, a user can find a needle in a haystack of more than 60,000 Civil War-era portrait photos in seconds rather than hours or days, but human discernment is still needed to draw the final conclusion.

More recently, CWPS launched a BackTrace feature (originally called “Backdrop Explorer,” see MI Winter 2024) focused on identifying painted backdrops in Civil War portraits. Using a form of AI called computer vision, BackTrace first removes the human subject from a Civil War photo and then compares the background to thousands of others in its reference database. The user can refine the search results by providing positive feedback on photos with similar-looking backdrops, such as those containing a flag or cannon that looks like the one in the mystery photo, causing the software to return a better set of matches. 

As mentioned above, all of these tools use AI for analysis, i.e., helping humans find or understand existing text or images. However, a new advance in AI technology called Large Language Models (LLMs) has greatly expanded AI’s capabilities for creating, not just analyzing, content. LLMs are trained on unimaginably huge data sets from the internet, books, and other sources, such that they can find patterns and relationships between words, and then use prediction to generate new text, images, and audio in response to human input. LLMs are part of a broader category called “generative AI,” which includes LLMs for text, as well as other AI models that can create images, audio, and even video. For example, generative AI can write stories, translate text, create images, speak in a variety of human voices, and even build new software, among much else.

Generative AI has taken the tech industry by storm, and is increasingly integrated into all sorts of products and services. You have probably noticed that search engines like Google now display a generative AI answer at the top of the search results page. Or perhaps you have seen summaries of email threads or suggested responses to text messages appearing on your smartphone. But the most powerful and flexible way to use generative AI is through conversation. Services like OpenAI’s ChatGPT, Google’s Gemini, and Microsoft’s CoPilot all allow users to interact with a generative AI “chatbot” using everyday natural language, either written or spoken, via a web page or phone app.

A full accounting of the impact of generative AI in just the few years it has been widely available is beyond the scope of this column. Suffice it to say it has already dramatically changed many industries. As a university professor, I have seen how students are increasingly using it to learn (and to avoid learning), and my colleagues and I have adapted our approaches to teaching and grading in response. Reactions to these changes vary greatly. Some people are enthusiastic about generative AI’s potential to improve productivity in the workplace, augment human intelligence, and improve our quality of life, while others express serious concerns about layoffs, copyright infringement, and environmental impacts. 

But what does generative AI mean for Civil War photo sleuthing? Like other fields, ours has also been greatly impacted in just a short time. I expect there are more to come as this technology becomes more widely adopted and integrated. In my opinion, some of these changes are positive and others negative, but generative AI per se is not good or evil. Like any tool, it depends greatly on who is using it, their skills, and their intentions. The upshot for Civil War photo sleuths is that there are more and less skillful ways to use generative AI to support our work. I can share some of these best practices based on the latest research in human-AI interaction, including some of my lab’s work at Virginia Tech. I can also make some recommendations about how not to use generative AI in a Civil War photo sleuthing context.

Best Practices for Generative AI

Compare Different Services

Generative AI services are not all created equal. There are many options available, including the aforementioned general-purpose ones from major tech companies such as ChatGPT, Gemini, and CoPilot, as well as more specialized versions tailored to specific tasks. For example, Ancestry.com offers a chatbot in the sidebar that is trained on genealogy sources to help with questions related to family history. Furthermore, each company often offers several different models optimized for different task types, with the more powerful ones costing more. Google’s Gemini has a free “Flash” model for “fast all-around help” and a paid Pro model ($20/month) for more complex advanced reasoning. Some models also offer special “modes” like deep research, which can generate 20-plus page reports, complete with bibliography, on a topic of your choosing.

Most experienced users of generative AI find that some models work much better than others and it requires continuous experimentation and comparison to find the best fit. For example, a student in my lab, Siddharth Rakshit, studied how well several commercial AI models generated biographical profiles of Civil War soldiers and found preliminary evidence that Perplexity AI was least likely to provide false information. Companies are also constantly changing and improving their models, so even if you didn’t have a good experience with an AI service six months or a year ago, it is worth trying again.

Learn to Write Effective Prompts

The saying “garbage in, garbage out” applies to generative AI. The quality of the input or instructions you give to the AI, known as “prompt writing” or “prompt engineering,” directly impacts the quality of the AI’s output. Fortunately, there are some useful guidelines available, and prompt writing is a skill that improves with practice. First, provide lots of context and detail to help the AI understand what you are looking for. Start by explaining what your goals are, whatever critical background information is needed, and what you already tried. Then specify exactly what you’re looking for in the response, including citations or web links to sources. If you can provide examples of content or format, that’s even better. 

Second, chatbots rarely ask follow-up questions unless you explicitly invite them. Instead, they will normally try to answer immediately based on as much (or little) information as you give them. It can be helpful to conclude your prompt with instructions for the chatbot to ask you for any critical missing pieces before it generates its response.

Third, while chatbots are trained on huge troves of data and have the ability to search the internet for more, their memories are surprisingly limited. As conversations grow longer, they tend to get more confused. If you are researching a particular photo or soldier or sailor, it’s best to limit that discussion to a single thread or session; this will also prevent having to provide background information again and again. 

Be Skeptical

Generative AI is essentially built on pattern recognition and prediction. The AI doesn’t really “know” anything, but it is good at generating text that sounds reasonable and is sometimes even accurate. However, it will also make up false information, a behavior known as “hallucination.” Worse, the AI doesn’t know what it doesn’t know, so it is often confidently wrong. This pervasive limitation means that you, the human photo sleuth, must remain vigilant in pursuing truth and skeptical of any AI-generated responses.

Even though generative AI is often wrong, it can still be useful. When Wikipedia, the free online encyclopedia first launched, many people doubted that an information resource open for anyone to edit, regardless of credentials, could provide value. In fact, later studies found that Wikipedia content, which is largely human-generated, is about as accurate as professionally written resources like Encyclopedia Britannica. Yet, readers were still correct to approach Wikipedia content with caution; teachers learned to warn students to verify any facts they read and to check for citations. Wikipedia was a place to start your research, never to end it.

We can bring a similar mindset to generative AI. It is often said that a chatbot is like a smart intern: helpful but by no means infallible. It can do an excellent job of suggesting and synthesizing information, but the burden is on us humans to check its work. 

AI-Augmented Photo Sleuthing

With these general principles in mind, we can consider some specific Civil War photo sleuth tasks for which generative AI is well- or ill-suited.

AI Can Suggest New Leads

Photo sleuthing, like many forms of investigative analysis, is fundamentally a creative process. We encounter a puzzle to be solved, and the straightforward solutions are rarely sufficient on their own. The obvious paths quickly lead to dead ends. When this happens, we rely on our creativity, our experience, and our intellect to think up alternatives. If we have exhausted the usual visual clues like insignia, uniforms, and weapons, what others might we have missed? If a regimental history doesn’t exist for this unit, what other, less common sources might we try for reference images? If the standard explanation for why these three soldiers might have sat for a group portrait doesn’t fit, what are some less likely, but still plausible reasons?

Generative AI can provide a brainstorming partner to help suggest new leads like these when our research process hits a brick wall. First, I will write a detailed prompt telling the chatbot what I’m hoping to learn and which techniques I already tried. Next, I ask what else I might try to move the investigation forward. If the initial suggestions aren’t helpful, I ask for more, encouraging the AI to get more creative and unorthodox. Even if the results aren’t feasible, there’s often a seed of a good idea I can build on.

AI Can Critique Your Research Process

Generative AI can help photo sleuths who are frustrated about getting stuck. But it can also help us when we are ecstatic about a potential breakthrough. In previous columns, I have written about confirmation bias, the tendency to seek out only information that supports a conclusion we reached prematurely, and ignoring any evidence to the contrary. Confirmation bias is a perennial concern for photo sleuths because the desire to identify a previously unknown Civil War portrait is so strong. My lab previously developed tools like Second Opinion (MI, Summer 2019) that harnessed human intelligence (via crowdsourcing) to provide more systematic, unbiased analysis of possible facial recognition matches and help combat confirmation bias. Beyond this, most of us have informally asked friends or social media to weigh in on a potential ID.

These are useful strategies verifying the results of an investigation. However, it’s more difficult to get feedback on one’s process, especially frequently or in a timely manner. Generative AI can help by critiquing the path you’ve taken during your photo sleuthing journey. Perhaps you skipped a step or overlooked an important caveat. Maybe there is a false assumption underpinning your reasoning. It could just be that your excitement for finally discovering a name for the unknown face is temporarily overpowering logic. For any of these cases, I find that asking AI to play the devil’s advocate and poke holes in my argument occasionally uncovers new problems, but at minimum helps me cover my bases.

AI Can Analyze Diverse Data Sets

Photo sleuthing requires sifting through piles of information, much of it found online, from records databases to image galleries to digitized books and letters. Websites like CWPS, the American Civil War Research Database (HDS), and the National Park Service’s Civil War Soldiers and Sailors System (CWSS) provide structured computational tools for navigating specific types of information. Against this backdrop, a unique benefit of generative AI is its immense flexibility. No matter what my research question is, as long as I can describe it using natural language, the chatbot will instantaneously make an effort to answer it. Thus, photo sleuths can leverage generative AI to quickly analyze data sets for which there is no dedicated website or software tool. 

For example, suppose I find a website with links to a large number of digitized photos or letters from the 19th century. Traditionally, my options would be to either painstakingly open each link, one at a time, to assess its relevance (least likely); skim the website’s contents by randomly clicking a small subset of links, potentially missing valuable content; or skip the site altogether given the low cost-benefit ratio (most likely). 

Instead, with generative AI, I can provide the chatbot with the website link and detailed instructions for finding relevant content, such as noting which images appear to be Civil War soldiers or which documents mention specific soldier names or military units. The key is to identify tasks that play to generative AI’s strengths and minimize its weaknesses. In this example, it’s likely that the AI will hallucinate some incorrect hits, but manual verification of that subset is much faster than examining the entire website, and false negatives are an acceptable cost compared to skipping everything.

Practices to Avoid

Having covered some best practices for photo sleuthing with generative AI, I conclude here with some caveats about how not to use it. First and foremost, it is rarely beneficial to directly ask the chatbot to identify an unknown Civil War portrait photo. With the exception of famous historical persons whose faces you would likely already recognize, the AI is far more likely to confidently suggest a wrong identification than the correct one. 

Second, I do not recommend asking generative AI to research a given soldier’s military service or write a biographical profile for them. Again, given AI’s propensity to hallucinate, the results are almost guaranteed to contain bits of false information which may be hard to detect amidst a generally accurate profile. The act of manually verifying each statement in the result may require more effort than simply researching and writing your own profile in the first place.

Third, I urge prudence in sharing any AI-generated results with others unless you have vetted and contextualized the information. Even then, content posted on the internet quickly gets separated from its source and distorted into misinformation. In Facebook groups for researching Civil War ancestors, I have seen well-meaning users post unvetted text straight from ChatGPT with a preface paraphrased as, “Here’s what AI said about your ancestor, hope it helps!” Within hours, that same text, rife with AI hallucinations, is taken for granted elsewhere in the comment thread.

Finally, I remind photo sleuths that while generative AI offers great promise, it’s just one tool in the toolbox. There is still no substitute for your expertise, experience, and eye for detail. The detective work of identifying unknown soldiers in Civil War-era photos benefits from a wide variety of techniques and resources, from classic research methods and community knowledge to new technologies like generative AI. Most of all, it requires qualities that are fundamentally human: creativity, passion, and persistence.

Kurt Luther is an associate professor of computer science and, by courtesy, history at Virginia Tech. He is the president of The Photo Sleuth Foundation, a non-profit organization with a mission to rediscover the names and stories of unknown people in historical photos through research, technology, and community. He is a MI Senior Editor.

---