Connect with us


I Unintentionally Created A Biased AI Algorithm 25 Years Ago – Tech Companies Are Still Making The Same Mistake



Facial recognition software misidentifies Black women more than other people. JLco – Ana Suanes/iStock via Getty Images

In 1998, I unintentionally created a racially biased artificial intelligence algorithm. There are lessons in that story that resonate even more strongly today.

The dangers of bias and errors in AI algorithms are now well known. Why, then, has there been a flurry of blunders by tech companies in recent months, especially in the world of AI chatbots and image generators? Initial versions of ChatGPT produced racist output. The DALL-E 2 and Stable Diffusion image generators both showed racial bias in the pictures they created.

My own epiphany as a white male computer scientist occurred while teaching a computer science class in 2021. The class had just viewed a video poem by Joy Buolamwini, AI researcher and artist and the self-described poet of code. Her 2019 video poem “AI, Ain’t I a Woman?” is a devastating three-minute exposé of racial and gender biases in automatic face recognition systems – systems developed by tech companies like Google and Microsoft.

The systems often fail on women of color, incorrectly labeling them as male. Some of the failures are particularly egregious: The hair of Black civil rights leader Ida B. Wells is labeled as a “coonskin cap”; another Black woman is labeled as possessing a “walrus mustache.”

Echoing through the years

I had a horrible déjà vu moment in that computer science class: I suddenly remembered that I, too, had once created a racially biased algorithm. In 1998, I was a doctoral student. My project involved tracking the movements of a person’s head based on input from a video camera. My doctoral adviser had already developed mathematical techniques for accurately following the head in certain situations, but the system needed to be much faster and more robust. Earlier in the 1990s, researchers in other labs had shown that skin-colored areas of an image could be extracted in real time. So we decided to focus on skin color as an additional cue for the tracker.

a color video frame showing a young man entering a room with a red curve overlaying the image outlining his head
The author’s 1998 head-tracking algorithm used skin color to distinguish a face from the background of an image.
Source: John MacCormick, CC BY-ND

I used a digital camera – still a rarity at that time – to take a few shots of my own hand and face, and I also snapped the hands and faces of two or three other people who happened to be in the building. It was easy to manually extract some of the skin-colored pixels from these images and construct a statistical model for the skin colors. After some tweaking and debugging, we had a surprisingly robust real-time head-tracking system.

Not long afterward, my adviser asked me to demonstrate the system to some visiting company executives. When they walked into the room, I was instantly flooded with anxiety: the executives were Japanese. In my casual experiment to see if a simple statistical model would work with our prototype, I had collected data from myself and a handful of others who happened to be in the building. But 100% of these subjects had “white” skin; the Japanese executives did not.

Miraculously, the system worked reasonably well on the executives anyway. But I was shocked by the realization that I had created a racially biased system that could have easily failed for other nonwhite people.

Privilege and priorities

How and why do well-educated, well-intentioned scientists produce biased AI systems? Sociological theories of privilege provide one useful lens.

Ten years before I created the head-tracking system, the scholar Peggy McIntosh proposed the idea of an “invisible knapsack” carried around by white people. Inside the knapsack is a treasure trove of privileges such as “I can do well in a challenging situation without being called a credit to my race,” and “I can criticize our government and talk about how much I fear its policies and behavior without being seen as a cultural outsider.”

In the age of AI, that knapsack needs some new items, such as “AI systems won’t give poor results because of my race.” The invisible knapsack of a white scientist would also need: “I can develop an AI system based on my own appearance, and know it will work well for most of my users.”

AI researcher and artist Joy Buolamwini’s video poem ‘AI, Ain’t I a Woman?’

One suggested remedy for white privilege is to be actively anti-racist. For the 1998 head-tracking system, it might seem obvious that the anti-racist remedy is to treat all skin colors equally. Certainly, we can and should ensure that the system’s training data represents the range of all skin colors as equally as possible.

Unfortunately, this does not guarantee that all skin colors observed by the system will be treated equally. The system must classify every possible color as skin or nonskin. Therefore, there exist colors right on the boundary between skin and nonskin – a region computer scientists call the decision boundary. A person whose skin color crosses over this decision boundary will be classified incorrectly.

Scientists also face a nasty subconscious dilemma when incorporating diversity into machine learning models: Diverse, inclusive models perform worse than narrow models.

A simple analogy can explain this. Imagine you are given a choice between two tasks. Task A is to identify one particular type of tree – say, elm trees. Task B is to identify five types of trees: elm, ash, locust, beech and walnut. It’s obvious that if you are given a fixed amount of time to practice, you will perform better on Task A than Task B.

In the same way, an algorithm that tracks only white skin will be more accurate than an algorithm that tracks the full range of human skin colors. Even if they are aware of the need for diversity and fairness, scientists can be subconsciously affected by this competing need for accuracy.

Hidden in the numbers

My creation of a biased algorithm was thoughtless and potentially offensive. Even more concerning, this incident demonstrates how bias can remain concealed deep within an AI system. To see why, consider a particular set of 12 numbers in a matrix of three rows and four columns. Do they seem racist? The head-tracking algorithm I developed in 1998 is controlled by a matrix like this, which describes the skin color model. But it’s impossible to tell from these numbers alone that this is in fact a racist matrix. They are just numbers, determined automatically by a computer program.

a matrix of numbers in three rows and four columns
This matrix is at the heart of the author’s 1998 skin color model. Can you spot the racism?
Source: John MacCormick, CC BY-ND

The problem of bias hiding in plain sight is much more severe in modern machine-learning systems. Deep neural networks – currently the most popular and powerful type of AI model – often have millions of numbers in which bias could be encoded. The biased face recognition systems critiqued in “AI, Ain’t I a Woman?” are all deep neural networks.

The good news is that a great deal of progress on AI fairness has already been made, both in academia and in industry. Microsoft, for example, has a research group known as FATE, devoted to Fairness, Accountability, Transparency and Ethics in AI. A leading machine-learning conference, NeurIPS, has detailed ethics guidelines, including an eight-point list of negative social impacts that must be considered by researchers who submit papers.

Who’s in the room is who’s at the table

On the other hand, even in 2023, fairness can still be the victim of competitive pressures in academia and industry. The flawed Bard and Bing chatbots from Google and Microsoft are recent evidence of this grim reality. The commercial necessity of building market share led to the premature release of these systems.

The systems suffer from exactly the same problems as my 1998 head tracker. Their training data is biased. They are designed by an unrepresentative group. They face the mathematical impossibility of treating all categories equally. They must somehow trade accuracy for fairness. And their biases are hiding behind millions of inscrutable numerical parameters.

So, how far has the AI field really come since it was possible, over 25 years ago, for a doctoral student to design and publish the results of a racially biased algorithm with no apparent oversight or consequences? It’s clear that biased AI systems can still be created unintentionally and easily. It’s also clear that the bias in these systems can be harmful, hard to detect and even harder to eliminate.

These days it’s a cliché to say industry and academia need diverse groups of people “in the room” designing these algorithms. It would be helpful if the field could reach that point. But in reality, with North American computer science doctoral programs graduating only about 23% female, and 3% Black and Latino students, there will continue to be many rooms and many algorithms in which underrepresented groups are not represented at all.

That’s why the fundamental lessons of my 1998 head tracker are even more important today: It’s easy to make a mistake, it’s easy for bias to enter undetected, and everyone in the room is responsible for preventing it.

The Conversation

I worked for Microsoft from 2003-2007.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *


'Mistaken, Misread, Misquoted, Mislabeled, And Mis-Spoken' — What Woody Guthrie Wrote About The National Debt Debate In Congress During The Depression




Guthrie questioned whether politicians really cared about the public interest — such as the welfare of these veterans demonstrating in front of Congress in 1932. Senate Historical Office

The debt ceiling debate between the House GOP and President Joe Biden could, if not solved, lead to economic chaos and destruction – so it might seem strangely lighthearted to wonder what a Great Depression-era singer and activist would think about this particular political moment.

Certainly, in all the research I did in putting together my book “Prophet Singer: The Voice and Vision of Woody Guthrie,” I never came across any comment Woody Guthrie made about the debt ceiling.

But he lived through the Great Depression and its aftermath. He also stood witness to legislators struggling to correct the direction that the nation was headed in during the 1930s and early ‘40s.

He had a lot to say about Congress in general and how it handled the national debt in particular.

He once made a folksy joke that suggests his feelings about this supposedly august body.

“The Housewives of the country are always afraid at nite, afraid they’s a Robber in the House. Nope, Milady most of em is in the Senate,” he wrote in his regular column for The People’s Daily, called “Woody Sez.”

Guthrie constantly railed against politicians, both Republican and Democrat, who he thought represented their own selfish interests rather than those of deserving working men and women.

What if he could survey today’s America? Would his comments on the state of the nation in the past suggest that he would have something to say in 2023?

In fact, some of his observations sound as if they were written about this political moment – rather than his own.

A man with a hat playing a guitar with a sticker attached that says, 'This Machine Kills Fascists.'
Guthrie, who was known as ‘the Dust Bowl troubador’ for his songs about the Dust Bowl and the Depression.
Library of Congress, World Telegram photo by Al Aumuller

‘Hearin’ the hens a cacklin’

When Guthrie visited Washington, D.C., in 1940, he managed to hear some Senate debates and provided his thoughts on their effectiveness.

“I gawthered the Reactionary Republicans was in love with the Reactionary Republicans; also that the Liberal Democrats was in love with th’ Liberal Demacrats. Each presented a brief case of statistics proving that the other brief cases of statistics, was mistaken, misread, misquoted, mislabeled, and mis-spoken,” he wrote in his column.

And just what were politicians arguing over then? The national debt.

Bipartisan legislative efforts raised the debt ceiling three times under President Donald Trump. Now, House Republicans are balking unless certain conditions are met, while the Democrats are demanding a clean bill with no restrictions.

Guthrie witnessed much the same situation in his era. During his visit to Washington, D.C., he listened to “senators a making speeches – on every conceivable subject under the sun, an’ though the manner in which they brought forth their arguments, their polished wit, and subtle maneuvers, were all very entertaining, I come out of it as empty handed as I went in,” he wrote in “Woody Sez.”

He then compared their debates to “hearin’ the hens a cacklin’ – and a runnin’ out to th barn.” Despite the scene’s being “loud, noisy, and plenty entertaining,” the result was “no eggs.”

There’s a lot of noise coming from Congress today also – but no results.

What could happen if the two sides cannot agree? A telling example occurred in 2011, when the bipartisan deal to raise the debt ceiling came so late that Standard & Poor’s downgraded the country’s credit rating – which hiked the interest that needed to be paid on the U.S. debt.

But if an agreement does not happen, Treasury Secretary Janet Yellen has warned that such a crisis would bring on “economic and financial catastrophe” on a national and global scale.

Guthrie would find this kind of brinkmanship troubling. Not because he was a political operative, with merely an intellectual understanding of the risks. Instead, he was driven by a personal knowledge of the day-to-day hardships, the human toll of such momentous political decisions. His family had fallen from middle-class safety into abject poverty even before the onset of the Great Depression.

A family on the road, standing next to a rickety truck with their belongings. Two boys in overalls are wearing no shirts.
Guthrie knew and sang about the needs of America’s poor, such as this Depression-era impoverished family of nine on a New Mexico highway.
Dorothea Lange, photographer; Library of Congress

Because of falling agricultural prices in the aftermath of World War I and his father’s real estate speculation in some small farms surrounding their hometown of Okemah, Oklahoma, the Guthries could not keep up with their mortgages. They were forced into foreclosure.

Guthrie joked that his father “was the only man in the world that lost a farm a day for thirty days.”

Foreclosures would likely be just one of the ruinous effects of default now, along with interest rates hikes, slashing of social programs, unemployment spikes and decimation of pension plans. All are negative results, but they are certain to hit the poor and working class the hardest.

Those are the people whom Woody Guthrie advocated for throughout his career. Those are the people whose hardships he lamented in such songs as “I Ain’t Got No Home” and “Dust Bowl Refugee.”

But he also expressed optimism about the power of those same people to make a positive change, such as in “Union Maid” and “Better World A-Comin’.” Individual and collective action was necessary, according to Guthrie, and he celebrated both. The union maid would “always get her way when she asked for better pay,” and in “Better World” he sings, “we’ll all be union and we’ll all be free.”

Perhaps his best-known comments about the nation appear in “This Land Is Your Land,” with the popular version praising the American landscape. But in his early version of that song, he ended it with his narrator surveying a line of hungry people lined up “by the relief office” and then asked, “Was this land made for you and me?”

That question could rise again in 2023: If congressional leaders debating over the debt ceiling fail to find common ground for the nation’s greater good, perhaps someone will challenge them and ask if the politicians are in office for the American people, or for themselves – just as Woody Guthrie would have.

The Conversation

Mark Allan Jackson does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

Continue Reading


Symbols Of The Confederacy Are Slowly Coming Down From US Military Bases: 3 Essential Reads




People pose next to a newly unveiled Fort Moore sign on May 11, 2023. Cheney Orr/AFP via Getty Images

Without much fanfare, a federal panel is removing the names of Confederate generals from U.S. military bases and replacing them with names that exemplify modern-day values and patriotism.

Most recently, on May 11, 2023, the U.S. Army base in Georgia originally named after Confederate Brig. Gen. Henry Benning was renamed Fort Moore after both Lt. Gen. Harold “Hal” Moore, who served in Vietnam, and his wife, Julia Moore, who had been an advocate for military families and reformed the military’s death notice procedures.

In stark contrast to the Moores, Benning was a leader in the South’s secession movement and strongly defended slavery.

Over the years, The Conversation US has published numerous stories exploring the legacy of Confederate nostalgia, everything from national monuments to U.S. military bases. Here are selections from those articles.

1. Reconsidering Confederate iconography

For decades, nine U.S. Army bases have carried the names of men who fought against the United States and its Union army – in a war waged to defend and perpetuate the slavery of people of African descent.

These military installations, all in Southern states, were named to honor such figures as Gen. Robert E. Lee, who commanded the Confederate Army, and John Bell Hood, an associate of Lee’s known for being both brave and impetuous.

Until recently, the military installations honoring Confederate leaders received little scrutiny from the media. As a newspaper reporter four decades ago, Jeff South gave the names a free pass. In 1981, South wrote, he covered the Boy Scouts Jamboree at Fort A.P. Hill in Virginia without mentioning that the base was named for a man who had turned against the United States and fought to defend slavery.

“In recent years, more Americans, including those living in the South, have reconsidered the use of Confederate iconography,” South wrote.

Read more:
US moves to rename Army bases honoring Confederate generals who fought to defend slavery

2. Memorializing modern-day values

As a professor of pop culture history who studies Black statues within mainstream society, Frederick Gooding Jr. wrote about America’s reckoning with its oppressive past.

“The nation (faces) the question of not just which statues and other images should be taken down,” Gooding explained, “but what else – if anything – should be put up in their place.”

Gooding pointed out that the lack of Black statues, for example, is an overlooked barometer of racial progress and “sends a clear message of exclusion.”

Read more:
Old statues of Confederate generals are slowly disappearing – will monuments honoring people of color replace them?

3. Memorials have expiration dates too?

Alan Marcus and Walter Woodward have been studying the role of Confederate monuments and other nostalgia in American memory.

“Historical monuments are intended to be timeless, but almost all have an expiration date,” they wrote. “As society’s values shift, the legitimacy of monuments can and often does erode.”

This is because monuments, including the names of U.S. military bases, reveal the values of the time in which they were created and advance the agendas of their creators.

Read more:
Monuments ‘expire’ – but offensive monuments can become powerful history lessons

Editor’s note: This story is a roundup of articles from The Conversation’s archives.

The Conversation

Continue Reading


How China Uses 'geostrategic Corruption' To Exert Its Influence In Latin America




The successful courting of Honduras is the latest example of China’s influence in Latin America. Lintao Zhang/Pool/Getty Images

Corruption has long been a scourge in parts of Latin America.

Traditionally, it has funneled down domestic routes, with local politicians, business interests and drug lords benefiting from graft and dodgy dealings. Indeed, a 2022 report from Transparency International found that 27 out of 30 countries in Latin America and the Caribbean have shown stagnant corruption levels with no improvement in recent years.

But over the last two decades, a new form of corruption has taken hold in countries in the region, a phenomenon we call “geostrategic corruption.”

It is characterized by external countries using corrupt methods – no-bid contracts, insider financial deals, special relations with those in power – to become stakeholders in multiple facets of the politics, economy and society of a country. China is a master of the art; the United States, less so.

As scholars of Latin American politics, we have seen how China has used geostrategic corruption to gain a foothold in the region as U.S. influence has waned.

What is geostrategic corruption?

Geostrategic corruption builds on traditional pervasive patterns of clientelism and patronage. In Latin America in particular, the growth of the drug gangs since the 1980s introduced “narco-corruption” in which police and local officials collude with organized gangs, which are able to “buy protection” from prosecution.

As a result, police, local governments and elected representatives are considered by watchdogs as among the most corrupt political entities in Latin America, with the region consistently scoring low in annual global corruption perception rating.

This pattern of corruption has coincided with a period in which the U.S. has turned its attention away from Latin America and toward first the Middle East and then Asia.

The vacuum has largely been filled by China. Trade between the region and China skyrocketed from US$10 billion worth of goods in 2000 to $450 billion in 2021. China is now the top trading partner of South America, making up to 34% of total trade in Chile, Brazil and Peru.

China’s expansion in the region is largely driven by the country’s search for resources such as cobalt, lithium, rare earths, hydrocarbons and access to foodstuffs, which are abundant in Latin America. In the past 20 years, China has also poured massive investments into infrastructure, energy and financial sectors of Latin America.

And China isn’t alone in upping its interest in Latin America. The last two decades have also seen an increase in investment and influence in the region from Russia and Iran.

These countries have found Latin America a fertile ground due in no small part to the region’s culture of corruption and weak institutions, we argue. Local criminal networks and the disregard of democratic norms on the ground have made it easier for countries that themselves are perceived to be dogged by corruption to gain a foothold in Latin America.

US-China global competition

China’s presence in the region forms part of the country’s long-term strategic objective to challenge U.S. influence across the globe through economic, military, financial and political means.

That process has been aided by global trends. Countries such as Brazil and Argentina have increasingly sought to diversify bilateral relationship and become less dependent on U.S. trade.

Meanwhile, Russian aggression in Ukraine has seemingly given China more weight on the international scene, with Beijing positioning itself as an alternative diplomatic force to Washington, especially to countries that feel nonaligned to the West. A recent example was seen in March, when Honduras announced it would establish diplomatic relations with Beijing and break off ties with Taiwan – a development that Taiwanese officials say followed the “bribing” of Honduran officials.

What gives China an added competitive edge as it extends its influence is that it is able to eschew constraints that bind many would-be investors in the West – such as environmental concerns or hesitation over a country’s labor rights and level of corruption. Chinese companies are judged by international watchdogs to be among the least transparent in the world, and bribery watchdogs have long noted Beijing’s reluctance to prosecute Chinese companies or individuals accused of bribery in regard to foreign contracts. A 2021 study found that 35% of China’s “Belt and Road” projects worldwide have been marked by environmental, labor and corruption problems.

The U.S. administration, in contrast, is more restricted by commitments to encourage democratic development as well as public pressure and international image. Washington does not have the same privilege of diplomatic pragmatism as China.

U.S. companies are, of course, not spotless when it comes to engaging in corrupt practices overseas. But unlike China, the U.S. government is bound to an international treaty prohibiting the use of bribes to win contracts. Moreover, the U.S. Foreign Corrupt Practices Act strictly prohibits American companies from bribing foreign officials; China has no such equivalent.

Chinese corruption in the region

Chinese investment has been easier where populist regimes govern and where the rule of law has long been undermined, such as Argentina, Bolivia and Venezuela.

For example, in Bolivia during the 14-year tenure of President Evo Morales, Chinese companies achieved a major foothold in key sectors of the economy that has translated into a monopoly over the lithium industry there, despite a strong anti-mining movement in the country.

Geostrategic corruption in Argentina is firmly rooted at the local level, in provinces and regions across the country, feudal-like governors have enabled a sophisticated corruption network that China has seemingly used to invest in everything from nuclear plants and building lithium battery plants to constructing a satellite-tracking deep-space ground station, railroads, hydroelectric plants, research facilities and maybe even fighter jets.

In Ecuador, such projects include a dam built in exchange for oil contracts; the Coca Codo Sinclair hydroelectric plant, which developed massive cracks soon after construction; and the Quijos hydroelectric project, which failed to generate promised volumes of power. Similarly, the Chinese-financed Interoceanic Grand Canal in Nicaragua was estimated by opponents of the project to irreversibly impact the ecosystem and displace about 120,000 people, while local activists faced harassment, violence and unlawful detention.

In Venezuela, China initiated but never completed construction of a multibillion dollar bullet train line, and an iron mining deal not only allowed the Asian country to buy Venezuela’s iron ore at a price 75% below market, but also turned out to be an instance of Chinese predatory financing, leaving Venezuela in a catastrophic $1 billion debt. Likewise, in Panama, port concessions and a high-speed train line were frozen or canceled, while the investor is under investigation in China.

Throughout the region, Chinese firms have been cited in numerous cases involving bribery and kickback schemes that have enriched local officials in return for contracts and access.

What does it mean for the US?

This use of geostrategic corruption works to the direct detriment of U.S. interests.

In Argentina and Bolivia, Chinese expansion means that sectors that are crucial for the success of the U.S.’s green energy goals are increasingly under Beijing’s hold. It also undermines U.S. efforts to counter corruption and human rights abuses in the region.

And U.S. companies are unable to compete. The Biden administration has set high standards for U.S. investment in the very sectors where the Chinese have a strong foothold. These include transparency and accountability, as well as commitments to environmental, labor and human rights standards.

President Joe Biden has stated that adherence to these standards is what distinguishes U.S. foreign investments from its competitors. But it does hamstring American companies when it comes to competing with China.

In the meantime, while the U.S. is looking for answers and trying to figure out how to reestablish influence in Latin America, China is quietly and pragmatically increasing its presence in the region.

The Conversation

As an academic and as director of a university research center, I’ve received funding from foundations, US government agencies, and multilateral institutions.

Valeriia Popova does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

Continue Reading