Apple launched its own branded MasterCard nationwide in August. In the months since, the digital-first payment system has won some fans for its easy integration into the iPhone and Apple ecosystem, and it more or less seemed to work about as well as any other credit card. Now, however, financial-services regulators want to know what’s going on under the hood amid accusations that the software determining the card’s terms has a sexist slant.

What happened?

Software developer and entrepreneur David Heinemeier Hansson took to Twitter late last week to complain about his wife Jamie Heinemeier Hansson’s experience with AppleCard.

“The @AppleCard is such a fucking sexist program,” his lengthy thread began. “My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.”

“It gets even worse,” he added, sharing a screenshot showing $0 owed on a limit of, apparently, $57.24. “Even when she pays off her ridiculously low limit in full, the card won’t approve any spending until the next billing period. Women apparently aren’t good credit risks even when they pay off the fucking balance in advance and in full.”

Speaking with Apple customer service did no good, he added, with representatives repeatedly deflecting blame to the black box that makes the determinations. Customer service representatives were, “very nice, courteous people representing an utterly broken and reprehensible system,” Hansson said. “The first person was like ‘I don’t know why, but I swear we’re not discriminating, IT’S JUST THE ALGORITHM.’ I shit you not. ‘IT’S JUST THE ALGORITHM!'”

Several other men on Twitter chimed in with replies outlining similar experiences. They said their wives, who on paper look like the better credit risks, received significantly less favorable terms on their Apple Cards than they did. One of the responses came from Apple co-founder Steve Wozniak, who tweeted that, although he and his wife have only joint bank accounts and assets, his Apple Card was given a limit 10 times higher than his wife’s.

As Hansson’s thread went viral and gained media attention, representatives of Apple VIP customer service stepped in. They bumped the credit limit on Jamie’s card up to match David’s and launched an internal investigation.

External investigation

Apple VIP support aren’t the only ones interested in figuring out if the company’s mysterious algorithm is behaving in discriminatory ways; regulators are investigating now, too.

Hansson’s tweets drew the attention of Linda Lacewell, head of the New York Department of Financial Services. “Here in New York State, we support innovation,” Lacewell wrote in a blog post Sunday, adding:

However, new technologies cannot leave certain consumers behind or entrench discrimination. We believe innovation can help solve many challenges, including making quality financial services more accessible and affordable. Yet, this can’t be accomplished without maintaining public confidence. For innovation to deliver lasting and sustained value, the consumers who use new products or services must be able to trust they are being treated fairly.

All financial products and services offered in New York State are required not to discriminate against protected groups. Those products include the Apple Card, which is backed by New York-based Goldman Sachs.

Goldman Sachs issued a statement Sunday saying the discrepancies happened because credit decisions are made on an individual basis, not taking family factors into account.

“We look at an individual’s income and an individual’s creditworthiness, which includes factors like personal credit scores, how much debt you have, and how that debt has been managed,” the company said. “Based on these factors, it is possible for two family members to receive significantly different credit decisions. In all cases, we have not and will not make decisions based on factors like gender.”

CNBC reports that Goldman was “aware of the potential issue” before the card launched in August but chose to move forward anyway. The bank says it is still considering ways of launching shared accounts, including adding multiple cardholders to a single account or allowing for co-signers.

The statement (and the potential for joint accounts or co-signers) does not specifically address why several users reported their wives—in some cases literal millionaires—were given significantly lower Apple Card credit limits and higher interest rates despite being the higher-income earners in the family, having higher credit scores, or both.

Unintended consequences

It’s unlikely in the extreme that someone at either Apple or Goldman Sachs sat down, twirled his mustache à la Snidely Whiplash, and said, “Ah ha! Let’s treat women more badly than men!” Doing so would be both morally and economically stupid, and nobody’s accusing the companies of doing it intentionally.

Decisions made by algorithm, though, have a way of reflecting good old-fashioned human biases—just with even less transparency. And it happens in almost every field. The examples are becoming countless.

About a year ago, Amazon had to stop using an AI tool for hiring and recruiting purposes after it turned out not to be advancing female candidates. Essentially, the software looked at the company’s current successful workforce, which skews male, and decided “male” must be a determinant of success.

In 2015, ProPublica discovered that Asian American families were likely to be charged significantly more for SAT test-prep services. The algorithm determining price wasn’t built expressly to discriminate by race; instead, it used ZIP code—but it charged higher rates in neighborhoods that turned out to be predominantly Asian.

Algorithms with systemic biases are also pervasive in the criminal justice system, where math tends to assign black criminals a higher chance of recidivism after serving their terms than white criminals, as well as higher cash bail, despite evidence showing the scores are unreliable and often wrong.

“The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants,” ProPublica wrote in 2016. “White defendants were mislabeled as low risk more often than black defendants.”

High-profile luck

The Hanssons are lucky in several ways. First, they’re at about the highest end of the consumer spectrum. Jamie wrote in a statement today that she has been financially successful, independent of her husband, for a number of years. She does currently hold a full-time job outside of the home while caring for their three children, she said, but “I am still a millionaire who contributes greatly to my household and pays off credit in full each month.” Both Hanssons have also repeatedly said in public that her credit score is not only excellent but also higher than his.

Beyond that, David has a high profile in the tech and business worlds, with lots of acquaintances and allies in all the right places and more than 350,000 Twitter followers. He can make a stink that will be both seen and taken seriously. The Apple Card is a luxury good, and the Hanssons got such a strong response, in short, because they have just about every privilege in the book—and they’re both keenly aware of it.

“This situation… does not matter for my livelihood,” Jamie wrote in her statement, acknowledging, “This is not merely a story about sexism and credit algorithm blackboxes, but about how rich people nearly always get their way. Justice for another rich white woman is not justice at all.”

Instead of being about her specifically, she wrote, it’s the principle of the thing: “We cannot bow down to the algorithms. We cannot keep sliding into a Black Mirror world. Apple can and should be better than this. We should all be better than this.”

“I hear the frustration of women and minorities who have already been beating this drum loudly and publicly for years without this level of attention,” she added. “I didn’t wish to be the subject matter that sparked these fires, but I’m glad they’re blazing.”

https://arstechnica.com/?p=1600217