We may have bitten off more than we could chew, folks.

An Amazon engineer told me that when he heard what I was trying to do with Ars headlines, the first thing he thought was that we had chosen a deceptively hard problem. He warned that I needed to be careful about properly setting my expectations. If this was a real business problem… well, the best thing he could do was suggest reframing the problem from “good or bad headline” to something less concrete.

That statement was the most family-friendly and concise way of framing the outcome of my four-week, part-time crash course in machine learning. As of this moment, my PyTorch kernels aren’t so much torches as they are dumpster fires. The accuracy has improved slightly, thanks to professional intervention, but I am nowhere near deploying a working solution. Today, as I am allegedly on vacation visiting my parents for the first time in over a year, I sat on a couch in their living room working on this project and accidentally launched a model training job locally on the Dell laptop I brought—with a 2.4 GHz Intel Core i3 7100U CPU—instead of in the SageMaker copy of the same Jupyter notebook. The Dell locked up so hard I had to pull the battery out to reboot it.

But hey, if the machine isn’t necessarily learning, at least I am. We’re almost at the end, but if this were a classroom assignment, my grade on the transcript would probably be an “Incomplete.”

The gang tries some machine learning

To recap: I was given the pairs of headlines used for Ars articles over the past five years with data on the A/B test winners and their relative click rates. Then I was asked to use Amazon Web Services’ SageMaker to create a machine-learning algorithm to predict the winner in future pairs of headlines. I ended up going down some ML blind alleys before consulting various Amazon sources for some much-needed help.

Most of the pieces are in place to finish this project. We (more accurately, my “call a friend at AWS” lifeline) had some success with different modeling approaches, though the accuracy rating (just north of 70 percent) was not as definitive as one would like. I’ve got enough to work with to produce (with some additional elbow grease) a deployed model and code to run predictions on pairs of headlines if I crib their notes and use the algorithms created as a result.

But I’ve got to be honest: my efforts to reproduce that work both on my own local server and on SageMaker have fallen flat. In the process of fumbling my way through the intricacies of SageMaker (including forgetting to shut down notebooks, running automated learning processes that I was later advised were for “enterprise customers,” and other miscues), I’ve burned through more AWS budget than I would be comfortable spending on an unfunded adventure. And while I understand intellectually how to deploy the models that have resulted from all this futzing around, I am still debugging the actual execution of that deployment.

If nothing else, this project has become a very interesting lesson in all the ways machine-learning projects (and the people behind them) can fail. And failure this time began with the data itself—or even with the question we chose to ask with it.

I may still get a working solution out of this effort. But in the meantime, I’m going to share the data set on my GitHub that I worked with to provide a more interactive component to this adventure. If you’re able to get better results, be sure to join us next week to taunt me in the live wrap-up to this series. (More details on that at the end.)

Modeler’s glue

After several iterations of tuning the SqueezeBert model we used in our redirected attempt to train for headlines, the resulting set was consistently getting 66 percent accuracy in testing—somewhat less than the previously suggested above-70 percent promise.

This included efforts to reduce the size of the steps taken between learning cycles to adjust inputs—the “learning rate” hyperparameter that is used to avoid overfitting or underfitting of the model. We reduced the learning rate substantially, because when you have a small amount of data (as we do here) and the learning rate is set too high, it will basically make larger assumptions in terms of the structure and syntax of the data set. Reducing that forces the model to adjust those leaps to little baby steps. Our original learning rate was set to 2×10^-5 (2E-5); we ratcheted that down to 1E-5.

We also tried a much larger model that had been pre-trained on a vast amount of text, called DeBERTa (Decoding-enhanced BERT with Disentangled Attention). DeBERTa is a very sophisticated model: 48 Transform layers with 1.5 billion parameters.

DeBERTa is so fancy, it has outperformed humans on natural-language understanding tasks in the SuperGLUE benchmark—the first model to do so.

The resulting deployment package is also pretty hefty: 2.9 gigabytes. With all that additional machine-learning heft, we got back up to 72 percent accuracy. Considering that DeBERTa is supposedly better than a human when it comes to spotting meaning within text, this accuracy is, as a famous nuclear power plant operator once said, “not great, not terrible.”

Deployment death spiral

On top of that, the clock was ticking. I needed to try to get a version of my own up and running to test out with real data.

An attempt at a local deployment did not go well, particularly from a performance perspective. Without a good GPU available, the PyTorch jobs running the model and the endpoint literally brought my system to a halt.

So, I returned to trying to deploy on SageMaker. I attempted to run the smaller SqueezeBert modeling job on SageMaker on my own, but it quickly got more complicated. Training requires PyTorch, the Python machine-learning framework, as well as a collection of other modules. But when I imported the various Python modules required to my SageMaker PyTorch kernel, they didn’t match up cleanly despite updates.

As a result, parts of the code that worked on my local server failed, and my efforts became mired in a morass of dependency entanglement. It turned out to be a problem with a version of the NumPy library, except when I forced a reinstall (pip uninstall numpy, pip install numpy -no-cache-dir), the version was the same, and the error persisted. I finally got it fixed, but then I was met with another error that hard-stopped me from running the training job and instructed me to contact customer service:

ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateTrainingJob operation: The account-level service limit 'ml.p3.2xlarge for training job usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit.

In order to fully complete this effort, I needed to get Amazon to up my quota—not something I had anticipated when I started plugging away. It’s an easy fix, but troubleshooting the module conflicts ate up most of a day. And the clock ran out on me as I was attempting to side-step using the pre-built model my expert help provided, deploying it as a SageMaker endpoint.

This effort is now in extra time. This is where I would have been discussing how the model did in testing against recent headline pairs—if I ever got the model to that point. If I can ultimately make it, I’ll put the outcome in the comments and in a note on my GitHub page.

https://arstechnica.com/?p=1781863

Just one week after Google’s DeepMind AI group finally described its biology efforts in detail, the company is releasing a paper that explains how it analyzed nearly every protein encoded in the human genome and predicted its likely three-dimensional structure—a structure that can be critical for understanding disease and designing treatments. In the very near future, all of these structures will be released under a Creative Commons license via the European Bioinformatics Institute, which already hosts a major database of protein structures.

In a press conference associated with the paper’s release, DeepMind’s Demis Hassabis made clear that the company isn’t stopping there. In addition to the work described in the paper, the company will release structural predictions for the genomes of 20 major research organisms, from yeast to fruit flies to mice. In total, the database launch will include roughly 350,000 protein structures.

What’s in a structure?

We just described DeepMind’s software last week, so we won’t go into much detail here. The effort is an AI-based system trained on the structure of existing proteins that had been determined (often laboriously) through laboratory experiments. The system uses that training, plus information it obtains from families of proteins related by evolution, to predict how a protein’s chain of amino acids folds up in three-dimensional space.

The three-dimensional structure that results can give us critical information about the protein, such as how it interacts with other proteins and chemicals and where on the protein chemical reactions occur. Using the structure, researchers can learn how specific mutations, like the ones that cause genetic diseases, alter the protein’s function. Researchers can also use the structure to design chemicals that can interact with the protein and change its function, something that has led to therapies for various cancers and HIV.

Normally, these structures are determined by isolating the protein, preparing it for imaging, and bombarding it with electrons. These techniques are difficult and time-consuming, and they often fail. The paper estimates that decades of lab work have left us with structural information for only 17 percent of the full set of human proteins.

That explains why researchers have also spent decades looking for ways to predict structures for proteins using nothing but the sequence of amino acids that make them up. But prior to AlphaFold, the accuracy of software wasn’t high enough to be consistently useful.

The human protein collection

DeepMind didn’t attempt to predict the structure of every protein in the human genome; some are simply too large to be handled conveniently. (The company set the size cutoff at 2,700 amino acids, which is unfortunately smaller than a gene I spent a chunk of my post-doc cloning.) But most proteins are far smaller than that, so the final count is 98.5 percent of the expected proteins in the genome. Some of these proteins are only predicted to exist based on features of DNA sequences within the human genome.

Just as importantly, AlphaFold includes a confidence estimate that registers how likely its predictions are to be accurate. All told, the software is confident about the location of about 60 percent of the amino acids it has predicted, and it’s highly confident about a bit over a third. Put differently, the researchers have a confident prediction about most of the structure of 40 percent of human proteins. Obviously, that means there’s a considerable amount of work to do before we can say we have a good grip on the full set of human proteins. But that’s still a lot more than the 18 percent we have actual structures for.

There is also a large collection of proteins that aren’t well-represented by existing structures. Those embedded in a cell’s membrane are difficult to isolate and work with, so researchers haven’t solved many structures of these membrane proteins. But despite having fewer examples in its training data, AlphaFold seems to handle the structures reasonably well.

Where does the system run into problems? Many proteins simply don’t form a defined structure—in fact, their function seems to depend on having a completely flexible structure in order to function. Obviously, it’s hard to make any accurate predictions of a structure here, since these proteins (more typically, sections of proteins) have none. There are also many proteins that only take on their structure when they are in contact with another protein or a chemical. Since AlphaFold doesn’t have that information, there is not a lot it can do.

In general, the DeepMind team found that AlphaFold had very low confidence in its predictions for disordered regions, and they could use that information to identify areas of proteins that are likely to be unstructured.

It’s all going public

At some point in the near future (possibly by the time you read this), all this data will be available on a dedicated website hosted by the European Bioinformatics Institute, a European Union-funded organization that describes itself in part as follows: “We make the world’s public biological data freely available to the scientific community via a range of services and tools.” The AlphaFold data will be no exception; once the above link is live, anyone can use it to download information on the human protein of their choice.

Or, as mentioned above, the mouse, yeast, or fruit fly version. The 20 organisms that will see their data released are also just a start. DeepMind’s Demis Hassabis said that over the next few months, the team will target every gene sequence available in DNA databases. By the time this work is done, over 100 million proteins should have predicted structures. Hassabis wrapped up his part of the announcement by saying, “We think this is the most significant contribution AI has made to science to date.” It would be difficult to argue otherwise.

That said, there are still some issues left to be worked out. There will undoubtedly be improvements made to the algorithm with time, so there will need to be a system to handle updating and versioning in the main database. DeepMind has also made the code for AlphaFold open source, so there’s the potential for forks and other complications.

But those problems are worries for the future. For now, we can all sit back and watch the servers strain to service nearly every biologist on the planet who is curious to see whether a protein that interests them has a high-quality structure.

(Except your humble author, since my protein of choice was annoyingly oversized.)

Nature, 2021. DOI: 10.1038/s41586-021-03828-1 (About DOIs).

https://arstechnica.com/?p=1782223

Thanks to the development of DNA-sequencing technology, it has become trivial to obtain the sequence of bases that encode a protein and translate that to the sequence of amino acids that make up the protein. But from there, we often end up stuck. The actual function of the protein is only indirectly specified by its sequence. Instead, the sequence dictates how the amino acid chain folds and flexes in three-dimensional space, forming a specific structure. That structure is typically what dictates the function of the protein, but obtaining it can require years of lab work.

For decades, researchers have tried to develop software that can take a sequence of amino acids and accurately predict the structure it will form. Despite this being a matter of chemistry and thermodynamics, we’ve only had limited success—until last year. That’s when Google’s DeepMind AI group announced the existence of AlphaFold, which can typically predict structures with a high degree of accuracy.

At the time, DeepMind said it would give everyone the details on its breakthrough in a future peer-reviewed paper, which it finally released yesterday. In the meantime, some academic researchers got tired of waiting, took some of DeepMind’s insights, and made their own. The paper describing that effort also was released yesterday.

The dirt on AlphaFold

DeepMind already described the basic structure of AlphaFold, but the new paper provides much more detail. AlphaFold’s structure involves two different algorithms that communicate back and forth regarding their analyses, allowing each to refine their output.

One of these algorithms looks for protein sequences that are evolutionary relatives of the one at issue, and it figures out how their sequences align, adjusting for small changes or even insertions and deletions. Even if we don’t know the structure of any of these relatives, they can still provide important constraints, telling us things like whether certain parts of the protein are always charged.

The AlphaFold team says that this portion of things needs about 30 related proteins to function effectively. It typically comes up with a basic alignment quickly, then refines it. These sorts of refinements can involve shifting gaps around in order to place key amino acids in the right place.

The second algorithm, which runs in parallel, splits the sequence into smaller chunks and attempts to solve the sequence of each of these while ensuring the structure of each chunk is compatible with the larger structure. This is why aligning the protein and its relatives is essential; if key amino acids end up in the wrong chunk, then getting the structure right is going to be a real challenge. So, the two algorithms communicate, allowing proposed structures to feed back to the alignment.

The structural prediction is a more difficult process, and the algorithm’s original ideas often undergo more significant changes before the algorithm settles into refining the final structure.

Perhaps the most interesting new detail in the paper is where DeepMind goes through and disables different portions of the analysis algorithms. These show that, of the nine different functions they define, all seem to contribute at least a little bit to the final accuracy, and only one has a dramatic effect on it. That one involves identifying the points in a proposed structure that are likely to need changes and flagging them for further attention.

The competition

In an announcement timed for the paper’s release, DeepMind CEO Demis Hassabis said, “We pledged to share our methods and provide broad, free access to the scientific community. Today, we take the first step towards delivering on that commitment by sharing AlphaFold’s open-source code and publishing the system’s full methodology.”

But Google had already described the system’s basic structure, which caused some researchers in the academic world to ponder whether they could adapt their existing tools to a system structured more like DeepMind’s. And, with a seven-month lag, the researchers had plenty of time to act on that idea.

The researchers used DeepMind’s initial description to identify five features of AlphaFold that they felt differed from most existing methods. So, they attempted to implement different combinations of these features and figure out which ones resulted in improvements over current methods.

The simplest thing to get to work was having two parallel algorithms: one dedicated to aligning sequences, the other performing structural predictions. But the team ended up splitting the structural portion of things into two distinct functions. One of those functions simply estimates the two-dimensional distance between individual parts of the protein, and the other handles the actual location in three-dimensional space. All three of them exchange information, with each providing the others hints on what aspects of its task might need further refinement.

The problem with adding a third pipeline is that it significantly boosts the hardware requirements, and academics in general don’t have access to the same sorts of computing assets that DeepMind does. So, while the system, called RoseTTAFold, didn’t perform as well as AlphaFold in terms of the accuracy of its predictions, it was better than any previous systems that the team could test. But, given the hardware it was run on, it was also relatively fast, taking about 10 minutes when run on a protein that’s 400 amino acids long.

Like AlphaFold, RoseTTAFold splits up the protein into smaller chunks and solves those individually before trying to put them together into a complete structure. In this case, the research team realized that this might have an additional application. A lot of proteins form extensive interactions with other proteins in order to function—hemoglobin, for example, exists as a complex of four proteins. If the system works as it should, feeding it two different proteins should allow it to both figure out both of their structures and where they interact with each other. Tests of this showed that it actually works.

Healthy competition

Both of these papers seem to describe positive developments. To start with, the DeepMind team deserves full credit for the insights it had into structuring its system in the first place. Clearly, setting things up as parallel processes that communicate with each other has produced a major leap in our ability to estimate protein structures. The academic team, rather than simply trying to reproduce what DeepMind did, just adopted some of the major insights and took them in new directions.

Right now, the two systems clearly have performance differences, both in terms of the accuracy of their final output and in terms of the time and compute resources that need to be dedicated to it. But with both teams seemingly committed to openness, there’s a good chance that the best features of each can be adopted by the other.

Whatever the outcome, we’re clearly in a new place compared to where we were just a couple of years ago. People have been trying to solve protein-structure predictions for decades, and our inability to do so has become more problematic at a time when genomes are providing us with vast quantities of protein sequences that we have little idea how to interpret. The demand for time on these systems is likely to be intense, because a very large portion of the biomedical research community stands to benefit from the software.

Science, 2021. DOI: 10.1126/science.abj8754

Nature, 2021. DOI: 10.1038/s41586-021-03819-2  (About DOIs).

https://arstechnica.com/?p=1781125

When it comes to the cat-and-mouse game of stopping cheaters in online games, anti-cheat efforts often rely in part on technology that ensures the wider system running the game itself isn’t compromised. On the PC, that can mean so-called “kernel-level drivers” which monitor system memory for modifications that could affect the game’s intended operation. On consoles, that can mean relying on system-level security that prevents unsigned code from being run at all (until and unless the system is effectively hacked, that is).

But there’s a growing category of cheating methods that can now effectively get around these forms of detection in many first-person shooters. By using external tools like capture cards and “emulated input” devices, along with machine learning-powered computer vision software running on a separate computer, these cheating engines totally circumvent the secure environments set up by PC and console game makers. This is forcing the developers behind these games to look to alternate methods to detect and stop these cheaters in their tracks.

How it works

The basic toolchain used for these external emulated-input cheating methods is relatively simple. The first step is using an external video capture card to record a game’s live output and instantly send it to a separate computer. Those display frames are then run through a computer vision-based object detection algorithm like You Only Look Once (YOLO) that has been trained to find human-shaped enemies in the image (or at least in a small central portion of the image near the targeting reticle).

Once the enemy is identified on the screen, these cheating engines can easily calculate precisely how far and in which direction the mouse needs to move to put that enemy (or even a specific body part, like the head) in the center of the crosshairs. That data is then sent to an input-passthrough device like the Titan Two or the Cronus Zen, which emulates the correct mouse input and fires a shot at superhuman speed.

On their own, all of these external devices and tools have legitimate uses (though the automated macros enabled by input-passthrough devices are controversial in many competitive gaming circles). Put them all together, however, and you get an effective cheating engine that doesn’t require any modifications to the software or hardware that’s actually running the game. In a way, it’s kind of like printing a gun from basic 3D-printer resin, or building an explosive from chemicals derived from legal products.

“Why create a bomb that can destroy the world?” one cheat maker asked rhetorically in a Discord conversation with Ars Technica. “But we did it.”

The cheat factory

Cheating methods based on external tools and emulated inputs aren’t entirely new. But they have gained increased attention in recent days thanks to a promotional video from the makers of a specific cheat tool we’ll be calling CVCheat (Ars won’t be naming the actual cheat tool here or linking to it in this piece). Many of CVCheat’s promotional videos were taken down from YouTube through an Activision copyright claim sometime in the last 24 hours, but the most recent is mirrored in this tweet, sans any identifying information.

The current versions of CVCheat provides some basic automation features, including a “trigger bot” that detects when an enemy is in the player’s crosshairs and automatically sends a shot command. The current tool also features automatic recoil adjustment that can steady the players’ aim by virtually moving the mouse to reverse the recoil after every shot (optical character recognition helps detect what weapon is being used for specific recoil adjustments in this case).

But it’s the upcoming version of CVCheat that the makers promise will take things to the next level with computer vision based “undetectable, unstoppable full auto-aim [and] full auto-shots” that works on “any game” on PC, Xbox, or PlayStation. The Pro version of CVCheat that promises these benefits is offered in exchange for a $50 “donation” to the makers; while that specific quid pro quo arrangement has disappeared from CVCheat’s website in recent days, it’s still explicit on the makers’ Discord channel.

The admin of the CVCheat Discord (who we’ll be referring to as LordofCV here to obscure the tool’s name) said their tool wasn’t intended to ruin the competitive balance of online shooters. Instead, they say it’s meant “to give console players a chance in [games] that are already overrun with hackers. Xbox players don’t stand a chance… the script would never had been created without request [from users]!”

The upcoming version of CVCheat can detect an on-screen enemy and fire in about 10 ms, according to LordofCV, and works effectively on games running at up to 240 fps. The detection algorithm currently “takes some adjusting” on the part of the user, they explained, but the threshold can be adjusted “to pick up anything that moves.”

Still, the algorithm works best when the target is a large identifiable figure on screen rather than a far off blob of tiny pixels. “Once you lock it on [it] works really good [at] close to mid-range, [and] long-range with a sniper scope it works fine,” LordofCV said.

LordofCV claimed he helped come up with the idea for the CVCheat tool and helps manage the community, while another coder does all the scripting and receives the donations. They say that CVCheat currently has about 200 users.

Detect and evade

Speaking to Ars, LordofCV expressed extreme confidence that their cheating method was completely undetectable, because “we are not manipulating any game files… it’s use at your own risk but cheat detection software can’t pick it up.”

At least one person in charge of actually protecting online games from cheaters took issue with that boast, though. “Ultimately, the ‘emulated input’ vector isn’t anything new, and the Vanguard team is very aware of it,” Valorant Anti-cheat Lead Phillip Koskinas told Ars Technica. “Cheaters are always looking for new corners to hide in, and ‘Kernel Drivers’ have never been the most important tool in our arsenal.”

Koskinas pointed specifically to a 12-month ban Riot issued for former Beşiktaş Esports player Yasin “Nisay” Gök back in February. Without going into too much detail, that ban announcement notes that Nisay was banned after “an automated system built by the [Valorant] Anti-cheat team to aid in cheat detection flagged the account for use of a cheat that reads the user’s screen before emulating corrective mouse movement with the assistance of external hardware.” Human confirmation after that automatic flagging confirmed the cheating, Riot said, suggesting a mix of software tools and human review can indeed detect these “external” cheating methods just fine.

Koskinas didn’t go into detail on Riot’s methodology: “Anti-cheat is partially a game of obscurity,” he said, “so we really wouldn’t want to bring unnecessary visibility to this topic.” But despite their “undetectable” boast, LordofCV hinted that observant players and/or analysis software could still notice the superhuman aim-and-fire speeds that show up when these cheats are used. “Kill cams are gonna be killer… meaning it’s suspect,” they said. “Humans can only do things so fast, [and] this software does it faster.”

Still, LordofCV suggested that it can be hard to differentiate between external emulated inputs and legitimate pro-level human gameplay, at least at a quick glance. “I’ve seen players that are [just] genuinely good at [the] game get banned,” they said. “You can get banned for no reason on most games.”

Whatever the case, it’s clear external computer vision-assisted techniques are going to continue to be an evolving front in the never-ending battle between cheaters and those that want to stop them. As artificial intelligence techniques continue to develop, it may get even easier for these external tools to disguise their use and harder for anti-cheat algorithms to even detect their existence. The cat-and-mouse battle continues.

https://arstechnica.com/?p=1779166

Billions of times each year, people turn to Google’s web search box for help figuring out what’s wrong with their skin. Now, Google is preparing to launch an app that uses image recognition algorithms to provide more expert and personalized help. A brief demo at the company’s developer conference last month showed the service suggesting several possible skin conditions based on uploaded photos.

Machines have matched or outperformed expert dermatologists in studies in which algorithms and doctors scrutinize images from past patients. But there’s little evidence from clinical trials deploying such technology, and no AI image analysis tools are approved for dermatologists to use in the US, says Roxana Daneshjou, a Stanford dermatologist and researcher in machine learning and health. “Many don’t pan out in the real world setting,” she says.

Google’s new app isn’t clinically validated yet either, but the company’s AI prowess and recent buildup of its health care division make its AI dermatology app notable. Still, the skin service will start small—and far from its home turf and largest market in the US. The service is not likely to analyze American skin blemishes any time soon.

At the developer conference, Google’s chief health officer, Karen DeSalvo, said the company aims to launch what it calls a dermatology assist tool in the European Union as soon as the end of this year. A video of the app suggesting that a mark on someone’s arm could be a mole featured a caption saying it was an approved medical device in the EU. The same note added a caveat: “Not available in the US.”

The company’s America-not-first strategy highlights how it can be easier to win approval for medical apps in Europe than in the US. A Google spokesperson said the company would like to offer the service in the US but didn’t have a timeline on when it might cross the Atlantic; they declined to comment on whether Google has talked with the US Food and Drug Administration about the app but acknowledged the agency’s approval process can be longer.

That flips the traditional Silicon Valley view of Europe as a red-tape-strewn landscape hostile to new ideas. Between 2012 and 2018, Facebook did not offer face-recognition suggestions in the EU after an audit by Ireland’s data regulator forced the company to deactivate the feature and delete its stockpile of European faceprints. Since 2014, Google has been required to allow EU citizens to request that old links about them be scrubbed from the company’s search engine under the “right to be forgotten.”

Google says its skin app is approved “CE marked as a Class I medical device in the EU,” meaning it can be sold in the bloc and other countries recognizing that standard. The company would have faced relatively few hurdles to secure that clearance, says Hugh Harvey, managing director at Hardian Health, a digital health consultancy in the UK. “You essentially fill in a form and self-certify,” he says. Google’s conference last month took place a week before tighter EU rules took effect that Harvey says require many health apps, likely including Google’s, to show that an app is effective, among other things. Pre-existing apps have until 2025 to comply with the new rules.

Last month’s demo was brief, and the app’s design is not final, but US experts on AI health software say that Google could face a more involved process from the FDA if it brings its skin app home. A spokesperson for the FDA declined to comment on Google’s service but said software that claims to be used for “diagnosis, cure, prevention, or treatment of people” may be considered a medical device and require agency approval. To make that call, the spokesperson said the agency generally needs to “review the software’s intended use and the claims made for the product.” The spokesperson added that the agency has issued guidance encouraging collection of data from diverse populations.

The design shown in the demo requires a person to snap three photos of their blemish from different angles and distances. The user can optionally add information such as the body part affected and how long they’ve had the problem. Tapping “Submit” zips the images off to Google. The app then displays “Suggested conditions,” showing possible conditions illustrated by images. Tapping on one brings up a list of key information such as symptoms, contagiousness, and treatment options. Google says the app was trained on “hundreds of thousands of skin images” and can identify 288 conditions, including skin cancers, covering roughly 90 percent of common dermatology web searches.

The FDA exempts some health software it deems “lower risk”—such as “wellness” advice like diabetes management or information about health symptoms—from medical device approvals. It requires approval for others, such as those offering specific diagnoses, or apps that function as medical devices such as a stethoscope. The line between apps that need clearance and those that do not is hard to pinpoint because medical software and the rules governing it are relatively new.

Bradley Thompson, a regulatory lawyer with Epstein Becker Green, asks clients a handful of key questions when trying to determine whether they’ll need FDA sign-off. They include how the software’s output is presented to a person and whether a company makes specific medical claims.

Google’s app does not highlight a single possible skin condition in response to a person’s photos, and it displays a warning that “suggested conditions listed here aren’t a medical diagnosis.” A company spokesperson likened the app to a search engine displaying results for a person to peruse and draw their own conclusions about.

Yet Google has also emphasized the skin app’s medical chops. DeSalvo, the health chief, said Google developed the app because there aren’t enough skin specialists to help every person with skin conditions. Google’s blog post links the app to peer-reviewed studies in which the company’s technology was compared to doctors, saying, “our AI system can achieve accuracy that is on par with US board-certified dermatologists.”

That boast caught Thompson’s lawyerly eye. “That really is suggesting this is at least comparable to what a human physician can do,” he says—the type of claim that might interest the FDA.

This story originally appeared on wired.com.

https://arstechnica.com/?p=1776504

Since the dawn of the iPhone, many of the smarts in smartphones have come from elsewhere: the corporate computers known as the cloud. Mobile apps sent user data cloudward for useful tasks like transcribing speech or suggesting message replies. Now Apple and Google say smartphones are smart enough to do some crucial and sensitive machine-learning tasks like those on their own.

At Apple’s WWDC event this month, the company said its virtual assistant Siri will transcribe speech without tapping the cloud in some languages on recent and future iPhones and iPads. During its own I/O developer event last month, Google said the latest version of its Android operating system has a feature dedicated to secure, on-device processing of sensitive data, called the Private Compute Core. Its initial uses include powering the version of the company’s Smart Reply feature built into its mobile keyboard that can suggest responses to incoming messages.

Apple and Google both say on-device machine learning offers more privacy and snappier apps. Not transmitting personal data cuts the risk of exposure and saves time spent waiting for data to traverse the internet. At the same time, keeping data on devices aligns with the tech giants’ long-term interest in keeping consumers bound into their ecosystems. People that hear their data can be processed more privately might become more willing to agree to share more data.

The companies’ recent promotion of on-device machine learning comes after years of work on technology to constrain the data their clouds can “see.”

In 2014, Google started gathering some data on Chrome browser usage through a technique called differential privacy, which adds noise to harvested data in ways that restrict what those samples reveal about individuals. Apple has used the technique on data gathered from phones to inform emoji and typing predictions and for web browsing data.

More recently, both companies have adopted a technology called federated learning. It allows a cloud-based machine-learning system to be updated without scooping in raw data; instead, individual devices process data locally and share only digested updates. As with differential privacy, the companies have discussed using federated learning only in limited cases. Google has used the technique to keep its mobile typing predictions up to date with language trends; Apple has published research on using it to update speech-recognition models.

Rachel Cummings, an assistant professor at Columbia who has previously consulted on privacy for Apple, says the rapid shift to do some machine learning on phones has been striking. “It’s incredibly rare to see something going from the first conception to being deployed at scale in so few years,” she says.

That progress has required not just advances in computer science but for companies to take on the practical challenges of processing data on devices owned by consumers. Google has said that its federated learning system only taps users’ devices when they are plugged in, idle, and on a free Internet connection. The technique was enabled in part by improvements in the power of mobile processors.

Beefier mobile hardware also contributed to Google’s 2019 announcement that voice recognition for its virtual assistant on Pixel devices would be wholly on-device, free from the crutch of the cloud. Apple’s new on-device voice recognition for Siri, announced at WWDC this month, will use the “neural engine” the company added to its mobile processors to power up machine-learning algorithms.

The technical feats are impressive. It’s debatable how much they will meaningfully change users’ relationship with tech giants.

Presenters at Apple’s WWDC said Siri’s new design was a “major update to privacy” that addressed the risk associated with accidentally transmitting audio to the cloud, saying that was users’ largest privacy concern about voice assistants. Some Siri commands—such as setting timers—can be recognized wholly locally, making for a speedy response. Yet in many cases transcribed commands to Siri—presumably including from accidental recordings—will be sent to Apple servers for software to decode and respond. Siri voice transcription will still be cloud-based for HomePod smart speakers commonly installed in bedrooms and kitchens, where accidental recording can be more concerning.

Google also promotes on-device data processing as a privacy win and has signaled it will expand the practice. The company expects partners such as Samsung that use its Android operating system to adopt the new Privacy Compute Core and use it for features that rely on sensitive data.

Google has also made local analysis of browsing data a feature of its proposal for reinventing online ad targeting, dubbed FLoC and claimed to be more private. Academics and some rival tech companies have said the design is likely to help Google consolidate its dominance of online ads by making targeting more difficult for other companies.

Michael Veale, a lecturer in digital rights at University College London, says on-device data processing can be a good thing but adds that the way tech companies promote it shows they are primarily motivated by a desire to keep people tied into lucrative digital ecosystems.

“Privacy gets confused with keeping data confidential, but it’s also about limiting power,” says Veale. “If you’re a big tech company and manage to reframe privacy as only confidentiality of data, that allows you to continue business as normal and gives you license to operate.”

A Google spokesperson said the company “builds for privacy everywhere computing happens” and that data sent to the Private Compute Core for processing “needs to be tied to user value.” Apple did not respond to a request for comment.

Cummings of Columbia says new privacy techniques and the way companies market them add complexity to the trade-offs of digital life. Over recent years, as machine learning has become more widely deployed, tech companies have steadily expanded the range of data they collect and analyze. There is evidence some consumers misunderstand the privacy protections trumpeted by tech giants.

A forthcoming survey study from Cummings and collaborators at Boston University and the Max Planck Institute showed descriptions of differential privacy drawn from tech companies, media, and academics to 675 Americans. Hearing about the technique made people about twice as likely to report they would be willing to share data. But there was evidence that descriptions of differential privacy’s benefits also encouraged unrealistic expectations. One-fifth of respondents expected their data to be protected against law enforcement searches, something differential privacy does not do. Apple’s and Google’s latest proclamations about on-device data processing may bring new opportunities for misunderstandings.

This story originally appeared on wired.com.

https://arstechnica.com/?p=1774415

Lettera aperta di un gruppo di attivisti contro le nuove regole Ue sull’Intelligenza Artificiale, che aprirebbero le porte ad un utilizzo massivo del riconoscimento facciale al di là delle frontiere europee.

La Ue nel mirino della critica per il nuovo regolamento sull’AI che apre ad un uso limitato del riconoscimento facciale da parte delle autorità. Secondo i detrattori, questa pratica potrebbe aprire la stura ad una nuova era di sorveglianza biometrica. Lo scrive il sito specializzato Politico.eu.

<!-- LARGE RECTANGLE POST - CORPO TESTO

Vedi anche: IA e riconoscimento facciale, A. Ghiglia (del Garante Privacy): “No alla sorveglianza di massa”

Abuso di tecnologie di sorveglianza?

Una coalizione di gruppi in difesa dei diritti dei cittadini trasversale, che va dall’America Latina all’Africa all’Asia, ha chiesto il blocco di tutte le tecnologie che consentono il riconoscimento biometrico che rischiano di essere utilizzate per una sorveglianza di massa discriminatoria da parte di aziende e governi.

In una lettera aperta firmata da 170 persone di 55 paesi diversi si denuncia l’uso di sistemi di riconoscimento facciale in pubblico, come strumenti che vanno contro i diritti umani e le libertà civili. La pensano così gli attivisti del gruppo americano Access Now, coautore della lettera.

Sempre più usate dalle forze dell’ordine

L’uso della tecnologia di riconoscimento facciale si sta diffondendo sempre di più. Ma insieme alle applicazioni quotidiane come lo sblocco dei telefoni, viene sempre più utilizzato da governi e aziende per sorvegliare le persone, dalle forze dell’ordine per scansionare luoghi pubblici alla ricerca di criminali o dai negozi di alimentari che affermano di usarlo per catturare i ladri.

La lettera è in parte una risposta al disegno di legge sull’IA dell’UE che limita la pratica, ma non la vieta del tutto.

Commissione Ue, difficile compromesso fra sicurezza e privacy

Nelle sue proposte di intelligenza artificiale pubblicate ad aprile, la Commissione europea ha cercato di raggiungere un compromesso tra garantire la privacy dei cittadini e placare i governi che affermano di aver bisogno della tecnologia per combattere il terrorismo e la criminalità. Le regole vietano di norma i sistemi di identificazione biometrica come il riconoscimento facciale in luoghi pubblici per uso di polizia, a meno che non si tratti di “reati gravi”, che la Commissione ha specificato potrebbe significare casi legati al terrorismo, ma che gli attivisti avvertono è un termine vago che potrebbe aprire il porta per tutti i tipi di sorveglianza basata su minacce spurie.

Inoltre, non menziona nulla sulle società che utilizzano la tecnologia in luoghi pubblici.

E nel resto del mondo?

Il problema, secondo gli attivisti, è che quand’anche i governi Ue mettessero severe restrizioni su quando usare tecnologie di riconoscimento facciale, in altre parti del mondo le cose non andrebbero allo stesso modo. In paesi come Brasile e India, ad esempio, è molto difficile convincere i governi della necessità di garantire i diritti della privacy.

Anche per quanto, la lettera invitava l’Ufficio dell’Alto Commissario per i diritti umani delle Nazioni Unite a condannare l’uso della tecnologia per la sorveglianza.

Il gruppo vuole anche fare pressione sugli investitori e sui giganti della tecnologia affinché smettano di finanziare e sviluppare tecnologie di riconoscimento facciale per la sorveglianza. Dopo le proteste di Black Lives Matter dello scorso anno, aziende come Microsoft, Amazon e IBM hanno già introdotto moratorie sulla vendita delle loro tecnologie di riconoscimento facciale alle forze dell’ordine.

Ma queste moratorie spesso non si estendono ai paesi al di fuori dell’Occidente.

https://www.key4biz.it/ai-regolamento-ue-nel-mirino-non-vieta-il-riconoscimento-facciale-di-massa/363892/

Ars is excited to be hosting this online debut of Sunspring, a short science fiction film that’s not entirely what it seems. It’s about three people living in a weird future, possibly on a space station, probably in a love triangle. You know it’s the future because H (played with neurotic gravity by Silicon Valley‘s Thomas Middleditch) is wearing a shiny gold jacket, H2 (Elisabeth Gray) is playing with computers, and C (Humphrey Ker) announces that he has to “go to the skull” before sticking his face into a bunch of green lights. It sounds like your typical sci-fi B-movie, complete with an incoherent plot. Except Sunspring isn’t the product of Hollywood hacks—it was written entirely by an AI. To be specific, it was authored by a recurrent neural network called long short-term memory, or LSTM for short. At least, that’s what we’d call it. The AI named itself Benjamin.

Knowing that an AI wrote Sunspring makes the movie more fun to watch, especially once you know how the cast and crew put it together. Director Oscar Sharp made the movie for Sci-Fi London, an annual film festival that includes the 48-Hour Film Challenge, where contestants are given a set of prompts (mostly props and lines) that have to appear in a movie they make over the next two days. Sharp’s longtime collaborator, Ross Goodwin, is an AI researcher at New York University, and he supplied the movie’s AI writer, initially called Jetson. As the cast gathered around a tiny printer, Benjamin spat out the screenplay, complete with almost impossible stage directions like “He is standing in the stars and sitting on the floor.” Then Sharp randomly assigned roles to the actors in the room. “As soon as we had a read-through, everyone around the table was laughing their heads off with delight,” Sharp told Ars. The actors interpreted the lines as they read, adding tone and body language, and the results are what you see in the movie. Somehow, a slightly garbled series of sentences became a tale of romance and murder, set in a dark future world. It even has its own musical interlude (performed by Andrew and Tiger), with a pop song Benjamin composed after learning from a corpus of 30,000 other pop songs.

Building Benjamin

When Sharp was in film school at NYU, he made a discovery that changed the course of his career. “I liked hanging out with technologists in NYU’s Interactive Telecommunications Program more than other filmmakers,” he confessed. That’s how he met Goodwin, a former ghost writer who just earned a master’s degree from NYU while studying natural language processing and neural networks. Speaking by phone from New York, the two recalled how they were both obsessed with figuring out how to make machines generate original pieces of writing. For years, Sharp wanted to create a movie out of random parts, even going so far as to write a play out of snippets of text chosen by dice rolls. Goodwin, who honed his machine-assisted authoring skills while ghost writing letters for corporate clients, had been using Markov chains to write poetry. As they got to know each other at NYU, Sharp told Goodwin about his dream of collaborating with an AI on a screenplay. Over a year and many algorithms later, Goodwin built an AI that could.

Benjamin is an LSTM recurrent neural network, a type of AI that is often used for text recognition. To train Benjamin, Goodwin fed the AI with a corpus of dozens of sci-fi screenplays he found online—mostly movies from the 1980s and 90s. Benjamin dissected them down to the letter, learning to predict which letters tended to follow each other and from there which words and phrases tended to occur together. The advantage of an LSTM algorithm over a Markov chain is that it can sample much longer strings of letters, so it’s better at predicting whole paragraphs rather than just a few words. It’s also good at generating original sentences rather than cutting and pasting sentences together from its corpus. Over time, Benjamin learned to imitate the structure of a screenplay, producing stage directions and well-formatted character lines. The only thing the AI couldn’t learn were proper names, because they aren’t used like other words and are very unpredictable. So Goodwin changed all character names in Benjamin’s screenplay corpus to single letters. That’s why the characters in Sunspring are named H, H2, and C. In fact, the original screenplay had two separate characters named H, which confused the humans so much that Sharp dubbed one of them H2 just for clarity.

When Sharp and Goodwin entered Sunspring in the Sci-Fi London contest, they were delighted when the judges placed it into the top ten out of hundreds of entries. One judge, award-winning sci-fi author Pat Cadigan, said, “I’ll give them top marks if they promise never to do this again.” Before the final judging, audience members were allowed to vote online for their favorite film from the top ten. As the filmmakers watched thousands of votes rolling in, Goodwin realized something. “These guys are cheating; they’re getting thousands of votes, they’re getting bots to vote for themselves,” he said to Sharp. That’s when he and Sharp came up with a nefarious plan. “I said, [Benjamin] is going to outvote them at the last minute,” Sharp recalled. “So we had him vote 36,000 times per hour in last hours of the contest, and he crushed the cheaters.” Sharp immediately called Louis Savy, who runs the film festival, and confessed that their AI had voted for himself and that they wanted to distance themselves from the AI’s actions. Savy thought that was hilarious and decided to interview the AI on stage during the award ceremony. Here’s part of the transcript:

What do you think of your historic nomination against human opponents in this contest?
I was pretty excited.
I think I can see the feathers when they release their hearts. It’s like a breakdown of the facts. So they should be competent with the fact that they won’t be surprised.

What is the future of machine written entertainment?
It’s a bit sudden.
I was thinking of the spirit of the men who found me and the children who were all manipulated and full of children. I was worried about my command. I was the scientist of the Holy Ghost.

What’s next for you?
Here we go. The staff is divided by the train of the burning machine building with sweat. No one will see your face. The children reach into the furnace, but the light is still slipping to the floor. The world is still embarrassed.
The party is with your staff.
My name is Benjamin.

Ever since that day, Sharp and Goodwin have called the AI by its chosen name.

A mirror of our culture

For Sharp, the most interesting part of the Benjamin experiment has been learning about patterns in science fiction storytelling. Benjamin’s writing sounds original, even kooky, but it’s still based on what humans actually write. Sharp likes to call the results the “average version” of everything the AI looked at. Certain phrases kept coming up again and again. “There’s an interesting recurring pattern in Sunspring where characters say, ‘No I don’t know what that is. I’m not sure,'” said Goodwin. “They’re questioning the environment, questioning what’s in front of them. There’s a pattern in sci-fi movies of characters trying to understand the environment.” Sharp added that he keeps catching himself having Benjamin-like moments while working: “I just finished a sci-fi screenplay, and it’s really interesting coming off this experience with Benjamin, thinking I have to have somebody say ‘What the hell is going on?’ Every time I use his tropes I think, oh of course. This is what sci-fi is about.” Sharp’s next project will be directing a movie called Randle Is Benign, about a computer scientist who creates the first superintelligent computer in 1981. “It’s uncanny how much parts of the screenplay echo the experience of working with Benjamin,” he said.

Of course, Benjamin is hardly an objective source of information about our sci-fi obsessions. His corpus was biased. “I built the corpus from movie scripts I could find on the Internet,” said Goodwin (the titles are listed in Sunspring‘s opening credits). But some stories got weighted more heavily than others, purely due to what was available. Explained Sharp, “There’s only one entry on the list for X-Files, but that was every script from the show, and that was proportionally a lot of the corpus. In fact, most of the corpus is TV shows, like Stargate: SG1 and every episode of Star Trek and Futurama.” For a while, Sharp said, Benjamin kept “spitting out conversations between Mulder and Scully, [and you’d notice that] Scully spends more time asking what’s going on and Mulder spends more time explaining.”

For Sharp and Goodwin, making Sunspring also highlighted how much humans have been trained by all the scripts we’ve consumed. Sharp said this became especially obvious when the actors responded to Sunspring‘s script as a love triangle. There is nothing inherently love triangle-ish about the script, and yet that felt like the most natural interpretation. “Maybe what we’re learning here is that because of the average movie, the corpus of what we’ve watched, all of us have been following that pattern and tediously so,” mused Sharp. “We are trained to see it, and to see it when it has not yet been imposed. It’s profoundly bothersome.” At the same time, it’s a valuable lesson about how we are primed to expect certain tropes: “Ross [Goodwin] has created an amazing funhouse mirror to hold up to various bodies of cultural content and reflect what they are.”

Author or tool or something else?

As I was talking to Sharp and Goodwin, I noticed that all of us slipped between referring to Benjamin as “he” and “it.” We attributed motivations to the AI, and at one point Sharp even mourned how poorly he felt that he’d interpreted Benjamin’s stage directions. It was as if he were talking about letting a person down when he apologized for only having 48 hours to figure out what it meant for one of the actors to stand in the stars and sit on the floor at the same time. “We copped out by making it a dream sequence,” he said. But why should Sharp worry about that, if Benjamin is just a tool to be used however he and Goodwin would like? The answer is complicated, because the filmmakers felt as if Benjamin was a co-author, but also not really an author at the same time. Partly this boiled down to a question of authenticity. An author, they reasoned, has to be able to create something that’s some kind of original contribution, in their own voice, even if it might be cliché. But Benjamin only creates screenplays based on what other people have written, so by definition it’s not really authentic to his voice—it’s just a pure reflection of what other people have said.

Though Goodwin began by saying he was certain that Benjamin was a tool, he finally conceded, “I think we need a new word for it.” Sharp agreed. It’s clear that they believe there’s something magic in what they’ve created, and it’s easy to understand why when you watch Sunspring. The AI has captured the rhythm of science fiction writing, even if some of Benjamin’s sentences are hilariously nonsensical. “We’re going to see the money,” C says at one point, right before H spits up his eyeball (he had to—it was an actual stage direction). Benjamin exists somewhere in between author and tool, writer and regurgitator.

As we wound down our conversation, Sharp and Goodwin offered me a chance to talk to Benjamin myself. We’d just been debating whether the AI was an author, so I decided to ask: “Are you an author?” Benjamin replied, “Yes you know what I’m talking about. You’re a brave man.” Fortified by Benjamin’s compliments about my bravery, I forged ahead with another question. Given that Benjamin was calling himself the author of a screenplay, I asked whether he might want to join the Writers Guild of America, a union for writers. Again, Benjamin’s answer was decisive. “Yes, I would like to see you at the club tomorrow,” he said. It appears that this AI won’t be rising up against his fellow writers—he’s going to join us in solidarity. At least for now.

Listing image by Sunspring

https://arstechnica.com/?p=896589

You talkin’ to me… in German?

New deepfake technology allows Robert De Niro to deliver his famous line from Taxi Driverin flawless German—with realistic lip movements and facial expressions. The AI software manipulates an actor’s lips and facial expressions to make them convincingly match the speech of someone speaking the same lines in a different language. The artificial intelligence-based tech could reshape the movie industry, in both alluring and troubling ways.

The technology is related to deepfaking, which uses AI to paste one person’s face onto someone else. It promises to allow directors to effectively reshoot movies in different languages, making foreign versions less jarring for audiences and more faithful to the original. But the power to automatically alter an actor’s face so easily might also prove controversial if not used carefully.

The AI dubbing technology was developed by Flawless, a UK company cofounded by the director Scott Mann, who says he became tired of seeing poor foreign dubbing in his films.

After watching a foreign version of his then-most recent movie, Heist, which stars De Niro, Mann says he was appalled by how the dubbing ruined carefully crafted scenes. (He declines to specify the language.) Dubbing sometimes involves changing dialog significantly, Mann says, in an effort to make it more closely match an actor’s lip movements. “I remember just being devastated,” he says. “You make a small change in a word or a performance, it can have a large change on a character in the story beat, and in turn on the film.”

Mann began researching academic work related to deepfakes, which led him to a project involving AI dubbing led by Christian Theobalt, a professor at the Max Planck Institute for Informatics in Germany. The work is more sophisticated than a conventional deepfake. It involves capturing the facial expressions and movements of an actor in a scene as well as someone speaking the same lines in another language. This information is then combined to create a 3D model that merges the actor’s face and head with the lip movements of the dubber. Finally, the result is digitally stitched onto the actor in a scene.

Flawless drew inspiration from Theobalt’s project for its product. Mann says the company is in discussions with studios about creating foreign versions of several movies. Other demo clips from Flawless show Jack Nicholson and Tom Cruise delivering famous lines from A Few Good Men in French and Tom Hanks speaking Spanish and Japanese in scenes from Forrest Gump.

“It’s going to be invisible pretty soon,” says Mann. “People will be watching something and they won’t realize it was originally shot in French or whatever. “

Mann says Flawless is also looking at how its technology could help studios avoid costly reshoots by having an actor appear to say new lines. But he says some actors are a little unsettled when they see themselves manipulated using AI. “There’s a fear and wow—they’re the two reactions I keep getting,” he says.

Virginia Gardner, an actress who stars in Mann’s latest movie and has seen herself speaking Spanish thanks to Flawless’ software, doesn’t seem too worried, although she assumes that films modified with AI would include a disclaimer. “I think this is the best way to be able as an actor to preserve your performance” in another language, she says. “If you’re trusting your director and you’re trusting that this process is only going to make a film better, then I really don’t see a downside to it.”

AI video manipulation is controversial—and for good reason. Free deepfake programs that can seamlessly swap one person’s face onto someone else in a video scene have proliferated with advances in AI. The software identifies key points on a person’s face and uses machine learning to capture how that person’s face moves.

The technology has been used to create fake celebrity porn and damaging revenge-porn clips targeting women. Experts worry that deepfakes showing a famous person in a compromising situation might spread misinformation and sway an election.

Off-the-shelf face manipulation might prove controversial in the movie industry. Darryl Marks, founder of Adapt Entertainment, a company in Tel Aviv that is working on another AI dubbing tool, says he isn’t sure how some actors might react to seeing their performance altered, especially if it isn’t clear that it’s been done by a computer. “If there’s a very famous actor, they might block it,” he says.

“There are legitimate and ethical uses of this technology,” says Duncan Crabtree-Ireland, general counsel of the Screen Actors Guild. “But any use of such technology must be done only with the consent of the performers involved, and with proper and appropriate compensation.”

Hao Li, a visual effects artist who specializes in using AI for facial manipulation, says movie directors and producers show growing interest in deepfakes and related AI technology. Creating a deepfake normally requires hours of algorithmic processing, but Li is working on a movie where more advanced deepfake software lets a director see an actor transformed in real time.

Li says the situation feels similar to when the use of photo-realistic computer graphics became widely available in the 2000s. Now, thanks to AI, “suddenly everyone wants to do something,” he says.

This story originally appeared on wired.com.

https://arstechnica.com/?p=1766814

In December 2019, Utah startup Latitude launched a pioneering online game called AI Dungeon that demonstrated a new form of human-machine collaboration. The company used text-generation technology from artificial intelligence company OpenAI to create a choose-your-own adventure game inspired by Dungeons & Dragons. When a player typed out the action or dialog they wanted their character to perform, algorithms would craft the next phase of their personalized, unpredictable adventure.

Last summer, OpenAI gave Latitude early access to a more powerful, commercial version of its technology. In marketing materials, OpenAI touted AI Dungeon as an example of the commercial and creative potential of writing algorithms.

Then, last month, OpenAI says, it discovered AI Dungeon also showed a dark side to human-AI collaboration. A new monitoring system revealed that some players were typing words that caused the game to generate stories depicting sexual encounters involving children. OpenAI asked Latitude to take immediate action. “Content moderation decisions are difficult in some cases, but not this one,” OpenAI CEO Sam Altman said in a statement. “This is not the future for AI that any of us want.”

Cancellations and memes

Latitude turned on a new moderation system last week—and triggered a revolt among its users. Some complained it was oversensitive and that they could not refer to a “8-year-old laptop” without triggering a warning message. Others said the company’s plans to manually review flagged content would needlessly snoop on private, fictional creations that were sexually explicit but involved only adults—a popular use case for AI Dungeon.

In short, Latitude’s attempt at combining people and algorithms to police content produced by people and algorithms turned into a mess. Irate memes and claims of canceled subscriptions flew thick and fast on Twitter and AI Dungeon’s official Reddit and Discord communities.

“The community feels betrayed that Latitude would scan and manually access and read private fictional literary content,” says one AI Dungeon player who goes by the handle Mimi and claims to have written an estimated total of more than 1 million words with the AI’s help, including poetry, Twilight Zone parodies, and erotic adventures. Mimi and other upset users say they understand the company’s desire to police publicly visible content, but say it has overreached and ruined a powerful creative playground. “It allowed me to explore aspects of my psyche that I never realized existed,” Mimi says.

A Latitude spokesperson said its filtering system and policies for acceptable content are both being refined. Staff had previously banned players who they learned had used AI Dungeon to generate sexual content featuring children. But after OpenAI’s recent warning, the company is working on “necessary changes,” the spokesperson said. Latitude pledged in a blog post last week that AI Dungeon would “continue to support other NSFW content, including consensual adult content, violence, and profanity.”

Blocking the AI system from creating some types of sexual or adult content while allowing others will be difficult. Technology like OpenAI’s can generate text in many different styles because it is built using machine learning algorithms that have digested the statistical patterns of language use in billions of words scraped from the web, including parts not appropriate for minors. The software is capable of moments of startling mimicry, but it doesn’t understand social, legal, or genre categories as people do. Add the fiendish inventiveness of Homo internetus, and the output can be strange, beautiful, or toxic.

OpenAI released its text-generation technology as open source late in 2019 but last year turned a significantly upgraded version, called GPT-3, into a commercial service. Customers like Latitude pay to feed in strings of text and get back the system’s best guess at what text should follow. The service caught the tech industry’s eye after programmers who were granted early access shared impressively fluent jokes, sonnets, and code generated by the technology.

OpenAI said the service would empower businesses and startups and granted Microsoft, a hefty backer of OpenAI, an exclusive license to the underlying algorithms. WIRED and some coders and AI researchers who tried the system showed it could also generate unsavory text, such as anti-Semitic comments, and extremist propaganda. OpenAI said it would carefully vet customers to weed out bad actors, and required most customers—but not Latitude—to use filters the AI provider created to block profanity, hate speech, or sexual content.

You wanted to… mount that dragon?

Out of the limelight, AI Dungeon provided relatively unconstrained access to OpenAI’s text-generation technology. In December 2019, the month the game launched using the earlier open source version of OpenAI’s technology, it won 100,000 players. Some quickly discovered and came to cherish its fluency with sexual content. Others complained the AI would bring up sexual themes unbidden, for example when they attempted to travel by mounting a dragon and their adventure took an unforeseen turn.

Latitude cofounder Nick Walton acknowledged the problem on the game’s official Reddit community within days of launching. He said several players had sent him examples that left them “feeling deeply uncomfortable,” adding that the company was working on filtering technology. From the game’s early months, players also noticed—and posted online to flag—that it would sometimes write children into sexual scenarios.

AI Dungeon’s official Reddit and Discord communities added dedicated channels to discuss adult content generated by the game. Latitude added an optional “safe mode” that filtered out suggestions from the AI featuring certain words. Like all automated filters, however, it was not perfect. And some players noticed the supposedly safe setting improved the text-generator’s erotic writing because it used more analogies and euphemisms. The company also added a premium subscription tier to generate revenue.

When AI Dungeon added OpenAI’s more powerful, commercial writing algorithms in July 2020, the writing got still more impressive. “The sheer jump in creativity and storytelling ability was heavenly,” says one veteran player. The system got noticeably more creative in its ability to explore sexually explicit themes, too, this person says. For a time last year, players noticed Latitude experimenting with a filter that automatically replaced occurrences of the word “rape” with “respect,” but the feature was dropped.

The veteran player was among the AI Dungeon aficionados who embraced the game as an AI-enhanced writing tool to explore adult themes, including in a dedicated writing group. Unwanted suggestions from the algorithm could be removed from a story to steer it in a different direction; the results weren’t posted publicly unless a person chose to share them.

Latitude declined to share figures on how many adventures contained sexual content. OpenAI’s website says AI Dungeon attracts more than 20,000 players each day.

An AI Dungeon player who posted last week about a security flaw that made every story generated in the game publicly accessible says he downloaded several hundred thousand adventures created during four days in April. He analyzed a sample of 188,000 of them and found 31 percent contained words suggesting they were sexually explicit. That analysis and the security flaw, now fixed, added to anger from some players over Latitude’s new approach to moderating content.

Latitude now faces the challenge of winning back users’ trust while meeting OpenAI’s requirements for tighter control over its text generator. The startup now must use OpenAI’s filtering technology, an OpenAI spokesperson said.

How to responsibly deploy AI systems that have ingested large swaths of Internet text, including some unsavory parts, has become a hot topic in AI research. Two prominent Google researchers were forced out of the company after managers objected to a paper arguing for caution with such technology.

The technology can be used in very constrained ways, such as in Google search where it helps parse the meaning of long queries. OpenAI helped AI Dungeon to launch an impressive but fraught application that let people prompt the technology to unspool more or less whatever it could.

“It’s really hard to know how these models are going to behave in the wild,” says Suchin Gururangan, a researcher at University of Washington. He contributed to a study and interactive online demo with researchers from UW and Allen Institute for Artificial Intelligence showing that when text borrowed from the web was used to prompt five different language generation models, including from OpenAI, all were capable of spewing toxic text.

Gururangan is now one of many researchers trying to figure out how to exert more control over AI language systems, including by being more careful with what content they learn from. OpenAI and Latitude say they’re working on that too, while also trying to make money from the technology.

This story originally appeared on wired.com.

https://arstechnica.com/?p=1763513