Embed Notice

After a period of relatively restrained handling of "AI" topics, my division at work decided that all the developers, designers, engineers, whatever, ... need to "use AI more in our everyday work". (Oh, joy.) This included a series of workshops designed to introduce everybody to some representative examples.

One workshop involved Github Copilot, and the following things happened to one development team, all senior developers:
- Copilot generated a unit test case that was hard to get to pass.
- When asked to generate empty test cases, Copilot generated the same (irrelevant) code over and over again.
- Copilot stopped giving suggestions to one developer after a while.
- Getting useful information out of Copilot frequently required a lot of fussy or non-obvious prompt editing and tweaking.

I won't supply direct quotes without the explicit consent of the people involved, but there was a very clear general sense that Copilot was not fit for purpose -- even when it did produce something not totally wrong, it was not a useful timesaver for the types of work this team was doing.

It wasn't just Copilot that seemed half baked. The workshop's guidelines (which are themselves part of a fairly polished Github repo) were poorly proofread. One example had a prominent typo in some HTML you were supposed to generate: '<button class=""btn" ...>' (note the extra double-quote). A newbie to web development would very likely add the spurious double quote mark to otherwise ok Copilot output to make sure it matched the instructions.

Finally, our IT department disallows results from Copilot that come from training on "public" code, for what should be fairly obvious legal concerns regarding copyright and similar issues. For one developer, Copilot repeatedly started to generate a result but then stopped, with an alert that the result appears to match known "public" code.

If it wasn't clear before that Copilot's basic mode (no "private code" option) is a copyright-laundering and license-laundering tool, it's really obvious now.