Mercifully for all within earshot, it happened smack in the middle of recorder practice. The day before, the chrysalis had started to darken; now, one of the kids­ distracted from mangling the chorus of “Skip to My Lou”—noticed he could see into it. Where once everything had been black and opaque, now were splashes of orange, and bright yellow spots gleamed through the translucent wall.

He gave a shout, and the cacophony died in a clatter of recorders dropping to the floor. Miss Brentnall didn’t even try to stop us. We crowded about the chrysalis, just as a proboscis unfurled. To us schoolkids, that segue seemed like sorcery.

The idea that something as unlovely as a caterpillar can weave about itself a magic cauldron and emerge two weeks later as a Monarch butterfly demands the suspension of belief. How can one genome possibly build two very different creatures?

Although the butterflies and moths are its most spectacular practitioners, metamor­phosis is more than just ercifully for all Monarchs. Beetles, flies, sawflies, wasps, ants and bees all undergo what we call complete, or holometabolous, metamorphosis, in which an egg hatches into a larva, which eventually goes through a pupal stage before emerging as an adult.

Hemimetabolous insects, on the other hand, practise what’s known as incomplete metamorphosis. They do not pupate: instead, immature stages called nymphs, which mostly resemble much smaller adults, periodically moult into bigger versions of themselves—called instars—on their journey to adulthood.

Among the hemimetabolous insects are cockroaches, crickets and grasshoppers, stick insects, praying mantises, termites, dragonflies and damselflies, earwigs, sucking bugs and parasitic lice. Insects are diverse and multitudinous, and many evolutionary biologists put their success down to meta­morphosis, which lets them exploit a plethora of different habitats and food sources at different stages of their life, then fly off to colonise new regions.

Complete or otherwise, the mechanism of metamorphosis has captivated scientists for centuries. Aristotle puzzled over it, and concluded—more out of expediency than conviction, probably—that a caterpillar was merely a continuation of the embryonic stage. Few came up with anything more plausible, until the 1930s, when Cambridge biologist Sir Vincent Wigglesworth, on an enlightened hunch, conducted a series of lurid experiments.

His unfortunate subject was the South American kissing bug, Rhodnius prolixus, the larvae of which go through five instars before adulthood. When he surgically severed certain blood vessels in their heads, he found that, as long as he performed his excisions before some critical phase, he could stop the larvae moulting and/or pupating.

He even grafted the heads of pre-moult larvae onto the dormant abdomens of others, and found that his transplants had bizarre results. When he beheaded a first-instar Rhod­nius larva and fused it to a moulting fifth-instar one, the tiny first instar developed an adult cuticle, body form and genitalia. This showed that blood-borne hormones are responsible for the induction of metamorphosis.

Thus he concluded that the insect brain directed metamorphosis with hormones carried in the blood, which was pretty close to the truth. We now know that the corpus cardiacum, an organ near—but discrete from—the insect brain, secretes prothoracicotropic hormone, or PTTH.

On certain environmental cues, PTTH directs an arthropod’s prothoracic glands to release ecdysteroids, or moulting hormones, into the haemolymph that passes for insect blood.

Later, Wigglesworth discovered that PTTH also stimulates the corpora allatum, an endocrine gland behind the creature’s brain, to produce another hormone, eventually known as juvenile hormone (JH), which keeps ecdysteroids in check—effectively preventing a headlong rush straight to adulthood. In R. prolixus, JHs prevent metamorphosis into the next instar until the larva is good and ready (with each instar, levels of juvenile hormone decrease), and suppress maturity until the larva has reached the appropriate—fourth instar—stage.

Again, Wigglesworth resorted to decapi­tation and recombination, and found that he could distort Rhodnius development by manipulating JH levels. When he excised the corpora allatum from a third-instar larva, it turned into a precocious adult at the next moult. Conversely, when he implanted the corpora allatum from a fourth-instar larva into a fifth-instar, it moulted into a Franken­stein “sixth-instar” rather than an adult.

Wigglesworth showed us that an insect’s genome can switch hormones on and off, controlling its development and morphology with precise timing.

But there’s more to metamorphosis than insects, or indeed that other classroom classic, the tadpole. It’s also practised by mosses, fungi, and no fewer than 15 known marine phyla. Many familiar fish—eels, salmon, flatfish such as flounder—go through striking physiological changes over their lifetime, and you wouldn’t recognise a larval rock lobster if it swam up to you.

Life began in the oceans, so that’s where the hunt for our common ancestor has led, on a hunch that metamorphosis is trying to tell us something.

In the 1870s, Francis Maitland Balfour, a friend and colleague of Charles Darwin’s, claimed the “primary larval form” of the earliest marine animals would point back to the progenitor of life itself. He spent years poring over the anatomy of marine larvae, trying to draft an evolutionary roadmap, but it was the very diversity and complexity of metamorphosis that thwarted him, as it’s done practically ever since.

The larvae of even closely related species are often strikingly different, which suggests that they’re secondary—not original—forms. The mists of time still obscure the nature of the earliest ancestor, but Balfour concluded that it would, in fact, look very much like a larval jellyfish does today—a simple ball of protoplasm, covered and propelled by minute hairs called cilia. Intriguingly, jellyfish—and their relations the hydra—are among the very earliest known life forms.

Balfour published his theory, but when he died in 1882 in a climbing accident, his ideas followed him into the void. It wasn’t until the 1980s that retired British planktonologist Don Williamson pointed out the same connection. Unaware of Balfour’s work, he sketched out two “trees of life”—one according to larval features, the other based on the physiology of corresponding adults. Like his predecessor, he was struck by some of the anachronisms he found.

“All these anomalies could be explained,” he told his local Isle of Man newspaper, “if larvae trans­ferred between one group of animals and another. It was only in the subsequent two years [that] I worked out it must have been done by hybridisation…by the sperm of one animal and the egg of another.”

Williamson reasoned that such a thing could easily happen at sea, where eggs and sperm are simply ejected in their millions on the off-chance of a specific correspondence. “From time to time, there is every chance eggs could be fertilised by foreign sperm. In most cases, it comes to nothing, but if it happens over millions of years, something will hatch out.”

His theory was greeted with derision, but Williamson was undaunted. Like Wiggles-worth, he performed a series of freakish experiments, crossing marine invertebrates from different phyla. In 1990, he fertilised sea squirt eggs with sperm from a sea urchin. Then in 2002, he reversed the cross. Both inseminations bore offspring, and most of the hybrid larvae duly metamorphosed into glob­ular adults. The first cross resulted in “spheroids” bearing a suction cup, by which they attached to the substrate. More intrigu­ingly, the second produced spheroids (they looked very much like Balfour’s ancestral larva) that reproduced asexually by budding to create a clone.

In a 2006 paper, Williamson proposed that some larva and adults are so very different because they were once, indeed, separate species. Today’s mercurial creatures are hybrids. “People have been trying to find one solu­tion that covers all of metamorphosis,” said Williamson. “I say it’s a change in taxon during development.”

He later published papers which proposed that such crosses may have powered spectacular bursts of evolutionary creativity like the Cambrian explosion, nearly half a billion years ago, which at its height may have boasted 100 animal phyla, as opposed to the 40 or so left living today.

Williamson’s theories were roundly ridiculed, and his papers relentlessly rejected, until 2009, when the Proceedings of the National Academy of Sciences finally accepted, after much peer review, a paper in which Williamson, by then 87 and wheelchair-bound, suggested that the ancestors of modern butterflies were mistakenly fertilised by velvet worms, also known as onychophorans.

On the tree of life, velvet worms lie some­where between worms and insects. They superficially resemble caterpillars, and males place sperm packets not on the female’s genital opening but on her skin, from where the sperm migrates to her ovaries. Thus it was, Williamson believes, that some ancient insect accidentally picked up onycophoran sperm, scribing two separate developmental programmes into the butterfly genome. One directs half the creature’s life as a velvet worm (caterpillar), and the other as a butterfly.

The publication prompted an outcry, and the rebuttals came thick and fast, but subsequent research by others has hinted that Williamson’s theories may yet have some merit. Metamorphosis continues to tantalise with clues to some of biology’s most obdu­rate mysteries.