With a mate and a nest to protect, the male threespined stickleback is a fierce fish, chasing and biting other males until they go away.

Now researchers are mapping the genetic underpinnings of the stickleback’s aggressive behavior. Armed with tools that allow them to see which genes are activated or deactivated in response to social encounters, a team from the University of Illinois has identified broad patterns of gene activity that correspond to aggression in this fish.

A paper describing their work appears in the Proceedings of the Royal Society: Biological Sciences.

“The molecular mechanisms underlying complex behaviors such as aggression are a challenge to study because hundreds of genes are involved, and in order to study them, we have to delve into arguably the most complex tissue: the brain,” said IGB affiliate and Illinois animal biology professor Alison Bell, who led the study.

The researchers looked at brain gene expression – the pattern of genes that were activated or deactivated – across four brain regions in the nesting stickleback fish shortly after it encountered an intruder. They compared the brains of nesting stickleback males that did and did not encounter an intruder, to identify how the experience of fending off a challenger changed gene expression in the brain.

“Territorial aggression and other behaviors in sticklebacks have been well-studied by astute observers of animal behavior for almost a century, but complex behaviors in wild animals require the use of powerful tools to understand them,” Bell said. “Until recently we have not had sophisticated computational and genomics tools to delve into the causes of aggression in real organisms in natural populations.”

The analysis revealed that hundreds of genes were upregulated (activated at higher levels than normal) or downregulated in different regions of the stickleback brain after it encountered an intruder. The upregulated genes were being transcribed and translated into proteins at higher levels to perform specific tasks within the brain.

An analysis of the types of genes that responded when a stickleback male faced an intruder revealed that many molecular and cellular processes were affected. Genes involved in immunity, metabolism and regulation of normal body states were recruited or put to bed. Many of these genes had never before been implicated in studies of aggression or territorial defense, Bell said.

Some of the genes that were downregulated are associated with metabolism and sexual behavior.

“For ages we’ve known that there are costs of aggression for things like immunity, and conflicts between aggression and other functions such as courtship behavior,” Bell said. “This study begins to identify some of the molecular mechanisms that mediate these tradeoffs.”

The greatest changes in gene expression were seen in the diencephalon (a region deep in the brain that is involved in relaying sensory information, emotions and motor signals to other brain regions and helps regulate consciousness, sleep, alertness and circadian rhythm, among other things) and the cerebellum (which receives sensory signals and plays a role in motor coordination).  A significant number of these genes were regulated in opposite directions in these two brain regions (up in one and down in the other), the researchers report.

One gene, which codes for a protein hormone known as CGA, was the most highly upregulated in the diencephalon and the most highly downregulated in the cerebellum. CGA is known to play a role in reproductive changes and is associated with aging in males and females.

“That the same gene was expressed in opposite directions in different brain regions suggests that there are complex patterns of gene regulation,” Bell said.

The researchers also found evidence that some proteins, called transcription factors, which regulate the expression of networks of genes, are regulated differently in different brain regions in response to a territorial threat.

“This suggests that complex transcription regulatory networks are involved in the behavioral response of territorial animals to an intrusion,” Bell said.

The new study offers a glimpse into the regulatory mechanisms that govern brain responses to perceived threats, Bell said.

“It lays the stepping-stones to the ultimate characterization of the neurogenomic states underlying complex decision-making in response to social challenges,” said postdoctoral researcher Yibayiri Osee Sanogo, the lead author on the paper.

“This study shows how computational approaches can help solve complex problems of molecular biology,” said IGB affiliate and computer science professor Saurabh Sinha, a co-author on the study. “Only powerful computational tools – in combination with new approaches in genomics – can begin to address the complexity of the brain and behavior.”