Drowning in data

That recent successful experiment to image a black hole produced an enormous amount of data.

It all had to be shipped to the processing facility; there was far too much to send over the Internet. It had to be shipped on boxes of hard drives. Then it all had to be analyzed.

It was a huge task. Astronomy has not always been like this.

Not that long ago, an observing session at a large telescope would involve whole nights making sure the telescope operated properly and stayed properly pointed at its target.

The result would be one or two images, a computer data file, or in some cases, just columns of numbers in a notebook. You would take the whole lot home to analyze.

If your attention slipped at all during the observations, the result could be an entire observing session wasted.

Today things are easier. There is no need to sit there adjusting telescope tracking all night, freezing in an unheated telescope dome.

Modern telescopes look after themselves and the instruments we attach to them have improved immeasurably.

Astronomical images may require hours of exposure to collect the required amount of light. A radio image made using our Synthesis Radio Telescope requires around 10 days of data collection.

Today, we can record the raw data as it is collected, and make images or otherwise process it afterwards. If we make mistakes we can go back to the original information, and start over.

That raw data can be archived and be made available to other researchers.

The latest generation of optical and radio telescopes is raising a new problem.

We want to keep recording the raw data, but it is coming out of the telescopes at such a rate that no current data handling system can handle it.

Even it if were somehow successfully recorded, analysis would be another serious challenge.

Two examples are the CHIME (Canadian Hydrogen Intensity Mapping Experiment) now operating at our observatory, and even more so, the Square Kilometre Array, the largest radio telescope in the world, which will generate a tsunami of data.

The only way we can deal with this is to have a computer program filter and partially process the data before it is recorded.

This raises a really nasty issue. How does such a program decide what is important and should be kept, as opposed to what should be let go?

Although new instruments almost inevitably yield new discoveries, they are primarily developed to address known problems of high scientific interest.

If we have to make a pre-processor program to weed out the uninteresting stuff from the raw data, how can we make sure it stays open minded? It is so easy to build in our prejudices, making it search most assiduously for what we expect to find.

What we really want is something to look at the data and to call our attention to anything at all that looks odd or stands out in some way.

Not long ago such a program would have been a science fiction dream; now it is not.  Artificial intelligence (AI) has arrived.

We don't tell our robot observing assistant what we are looking for. Using new concepts like "computer learning" and "neural networks" we just let the program look at huge amounts of past data.

Without telling it anything, it can identify patterns, what usually turns up in the data, learn about things that can corrupt it, and then report what stands out or what most closely fits what we ask it for.

This will revolutionize our ability to squeeze information out of huge amounts of data. When observational discoveries are being made using AI, how long it will be before the AI becomes a co-author on the published articles?

  • Mars lies very low, getting lost in sunset glow as it approaches the other side of the sun.
  • Jupiter, shining like a searchlight, rises around midnight
  • Saturn is up at 1 a.m.
  • The moon will reach Last Quarter on the 26th.


Comments are pre-moderated to ensure they meet our guidelines. Approval times will vary. Keep it civil, and stay on topic. If you see an inappropriate comment, please use the ‘flag’ feature. Comments are the opinions of the comment writer, not of Castanet. Comments remain open for one day after a story is published and are closed on weekends. Visit Castanet’s Forums to start or join a discussion about this story.

More Skywatching articles

About the Author

Ken Tapping is an astronomer born in the U.K. He has been with the National Research Council since 1975 and moved to the Okanagan in 1990.  

He plays guitar with a couple of local jazz bands and has written weekly astronomy articles since 1992. 

Tapping has a doctorate from the University of Utrecht in The Netherlands.

[email protected]

The views expressed are strictly those of the author and not necessarily those of Castanet. Castanet does not warrant the contents.

Previous Stories