The 70-year-old man from Salzburg, who wasn't named, allegedly provided a Russian intelligence agency with information about Austria's military and political situation, officials said.
"According to our information this activity may have begun in the 1990s and continued until the year 2018," Austrian Chancellor Sebastian Kurz told reporters. "That means there's a case of espionage here."
Kurz said the Russian charge d'affaires in Vienna was summoned to the foreign ministry.
"At the moment we're demanding transparent information from the Russian side," he added.
In Moscow, Russian Foreign Minister Sergey Lavrov accused Austria of "microphone diplomacy" for making what he called "public accusations and demanding an explanation over an issue we know nothing about."
"If a country has concerns or suspicions about another country's actions and its alleged involvement seen as a threat, it should directly ask for an explanation in line with the international law," said Lavrov.
The Russian Foreign Ministry later said it summoned the Austrian ambassador to tell him that "Vienna's steps based on suspicions not backed by any proof already have exacerbated our previously positive ties."
Austria's Defense Minister Mario Kunasek said Vienna was warned several weeks ago by a friendly European intelligence agency about the suspected spy.
Kunasek said the colonel in question retired five years ago. He said the man, whose electronic devices, including a laptop, were being examined, had told investigators that his handlers were interested in weapons systems, recent migration to Austria and information about Austrian personalities.
Salzburg prosecutors, who are handling the case, declined to say whether the man is in custody.
"Whether this was an isolated case can't be said at the moment," said Kunasek.
Kurz, the Austrian chancellor, said that based on the information available so far "we can indeed assume at the moment that the suspicion (against the colonel) will be confirmed."
If that occurs, the case in Austria and recent allegations of Russian espionage in the Netherlands "don't improve relations between Russia and the European Union," said Kurz, whose country holds the rotating presidency of the 28-nation bloc.
He added that Austria's foreign minister, Karin Kneissl, has canceled a planned trip to Russia. Kneissl was criticized by Austria's opposition for inviting Russian President Vladimir Putin to her wedding in August.
Vladimir Isachenkov contributed from Moscow.
Copyright 2019 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.