The Ukrainian government has intensified efforts to keep people connected to the Internet while Russian invaders try to destroy the communications infrastructure that keeps it going.
Ukrainian armed forces have taken control of civil communications infrastructure in regions where they are at battle with the invading Russian army. Elsewhere, Ukrainian security services and civil administrations have become dependent on mobile communications to muster the country's response to Russian attacks.
Russian forces have been trying to dismantle mobile phone cell communications towers in cities they have taken, according to Ukrainian state sources. Meanwhile, Russian artillery fire has been destroying network cables and knocking communications out across entire cities and regions. Ukrainian engineers have been going out when the shelling stops to make repairs and restore services.
The unsung heroes
Alla Struchaieva, chief executive of LLC MaxNet, an Internet service provider that operates one of the primary European Internet communications exchanges in Karkhiv, Ukraine’s second largest city, where intense Russian bombing has caused most of its 1.8 million inhabitants to flee, said some of her engineers had chosen to stay to keep the data center running.
Struchaieva told Data Center Knowledge that she had helped their families escape to regions of Ukraine that were still safe from shelling.
"Our guys go out to repair the network despite the fact that it is life-threatening," she wrote over a secure Telegram chat connection. "This is incredibly difficult. Firing and bombardment continues for days on end in Kharkiv.”
"We need to work. A lot depends on it now. We are not just a data center. We are a telecoms operator. Internet, telephone and TV," she said.
On Tuesday, when it seemed safe to go out again after last week's bombing in central Kharkiv, MaxNet engineers were repairing torn Internet cables that had knocked communications out at their data center and surrounding streets.
They took pictures and video of the bombed street where they did the repairs outside the Karkhiv Regional Administration, showing how a beautiful metropolitan boulevard, Myronosytska Street, had been turned into an apocalyptic scene.
The engineering team repaired network cables between manhole covers, where fire had caused them to fuse. It restored about a third of the network, Struchaieva said. Now they were trying to work out where else they needed to repair. MaxNet suspended services for non-Ukrainians.
The destroyed building beside them was the site of Ukrainian Host, one of the country's largest web hosts, with a clutch of government and corporate customers. It has other data centers, according to its website. Ukrainian Host’s own website was still running this week. It did not respond to a request for comment.
Struchaieva joined the refugees fleeing the city that day, her home having been damaged, along with the home of her parents.
"We had thirteen days without electricity, without heating. Eight days without water, making water out of snow. Five days without food. But the main thing is that we are alive.
"We got in our car and left through Russia, because we couldn't get through Kharkiv. My husband drove the car like a robot for 28 hours without stop," she said.
"We pray for Ukraine and the whole world as this horrible war has affected all of us."
Dmitry Deineka, chief executive of another hosting company with a data center in bombed-out Karhkiv, told Data Center Knowledge last week that he had six technicians and support engineers still there, in shelters with their families.
"Some of my colleagues managed to travel to Western Ukraine. Some others, my relatives, are planning to leave the city. We must help them," he said from his base in Bulgaria.
"Today's strike on the city was 500 meters from one of our nodes, and a kilometer from the data center. As long as there is electricity, connectivity, and the cooling systems are not damaged, I hope the site will be online. But I have no illusions. This could end at any moment.
"But I believe we will push them back behind a new iron curtain. We have a highly motivated army and our local civil defense. I am angry," he said.
Ukraine’s government agencies began working with telcos and local authorities this week to run Internet cables to the basements that people had turned into bomb shelters, in cities under Russian attack in the South, East, and North of the country.
The state is trying to keep Ukrainians connected to government services, and to keep them informed about the war; this is an information war as much as it is a physical one. Russians were broadcasting propaganda in areas they had occupied, to persuade the Ukrainians there to concede to their rule, the Ukrainian Armed Forces said today.
The executive of one telecommunications provider helping get Internet connections to bomb shelters, who asked not to be named, said many residents were using the basements of apartment buildings that already had Internet connections. To make a connection to the basement, they just needed to run an Ethernet cable down.
"It's like connecting an apartment," he said. For shelters below buildings with no connection, whichever local network provider had the closest point of connection did the work of laying the cable from the outside.
"In Ukraine, five to ten providers operate in every city,” he said. "So it's not so difficult".
In Kyiv, Ukraine's capital, the local authority had built a website through which residents could request a connection to their bomb shelter. It had built a mobile phone app that showed residents maps of places to get things like food and medicine. It was also soliciting feedback from residents through the app.
The Ukrainian security services were using mobile phone apps, a chatbot, email, and secure messaging through WhatsApp, Telegram, and Signal, to solicit reports of Russian troop movements, and collect evidence of potential war crimes that could be used to mount a prosecution of Russian forces under international law.
This week, once again, saw a struggle between Russian attempts to destroy communications infrastructure, and Ukrainian attempts to repair it.
Russian damage had cut off the main communication lines to cities they had occupied, the Ukrainian State Special Communications Service said in a statement. Mobile phone network operators were trying to rebuild services in Kharkiv. They had restored services in Chernihiv, a northern city that, like Kharkiv, had been bombed heavily. Kyiv's communications had been partially cut and then restored. Remarkably, three mobile operators had restored services almost completely in Kherson, a city under Russian control in the South. Engineers were working to restore services to other occupied cities.
Three quarters of Ukraine's Internet nodes were still working, the agency said. But the war had done $100bn worth of damage to Ukraine's civil infrastructure, and shutdown half its business, according to one of the Ukrainian president's advisors. The government asked people to ration their mobile phone use, because of the damage done to the network. Since Russians had been destroying Ukraine's television broadcast towers, the government had arranged to send TV signals by satellite and mobile networks.
Meanwhile, Russia has been mounting intense cyber attacks on Ukrainian government web services and infrastructure, according to security agencies. Ukraine responded by lobbying for sanctions to be imposed on Russia at the International Telecommunications Union, the international organization through which countries agree terms of cooperation in communications.