History of Water/Wastewater SCADA Expenses

Since we suddenly need to spend over $40,000 for yet another SCADA (Supervisory Control and Data Acquisition) computer system for the water/wastewater system, I did some research to find out just what the history of expenses are for this.

Here is what I found going back to 2013 (as far as I was able to search):

August 9th, 2013: City places an ad in the Dispatch asking for proposals and qualified bids for a new SCADA unit. Yes, they put this out for bid back then. The public post was again made in the Dispatch on August 13th, 2013.

Records are incomplete and sketchysee page 105 as my proof. No numbers were given but “Phil Andreas, Public Works Director, explained that proposals were sent out for the installation of a SCADA system for the Electric Department and for an upgrade of the W/WW SCADA system. The proposal costs came in considerably higher than expected. City staff is asking that Council reject the W/WW bids received” [from page 14]

The closet thing to “W/WW SCADA” I can find is at the end of 2013 [page 67] – awarding a $18,252 contract to AWD for “lift station monitoring” and “SCADA additions”. They talk of integrating the equipment into the City’s “existing Wonderware SCADA system”.

July 14, 2017: Council unanimously approves the installation of Supervisory Control and Data Acquisition equipment at the sewage lift station on FM 580 East, just past the Key Avenue intersection. The cost of labor, materials, programming and installation is $25,250 through Trac-N-Trol, the city’s SCADA system contractor. [see page 127 for invoice and explanation]

March 30, 2018: According to the Dispatch: “Council voted 6-0 to replace Supervisory Control and Data Acquisition system equipment at four lift stations”

“In 2013, the city approved a bid for SCADA equipment that did not exactly match the existing SCADA system for the sewage treatment plant and the water system, Water/Wastewater Operations Manager Van Sims said. He said the two systems have not been communicating properly, and he said the only way to fix the problem is to ensure all units are the same brand. The council awarded the equipment replacement project to Trac-N-Trol for $15,900.”

According to the invoice [see page 151], they had to “remove existing Siemens PLC” and “install Allen Bradley PLC” instead. So, rip out all the old stuff that was previously bought and put in new stuff.

[Whoops! Sounds like a pretty big fuckup. Did anyone’s head roll for this? Did the contractor make things good for free? Did the City get any money back for the mistake? I’m guessing not]

August 2019: A mere 17 months after everything was now supposedly running smoothly thanks to $15,900 spent in March of 2018, a ransomware attack was allowed by Monica Wright’s IT Department which rendered the W/WW SCADA unit “inoperable”.

The City (Finley) informed us that the SCADA was “approaching end of life” anyways. Yes, you read that properly: the brand new SCADA that was just installed in April 2018 was now “end of life” in August of 2019.

Ok…if you say so, Finley.

The City then spent ANOTHER $15,990 in September of 2019 to install an ENTRIELY NEW SCADA under the premise that the previous one was “old” and only had Windows 7, when actually they needed Windows 10.

Fun fact: Windows 10 was released way back in 2015 – THREE YEARS before the March 2018 SCADA installation. So it begs the question: why didn’t the idiots at TraC-n-trol install a Windows-10-supported SCADA system in March 2018??

October 2021: Barely two years after the City paid $15,990 (a second time) to upgrade everything to Windows 10-supported SCADA, they are back asking for $40,530 for a new system. This is now supposedly an “upgrade” for an entirely new system.

How long will this one last before disaster strikes?

Don’t get me wrong – water treatment is right up there with law enforcement as an important and necessary function of local government. I’d rather they spend a million dollars on a proper water treatment system than spend $1 on stupid shit like a “business” park or water tanks for a community garden that already has water.

But heads should have rolled big time back in 2019 when a BRAND NEW SYSTEM was wrecked by a ransomware attack. Instead, Finley and Talbert brushed it under the rug as a Windows 10 upgrade and then THANKED the IT Department for their hard work!!!

City Wants $40,530 for ANOTHER SCADA for Wastewater Plant – Spent $16,000 in 2019 for EXACT Same Thing

One of the items on tonight’s agenda is the request for $40,530 to purchase a SCADA for the wastewater plant [page 3, Item 7.6 and also page 207].

This would all be fine and dandy except for one thing. Almost exactly two years ago, the City ALREADY bought a brand new SCADA for $16,000 after the old one was wrecked due to our IT Department allowing ransomware to run amok on the City’s servers.

I’m sure you all remember that. Here is the article I wrote at the time:

Back then, here is how they justified the expense of $16,000 for a new control unit for the water/wastewater plant. This is lifted DIRECTLY from the City council packets [page 131]:

“The current SCADA computer was compromised in the recent ransomware incident and is not operational. While the computer was not planned for replacement in the current fiscal year, staff has been informed that due to the age of the device and the operating system (Windows 7) reaching end of life and no longer supported, it would be more cost effective to replace versus attempting to restore the existing device.

“The bulk of the costs of the replacement device result from a newer version of the SCADA software, which is required due to the update in the operating system to Windows 10

So, they basically made it sound like Monica Wright’s IT Department fuckup was no big deal – because they needed to switch to Windows 10 anyways, right? They pretty much fobbed the whole thing off on Windows 10.

That was a mere two years ago. Here is a screenshot of the invoice for the August 2019 replacement from Trac-N-Trol:

Wow! A three-year warranty and tech support for the life of the system! And the much-ballyhooed upgrade to Windows 10. All for $16,000.

OK, we all know that this was a convenient excuse to cover for the IT Department fuckup, but whatever. The water/wastewater plant is pretty important and it was “only” $16,000. It should last a long time, right?

Wrong.

Tonight, the City will request yet ANOTHER SCADA unit! From tonight’s packet [page 207]:

Trac-n-Trol will provide (for $40,530) the water department with a Windows 10 operating system, burn in and setup including all Microsoft updates,”...blah, blah, blah.

Whoah – sounds like they are using the “Windows 10” bullshit AGAIN. Sorry, but you ALREADY did all that a mere 25 months ago! Not to mention, the price now is more that DOUBLE what it was then! Hell, the invoices even look IDENTICAL!! Here is the screenshot for THIS YEAR’S SCADA:

They are virtually IDENTICAL! WTF? I don’t see any explanation in the packet as to WHY yet ANOTHER SCADA is needed just 24 months after the last one was installed (at less than half the price, I might add).

The only differences I see are very minor. The Xeon W-2123 versus the W-2223 chip – which I looked up, since I’m not a computer chip nerd – looks pretty much the same. The video card (going from the Radeon Pro WX 4100 to the WX 3200) looks WORSE, if you look at THAT comparison!

The hard drive for the NEW one is 512GB while the old one is a terabyte!

So…..the questions are (1) why in hell do we need a new one already and (2) why does the exact same thing cost over $40,000 when it was $16,000 two years ago?

Hell, the one from August 2019 is STILL UNDER WARRANTY!!!!

Hey City council….you paying attention yet?