DaaSle, Inc., is a trusted technology consulting and management firm delivering successful solutions for our clients by helping companies select, acquire and manage their technology needs and allowing them to focus on their core business. DaaSle offers technology products, Cloud and Managed Services including building and supporting private, public, and hybrid-cloud solutions on site or at our secure international service delivery centers.
Click here for our InfoGraphic - made with Microsoft Sway!
Please enjoy the following article by DaaSle Tech Team:
An ounce of prevention? It’s worth a great deal more than a pound of cure when it comes to your corporate information systems.
Old adages stick around for a reason; they are usually filled with profound common sense. I feel compelled to share some experiences from the past few weeks that this particular gem fits like a glove. But first, let’s set the context.
We are DaaSle. We are in the IT business to bring value to our clients. We are an elite group of experienced technical ninjas, adept at the art of engineering new and innovative ways to meet IT challenges. Our solutions are comprised of technologies from leading vendors and our overall process is designed to increase revenue, avoid and/or reduce cost, leverage our clients existing IT investments, minimize risk and ensure alignment of IT to achieve business goals.
One of our marquee services is the design and implementation of on-premises, cloud-based and hybrid data protection systems (backups, for short) and as a result, we have some experience in helping our customers recover their data when disaster strikes.
Now, on with the story… In the last month we have spent a lot of human brain cycles getting 3 “new” customers back into operation. The reason I say human brain cycles is that I don’t think AI is going to be able to solve this problem any time soon – AI has no “gut feeling”, no intuition. It can be a big help when it comes to scouring petabytes of data in search of valuable patterns, but it will be a while till an AI can match the creative energy and overall synergy a team of experienced humans engaged in a collaborative effort and working toward a common goal. So, a quick tip of the hat to the leadership of the CIO’s involved, the helpfulness of the vendors involved and the tenacity and hard work of the client IT staff and DaaSle ninja’s. Your efforts literally saved three companies, kudos to you all!
The following three cases have one thing in common: they were all avoidable situations.
I got a call at 7am on Thursday morning, when the tired CIO informed me he needed immediate IT support, since his financial application was “down”. He indicated he and his IT person had been up all night, since, without a place to enter orders and create invoices, their business was effectively shut-down. He had been on manufacturer tech support, escalated tickets and all the basic checklists had been exhausted. We arranged a quick web conference call, and soon had our ninja’s trouble-shooting along-side this customer for most of the day and half of the next. This could have easily been an RPE (resume producing event) for the CIO, so we worked as part of his team until we were able to restore the financial system. While he didn’t lose his job, this CIO lost 2 full days of production, 2 nights of sleep and his credibility with management and the user base. We are currently in the process of designing and implementing a remote fail over configuration for his critical applications, off-site replication of his entire VMware infrastructure, a well-designed backup system and a DR Plan.
Similar call at 9pm on a Monday night, this client had experienced a system crash that had resulted in corrupt and missing data as well as their primary file server (the life-blood of their business) being unable to boot. Users were complaining about slow and/or no response from several other critical business systems and in the process of attempting to recover they found that both local and off-site backups were corrupted and mostly unusable. As the client was in close proximity we were able to have 2 ninjas on site by 8am the next day. It was unclear at the initial triage meeting what had happened, when it happened and what the extent of the damage was. We did not know what we did not know.
We spent the rest of the week, the long weekend and a couple of sleepless nights with the client before the root-cause(s) became clear. First, a combination of low-level hardware and firmware configuration errors had been sporadically introducing unnoticed data corruption into their environment for months. Second, a poorly implemented backup scheme had allowed the corruption to infiltrate and effectively destroy the entire chain of local and off-site backup files. And finally, the system crash that brought down their infrastructure was not the cause of the file corruption and data loss, it was actually a symptom! This CIO found himself living his own worst nightmare: extended downtime, facing an imminent RPE, the C-Suite and upper management on his back, users fuming and years of work that could be lost forever. Fortunately, in concert with his staff, our ninja’s were able to find and fix the cause of the problem and restore operations in a timely manner. Due to the cause and the nature of the data corruption, the data recovery effort is still ongoing.
We are now designing and implementing a disaster recovery/business continuity system, which includes off-site replication of all IT systems, enabling fail over and fail back of all business operations to the cloud as well as a backup scheme that complies with the 3-2-1 principle and aligns with RPO’s & RTO’s established by the company.
This client is in the media and entertainment industry and was one of the many that were hit in the global “wannacry” ransomware attack. Their systems were compromised, their files were encrypted and hackers at the other end of the bitcoin account had a stranglehold on their business. Since this customer did not want publicity, they acquiesced and paid the ransom. Of course, the ransom was to be paid in bit coin, which took them a few days to get, because not everyone has 3,600 in bit coin ready to go. Luckily, once the ransom was paid, they were able to decrypt and regain access to their files.
Our ninjas are currently assisting in the company-wide remediation and cleanup effort which includes a “if you’re not 100% sure it’s clean – wipe it out & start over” clause. The project is going to take some time and it will not be cheap, but let’s face it: the last thing the client wants is to find out that 6 months down the road some seemingly benign code tucked away on an infrequently used drive is “calling home” to initiate a new crypto-locking scheme and start the nightmare in motion again.
This was not a fire drill emergency, or a potential RPE as in the previous two examples, but it was definitely a wake-up call. Along with the cleanup effort, we will be fast-tracking a disaster recovery project that was slated for Q4 into this quarter and implementing a zero-day threat management system to watchdog his freshly scrubbed infrastructure. As an added measure of security for the media files that were the focus of the ransomware attack, they are being copied to IBM cloud object storage (ICOS) where the data will be erasure coded, dispersed across multiple geographic locations and encrypted at rest. Think of it as a RAID array that has self-encrypting disks in 3 different data centers that are in 3 different cities. Take that, you bitcoin demanding cyber-criminals!
Do you see the common thread, how all of these situations could have been avoided? Granted, we have no control over such things as rogue actors on the cyber stage, or data disasters caused by nature or human error. However, it is well within our control to prevent the types of disasters I’ve just described and it all starts with a well thought out and properly implemented Disaster Recovery Plan - one that leverages a Cloud Services Provider and takes full advantage of Disaster Recovery as a Service (DRaaS).
Complexity and cost have traditionally kept DR initiatives from being fully realized. Most companies simply can’t afford the money it takes to:
1. rent another building or colo facility
2. buy enough hardware and software to re-create their entire computing environment
3. spend weeks or months of man-hours setting it up and testing it
4. let that entire investment sit idle and maintain it “just in case”
DRaaS eliminates need to build & maintain a secondary site, it uses a cloud providers resources for compute, network and storage and acts as the target site for replication and recovery of your company’s critical systems, data and applications. DRaaS will reduce recovery costs, cut infrastructure investment, limit staffing needs, increase flexibility, simplify testing, speed recovery and improve overall Disaster Recovery efficiency and effectiveness!
DRaaS empowers the CIO to stand tall in the Board Room and explain that an ounce of prevention is “way cheaper” than the cure and then present a dollar amount that proves it. DRaaS leverages the best that the cloud has to offer - against the most important function the IT department has - and provides the CIO with the ability to sleep easy, knowing that they can reliably recover from a disaster and that there are no potential RPE’s on the horizon.
The early bird gets the worm.
Getting up early, usually means getting an early start, getting a good seat on the commute train or if you live in LA, beating the traffic into the office!
In our context I would like to address how we help our customers with zero day threats. Worms are bad actors on computer systems in that they propagate to other systems unbeknown to any basic monitoring systems that may be in place. Worms, virus, ransom ware….it seems our vocabulary is growing faster each millisecond. Bots rule the Internet or so it would seem. We can’t seem to patch fast enough, systems that are on-site, and attackers are persistent and smart. This is where AI helps and we provide this capability.
Detecting and stopping the threats that matter and containing these threats in real time is absolutely a requirement to play on the Internet today. This means monitoring, quarantining and eliminating worms and all malware threats as early as possible.
A bird in the hand is worth two in the bush.
Untested backups are the two in the bush. When we go in and ask customers about their DR Plan, the answer is usually not good. They have copies on a remote server, or some-one copies files on to tape, and then takes them home, or users keep copies on their hard drives of data files only, and system files don’t get backed up. This is the Humpy Dumpty scenario – and putting Humpty back together again after the fall takes time. He does not look quite the same with all those broken eggshells glued together.
The bird in the hand is verified backup of the entire system – virtual and physical machine – and the data, so you are able to quickly resume business operation. One copy of the VM should be kept locally on a separate media platform and one copy kept on cloud to spin up as needed. For good measure, we recommend a periodic full backup into inexpensive online secure object storage, on multiple clouds based on value of data.
You can lead the horse to water, but you can’t make the horse drink.
This is the ‘I told you so’ adage, just doesn’t sound as condescending. IT guys are supposed to have backups. However some IT teams are so caught up on keeping the day-to-day operations going that they may not have time or staffing to make time to solidify the infrastructure and operational processes. For these folks, and because of these three recent incidents, we have started DaaSle Rescue Services for IT. This is an emergency response team that will help you get your systems back in operation, should you have a system failure or cyber attack. We will provide this on a best effort basis, and the service is subject to resource availability.
Failure to plan is planning to fail.
If you have read down this far, and are still not convinced that data protection and security should be the first priority as CIO, then I have one last adage for you – “pay now or pay later “(>16X?), the cost for rescue service for these 3 companies was not as high as it could have been. Seriously, they came close to having their business shut down entirely.
The best laid plans of mice and men.
No matter how carefully a project is planned, something may still go wrong with it. The IT leadership at each of these companies was able to stay focused and leveraged all of the resources that were available and opened up shop the following business day. We were all quite lucky and being prepared helped. CIO leadership, dedication, co-ordination and timely response by the rescue team combined for the positive result, this time. Time now to sharpen the saw, and stay vigilant.
Water off a ducks back.
When cyber-attacked, then for the prepared IT guy, it’s like “water off a ducks back.” In modern data protection schemes, systems can automatically fail-over to alternate data centers or to our cloud. Bad guys get nothing, and IT guy keeps systems running.
Living and learning by these timeless adages is just plain common sense and applicable for us in IT. Getting your backups and security strategy created and implemented is best done immediately since “hope is not a strategy”. To get ahead of the game contact me at this link, we can get your hybrid cloud replicating, and that will be as close as you can get to 100% uptime. Link to DaaSle Service Request Form.
Ok, so that’s all I got today. Thanks for reading, and I hope you are feeling more prepared for the inevitable downtime challenges. Now, from the songwriter Ian Anderson of Jethro Tull fame, it’s time to go “skating away on the thin ice of a new day”.
Written by: James Mal, CEO with Chris Radigan, Director of Systems Engineering, Daasle, Inc.