Month: September 2019

The end of life for Windows 7 and Windows Server 2008 and 2008 R2 is coming January 15, 2020. This means that support, is coming to an end. On this date, these OS’s no longer check certain compliance checkboxes for safe usage.

The biggest impact is going to be on security for the OS itself, but this impacts the whole site. Once these reach end of life, there will be no support whatsoever for critical bugs or massive security holes. That newest zero-day? There’s nothing that can be done about it most likely.

What Should You Do?

You need to be ready so your clients can stay safe. You and your client are sitting on a ticking time bomb if you haven’t started preparing to move already, but at least you have plenty of time to research and get ready, but only if you start as soon as possible. Moving from Windows 7 and Server 2008 (R2) can be difficult, but not with a little bit of planning.

Workstations are easy to get away from for most cases, but servers are a bit harder. This is a great opportunity to also sell the client on newer hardware if you can to get out of the predicament and to outpace the natural inflation of hardware requirements for software. Naturally, this isn’t always possible depending on the client and depending on how new the hardware they’re running is. If the machines have been sitting around since much earlier in the support cycle for Windows 7, your users will probably welcome an upgrade.

What’s Involved?

Licensing and software compatibility are two factors to look into heavily for migrating. A great server from a few years ago can be a huge expense to license a newer version of Windows Server on, but an okay server with a cheaper license (OEM discount, fewer cores, etc.) can even end up cheaper than what you get back selling the old server. SQL Server and Exchange can further convolute the licensing situation however.

Software compatibility is another huge factor in the migration process. Some specialty software just plain doesn’t run on newer OS versions. There are complex ecosystems which are centered on a specific OS version and require an almost complete repurchase of every piece of the ecosystem to upgrade to a new OS in the first place.

These limitations can impact budgets pretty heavily depending on the size and scope of upgrades required. This is something which should be planned from day one of deploying a server (specifically, how to plan a budget around the next jump and when it should be), but is often overlooked. Computers, despite their relative upgradability, are not one time purchases.

Making a Plan

If you or your clients care about security. You will move or at least limit the damage an older box can do. If you haven’t built upgrade cycle budgets into hardware budgets, you need to start as soon as possible. A server or workstation should have a planned lifespan, and the money should be allocated for the replacement as soon as it hits end of life. “If it ain’t broke, don’t fix it,” doesn’t quite cut it for security or future-proofing.

Staging upgrades in over the next few months can also help your clients. This can reduce the perceived cost and make upgrades a bit more predictable. It also gives the clients time to get used to the change. Staging the upgrades in with the most technical or least impactful employees (e.g. interns) at the company to the least technical or most impactful (e.g. C-suite) can help build inertia for deployment and help the company adjust without as much impact.

Overcoming the Limitations

There are machines which cannot sanely be upgraded. There are several methods to overcome the limitations of the upgrade cycle. The two most common tactics are virtualization or partial air-gapping (or getting as close as possible) for the affected machines. These are not completely isolated tactics however and are best combined if possible.

Virtualization

This is the most common tactic to get around upgrades and the safest. There are still many Windows XP VM’s floating around. From old accounting software to legacy industrial systems, there are plenty of reasons to keep XP around. The more specialized the environment, the harder it is to move away from it or even upgrade it depending on the upstream vendor or cost.

For software which just won’t work outside of Windows 7 or Server 2008, most of which actually predates Windows 7 or Server 2008, virtualization is an easy step with modern Windows and decently modern hardware. A P2V migration may be a good idea for these scenarios. For workstations, this is pretty straightforward, especially when the machine is being upgraded because it’s usually too old for Windows 10 to be practical, but it can get a little harder with servers.

For servers, you want to make sure you have a suitable host, and you want to strip the server of as many roles as possible. The less access and privilege this server has on your network, the better. Even if it is less than ideal, it is also a good idea to try and avoid consolidating these servers too much. The more specialized they are, the more exact privileges they can have which limits security holes when intelligently applied.

Partial Air-gapping (Or Getting As Close As Possible)

Air-gapping is the practice of separating a machine entirely from the outside world. While complete air-gapping probably isn’t going to be too practical in most cases, the general principle should be followed as much as possible to partially air-gap a machine. A box which is inaccessible is not going to be practical to compromise. Every layer of convenience is a face to the attack surface for these weak-points.

Block as much traffic as possible to the given machine. If it was on a domain, take it off. If it has to be on a domain, spin up a secondary domain specifically for it. This limits the attack surface substantially and reduces what a successful attack can do.

If you need file shares, use a clean machine as an intermediary. Have multiple shares and use the intermediary as a jump box of sorts for transfers. Have a limited share between the intermediary and the old agent, and a share between the intermediary and the rest of the network. This adds a layer of complexity, but helps with safety.

How Many Are Out There?

Windows 7 usage sits at about 30%. A subset of our environment (just over 27,000 Windows agents for this example) shows that Windows 7 and all Server 2008 derivatives are sitting at around 30% as well. The general trend seems to remain the same for both business and overall usage. The overall number is in free fall, but still has a ways to go. Enterprise is a bit harder to peg down exactly what is going on.

Obstacles to Upgrades

The only thing which is really holding the numbers back is the lack of a viable alternative to most users. Windows 10 tries to be Windows 7, but misses the mark with both IT professionals and users. The majority of shifts happened during the free upgrade period, and newer shifts to Windows 10 are from machines dying rather than planned upgrades. Some clients even lament the loss of their Windows 7 machines. Some businesses were even buying old keys from salvage machines up until a few months ago. The Windows Update and upgrade system is maddening without moving to Windows 10 Enterprise.

From a server perspective, it doesn’t really offer enough to compel upgrading perfectly functional servers either. The licensing nightmare that is Windows Server further exacerbates the problem. Hopefully, Microsoft thinks to implement a smoother, more transparent plan to move servers (besides their push to Azure). I personally doubt they will as a power play, since they know many business’s hands are tied due to compliance.

Moving Forward

Ultimately, servers may hang on due to licensing, but the vast majority of workstations are going to have to be upgraded for both security purposes as well as pragmatic purposes. Newer software updates will begin shunning Windows 7 and Server 2008 the same as Windows XP back in 2014. It won’t start all at once, but within a year or two, the vast majority of applications which work on Windows 7 will work by lack of change rather than support.

It can be pricey and painful, but it is ultimately necessary. Try to amortize it out where possible and be ready to keep key infrastructure pieces secure which cannot be upgraded. If a client refuses to upgrade, they open themselves up to more and more security compromises which can bring down their business which hurts both them and you. There really isn’t much of a choice but to upgrade, or try to continue supporting a device past the point of obsolescence which weakens their business and yours.

by Sage Driskell

MSPs large and small are systematically being targeted over and over in the news. It’s almost weekly a new article comes out about a given large provider being targeted. Many of these attacks come from API weaknesses. You can’t control the provider or service, but you can minimize the chance of these attacks impacting you and your customers.

Leaky APIs

Leaky APIs are APIs which allow easy exfiltration of data from a service. These exploits often stem from deprecated APIs or privilege escalations. Many deprecated APIs exist in products for backwards compatibility, but they often come with caveats and holes.

Privilege escalations can happen on APIs due to loose queries which can return data from outside of their scope. Other escalations rely on multiple APIs which accidentally return data outside the scope a user can see normally due to their interactions. A deprecated API may help break another API when glued together with the wrong product.

These types of attacks are often used to harvest credentials or information for attacks later. Stolen passwords can be used on any similar account on multiple platforms to see what is shared. Hackers glean data which makes their later attacks easier either for traditional attacks or for things like phishing attacks.

Weaponizing APIs

With the creation of things like fileless malware and easy, privileged access via RMM tools, weaponizing an API has never been easier. Other products like Webroot have been had similar incidents from adding the feature to run commands remotely. This feature creep combined with API access makes these tools further targets.

Most products rely on various SQL products for databases. Many APIs where the developers are not security conscious will be a thin layer between the user and raw SQL queries. These can be weaponized to poison the data allowing greater access or to exfiltrate useful data. Depending on the product, it may be possible to insert hostile, arbitrary code which gets run by something within the API. Some RMMs even store scripts as blobs in the database.

How Do These Attacks Happen?

These attacks happen because of lax security policies on both sides of the equation. Many vendors do not take into account the ramifications of the access their API can provide. Vendors which integrate their products may ask for more permissions than they should need to function. A lot of permission sets are too permissive in general because its easier for the developer and the user to set up.

Clients of these products often fail to limit users enough. API users floating around provide an easy in to a company if they are compromised. Sometimes, the way multiple APIs talk to one another may be targeted as well. A simple return status from a query in a limited API may provide information that the other API would not normally have given. A simple boolean reply may provide a necessary bit of information for a malicious actor to work off of.

What Can You Do?

Removing unnecessary API users or users which may have API access is one of the easiest steps to protecting yourself, even from APIs outside of your control. Turning obsolete or unnecessary API versions, or even entire APIs off, is another great step. Use it or lose it. Trim off enough of the fat, and the hunter will target easier, more profitable prey.

Shrink your attack surface to shrink what you have to keep safe. Besides just trimming off the obsolete, scope your API users. An API user which only reports from a product doesn’t need write access. Your users need Two-Factor Authentication (2FA) everywhere possible. Do not share credentials between API users and do not recycle user names if you can help it. These basic steps have headed off many attacks before they even have the chance to become a threat with very little imposition on our technicians.

If you run products in house which have an API you use, try blocking traffic based on IP for whatever is using it. This isn’t always possible, but can often be used to limit certain service APIs to specific, known entities which limits the impact of a leaky or broken API. Rate limiting connections is another great step. If your average client hits 5 requests per hour, why not set a limit of 10 requests per hour so that brute force attempts take significantly longer? Alerting on these thresholds is another great step, especially if you control something in the stack which can see this.

Researching Products

No product is going to be perfect, but you can shop around to minimize damage. How does your current product handle exploitation? Are they quick to report it or do they take their time? A vendor which reacts fast, may still get hit, but at least you’ll know before your clients do and be able to protect yourself.

A vendor which tells you about an exploit quickly is also a vendor which works on fixing it quickly. Look at the vendor’s response history and how long it takes them to clear out serious CVEs to know how big a threat they are to your business. If they can’t keep up with serious vulnerabilities which are reported, what else are the missing that’s not reported yet?

Always be on the lookout to how a vendor impacts you and your clients. A vendor which never has real access is easier to trust than one which can make system level changes. Look out for how they handle older APIs too. A vendor which leaves deprecated features in too long runs the risk of being exploited down the line.

Ask your vendor what they do about older versions and whether or not they rate limit requests and accounts. See what the scope of their API access is. The irony is that those proudest of their APIs open access will usually be the first to tell you about it. Weigh this with your other options and the impact on your client before signing.

Our Strategy

We minimize unnecessary API interaction and work to maintain best practices to prevent exploits. When an API becomes obsolete, it is removed from our system where possible. API access is also further limited for fixed entities to prevent more wholesale access from being available off premise. Users need 2FA to get into basically anything. These patterns heavily minimize the attack surface with very little maintenance. Our large community contributes to helping make sure every potential exploit is known as soon as possible.

Our Security Focus

We focus on a holistic approach to security, and try to stay ahead of exploits and reduce the risk of any given component. Your security is only as strong as its weakest link, so you must be vigilant. Prevent unauthorized API access by preventing any access unless necessary. We want to know about an exploit as soon as it is public, if not before and be able to react to it.

Cutting off your finger is better than losing your arm, but not having to lose either is best. Prioritization of exploits is extremely important to surviving in the modern security landscape. We’re well past the days of “perfect security” even being a pipe dream, let alone realistic. We work to hedge our bets and make our platform the least ideal for hackers without sacrificing functionality. An ounce of prevention, even if it’s bitter, is a lot better than a pound of cure.

Going Forward

Stay ahead of hackers by locking down every aspect of your security. APIs are one of the most often overlooked, easily exploited part of many products. Almost every major software product is going to have an API of some kind too. Know what you’re dealing with and limit the damage where you can. MSPs have become low hanging fruit to many hackers, elevate your security and elevate yourself from being next.

by Sage Driskell