Ceph & Security, OpenStack best Practices For Storage

Abhishek Singh, OpenStack Best Practices

This article attempts to throw some light on community Openstack best practices for Ceph storage. It aims to assist storage administrators and operations engineer that are engaged in deploying multi-mode Openstack clusters.

Firstly, the author would discuss about Ceph and its benefits, and then the best practices of Ceph from different scenarios looking into the infrastructure peripheries. The author assumes that the readers have fundamental knowledge on Openstack and its deployment. Ceph is a software defined storage solution for Openstack and it is used for aggregating different storage devices including commodity storages to give an intelligent storage pool to various end-users. A properly designed Ceph can provide High availability too. Openstack Cinder is used to provide volumes and Glance provides image service. Like other object storages Ceph also needs a gateway which is an intelligent service to categorize the defined data and place into object storage and it is RadosGW. Ceph integrates into Openstack Nova and Cinder by Rados block devices. A benefit one can see in cinder with Ceph over the default volume back end local volumes managed by LVM and cinder is it is a distributed and network available solution.

Another advantageous feature that comes along with Ceph is copy-on-write that allows existing volume as a source for unmodified data of another volume. It significantly consumes less space for new virtual machine based on templates and snapshots. Using network availability and distributed storage, live migration is possible for even ephemeral disks. This proves to be handy dealing with failing hosts and during infrastructure upgrades. Ceph’s integration with QEMU also gives space to use Cinder QoS feature to control virtual machines from consuming all IOPS and storage resources.

The purpose of this article is to emphasise that cloud deployment is more exposed to threats than traditional environment. This is because storages are accessible on internet, and in addition, Ceph and other Openstack services are installed on servers and mostly with default options.

This article will now discuss on securing block and object storage using Ceph and then move to the topic of securing connectivity between Openstack and SAN/NAS solutions. RadosGW is vulnerable component in object storage because it is exposed to HTTP Restful requests. It was also suggested in Openstack Summit at Vancouver to have a proxy appliance that has a separate network with “SSL termination” enabled with proxy forwarding and web authentication filtering between virtual machines and RadosGW. Ceph is not having centralized mechanism for managing object storages, it is managed using CephX with each device. This means that clients can directly interact with the Object storage devices (OSDs), Cephx works like kerberos. Here’s the catch, CephX authentication is only between Ceph client and Ceph server hosts, it is not extended beyond the Ceph client and CephX policy doesn’t work when someone access Ceph client from remote host.

In order to exercise the functionality of monitoring, OSDs and metadata servers’ Ceph has another authentication mechanism called “Caps”. Caps also restricts access among pools, with this said users can have access to certain pools and have no access for some. In other word this authentication helps in building policies for authorization.

It is very important to understand how Ceph authenticates and the vulnerability attached to it. Ceph use keys for communication. These keys used to authenticate Ceph client are stored on server and are in plain text files which is a vulnerability for any environment. If one hacks the server the keys are exposed. In order to control this, arbitrary users, portable machines and laptops should not be configured to talk with Ceph directly because it would then require storing plaintext files to be stored in more vulnerable machines and compromise the security. As a best practice, users can login to a trusted machine with proper hardening and security and use that machine to store plaintext authentication files. So far Ceph does not include options to encrypt user data in object storage. There is a need for out-of-the-box solution to encrypt the data. Apart from these, one can implement best practices of DoS, for example: limit load from client using QEMU IO throttling features, limit the max open socket per object storage disks (OSD), limit the max open sockets per source IP and use throttling per client.

Moving forward to the second section, this article will now discuss on securing connectivity between Openstack and SAN/NAS solution which is equally important as securing block and object storage. Cinder and storage communicates through management path. Communication uses SSH, or through REST by SSL. It is advisable to keep management interface on secure LANs and use strong passwords for management accounts. Try avoid default vendor passwords, and role based security and accountability can be helpful forensic tools. Now the readers might be thinking about the efforts that can be used in securing the data path. There are many ways to do it and a strict checklist with specifications for setting hardening parameters can be used for devices and components, let say for NFS: a stricter configuration options for exports and user management can be practiced, proper access control lists (ACLs) to limit only authenticated users to see IP SAN and all other setting that can reduce the vulnerability list. The reason why proper ACL is important is because any server that resides on same IP segment as that of isci storage can access the storage theoretically and perform read/write operations. Proper control on file owned by root with permissions 600 is also advisable.

There are other ways for securing communications like CHAP that assists identifying client through username and password. When Cinder communicates with storage it generates a random key and it is used by Nova when it connects with iscsi and thus a secure connection is established. Another important area to consider is encrypting exposed traffic using “transport” and “tunnel” encryption. there are two ways to encrypt the data- Transport mode and tunnel mode. On transmitter side, transport mode only encrypts data portion and not the header whereas tunnel encrypts both header and data portion. On receiving side, IPSEC-complaint device should be able to decrypt the data packets and for that to work transmitter and receiver should share a public key that gives a secure connectivity, however, it can put some load on network in return. For those volume that uses block storages through fibre channel volume should have Zone managers which can be configured through cinder.conf for proper control.

To conclude, Openstack ecosystem is quite vulnerable when typically installed and a lot of improvements can be seen in terms of security.

Hypercompetition: Why it’s not Profitable?

blue targets and arrow

many blue targets and three arrows hitting the center of the first one

Hypercompetition coined by Richard D’Aveni considers competitive analysis under conditions in which competitive advantages are quickly negated. In addition, guiding policies are repeatedly flouted by iconoclastic rivals competing in border-less boundaries, and customer loyalty is constantly fickle. Can we have hypercompetition between two industries? Yes, because hypercompetition depends upon the mind-set and actions of hypercompetitive partners not on number of competitors.

Temporary advantage is with one who is slightly ahead of competitors, it is the one who breaks the silence first similar to prisoner’s dilemma. It always needs action and that creates issue on profit level for an organization, mostly hypercompetitive partners rely on margins and at times cannot withstand with the demand of innovation and R&D. Oligopolistic cooperation does not produce excess profits because of easy barriers to entry (Porter’s 5). Firms mostly compete with each other in four arenas and they are cost, quality, timing and know-how, strongholds, and deep pockets. In cost and quality arena, the trade-off between price and quality is eliminated, forcing the industry’s product offerings toward a point of ultimate value for consumers, combing low prices with high quality. This can only give minimum of marginal profit scavenging options for hyper competition. So, to overcome these most competitors either redefine quality and increase prices of products/services or force competitors towards timing and know-how arena.

The one who breaks the silence requires a uniquely different set of skills than that of followers, and that confers several advantages. The competitive moves can be easing through the know-how creating/altering resource base with speed, agility and timings. It also escalates when one competitor goes for creating a new resource base or transform strategy to imitate/replicate the one with competitor (let’s say successful competitor). This whole notion is called leapfrog strategy, and it is suggested that it is expensive, and competitors can imitate quickly, and competitions try to create strongholds. Under hypercomeptition, entry barriers can easily be circumnavigated. When Stronghold erodes through the entry of increasing rivals, the rivalry shifts to who can outlast the others based on their deep pockets. Big fishes as I’ve mentioned in my previous article in superior financial conditions can neutralize hypercompetition through tatics such as strategic alliances, acquisitions, franchising, niching, and swarming (moving is large numbers).

Hypercompetition can be in multiple arenas, or it could have stuck in a particular arena for a long period. Here comes proper project management, suppose the delay between moves are longer, and this delay is purposefully done so that enough resource can be produced to give a slack in the project for a recoup. Perfect competition gives profits significantly but hypercompetition doesn’t give profits, and if some then it’d be temporarily until the firm’s advantages are neutralized or eroded by other competitors.

Hypercompetion takes following points as assumptions:

  1. Firms mostly destroy their own competitive advantages and create new-new products. They deliberately cannibalize their own leading product before it goes through full product life cycle. Moving to new products needs capital for innovation, production costs etc decreasing margins.
  2. The one who are determined will likely to break the barrier but exiting barriers only provide a false sense of security, lulling incumbents into complacency.
  1. Consistency and logical thinking are the easiest thing to understand in competition. So, it is always advisable to be unpredictable.
  1. Long-term planning will only help to sustain and exploit existing advantage. Hypercompetitive is all about eroding existing advantage of competitors.
  1. From SWOT, one gets weakness of competitors. Studies suggest that targeting ones’ weakness consistently will make them to work on that and can improve turning into stronger instead.

Thanks for reading…

How Big Fish Can Mislead: A Thought On Technological Progress


Let’s start with digital disruption to explain how new technology disrupts everything else and prevails. Then we can see how the opposite happens in energy industry. When will disruption prevail in an energy sector?

Digitization and Digital Disruption are a few common phrases used in techno talks, and a lot of speculation on future based upon them. We’ve seen ipad disrupting physical libraries when it enhanced e-books sales closing over 30% physical libraries in United states in an year. Mobile phones disrupted almost everything including mp3 players, cameras, books, watches and sometime ornaments (studies suggest people gave away a few rings while carrying smartphones). Other examples include that of revolutionary apps like Uber, Skyscanner, Airbnb. So far it is completely understood how it changed the way of business, but it’s equally displaced the equilibrium of the business ecosystem. Let’s suppose when there were 100 smaller companies generating a billion dollar, and now we’ve just one company generating a billion dollar leaving rest 99 out of the league. In this case, what would they do? one option is to leave the current line of business and try something else, another would be to adopt the change and do the same thing like copying what the competitors doings, or leave the business. Out of those most viable option for me is to stay in the market imitate what competitors doing and target on a smaller scale than before to keep going, but the question is how long one shall keep going? In this world of Entrepreneurs, and in the society with technology change, and people appreciating innovation how long one can survive without swaying towards the current of change? What would be the plan of action for the small organization to help get bounced back to what they were, this is for me a mathematics question that has an answer within the equation. It is very important to understand the equilibrium and the shift from normal, it might be too impossible in some cases if the displacement is too much and the off set price to bring it to the normal would be either too much or impossible. There is nothing like a crystal ball with any strategist but yes envision technological change, and when everyone goes behind the same goal can lead to severe issues, it mostly gives one winner and rest all losers. The winner is often the most fearless and also the most adaptive of all, and losers mostly overconfident status-quo addict. Why I’m writing these all when everyone knows them all? In more than a century ago, the industry suffered from dominance by a large western market and competitiveness among a few private companies for the world’s oil reserves. Oil price saw the highest and indeed permanent increase in 2005. In respond to the trend in the oil and gas industry, other companies are now producing efficient vehicles, engines and aircraft and also going for an alternative source of fuels as a measure to capture the demands for transportation. There are new technologies available today which provide different opportunities for unconventional oil and gas in most parts of the world, however, the production of this unconventional energy is uncertain. So far I just wrote against the big fish in the market, but non-agile, non-adaptive or whatever are the most powerful among all most of the time, except a few industries these status-quo fishes can haul innovations or guide the whole world in a completely different way influencing interest of entrepreneurs. When such big firms, lets say Exxon Mobil and RDS, they are the biggest of all companies in the world in terms of sale, fortunately, there is no disruption for them except for oil price fluctuations that also wouldn’t have affected them much if they’d hedged at a safer rate. These organizations are completely dependent on one source of production and survival, and currently pumping crazily every second to suck as much as they can. While making their future and business model running, they might be preaching a different story, and wouldn’t like to see negative curbing for oil prices. Russia (loses about $2bn in revenues for every dollar fall in the oil price) is losing too along with United states with price slumps, but how far these companies have distracted others of not inventing alternate, source of energy. When big companies like RDS and Exxon focuses one source of energy it’ll for sure negate the interest of others, and of course, a lot of capital needed for research and development act as a super barrier stopping innovation. The reasons, Wharton professors and others note, are many. Fossil fuels have been in use for decades, a reality that makes a switch to another energy source very difficult, they point out. Oil as an energy source is not only practical, it has an entire infrastructure built around its use, they note. Politics has played a varied role too, sometimes helping energy innovation, but often hindering it, they add. But by far the biggest hurdle to alternative energy innovation is that oil wins as a simple matter of cost. “Innovation consists of matching a solution to a need”.

My question is how legal is to only think of profit and comfort, and not contributing in any innovation toward environmental risks. Unfortunately, there is no such regulation in United states and Europe to put the blame on such big fishes, since an environmental issue is a border-less phenomenon, and such regulations will promote technological innovations.

To be continued…

Disiuption of the status quo to acquire a temporary advantage is accomplished through exploiting the new 7-S framework (which replaces
McKinsey’s older 7-Ss):
1. Superior stakeholder satisfaction
2. Strategic soothsaying
3. Capabilities for speed
4. Capabilities for surprise
5. Shifting the rules
6. Signaling strategic intent
7. Simultaneous and sequential strategic thrusts

How To Build A Strategy? An Overview Of Strategy Tools

I thought of writing something to refresh memories on strategic tools, options, foresight, intent etc….Not sure whether I’d be writing them exactly how I perceived during lectures back in 2012. The reason behind saying this is because I personally feel that my understanding is differing now as I’m becoming more mature. Not sure how others think but as I am growing older I tend to focus more on the resultant force that would influence more than other forces. It might be a priori within since I’m an engineer to consider that major contributing resultant force (say top 40%), but of course it comes with risks. The only thing I can do at work is to be logical, and there is no room for philosophy or arguing about risk management or about Beck or anyone because organization are all rational to a major extent. They’d rather fail being rational than trying to spend a lot for a trade-off. Beck said so much, for example – “Science’s rationality claims to be able to investigate objectively the hazardousness of a risk permanently refutes itself” – Ulrich Beck (1986), but who cares? Leave it, let’s quickly move to strategy and exhale the frustration….

What is Strategy?

Let’s start with Strategy definition, it could be dicey to comment anything now so let’s figure out what is not a strategy first because that is very important to know what shouldn’t be considered as a strategy statement- Maximizing profit, reducing operating cost, improving quality are not a strategy. Becoming no #1 airline in the world, no #1 X in the world, maximize sale is not a strategy. Regulating money, giving the best class loan or maximizing number of accounts for a bank is not a strategy but they are goals and goals cannot be a strategy. Any strategy that is not imperfectly imitable is not a strategy.

Strategy is about doing things so differently at every step of production/services that competitors cannot imitate, and that gives you a value proposition in overall process resulting in maximizing profit or X ~ AB

Other wrote something similar to mine: P

A long-term direction which seeks to meet the expectations of, and create value for, stakeholders

  •  Johnson, Whittington and Scholes

However beautiful the strategy, you should occasionally look at the results.

  • Winston Churchill

If you’re serious about strategy work you must always do your own analysis

  • Richard Rumelt

Building Strategy: Tools

We’ve so far covered strategy definitions, then comes external audit, internal audit, corporate and competitive strategies, and developing and evaluating options.

External audit is done mostly through tools like SWOT, PEST, Porter’s diamond, Cause and Effect diagram, scenario implication etc. These tools are to access outside environment and competitors. Let me talk about the tools I use at work. So far I’ve used SWOT, PESTLE, BCG Matrix, and sometimes in fact just once Porter’s 5 Forces. However, topics from operations management are more valid for my role than any other subject including TQM, Balance scorecard, FTE calculations, Break-even, production optimization, fixed and variable costs etc…

Internal audit is equally useful because understanding and analysing own footprint matters the most. It can be done through positioning analysis, SWOT, RBV logic, VIRO, core competencies, capability resources, investment and transfer-ability, useful questions from Javidans (given below), 5 whys etc..

We know how to __________ very well.

In one part, or all, of our business(es) ?

We are better at _______ than our competitors ?

Then comes diagnosis before making a strategy, as per Rumelt’s 3 steps strategy process: Diagnosis, Setting guiding principles and coherent set of action. Diagnosis can be done through tools like open framing and closed framing (WXYZ) statements:

W …       the objective or goal of the change process
…        the process by which change will be delivered
…        the resources or skillset involved
…        the time frame for the change to be completed

In addition, tools like porters generic strategy helps to diagnose and identify the footprint, market share and other business attributes to a certain extent. Bowman’s Strategy Clock can assists with positioning and target. Strategic groups and strategic space is the most common type of graph used for representing companies of similar industry with bigger, medium or smaller circle, or using matrix visualization putting significant x/y axis for market share and responses. Porter 5 forces of course is a powerful tool but it is for me a tool with no use upfront, but a good tool to use in classroom. Other tools include Blue ocean strategies, BCG matrix, Ashridge visuals etc..

Now comes building strategy(options), to start with there is a need for development directions and tools like Ansoff Matrix can help with it revealing Existing and new markets and products, and they’re resultants from internal/external audits, 5 forces, competitive, cooperativeness analyses etc. TOWS analysis can assist with creating options through prioritizing the SO, WO combinations. And at last, choosing SFA – suitable, feasible and acceptable option(s) can give a strategy.

How To Build A Strategy? An Overview Of Strategy Tools

Related Article:

The Side Effects of an MBA

Its been a while that I’ve written something on my blog. Like always I’ve been busy with some necessary hindrances, in fact, my days are driven mostly by such priorities that are never urgent and important. Yeah, I normally drive myself, I create my own work, I mostly try innovating work because I generally get bored of mundane tasks. So, the reason for writing this article is how one should exercise stuffs/skills continuing the regular job, however, it amplifies risk in practicing with activities that have short timelines, but it helps in attenuating the side effects of MBA.

Every problem is an opportunity and there are multiple ways to solve them. Uncertainty is more predominant in this era than before and reduces dichotomy as a solution. There is no right or wrong answer is a common phrase everywhere, so who wins at the end of the day? The answer is no one but if you see opportunity in it, you consider any job, except the mighty “Button pressing” job that we use as an example all the time, there are ample opportunities to work in your own way. I don’t mind altering the way I present solutions when given the similar problem day after day. I remember solving integration problem in my school days and in engineering that has many solutions and ways to solve. This is what creativity is in my terms, solving in different ways will help to integrate different models together and you learn doing it every-time. This is how you can leverage your daily task into an enjoyable challenge-able assignment.

Now coming to what MBA thought us, I wouldn’t say I was stupid before MBA but the only one thing I can feel MBA reminds me every-time is that you can be equal to the one in the market who are in the better position or with better pay. It’s all up-to to the leverage level, the extent you market yourself at the right time and at the right place. There is nothing that we missed during MBA, covering political affairs, scientific theories, models, trading, finance, economics, medicines, arguments on issues, sales, negotiations, time management, money management, wealth management etc…you just name it and somehow knowingly or unknowingly it would’ve been covered, and again as stated earlier it lies in the word “leverage”. Now coming to the side effects of MBA, there are many, the most common one is that it makes you arrogant at times when someone talk nonsense, and with poor managers. Other side effects are ending up losing a job due to overconfidence and high expectations. It enables you to challenge and take pressure  all the time, but sometimes you don’t really need to. One very peculiar change is observed in fellow MBA grads is the ever declining ratio of emotional quotient. People lose their sensitivity and become hardened; which may be appropriately called stress hardening in pure engineering terms. Personal and intangible issues take a backseat and life of almost everyone becomes mechanistic to its extreme point. Humans turn into machines churning out PowerPoints and word reports. In addition to this the most deadliest side effect I think is that it demands you to be updated with time, and highly demands practicing tools and subjects, else there will be only artifacts on your name and slowly only “obsolete you” will be left behind.