Few days back the Zerodha platform had an outagefor more than half an hour. Zerodha is India’s largest retail and discount broking firm. It has around 8.47 lakh userscompared to the other brokerage firms like HDFC Securities, ICICI Securities , Sharekhan etc. The users were additionally miffed because this is the second issue in a span of 6 months and it was the date of expiry for some of the F&O contracts.
The issue with their order management system resulted in a system outage of about 40 mins and as a result orders weren’t fulfilled leading to brokers incurring losses, some to the tune of few lakhs (1 lakh = 0.1 million).
Photo by Adam Nowakowski on Unsplash
The actual issue was caused by a huge purchase of a Rs 1 penny stock resulting in a great number of (1 lakh+) individual trades that led to an overload and crash of the Order Management System. In my short stint as a Product Line Manager at Cognizant, we experienced similar issues that would result in us receiving a late night call about servers affected and our internal apps hosted on those servers getting affected. In a global organization there can be no lag in support to the app platform or the apps on it, especially the key applications, of which some were in the Talent portfolio that I handled. On one particular occasion, some bad database calls in one particular app brought down the the other apps who had their database on the same server. The root cause was a similar edge case that had not been identified earlier.
During this occasion, the relevant teams were woken up and they did the needful during the middle of the night, because Support was based out of India . The commonality between this kind of problem , the Zerodha outage, and many others that affect life and usage of numerous applications are the lack of inclusion of boundary cases. These are also referred to as corner cases or edge cases . Many lump these names together but some of these terminologies are different from each other. While edge and boundary cases are used interchangeably, corner cases involve more than 1 edge case.
One of my favourite apps in Cognizant , and one that we built from scratch, was a rewards and recognition app. Among several features, it had a feed of updated recognitions and a carousel of the people who had been recently recognized. It was popular because it was the only app in the organization that allowed peer recognition which was earlier delegated to emails and preceeded the social apps like Yammer, which were available in the organisation. Also a hierarchical culture ensured that people were focused on recognition of any kind by senior colleagues and leaders as they were instrumental in bettering one’s career. Naturally our app was a refreshing difference and especially appreciated by the junior colleagues who were yet to get worn down by the cynicism that wears one down as you progress .
One fine day we discovered that one of the users had posted 100+ identical recognitions causing the feed, carousel and notifications to get impacted. While we worked to figure out how to handle this including reaching out to the user and understanding the reasons behind his specific actions, I still remember one of our developers asking me as I was constructing the use case that had been missed earlier , if this one-off scenario really mattered considering we would rarely come across someone with a similar motivation. The truth is that it does matter.
One cannot predict how some users will test our boundary/edge cases but that does not mean that we do not try to design the system around it.
In some cases the impact from such misses may have far reaching impact. The MyGate app used in our apartment complex and many others involves approving visitors to our apartment. This includes frequent vendors for milk, newspaper, etc and one off visitors like those delivering packages, food etc. One of the features involved MyGate working with different partners to request residents for a pre-approval to the apartment complexe.g. if the resident was expecting someone to arrive for a delivery. It was helpful for both parties, because the resident had enough time to approve the entry before the delivery person had even reached the address. Once the delivery guy reached, he could show the pre-approval to the security and walk in without spending time in registering and additional entries etc. The additional and most important advantage was that the delivery person did not have to wait at the entrance till the resident approved. (Delays happened for various reasons including network issues resulting in the delivery person waiting longer to just deliver his package.) On one such day, the delivery person declined to deliver my food order. Since he was pre-approved, I proceeded to reject the approval and realised that the application did not allow me to do so.
The support person who responded quickly to my request quite naively assumed and informed that this could be handled by just informing the security at the gate. The problem is that security personnel work in shifts. They also have loads of other issues to deal with continually, than remember that a food delivery person for a particular flat amongst hundreds of others was not to be allowed entry despite prior approval. And even if one person did remember that person may not be around to prevent the delivery person from entering. In 99 out of 100 cases we would not need to worry because the delivery person would visit the apartment where he actually does not need to deliver anything. In the 1% scenario he could use the prior approval for entry for various reasons including ones where he simply wants to meet another friend working there. In an extremely exaggerated but not unforeseen scenario that could work in various ways in a gated community and result in an unwanted security situation. The problem here is two fold. The first involves not handling this use case properly and the second is missing the boundary cases of how it could be misused.
Such mistakes can cost a lot. An unwanted situation in case of MyGate, or loss of customers ( existing and new) in case of Zerodha. I was looking around for changing my current Demat account and Zerodha was one of the firms in my radar. With the current outage and its history of issues, any new customer would think twice. Like myself, there are others who are extremely key to the business of any company, and not accounting for such boundary issues might affect its business negatively. Hence it is imperative to design such boundary cases and proactively discover the others.
- Source for title image : Photo by Joshua Earle on Unsplash
Koel's Blog 