6 Lessons On Work Ethic I Learned In One Year Of Professional Career

vagnini_101317_4227_hero_lg

Time flies. Recently I had completed one year as a full-time employee at my current employer Squad. A year has passed, and I decided it was time to revisit instances, memories, and experiences and to recollect what I had learned as a professional in this past year.

It was also a wake-up call to reassess and redirect the ship named professional career to make sure it doesn’t get stuck in a whirlpool.

After all, our career is our responsibility and we all should make efforts to “Own our story!”.

This is going to be a list of 6 important lessons in professionalism and work ethic that I learned working as a Product Engineer at one of the most innovative startups in India.

1. Know your field

Do you know what a facade pattern is? Do you know what sprint and story points are? Do you know how to work your way with the debugger your IDE provides?

A wealth of ideas, disciplines, techniques, tools, and terminologies have decorated the last fifty years of our field. And if we want to be a professional we want to know a sizeable chunk of it.

The motto that I believe in is, “If you want to see far, stand on the shoulder of the giants”.

 

2. Continuous Learning

The frenetic range with which the industry is changing, it means that we as engineers also need to learn colossal amount just to keep up.

Read books, read articles, watch talks. Keep adding deltas to your learning daily.

 

3. Practice

This stuck me around 2 months back. Tennis is my favorite sport, and players believe that playing in the tournaments continuously actually makes their game less polished.

It’s the deliberate practice of doing things right, which makes it gleamy again.

Is it true about our jobs also? Cutting corners to meet deadlines, working on the outdated stack at the company, working with legacy code, can all this make us less sharp.

At least I find it to be true, and practice to deliberately improve your craft is a vital component of one’s work ethic.

 

4. Know your domain

It is the duty of every software developer to understand the domain of the solutions that they are programming. If you are writing software for healthcare, get at least a basic understanding of healthcare, if you are writing for sales, know about sales.

Read a book or two on the domain, ask the domain experts.

We should be able to know enough about the domain to question the product direction and feature requests that we get.

 

5. Collaboration and mentoring

We all must make special efforts to collaborate, mentor and get mentored by other developers.

Whatever I have learned in this course of one year, a major portion of it due to learning from others.

 

6. Humility

Programming is an act of creation. It feels like magic when we can write code that can do things that can produce tremendous value.

We all should take pride in our work, but never be arrogant about it. We should be confident in our abilities and take risks.

But we must know that we will fail. Things that we create will break, risks will be proven wrong and we will be called upon for these mistakes.

And when all this will happen, all we can do is be humble and take Howard’s advice, laugh and move on.

 

That’s all, folks!

 

Advertisements

Organization Archetypes And The Concept Of Market-Oriented “Solver Teams”

depositphotos_44074529-stock-illustration-flat-design-illustration-concept-of

Organizations which designs systems are constrained to produce designs which are copies of the communication structure of the organization.

In other words, how we organize our teams has a powerful effect on the software we produce, as well as our resulting architectural and production outcomes.

Thus, in order to get a fast flow of work from Development to Operations, with high quality, great customer outcomes and fast speed of delivery,  we must organize our teams to bring the team structure to our advantage.

Done poorly, this can prevent teams from working safely and independently, instead, they’ll be tightly coupled, all waiting on each other for work to be done.

At SQUAD, teams are structured as market-oriented teams, to quickly respond and solve customer needs. At SQUAD, we call them “Solver Teams”.

 

Organizational Archetypes

meaning-of-life

There are primarily three types of organization structures that inform how we design our DevOps value stream: functional, matrix and market.

Functional-oriented:

Organizations optimize for expertise, division and reducing cost. These organizations centralize expertise and have tall hierarchical structures.

Ex. Server admins, SREs, Data admins

Matrix-oriented:

Organizations attempt to combine functional and market orientation. This results in complicated organization structures like a single person reporting to multiple managers etc.

Market-oriented:

Organizations optimize for responding quickly to customer needs. These organizations tend to be flat, composed of multiple cross-functional disciplines (ex. marketing, engineering, machine learning).

Each market-oriented team is responsible for feature delivery, operational tracking and service support.

Market-Oriented “Solver Teams” at SQUAD

At SQUAD, we have a bunch of interesting problems to solve that highly impact and solve customer needs.

Broadly speaking, solver teams at SQUAD are market-oriented teams, composed of cross-disciplinary work like engineering, marketing, machine learning, data analysis etc to “solve” a customer problem.

These teams are responsible not only for feature development but also for user experiments, testing, optimizations, deployment and operational tracking of services, from idea conception to, successful launch, to retirement, all without dependencies on other solver teams.

Advantages of Market-Oriented Teams

  1. Small teams working independently and safely.
  2. Faster execution and delivery of work.
  3. Enables team members to be “E-Shaped” specialists.

 

Enable every team member to be a generalist

Screenshot from 2018-05-05 21-03-30

As we rely on ever increasing number of technologies. We want engineers who can contribute to multiple areas of value stream.

Another major advantage of the market-oriented teams is that, because of their innate nature of being cross-disciplinary and covering entire value stream from development to operations, it provides opportunities for the team members to develop and multi-specialist capabilities, also called as E-Shaped specialists.

When team members start becoming “E-Shaped” experts, business benefits of enabling faster flow are overwhelming.

As the same team member is able to contribute to multiple points in the value stream, the flow of the stream is much smoother and faster than a specialist working on a single point in the stream without having comprehensive knowledge of the entire value stream.

 

Conclusion

We saw how organization architecture dramatically improves our outcomes. Done well, organizational structure plays as an advantage and helps teams move and deliver faster.

At SQUAD, we structure teams as “solver teams”, which are responsible to own the entire value stream of the problem they are solving.

Solver teams are small and can move fast and safely without having dependencies on other solver teams.

That’s all, folks!

 

Devops and The Principle Of Flow

lean-software-development-1-728

In the technology value stream, work typically flows from Development to Operations, steps consisting of functional areas between our business and our customers.

As stated in the lean principles developed by Toyota, we should optimize to get a single-piece fast and smooth flow for our releases.

We increase flow by:

  1. Making work visible,
  2. Reducing batch sizes and intervals of work
  3. Building in the quality, preventing defects from being passed to downstream work centers.

Why a fast flow is needed?

By speeding up the flow through the technology value stream, we reduce the lead time required to fulfill internal and external customer requests, further increasing the quality of the work while making us more agile.

Our goal is to decrease the amount of time required to deploy the changes into production and increase the reliability of those services.

Make our work visible

agile-pm-kanban-board

A significant difference between manufacturing and technology value streams is that our work is invisible.

It’s so easy for work to keep bouncing off between teams and yet have no visual control over it.

To prevent this and make out work more visible, we can use something like a Kanban board. (I prefer Trello for this).

Ideally, our Kanban board will span the entire value stream, defining work as completed only when it reaches the right side of the board.

Work is not done when development completes, but only when our application is running successfully in production.

Limit Work In Progress (WIP)

In technology, our work is far more dynamic than manufacturing. Teams have to satisfy demands of multiple stakeholders. As a result daily work gets dominated by urgent requests for work coming through every communication channel possible.

We can limit multi-tasking by using Kanban board, such as by codifying and enforcing WIP limits for each column on the board.

For example, we may set a WIP limit of three cards of testing. When there are already three cards in the testing column, no new cards can be added.

Using Kanban ensures that work is visible and WIP doesn’t get piled up.

Reduce Batch Sizes

one-piece-flow

Another key component to creating smooth and fast-flow is performing work in small batch sizes. Prior to the lean manufacturing revolution, it was common practice to manufacture work in large batches.

However, large batch sizes result in skyrocketing levels of WIP. According to lean principles, the ideal is a single piece flow, where each batch size is of just one.

Let’s take an example:

Suppose we have ten brochures to mail and mailing each one of them requires 4 steps:

  1. fold the paper
  2. insert the paper into the envelope
  3. seal the envelope
  4. stamp the envelope

Now in the traditional batch processing flow, we will perform each step sequentially for all ten envelopes.

In the lean one-piece flow, only one envelope can be at any given step. In other words, we fold the paper, insert it into the envelope, seal the envelope and stamp it before starting the next one.

How is one-piece flow dramatically better?

In the above example, suppose each step takes 10 seconds. In batch processing, we get our first complete envelope after 310 seconds, but with the one-piece flow we get it just after 40 seconds.

Worst, what if we find that the way we have folded the paper, doesn’t allow the envelope to be sealed. In which case we’ll be in a bigger trouble?

Eliminating hardships and wastes in the value stream

According to the Toyota Production System pioneer Shiego Shingo, a waste is:

The use of any material or resource beyond what the customer required or is willing to pay for

In software development value stream, a waste is anything that causes a delay for the customer, such as activities that can be bypassed without affecting the result.

The following are some common categories of waste that we encounter when implementing lean in software value stream.

  1. Partially done work
  2. Extra processes
  3. Extra features
  4. Task switching
  5. Waiting on QA or testing or acceptance testing
  6. Defects and bugs
  7. Non-standard or manual work

Explaining each of the above point deserves a post of its own. Will do that soon.

Conclusion

Improving flow through the technology value stream is essential to achieving DevOps outcomes. We do this by making work visible, limiting WIP, reducing batch sizes and eliminating wastes from our processes.

All of this will allow us to become more agile and will help in reducing lead times dramatically, and at the same time increasing the quality of releases.

That’s all, folks!

 

7 Tips On Making Your Engineering Workflow Faster

One of the most important thing that I like about pair programming with other awesome engineers is you get to see their workflows. How they get things done? How they get find and make their way around tools, terminal, and editors?

After witnessing and getting awestruck by many such experiences I realized that having an effective workflow can increase your day to productivity many folds.

The next step was to take action, and while doing so, I have compiled few tips to make your engineering flow faster too.

1. Identify waste:

lean-manufacturing-identifes-waste-in-workflows

According to the Toyota Principles:

A waste is any hardship and drudgery that doesn’t align itself with what customer requires and is willing to pay for.

The First step towards developing a ninja workflow is to do an audit and identify waste.

Identifying waste is the first step towards eliminating it.

Ex. My workflow had many wastes like not automating enough, not harnessing the power of IDEs, waiting for the large codebase to reload etc.

2. Get proficient with your IDE:

8rljtzts_400x400

Today, IDEs are super power packed. To develop an awesome workflow, we must learn how to harness the power of our IDEs.

I use PyCharm for my day to day work and have witnessed very considerable efficiency increase once I took steps to become proficient with it.

Features like:

  • Custom live templates
  • Smarter code navigation
  • Custom keyboard shortcuts
  • Debugger
  • Distraction free mode
  • Plugins

All make you a power user. It’s worth investing time given the return on investment.

3. Get familiar with Unix shell commands:

Getting familiar with Unix shell commands is a game changer. First, it makes you look smart and second it helps you automate stuff.

I noticed that I used to do a set of repeated tasks every day when I started working.

  1. Login to office WiFi network
  2. Open browser
  3. Open Slack in tab
  4. Open mail in a tab
  5. Start the IDE
  6. Activate virtual environment
  7. Checkout VCS repository etc.

And now all it looks like is:

~$ startwork

You get the idea. Invest some time and become a Unix power user.

4. Automate your manual workflows:

Developing skills to automate takes time. Whether they are using shell scripts, browser extensions or little code snippets.

Investing time in automating workflows is a high leverage activity.

Ex. You don’t need to manually follow a certain flow of the app to test out something every time, which takes 2-3 mins.

Can we automate this and do this in 2-3 seconds.

How much time will it save if suppose we do this 25-30 times on average?

Ex. Or automating generating search tags for blog posts 😛

5. Prefer keyboard over mouse:

Screenshot from 2018-04-23 01-02-17

We all will agree. Using mouse is slow. Using keyboard over mouse helps decrease the time it takes to perform actions by many folds.

The action that was buried under 3 sub-menus can now be performed just by pressing a key combination.

Personally, for this, I would highly recommend a plugin for JetBrains IDEs called key promoter. It’ll help big time in getting over our mouse addiction.

6. Learn at least one productive high-level language:

Getting things done in a language like Python is way faster than something like say,  Scala.

Learning at least one high-level language allows us to quickly test out ideas and implement them.

No more resistance of writing a 20 line class just to test out an API call to a service.

Move fast and test out ideas on an interactive interpreter instead of compiling code files.

7. Make it easy and fast to run the unit tests associated with just your current changes:

Screenshot from 2018-04-23 01-22-07

Running the entire test suite or even the test suite of the module you touched can be time-consuming. Life is too short for that.

To quickly validate things, make it super easy and fast to run the unit tests of just your current changes.

Personally for this, I use the copy reference feature of PyCharm a lot, and obviously, I use the keyboard shortcuts to do so.

Conclusion

With this post, I wanted to share some ways that I’ve been consciously working on to making my engineering workflow as efficient as possible. Still, there is a big room for improvement, but hopefully, this article would’ve been of a little help at least.

That’s all, folks!

 

Fail Fast: Hone Your Ability to Recover and Respond Quickly

veteran-turned-software-engineer-e1485204975427

It’s close to midnight and you are about to wrap your day off. Suddenly you get a pager-duty to resolve a critical bug that’s failing some of the automated reporting emails.

You go on to check the logs in the log management tool. This is not the ideal time to find out that logs are not getting streamed to the log management service properly.

Next, you decide to check the performance metrics of the email API and you realize that you don’t know the new monitoring tool well enough to get the right metrics quickly.

That sets the theme to why as effective engineers we should fail fast and hone our abilities to recover and respond quickly to failures.

Another post I wrote on failing fast:

https://priyankvex.wordpress.com/2017/07/08/philosophy-behind-the-offensive-programming/

“The best defense against major unexpected failures is to fail often.”

Netflix knows its way around when we talk about creating reliable systems. What engineers at Netflix have done may sound counter-intuitive, but they have made a tool called Chaos Monkey. It randomly kills services in their own infrastructure.

It turns out that this strategy helps Netflix to increase site’s reliability. Failing services during office hours when all the engineers are available, helps them perform recovery drills effectively and prepares them well enough for actual emergencies.

Why is it so important to prepare for failures?

As software engineers, our systems are bound to fail at some point and some releases certainly will have some bugs. In such scenarios, learning and investing time in the ability to recover quickly becomes a high leverage activity. It gives you the confidence to move fast with your product having peace of mind that you are ready to tackle problems if they arise.

Few reasons to invest time in recovering from failures:

  1. Prepares the team to write scripts for success via mock drills.
  2. Surfaces gaping holes in the systems used for monitoring and debugging.
  3. Helps develop better tools and processes to handle emergencies.
  4. Helps control stress and panic in the cases of actual failures.

Write your contingency plans

what-if-800x435

Ask yourself “what if” questions and work-through contingency plans:

  1. What if a critical bug gets deployed with a release?
  2. What if a user raises an urgent ticket?
  3. What if my message broker goes down?
  4. What if my systems face a spike in usage?

This can be applied to even more aspects of software engineering:

  1. What if the due date for a feature gets preponed?
  2. What if a critical team member goes sick?
  3. What if there is a dilemma in the product plan and prioritization?

Conclusion

No matter how careful we are and what we are working on, things will go wrong some of the time.

The better our tools and processes for recovering quickly from failures, and the more we practice using them, the higher our confidence and the lower our stress levels will be. This allows us to move forward much more quickly.

That’s all, folks!

 

8 System Design Principles I learned After Doing It Wrong More than 50 Times!

laptop-3190194_960_720

 

At Squad, we strive to build awesome products to solve customer(internal and external) needs. As a product engineer, paramount part of your job is to design and build products. Dig deep into the root cause of the problems, design solutions and implement them as the end product.

Over the course of my journey so far, here are the 8 system and product design principles that I’ve learned from other awesome people at Squad, from feedback and simply doing it not right enough multiple times.

1. What is the underlying problem that led to the feature request?

At Squad, you don’t just code the requirements into the software. As a product engineer, it’s your responsibility to remove the layers and expose the root problem that led to the feature requirement.

Get to know the root cause of the problem that you are trying to solve. Or even better, as the lean principles say “genchi genbutsu” i.e go and see it yourself.

2. How can you make the feature more robust, reliable and usable?

Once the essential feature requirements are finalized, we must press on how can we make the feature more robust, reliable and usable?

Things to ponder upon and take into consideration can be :

  1. The persona of the users that’s going to use that.
  2.  Scenarios in which that feature would be used. Ex, if in the case of fires, than show more data than needed for faster resolutions.
  3. Building in the quality in the product itself or “Jidoka” as said in lean.

3. What is the first iteration going to be?

Given the time and resources you have., what is the best possible first iteration of the product going to be? If it’s a large system or something you are building from scratch, there are always going to be iterations.

The main idea here should be to move fast and get things shipped. Good enough and shipped on time is always better than perfect and in-development forever.

4. How easy will it be to make iterations on the current feature?

The design should incorporate all the non-functional requirements to make future iterations easy.

Scale the feature? Change a component? Use a different 3rd party service? Your implementation should be flexible enough to incorporate and encourage these enhancements.

Design patterns are your best friend here.

5. What are the potential bottlenecks with scale?

Scale-land is where everyone wants to be, but it is scary. It breaks what was not supposed to break and has witnessed more horror stories than a haunted castle.

What are the potential bottlenecks that are not a problem now, but will break at 5X, 10X or 100X scale?

List them down on the feature ticket, or better document it in the code itself.

6. What’s the data that has to be captured and how will it be consumed?

Every feature in the product will need some data that needs to be captured to track it. It can be but not limited to:

  1. Action logs.
  2. Event logs.
  3. Metrics
  4. Failures.
  5. Anamolies.

What affects this majorly is how that data will be consumed? Store it in a structure that will make the consumption of data easy and efficient. Afterall, the only motive to store data is to use it.

7. How good the developer experience will be when interacting with the code base of that feature?

There can be many developers who’ll use or modify the code that you are going to write.

How will be their experience when doing that? Ex. Will the test cases you wrote, make them feel confident enough to make changes fast?

Few points to consider:

  1. Is the code well documented?
  2. Are test cases strong enough?
  3. Is the code, re-usable where it makes sense?
  4. Are functions small and code, simple to read?

8. What metrics will determine that the feature has been implemented successfully?

Finally, after all the fun-time you had creating the feature, what will determine that the feature has been implemented successfully?

The data you tracked will be of paramount importance here.

It can be the case that to track this quantitatively is not possible, but can you track the qualitatively in that case?

The idea here is that you can’t improve what you can’t measure?

Processing 100,000 requests? Fewer errors by the users? 95% work done by the new system instead of old one?

This can and will involve more stakeholders of the team and not just the developer.

Conclusion

Obviously, this is not the exhaustive things to take into consideration while designing a system or a product as an engineer. This just covered what I have learned so far by just doing things wrong or not right enough multiple-times.

It’s fun to build stuff! Continuously improve (“Kaizen” in lean)! Keep iterating! Keep shipping!

 

 

That’s all, folks!

 

Deploying a nginx application using Kubernetes for Self-Healing and Scaling

Kubernetes is an open source system for automating deployment, scaling and management of containerized applications. A more technical term for it is, container orchestrator which is used to manage large fleets of containers.

Minikube is an all-in-one single node installation for trying out kubernetes on local machines. And the following post covers deploying a nginx application container using kubernetes in minikube.

If you don’t have, then this link has it all to install minikube and kubectl (command line tool to access minikube) : Download and install minikube and kubectl

Step 1 : Making minikube up and running

Ensure that minikube is running.

starting_minikube

Step 2 : Open the minikube dashboard

Minikube comes with a GUI tool that opens in the web browser. Open the minikube dashboard with following command :

opening_dashboard

It should open the dashboard in a browser window and it’ll look something like this:

first_look_dashboard

Looks cool! No?

Step 3 : Deploy a webserver using the nginx:alpine image

Alpine linux is preferred for containers because of its small size. We’ll be using the nginx:alpine docker image to deploy a nginx powered webserver.

Now, go the deployments section and click the create button, which will open an interface like below.

create_app_filled

Fill in the details as shown in the image.

We can either provide the application details here, or we can upload a YAML file with our Deployment details.

As shown, we are asking kubernetes to create a deployment with nginx:alpine image as container and that we want 3 pods (or simply instances) of that.

A pod in kubernetes is a scheduling unit, a logical collection of one or more containers that are always scheduled together.

Go on and click that awesome deploy button!

Step 4 : Analyzing the deployment

Once we click the deploy button. Kubernetes will trigger the deployment. Deployment will create a ReplicaSet. A ReplicaSet is a replication controller that ensures that specified number of replicas for a pod are running at any given point of time.

Flow is something like this:

Deployment create ReplicaSets, ReplicaSets create Pods. Pods is where the real application resides.

deployment_overview

As expected, we have our deployment, replica set and pods in place.

We can also, check our deployment via command line using kubectl.

deployment_overview_cli

Step 5 : Create a Service and expose it to the external world with NodePort

So far, we have our pods up and running. But how do we access them?

This is where a service comes into play. K8S provides a higher level abstraction called as a service that logically groups pods and policy to access them. This grouping is done via labels and selectors.

Then we expose the service to the world by defining its service type and service redirects our request to one of the pod and load balances them.

Create a my-nginx-webserver.yaml file with the following content:

https://gist.github.com/priyankvex/3b34ec02c82934b84c8dfb68272ed4f1

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-web-service
  labels:
    run: my-nginx-web-service
spec:
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: my-nginx-webserver

Enter the following commands to create a service name my-nginx-web-service

creating_service_cli

We can now verify that our service is running :

service_running

Step 6 : Accessing the application

Our application is running inside the minikube VM. To access the application from our workstation, let’s first get the IP address of the minikube VM:

minikube_ip

Now head to the address and port number of the service we got in above step.

application_running

And our app is running! Amazing, give yourself a pat now!

Taste of self-healing feature of the kubernetes system :

One of the most powerful feature of kubernetes is self-healing capabilities (just like Piccolo. DBZ, anyone?). While defining our app we created a replica set with 3 pods. Let’s go ahead and kill one pod and kubernetes wil create another one to maintain the running pod count 3.

self-healing.png

As we can see in the image. We deleted the bottom-most pod and K8S created a new one instantly.

Such kubernetes! Much HA (High Availability)!

Taste of scaling with Kubernetes:

Now, our app is receiving a crazy amount of traffic and three nginx pods are not enough to handle the load. Kubernetes allows us to scale our deployments with almost zero effort.

Let’s go ahead and spin up a new pod.

scaling_menu_option.png

scaling_to_4.png

Click OK. Now let’s go and check our pods.

scaled_deployment.png

As we can see in the image, we have now 4 pods running to handle the increased traffic.

Isn’t it amazing? We just horizontally scaled our application with the power of kubernetes.

This was just the tip of the iceberg what Kubernetes can do. I am also exploring the kubernetes and containerized architecture just like you, hopefully we’ll be back with another post soon with more kubernetes stuff!

That’s all, folks!