API test automation using Postman simplified: Part 1

Every application you build today relies on APIs. This means it’s crucial to thoroughly verify APIs before rolling out your product to the client or end-users. Although multiple tools for automating API testing are available and known to QAs, we have to decide on the tool that best suites our project requirements and can scale and run without much maintenance/upgrade while creating a test suite. And once the tool is finalized, we have to design an end to end flow with the tool which is easy to use, and we can get most benefits out of automation efforts.

I am writing this two blog series to talk about end to end flow of API automation with Postman from deciding on the tool to implementing the suite till integration with CI tool and report generation .thus the content of these blogs is totally based on our experience and learning while setting it up in our project.

In the first blog of the series, we’ll talk about the phases till implementation while in the next blog we’ll discuss integration with Jenkins, monitoring etc.

Thus I have structured this blog into following phases:

• Doing POC and Deciding on the tool depending on its suitability to meet project requirement
• Checking tool’s scalability
• Suite creation with basic components
• Testing the Suite using Mock servers

Before moving on to talk on these topics in detail , For those who are new to postman, in brief, Postman is one of the most renowned tools for testing the APIs and is most commonly used by developers and testers. It allows for repeatable, reliable tests that can be automated and used in a variety of environments like Dev, Staging, and Production etc. It presents you with a friendly GUI for constructing requests and reading responses, easy for anyone to get started without any prior knowledge of any scripting language since Postman also has a feature called ‘Snippets’. By default, they are present in Java script but by using it you can generate code snippets in a variety of languages and frameworks such as Java, Python, C, CURL and many others.

Let’s now move to each phase one by one wherein I’ll talk about our project specific criteria and examples in detail.

Doing POC and Deciding on the tool

A few months back when we came up with a plan to automate API tests of our application, first question in mind was, what tool to use which will best suit our requirement?

Mostly the team was briefly familiar with Postman, JMETER, rest assured and FitNesse tools/frameworks for API automation. The main criteria for selecting the tool was to have an open source option which helps to quickly get started with the test automation task, is easy to use,  gives a nice and detailed reporting and is easy to integrate with CI tool.

We could quickly create a POC of complete end to end flow using postman and a similar POC for comparison purpose on JMETER. However, Postman came out as a better option in terms of reporting and user-friendliness since it does not require much of scripting knowledge and hence anyone in the team can pitch in anytime and contribute in the automation effort.

Checking tool’s scalability

Now since we had liked the tool and wanted to go ahead with the same to build a complete automation suite, next set of questions on our mind was related to limitations and scalability of Postman free version,

This was important to be evaluated first and foremost before starting the actual automation effort as we wanted to avoid any unnecessary rework. Thus we started finding answers to our questions. While we could find few answers through the web searches, for some of the clarifications we had to reach out to postman customer support to be double sure on the availability and limitations of the tool.

As a gist, it is important to know that if you are using postman free version then:

• While using personal workspace there is no upper limit on the number of collections, variables, environments, assertions and collection runs but if you want to use shared/team workspace then there is a limit of 25 requests.
• If you are using postman’s API for any purposes (for ex. add/update collections, update environments, or add and run monitors) then limit of 1000 request and rate limit of 60 applies.
• Postman’s execution performance is not actually dependent on number of request but mainly depends on how large computations are being performed in the scripts

This helped us to understand whether free version suffices our requirements or not. Since we were not planning to use postman APIs or monitoring services, we were good to go ahead with postman free version.

Suite creation with basic components

Creating an automation suite with postman requires understanding of following building blocks (Refer Figure 1)

• Collections & Folders: Postman Collections are a group of saved requests you can organize into folders. This helps in achieving the nice readable hierarchies of requests.
• Global/Environment variables: An environment is a set of key-value pairs. It lets you customize requests using variables so you can easily switch between different setups without changing your requests. Global variables allow you to access data between collections, requests, test scripts, and environments. Environment variables have a little narrow scope and are applicable only for selected environment. For instance, we have multiple test environments like Integration, Staging, Production. So we can run same collection in all three environments without requiring any changes to collection but by just maintaining 3 environments with environment-specific values for the same keys.
• Authentication options: APIs use authorization to ensure that client requests access data securely. Postman is equipped with various authorization methods from simple Basic Auth to special AWS signature to  OAuth and NTLM Authentication
• Pre-Request: Pre-request scripts are snippets of code associated with a collection request that is executed before the request is sent. Some of the common use cases for pre-request scripts are Generating values and injecting them in requests through environment variables, converting data type/format before passing to test script etc.,
• Tests: Tests are scripts written in JavaScript that are executed after a response is received. Tests can be run as part of a single request or run with a collection of requests.
• Postman in-built js snippets for creating assertions: Postman allows us to write JavaScript code which can assert on the responses and automatically check the response. We can use the Snippets feature of the Tests tab to write assertions.

Figure 1: Basic Building Blocks of Postman

Testing the Suite using Mock servers

Using the base framework that we created during POC we were able to extend it to add multiple request and multiple tests around each request’s response to having a full-fledged automation suite ready with us and running daily.

In our case our first problem statement to be achieved in API automation was a set of APIs for a reporting module.

Since report contains dynamic data and generating a set of test data is also very tough due to multiple environmental factors, it was not possible for us to apply fixed assertions to validate data accuracy. That’s why we had to come up with other ways to test that don’t exactly match the correctness of the data but still are thorough enough to check the validity of the data and report actual failures.

Thus while doing this exercise what we followed and that really turned out quite beneficial for us was that before starting to write the tests in the tool itself, it is very important to clearly list down everything in detail as to what exactly we want to assert.

For simple APIs with static responses, this list might be pretty straightforward to define. But In our example, it required a good amount of brainstorming to come up with list of assertions which can actually check the validity of the response without knowing the data value itself.

So we thoroughly studied the API responses, came up with our PASS/FAIL criteria, listed down each assertion in own words in our plan  and then went ahead with converting them into actual postman assertions. For ex:

As we see in above list, we have a number of positive and negative tests covered. While we have such assertions in place in Postman, we can’t say it will work when such response gets generated in actual until we test it thoroughly. If we are testing postman collection of APIs with actual environment response, we might not get each type of failed responses.

And thus to test it we need a way to mock the API request and response which is very similar to the actual response but has some values modified to invalid values to test whether out script and assertion catch them as failure or not. This is possible in postman through mock servers. You can add a mock server for a new or existing collection by navigating to New->Mock server in postman (Refer to Figure 2)

A Postman mock server lets you mock a server response, allowing a team to develop or write tests against a service that is not yet complete or is unstable so that instead of hitting the actual endpoint URL, request is made to specified request path given in mock server and accordingly mocked test responses are returned and we can see how our script behaves for such requests and responses. Thus during actual execution of live endpoints if similar scenarios occur, we already know how our script is going to handle them.

Figure 2: Adding mocked request/response using Mock Server in Postman

Now once we have our test suite ready with required cases added and tested, we are good to start scheduling it to run daily for our test environments so that it checks the API health and reports failures.

In our next blog we will discuss these phases in detail. Stay tuned.