Testing World

Pink Ninja Tester

Category: Automation

It is not only about End to End Automation

pinkninjatester

How many times have you encountered when you are discussing automation in software testing or test (check) automation and the other party always refers it to e2e (end to end) automation? In my case, almost all the time.

It is true, for some reason recruiters, testers, interviewers, managers, developers, and maybe others thought that automation is merely about simulating the end user’s behaviors from the application.

So therefore when they put a job offer as a test automation engineer, it mostly contains a skill list of selenium and BDD (Behaviour Driven Development). Is it because of the name? (what is in the name?) 

Test Automation Engineer is more than just building an End to End automation. I personally think the test automation engineer’s job is to make the work of testing easier and more efficient. It can include preparing test and staging environment, preparing test data,  make sure that the data are consistent through the whole chain, make a deployment script, make a database script, automate notifications, automate test reports, and a lot of others. 

Let me give you an example: several years ago (in 2013) I was a software tester for a company that offers a framework to build a portal or dashboard or application which enables the non-technical person to build it themselves with minimum help from developers. The framework is required to be deployed to the application/web server chosen by the client. The list of application servers is huge including WebSphere, WebLogic, JBoss, and Tomcat (and some more with different versions). It also needs to run a database script to database server which is supported such as MySQL, MSSQL Server, DB2, and Oracle. And we can start testing the application just after we deployed the framework, run the database script, and set the configuration. So we can see all the possible combination matrixes and if you have experienced deploying packages to WebLogic, WebSphere, and JBoss, you will understand the hell of it. 

In this case, automation is really needed to automate the deployment, the database script execution, and to set the configurations. In this case, the tester will be able to focus on testing the application on the different possibilities of combination without being bothered by all of the nitty-gritty of the pre-requisite work.  

Another example I had when I work as a Chain tester in 2019.  It is not an easy test such as you access the web application, do something, and then see the output from the application itself. It was a complex systems consisting multiple applications and backends. It has queueing mechanism and it also has several front applications.

In order to start the actual testing, I had to create many test data in several systems and these data needs to be valid and synchronised. The test preparation itself when it is done by hand took several days (and dont forget that different systems are owned by different department – which made it even longer to prepare). In this case, automation comes in handy to prepare all of those data in different systems and check if they are valid and synchronised. So the tester will be able to focus to the actual test execution.

I used these two examples are representative to show that automation in software testing is much more than just End to End Automation. In fact, IMHO E2E automation is not the biggest picture of automation but people made it so. 

What do you think?

How to show SpecFlow feature files in VS2019 MacOS?

pinkninjatester

Problem: SpecFlow feature files in VisualStudio 2019 for macOS don’t show up in the solution explorer. Rebuild, Unload, and Reload do not help. It just shows the generated .cs file but not the features file.

Solution: It turns out it is a common problem and people are still looking for the answer. This is the workaround I found and so far it is working for me.  Right-click the project > Display Options > Show All Files

After that, I am able to see my feature file.

Setup SpecFlow in MacOS with VisualStudio 2019

pinkninjatester

I was excited when .NET finally released VisualStudio 2019 for macOS. So I installed it in order to be able to work with SpecFlow. Unfortunately, SpecFlow does not have a plugin for macOS.

After some search, I finally found there is a plugin for SpecFlow for macOS. It is just not officially built by the SpecFlow itself. 

Create a project in MacOS with VS2019:

  1. Create a new Project with type NUnit
  2. Choose the version of .NET core ( I chose 3.1)
  3. Set the project name and location

Install the SpecFlow extension

  1. Download the extension from in https://github.com/straighteight/SpecFlow-VS-Mac-Integration
  2. go to menu Visual Studio > Extension. 
  3. The extension manager will be displayed and then click the “Install from file”. It will install the SpecFlow extension.

Add SpecFlow NuGet packages to Project:

  1. Right-click the project > Manage Nuget Packages
  2. Type SpecFlow in the search bar
  3. Choose SpecFlow, SpecFlow NUnit, SpecFlow.Tools.MSBuild.Generation

Add Feature Files:

The only thing that I am missing here is the auto-generation of a step definition. The extension does not have that capability. Therefore, we need to make it ourselves. 

Knowing Test Automation Tools does not make you a tester

andyssundaypink

Lately the demand for automation testers is high compared to the demand of functional testers. Most of the time, a company only wants to have someone technical so they can automate their testing. So they ignore the fact that functional tester might not be needed since everyone can test (that is what they think or assume). And sometimes they just think that the tester can do both, which are testing their application at the same time as increasing their automation testing coverage.

I think the interviewer needs to know which kind of tester they want and focus on getting the tester with the right speciality. There are several different goals:

  1. Your application is in the end of the development phase and there are sometimes new feature requests or bug-fixes. So it is needed to do some regression tests after each new implementation of a feature or a bug-fix.
  2. Your application is a work in progress and still has long way to go to actually be a the end state. In other words, features implemented can change a lot of times till the end product. So you want someone to challenge your application and find obvious bugs and unknown bugs so that you have a bigger level of confidence in the application.
  3. Your application is a work in progress and you need someone to automate the regression test so there is no need to have a elaborate manual regression tester for any features that have been implemented in the past.

So which one is your current situation and which one is your desired goal?

If you are in the first situation then go ahead, find someone that can automate the regression tests. He or she does not have to be a tester as long as he or she knows automation tools. It means that there are several regression test cases so the automation engineer can just pick it up and create script for it.

If you are in the second situation then you need a specialized tester. Someone that can really challenge your application from different perspectives. Someone that can question your user stories. Someone that can break your application. Trust me, finding a good tester is much more difficult than finding an automation test engineer.

If you are in the third situation then…dude seriously…get two people. One for the functional testing, requirement analysis, and some other type of testing and another one for the automation part. If you only hire one person then don’t expect much about the amount of coverage for your automation testing or the confidence level in your application. You need to sacrifice one of them. She or he is not a super(wo)man.

It will be useless if you ask someone that has knowledge of automation tools but they are not good in testing itself or even they think they have the knowledge but they don’t actually have it. Some people think testing is just following requirements and clicking stuff and some people think that if they have ISTQB or TMap Certification then they are eligible to know what testing is. And why it is important to know what testing is? Well you can read about it in my blogpost here.

Several times I have seen good candidates being rejected from hiring just because they don’t know about Robot Framework or Python or Protractor or TestNG or any other testing tools even though they have some (adequate) technical understanding. What I consider a good candidate is someone who knows what the actual testing is (not just the tools) and someone who knows how to proceed with it.

So in my opinion, it is non-sense if you just curious about his or her automation skill and even non-sense if you just ask for a particular specific automation framework. In my opinion, if someone knows how to work with Protractor or Cucumber, they will also be able to work with other framework such as Robot Framework or TestNG or others.

Thus, if you want to have bug-free (minimum bug) product then find a good tester, but if you want to accommodate Devops then find automation test engineer. If you want to accommodate Agile, please get two testers (especially if it is more than 5 developers and 2 week sprints).

 

Happy Testing!

 

 

 

How to treat Automation in Software testing

andyssundaypink

In order to create good and maintainable test script automation, we need to treat it as a project. If it is handled as a side task for tester then it will not work. I can assure you it will create chaos and un-maintainable scripts. It should be treated as test development. There should be discussion and deep thought about the test architecture or test structure. There should be peer and code review to keep it clean. There should be a release plan or milestone about it. It should have maintenance time and bug fixing for the scripts automation itself. Yes indeed, there are sometimes bugs in the scripts itself.

I always imagine the process is like ‘building a house’. There will be initial process which is designing the architecture of the house such as the blueprint, the floor plan, and the materials that it will be used. When the structure of the house has been finished then it is shifted to designing the interior of the house such as painting, choosing the furniture, filling up the cabinets, etc. Even after that there will be other steps like maintaining the pipe, the floor, fixing the broken window, etc. So the process is most likely the same as Automation in software testing or Development in a project.

This kind of things are usually overlooked by a lot of company or teams. They keep thinking that automation can be built in like several days. Automation does not come as a free object. Even though it is called automation, it still needs resources to implement things to make other thing automated. It is the same as a development project. You need resources to develop or implement things to make another thing visible. For example; you want to automate the payment therefore you need resources to develop payment automation to create it. The code is not created automatically out of the blue just by saying it.

Most people keep thinking that it will be automated automatically and don’t need to be implemented correctly as long as it does what it needs to do. Well, forgive me my friend, What happens if I say the same thing with the development codes that you have written?

 

Automation in Software Testing

andyssundaypink

First of all I prefer to use the term: Automated Checking rather than Automated Testing. Why do I prefer automated checking than automated testing? It is because most of the time it is not doing testing, it is just checking whether the actions that you have scripted give an output as expected (because you command it as expected) is equal with the actual output. It does not really do the actual test. It is just doing what you have been told it to do. Testing is more than just executing test scripts or test cases.

Testing in my definition is questioning the test mind map object that you have created in your mind, whether it is accepted, whether it has an added value to the software, whether it does not give a blocking error etc. It is repetitive searching and questioning the actual values of the application or the object to test. Testing is continuous and expanding action to find out what is good, what is bad and what is probably in the middle. But check automation can not create or expand its own script. Automated checking can not determine whether a scenario is good or bad. Automated checking can not choose another path or script which is not written by humans.

Therefore, I prefer to use the term of automated checking if it is only about checking whether the actions it has carried out are working as expected. It is only checking whether the changed development code bring bugs to the other current code. If there is any tools that can create its own test script just by adding a plugin or a connector to the system under test without any human interference, then I will be happy to call it as Autonomous testing.

However, there is also another perspective of automation in software testing team. In my opinion, automation in testing team should actually help them to reduce the annoying or repetitive task that they have to do without the necessity of using their brain.

Some examples of these are:

  1. Create a script that helps them to directly go to the intended testing module without have to be bothered with login and navigating through several things before they can actually do the real testing.
  2. Create a test script for the previous released functionality. It will be used as the regression test script.
  3. Create a test script to cover the bugs reported by the testing team and has been fixed by developers. Therefore the testing team does not need to revisit this all the time. It will be good as example for sanity testing.
  4. Helping tester or team to create a bunch of test data.
  5. Scripts that can help to do smoke test.

 

Happy testing!

 

Setup Selenium – TestNG

andyssundaypink

You can follow the instructions via this site.

The instruction in this site is using selenium version 2.43.0. However, this gives me trouble. When I run the test, it opened the browser twice and complained that it could not find the FirefoxBinary.

The error message does not really refer to the real issue since it is able to open the Firefox browser. I then tried to update my firefox version and also changed the selenium version to 2.49.1. It works!

Another thing, I run this test and then there was another issue with the test code itself.

driver.findElement(By.cssSelector(“#s”)).clear();

However, it will not able to find the element if the window is not maximised since the menu layout is changed regarding the window size. Therefore, I added extra code before this specific line of code.

driver.manage().window().maximize();