ISE Blog

How to Use Calabash for Cross-Platform Mobile Testing

Now that we've established automated testing is almost always worth the investment, what is the best way to go about it? In this post, I'll give a brief summary of the automated acceptance testing tool, Calabash, and how to set it up for cross platform testing, then lay out a few of the most important best practices we employ. 

What is Calabash?

Calabash is a behavior driven development (BDD) testing framework that tests iOS and Android apps by driving the user interface. It uses Cucumber to define the test scenarios in natural language, and Ruby to create test step definitions which use the calabash API to inspect and interact with your Android or iOS apps. Our experience has been in using it to interact with native and Xamarin based apps, though it is relatively flexible.

For more information and the basics go to https://developer.xamarin.com/guides/testcloud/calabash/introduction-to-calabash/

It's also helpful to note that you can use physical devices or AVD/simulators to run Calabash tests on both Android and iOS. For iOS it tends to be more difficult to get a physical device setup working, and for Android it tends to be (slightly) more difficult to get an AVD setup working, but your mileage may vary.

The Setup

When we first tried Calabash we were using the predefined steps packaged with it to drive our app. We quickly found that though this is a good way to try out the tool, it didn't provide the readability, flexibility or reliability we needed in our tests. So, one of our amazing interns set up a cross-platform test infrastructure by using the guide provided by Xamarin.

The guide already does a good job of detailing how to set things up in features, step_definitions and page objects, so I won't go into detail here. Just keep in mind, to run your tests on both platforms you should have an app which has a similar UI workflow across Android and iOS.

The basic idea is that you only use custom steps that you write. You share step_definitions between platforms, and those definitions only use page objects to interact with the app, which you keep a platform specific version of. We've also successfully used this pattern with apps which have tablet and phone specific versions of the interface.

Best Practices

Here are a few of the best practices we employ with our tests to get you headed in the right direction.

1. Think like a user

When you write your feature files, make sure your steps are written from a user's perspective, not a developer or a tester's. Ask yourself "will my boss know what this means?" A test step should be something like: 

When I login to the app as "Bob"

NOT

When my fake server is running
And the app sends a login message to it

 
2. Test all the time

Run your tests automatically - use Jenkins or another Continuous Integration (CI) solution to kick off tests automatically on a build box. Use the Cucumber Reports Jenkins plugin to display your results nicely, with inline errors and screenshots. Unless you need something like bluetooth that simulator/AVD don't support, you should be able to use those to run the tests on the build box. If you do need a physical device to run tests on, make sure there's a way to connect it via USB to your build machine. Also keep in mind for building and running iOS apps/tests, your build machine will need to be a Mac.

3. Test in isolation

For reliability's sake, test your app in isolation from the rest of the system as much as possible. Create fakes and simulators to stand in for network endpoints, connected devices, etc. There are any number of options for faking your REST endpoints, so don't run your auto tests with a real server when a fake can be more reliable.

4. Choose your battles

Don't write automated tests for everything - start with the most repetitive, least human interpretation necessary test scenarios. For a given user story expect there may still be 1-2 manual tests. Some tests may be possible but not cost effective or reliable to auto test. It takes practice to develop this judgement, but start with the easy ones.

5. Mind your runtimes

Number 5 goes nicely with number 4 - Try to keep the running time of your test suite down. One of the major advantages of creating automated acceptance tests is a quick feedback loop for regression testing. If it takes longer than 30-45 minutes to run the full suite of tests, the delay time required will slow down the entire development process. Keep your test suite short and run it on each pull request and merge.

6. Timing is everything

When working with automated acceptance tests that drive a user interface, it can be easy to just add a "wait 1" into a test when you notice it just needed to wait 1 second more and it would have worked. Resist this temptation! Though it might have needed one second right now, when the code changes or you run on a different device it might need more. Instead, wait until something is visible on the interface that indicates it is safe to move on.

7. Don't repeat yourself

Just as in your code, avoid this in your test scenarios. You can write a step that runs multiple other steps in Ruby like so, saving you from repeating a series of steps over and over, and making your tests more readable to the user! 

When /^I am selling something as "(.*)"$/ do |user|
steps %Q{
When I login as "#{user}"
And I switch to the "Marketplace" tab
And I switch to the "Sell" tab
}
end

Download the Ebook on Creating iOS App Themes using UIAppearance


How did it go?

Now that you're excited to try all this in your Calabash testing, don't forget to let us know how it goes in the comments! And as always, feel free to contact us if you're interested in learning more about how we can solve your team's challenges, or if you're interested in joining our team!

Clay Schumacher, Senior Software Engineer

Clay Schumacher, Senior Software Engineer

Clay Schumacher is a Senior Software Engineer and Practice Lead of Mobile Development at ISE. Clay lives in Normal, IL where he works from his home office and Slingshot Cowork. For the last four years he has worked to develop mobile solutions that delight our clients and end users. Clay enjoys agile/lean software development, and treats each project team as a startup bent on delivering the best app for each unique problem. He enjoys traveling and playing games with his wife and two daughters.