UI Testing over the years - Peter Steinberger - App Builders 2020

The speaker is excited to be back at Abdullah and acknowledges that the situation is different than planned. The talk will cover testing a topic that combines the boring part of testing with the frustrating part of user interfaces. The speaker aims to provide inspiration and new ideas for career advancement in testing, based on their experience at Peace PDF Kit. They emphasize that there is no one-size-fits-all strategy for testing and it's not a tutorial for beginners.

PeacePDF is a small distributed company that specializes in solutions for showing and annotating documents, particularly PDF files. They offer features such as annotating, filling forms, signing, merging, and flattening.

The testing pyramid is a classification system for different types of tests. The majority of your tests should be fast and independent unit tests, which can also become integration tests when testing different blocks in combination. Snapshot tests are useful for specific cases like building a PDF library with a renderer. User interface (UI) tests are important but cannot replace manual testing.

The iOS developer community survey by Dave provides valuable insights into building apps. The survey reveals that 60% of people write unit tests for hobby projects, while UI tests are less common with less than 25% for hobby projects and less than 40% for companies.

In the survey, one interesting question was about the frameworks used by companies. Some popular frameworks in the market today include XUI, Uncaf, and Unload Grey. Other notable ones are IBM Calabash and various others.

The reliability of tests is a significant concern, as indicated by discussions on Twitter. However, it is important to consider multiple dimensions when evaluating test reliability. For instance, if a test takes an excessive amount of time to build (e.g., 30 hours), it may not be ideal despite its reliability.

To address the issue of slow UI tests, a common strategy is to parallelize them by using multiple simulators and nodes. This allows for faster execution of tests. It is recommended to utilize both approaches - having multiple simulators and multiple nodes - for optimal performance.

Funding Circle has been writing UI tests since the beginning, focusing on frequent paths that take around 30 minutes to run. Despite using an old framework called KIF and dealing with flaky maintenance, they have managed to keep their automated tests under control.

Creating accessibility tests can be challenging, especially when it comes to touch interactions. While KIF is a popular framework for creating such tests in Objective-C, it is also possible to create them in Swift. However, there may be some complexities involved due to the language and changes in iOS versions. For example, Apple made changes to how touches work internally starting from iOS 9, which affected the functionality of KIF temporarily. To overcome this issue, developers had to resort to hackery by creating digitizer events with IOKit and attaching finger events accordingly.

Accessibility is an important aspect of iOS development. It involves making the app usable for people with disabilities, such as those who use assistive technology like VoiceOver. One key concept is the accessibility label, which describes what users hear when using assistive technology. Another concept is accessibility identifiers, which are non-localized tokens used to identify specific views or areas in the app.

Ixia is a useful tool for writing tests and comes with a test recorder. However, the test recorder in Ixia may not work well, especially when using Swift.

Introduction to Hue Automation Hue Automation was introduced by Apple in iOS 4, but it was removed in iOS 10. It allowed for JavaScript-based testing using instruments. Although poorly documented, it provided context for the accessibility identifier documentation.

"Kif" Testing Framework "Kif" is a white box testing framework that revolutionized blackbox testing during its time. Unlike Hue Automation, Kif allows access and mocking of tested parts within one process.

In this chapter, the speaker discusses how they were able to create a fast video for testing purposes. They used two main techniques: increasing animation speed and using busy waiting. By accelerating the speed property of the base layer, they were able to make animations appear faster. Additionally, they mixed native code with simulating taps and checking if certain conditions were met.

Busy waiting is a technique where the program flow doesn't continue until a certain condition is true or a timeout is triggered. It involves creating a run loop to allow other events in the system to happen. While it can be useful for making tests more stable, using busy waiting in your actual program can be both horrible and amazing.

NSAttributed String and HTML Initialization - When initializing an NSAttributedString with HTML, it uses WebKit. - Browsers are multi-threaded, so Apple spins up a run loop to make a fake super call. - Doing this in a table view cellForRowAt index path can create recursion and crash the app.

Async Version in iOS 13 - In iOS 13, Apple added a real async version of NSAttributedString initialization with completion handler.

The Power of DIY Testing Kif is an amazing tool developed by the people at Squared. They didn't wait for Apple to offer solutions and instead built their own UI testing tooling, which they open-sourced for the community. Many good projects have benefited from Kif's existence.

"Big Players" Approach to Testing "Testing Cabinets" at companies like Netflix, Snapchat, and Facebook consist of on-site labs with thousands of tests running on devices attached to test runners. These companies have dedicated teams that manage custom infrastructure and software for testing purposes.

Challenges Faced by Small Companies For small companies like Which?, implementing effective UI testing can be challenging due to limited resources. Sevilla Saudi shared that they struggled with flakiness when running tests on simulators or devices using Jenkins.

In Xcode 8, Apple introduced a new parameter called 'only testing' for the Xcode build. This parameter allows you to selectively run a subset of tests by specifying either a class or an individual test. It provides an easy way to execute specific tests without running all the tests.

The Test Center Plugin Multi Scan, developed by Lindsey Ferguson, is a popular tool for running tests and automatically parsing the output to identify failed tests. It allows for retesting of these failed tests multiple times. This feature is particularly useful when dealing with flaky network-dependent tests.

Parallel testing increases test confidence by enabling faster and more stable tests. Xcode 10 introduced this feature, but it was unreliable. However, with the release of Xcode 11, parallel testing has become much more stable and efficient.

Blue Pill is a reliable iOS testing tool created in 2017. It aims to make test execution faster by running UI tests using multiple simulators on a single machine. Despite the lack of public API from Apple, Blue Pill utilizes internal frameworks and various techniques to achieve its goal.

XE Pretty is a Ruby script that uses regular expressions to make the log output of Xcode builds more manageable. It was created in 2013 and has since become an essential tool for developers.

King Uzziah's influence is widespread, even in Fastlane. The song 'Supermarine' was written by him and he has a strong presence on Twitter.

Horta, the creator of a great tool, deserves praise for his amazing work. Although he is no longer working on iris projects, he is now involved in building impressive things for TypeScript at Microsoft.

Using Danger for Pull Request Testing Danger is a tool that runs on pull requests and can comment on certain conditions. For example, it can detect when there is a large code change but no corresponding UI tests, indicating the need to run UI tests for new features. By automating these checks, Danger helps codify team norms and allows developers to focus on more challenging problems.

"Accept Pretty" Functionality in Log Output "Accept Pretty" function in log output translates the log output into an XML report while running tests with X. This feature enhances readability of logs and provides useful information during testing.

There was a problem with the Jenkins CI system we were using from 2013 to 2016. Our tests would sometimes freeze due to a raised condition that caused failures and generated huge log outputs, slowing down the system. We considered rewriting XE pretty but found an alternative converter that worked faster and more reliably. Other teams also faced similar issues and built their own tools like exceed beautify in Swift.

"Flaky tests" is a term used to describe unreliable or inconsistent test results. A funny comment referred to them as "indeterministic", drawing a comparison with flaky croissants. Our UI tests also cover networking, which adds complexity but allows us to implicitly test various scenarios. In a recent pull request, there were 854 features being tested across multiple nodes, taking 35 minutes.

Mendoza is a distributed testing tool written in Swift. It allows you to run tests on multiple nodes, with each node running a subset of the tests. This can greatly accelerate your test setup by utilizing multiple simulators across different machines.

Accelerating Simulators Thomas uses four simulators and can accelerate them by almost a factor of 40.

'Savvy Tow' Test Shield "Savvy Tow", the Italian company where Thomas works, has a large test shield that reduces release time to one week. It is particularly useful for manual verification and large underhood refactorings.

The SPT UI test tunnels tool allows for communication between your test project and app project by opening a web server. It helps with stubbing network calls, setting defaults, and running locations.

Swift localhost is a tool focused on mocking network requests by calling against localhost instead of the API.

Distant Object is a remote invocation library for iOS, developed by Google. It allows you to call code across processors using proxies or NS coding. This means you can create a configuration class in your test process and send it over to be used in another process.

In Belgrade, not everyone is successful with their tests. Alonso from Mind/Body shared that his team faced difficulties in implementing UI tests. Even the most basic test would sometimes fail because they couldn't find a field or label.

Using accessibility identifiers is crucial to ensure stability and avoid issues with language changes. Additionally, always use 'wait for existence' to handle background processes or high system load. When constructing tests, think like a programmer and consider each test as its own entity. Utilize patterns like page object or robot pattern to package common functionality into objects.

- Log statements are a powerful tool for writing tests quickly and gaining confidence in their functionality. - Xcode provides excellent visualization of log statements, making it easier to understand the flow of code execution. - By using log statements generously, developers can gain valuable insights into the behavior and performance of their code.

Having more data when a test fails is helpful to identify the reasons behind the failure. Sometimes tests fail due to frustrating issues, such as new tutorials or changes in keyboard layouts. If you encounter failures related to these issues, it is recommended to use all grey and copy the decoder from another source.

Peace PDF Kit is an SDK that offers various view controllers for different use cases. It doesn't have a clear entrance point, but L gray is the recommended choice. L gray is named after Google's espresso framework and it uses version one of L gray, which is similar to kif but tracks system state. There's also a second version based on XE UI.

Elebrate is a tool that helps track busy resources in your app. It monitors various aspects such as gcd, timers, layer passes, view appearing, keyboard animations, video transitions and network requests.

In this chapter, we explore the categories that recognize scroll views and system body interactions. These categories consist of Objective-C functions methods that can be Swizzled.

Grey is a collection that tracks and delays functions. By creating an object to keep track of the system state, you can easily know when a function has completed.

Dispatch async is a powerful feature in Swift that allows for asynchronous execution of code. It is based on the underlying dispatch queue system. Whether using dispatch_async or its function-based version, it's important to know how to override and customize these functions when needed.

The DialD open source project is still available. One of the projects within DialD is focused on developing a dynamic linker.

DoD has a function called Dynamic Interpols that updates bindings on specified images. It replaces certain functions with custom ones and operates on a framework or binary.

Google Chrome has been using this code for years to work around a bug on Mac. Despite Apple not fixing the bug, Google Chrome has found success with it.

Despite its intimidating name, a cell array is not as scary as it seems. Surprisingly, it doesn't use the second 'e' for some unknown reason. Instead, there's another approach called Fish Hook that Facebook uses. Fish Hook is a straightforward library that allows dynamic rebinding of symbols in Mac or iOS binaries running on simulators or devices.

All frameworks, including KIF and Earl Grey, work with SwiftUI. The principles remain the same: views have accessibility identifiers which can be used for UI testing. To demonstrate this compatibility, I made a pull request on an Animal Crossing app written in SwiftUI and added tests using both XCUI and Earl Grey.

Today, we discussed various topics related to UI frameworks and automation. We talked about Keef, the oldest working UI framework. It is important to always use accessibility identifiers as an inspector for automation purposes. Additionally, we explored the history of automation and learned how to accelerate animations using busy waiting techniques. Running individual tests with Xcode build was also covered along with parallel testing using multiple nodes or simulators in DOS mode. We compared version 1 and version 2 of Earl Grey, highlighting differences such as dynamic interpolations used by Google Chrome. Lastly, we emphasized that everything learned today is applicable to Swift UI.

The speaker expresses gratitude for the opportunity to share information and thanks everyone who answered their questions on Twitter. They mention being available in a chatroom and on Twitter to discuss testing strategies.