This blog post was authored by Craig Horsfield, a Senior SDET on the Windows Phone Test and Operations team.
– Adam
Testing your app throughout the development process can help you create a really great Windows Phone app. Testing helps ensure that your app is effectively represented in the Windows Phone Store as an app that offers Windows Phone users a high level of performance and quality. A small investment in the key areas described in this post can help you bypass common errors early in the development process, and help you get positive results in the long term. This post is part 2 of a 3-part series that outlines key test areas that you should consider before submitting your app to the Store. See part 1 for additional details.
Push notification and Live Tiles
Live Tiles
Live Tiles are updated through push notification or through an app’s periodic background agent. When testing these areas, you should accelerate the update time so that you can test more rapidly. For more info, see Tiles for Windows Phone.
Test scenario |
Details |
1. Verify that your Live Tile updates. |
Verify that the Live Tile updates after it has been pinned to Start. |
2. Verify that your Live Tile stops updating. |
If this setting is disabled in the app, make sure the Live Tile stops updating. |
3. Verify that the Live Tile updates via a periodic agent. |
If the Live Tile is updated via a periodic agent, verify the update on all network types, and verify that the Live Tile updates when there is no network, for example, in Airplane mode. |
4. Verify that the Live Tile is working and present after an app upgrade. |
None. |
5. Verify that the Live Tile is working and present after an app upgrade and subsequent restarting the device. |
If updating the app, make sure that you don’t change the TokenID in the WMAppManifest.xml file. This results in your Live Tile being removed from Start when the device is restarted. |
6. If using a background agent, verify that the agent doesn’t crash or terminate. |
This results in disabling the agent and Tile updates will fail. |
Notifications
Apps that use notifications normally are used within background agents. Test these notifications to ensure that they work properly. For more info, see Notifications for Windows Phone.
Test scenario |
Details |
1. Verify that notifications are received. |
None. |
2. Verify what happens when you tap the notification. |
Tapping the notification launches the user into the app in the correct state. |
3. Verify that the app doesn’t overuse toast notifications. |
None. |
Agents
Background agents provide key abilities for an app, but they also introduce some specific test considerations. Agents can be disabled and enabled in the phone’s settings, on the Settings > Application > Background Tasks screen. The app needs to be aware of the state of the agent. Resources available to agents also are restricted. A key point to remember is that when running in the debugger, these restrictions are not enforced so it’s important to test your app outside of the debugger and track the resource that it is using. For more info, see Background agents for Windows Phone.
Test scenario |
Details |
1. Verify initial app start. |
The agent starts when it’s needed. |
2. Verify that the app handles state in which the agent has been disabled by the user in Settings\Application\Background Tasks. |
Additionally, verify that the app performs as follows:
|
3. Agents are disabled in low-battery conditions – test that the app can handle these states. |
None. |
4. Agents that crash or are terminated by the OS for exceeding resources, on two successive crashes will be disabled. Ensure your app handles this state. |
In this state the foreground app has to reschedule the agent. This state can be queried from the agent API. For more info, see Background agent best practices for Windows Phone. |
5. Resource-intensive agent only runs when on power and Wi-Fi. Ensure app handles this correctly. |
None. |
6. Verify that the agent stays below the required CPU and memory caps. |
None. |
This table lists the required CPU and memory caps, by agent on Windows Phone 8:
Agent |
CPU |
Memory |
CBE |
10% |
N/A |
GBA |
10% |
11 MB |
BAP |
10% |
20 MB |
FA |
5% |
10 MB |
VoIP agent |
20% |
60 MB |
Media, audio, and video
Media, audio, and video throughout your app should be tested. Consider these test scenarios in the table below. For more info, see Media for Windows Phone.
Test scenario |
Details |
1. Preserve audio state. |
|
2. Verify the Universal Audio Control during audio playback. |
|
3. Verify audio playback via a background audio agent. |
|
4. Media Source |
|
5. Video playback |
|
6. S-Video playback in FAS scenarios |
|
7. Bandwidth |
Visual studio phone emulator can manipulate the network quality and type to aid in this testing. |
|
|
9. FM Radio |
|
Geolocation
With Windows Phone 8, you can create apps that use info about the phone’s physical location. Scenarios for location-aware apps include checking the user into a web service using the user’s instantaneous location, and tracking the user’s location as it changes over a period of time. The location data the phone provides comes from multiple sources, including GPS, Wi-Fi, and cellular. The visual studio phone emulator can be used to simulate these location changes. The location can be moved manually or simulate a sequence of location changes. For more info, see Location for Windows Phone 8.
Test scenario |
Details |
1. Prevent PositionChange events from firing too often and placing CPU processing load on the app. |
Set the MovementThreshold property to the appropriate value for the app needs and make sure that events fire only outside of that threshold. |
2. Handle the unknown location state. |
Ensure the app can handle NA values. Test the app when it has no location state. |
3. Test a large change in location data so that any internal calculations in the app do not fail in these cases. |
For example, position changes greater than 1 degree. This can happen if the phone has had no valid location data for some period of time. |
4. Test app in all hemispheres: North/South and West/East. |
Ensure that your calculations on negative degree values are correct. |
5. Test 0-degree and 180-degree location for longitude and latitude. |
None. |
Design considerations when using location:
Test scenario |
Details |
1. Use a lower level of accuracy to save on battery power if applicable. |
None. |
2. Check the Position.Location.IsUnknown and GeoPositionStatus properties to ensure that the location is valid. |
None. |
Resource usage and performance
Windows Phone apps need to be designed to efficiently use and preserve the limited resources of the phone platform. You want to design your apps to use the least possible CPU cycles, to access networks efficiently and purposefully, and to make the best use of visual components—graphics, bright colors, and themes use more power than a simpler UI.
Test areas to consider:
Test scenario |
Details |
1. Check for a nonresponsive or jerky UI. |
This could be caused by long-running activity on the UI thread. |
1. Check for Memory leaks – repeat scenarios multiple times to detect memory leaks during specific sequences. |
Repeat page navigation could increase app memory. |
2. Check for rapid battery drain. |
Can be caused by using the app for long periods of time. |
Tips and tools:
- The Visual Studio app profiler, on the debug menu, is a key tool for looking into app memory and CPU usage.
- Using the EnabledFrameRateCounter/EnableRedrawRegionscan be useful when app testing.
- Use APIs in the DeviceStatus class track memory usage in the app, especially ApplicationCurrentMemoryUsage and ApplicationPeakMemoryUsage.
- See App performance considerations for Windows Phone for an overview of app performance for Windows Phone.
In part 3 of this series, we’ll discuss additional areas and testing approaches to consider, including to network resources, device-specific tests for hardware variations, display resolution, app upgrade, common Store test cases, and real-world testing.