Testing your Windows Phone app – Part 2

This blog post was authored by Craig Horsfield, a Senior SDET on the Windows Phone Test and Operations team.

– Adam

Testing your app throughout the development process can help you create a really great Windows Phone app. Testing helps ensure that your app is effectively represented in the Windows Phone Store as an app that offers Windows Phone users a high level of performance and quality. A small investment in the key areas described in this post can help you bypass common errors early in the development process, and help you get positive results in the long term. This post is part 2 of a 3-part series that outlines key test areas that you should consider before submitting your app to the Store. See part 1 for additional details.

Push notification and Live Tiles

Live Tiles

Live Tiles are updated through push notification or through an app’s periodic background agent. When testing these areas, you should accelerate the update time so that you can test more rapidly. For more info, see Tiles for Windows Phone.

Test scenario	Details
1. Verify that your Live Tile updates.	Verify that the Live Tile updates after it has been pinned to Start.
2. Verify that your Live Tile stops updating.	If this setting is disabled in the app, make sure the Live Tile stops updating.
3. Verify that the Live Tile updates via a periodic agent.	If the Live Tile is updated via a periodic agent, verify the update on all network types, and verify that the Live Tile updates when there is no network, for example, in Airplane mode.
4. Verify that the Live Tile is working and present after an app upgrade.	None.
5. Verify that the Live Tile is working and present after an app upgrade and subsequent restarting the device.	If updating the app, make sure that you don’t change the TokenID in the WMAppManifest.xml file. This results in your Live Tile being removed from Start when the device is restarted.
6. If using a background agent, verify that the agent doesn’t crash or terminate.	This results in disabling the agent and Tile updates will fail.

Notifications

Apps that use notifications normally are used within background agents. Test these notifications to ensure that they work properly. For more info, see Notifications for Windows Phone.

Test scenario	Details
1. Verify that notifications are received.	None.
2. Verify what happens when you tap the notification.	Tapping the notification launches the user into the app in the correct state.
3. Verify that the app doesn’t overuse toast notifications.	None.

Agents

Background agents provide key abilities for an app, but they also introduce some specific test considerations. Agents can be disabled and enabled in the phone’s settings, on the Settings > Application > Background Tasks screen. The app needs to be aware of the state of the agent. Resources available to agents also are restricted. A key point to remember is that when running in the debugger, these restrictions are not enforced so it’s important to test your app outside of the debugger and track the resource that it is using. For more info, see Background agents for Windows Phone.

Test scenario	Details
1. Verify initial app start.	The agent starts when it’s needed.
2. Verify that the app handles state in which the agent has been disabled by the user in Settings\Application\Background Tasks.	Additionally, verify that the app performs as follows: App notifies the user that it’s not available and continues to work as expected. App notifies that the agent is needed and re-enables it.
3. Agents are disabled in low-battery conditions – test that the app can handle these states.	None.
4. Agents that crash or are terminated by the OS for exceeding resources, on two successive crashes will be disabled. Ensure your app handles this state.	In this state the foreground app has to reschedule the agent. This state can be queried from the agent API. For more info, see Background agent best practices for Windows Phone.
5. Resource-intensive agent only runs when on power and Wi-Fi. Ensure app handles this correctly.	None.
6. Verify that the agent stays below the required CPU and memory caps.	None.

This table lists the required CPU and memory caps, by agent on Windows Phone 8:

Agent	CPU	Memory
CBE	10%	N/A
GBA	10%	11 MB
BAP	10%	20 MB
FA	5%	10 MB
VoIP agent	20%	60 MB

Media, audio, and video

Media, audio, and video throughout your app should be tested. Consider these test scenarios in the table below. For more info, see Media for Windows Phone.

Test scenario	Details
1. Preserve audio state.	If your app plays an audio sound (clip) when it starts, your app should not pause the currently playing audio. The app should preserve and not interfere with the currently audio playing on the device. If your app plays back audio content from a background agent or a foreground app, your app should pause any currently playing audio.
2. Verify the Universal Audio Control during audio playback.	Verify Play. Verify Pause / Resume. Verify Skip Next / Skip Previous. Skip to last and back to the first track, etc. Some apps may take time to process these calls. Verify that the UI is set to disabled while the app is processing these calls to prevent multiple invokes. Verify volume controls. Audio continues under lock. Audio continues when app is on the back stack. Verify all expected codecsthat the app offers can be played back. Track info is displayed in the UI.
3. Verify audio playback via a background audio agent.	Test all of the above test cases. Audio continues when the app is terminated but the background agent is allowed to continue. Audio continues to play when device screen is locked. App agent remains below the app 20-MB cap.
4. Media Source	Verify media from a network stream. Verify media from the app’s ISO store. Verify media playing from the media library on the phone. (This case requires the correct capability in the app manifest.)
5. Video playback	Verify all states: Play from start Pause, resume Skip forward and back Change states rapidly
6. S-Video playback in FAS scenarios	On navigate away event, the app needs to record the current media stream location. Play media stream return to the phone start page and then switch back to the app. Verify the stream is preserved, should continue from the previous point. Play media stream and force the app to tombstone via Visual Studio. In this case, the app should continue from previous point but will have to load the stream and forward it to the location saved in navigate away and deactivated events.
7. Bandwidth	Verify media stream playback on different networks and bandwidth. App should adjust playback quality and codec as need. Verify network dropped scenario and the app handles this and informs the user correctly. Visual studio phone emulator can manipulate the network quality and type to aid in this testing.
8. Media hub integration	Verify app is visible in the Media hub, manifest must have HubType=1. App has to use MediaHistory and MediaHistoryItemclasses in this scenario. Verify post market place ingestion where the correct capabilities are set on the app manifest to enable it in the hub. (Use the App Beta submission process to test this.) App must update the, ‘now playing’ tile, ‘history’ and ‘new’ list. Verify app plays correct stream when launched from history or new list.
9. FM Radio	App is not compatible with all hardware and versions of Windows Phone. Test app on the correct platform. Verify that app sets the correct region based on the phone location; this enables the correct frequency stepping. If incorrect it will not tune well or find stations. Set the correct power modes. For example, verify that the user is not playing media from the FM radio, then set RadioPowerMode to off.

Geolocation

With Windows Phone 8, you can create apps that use info about the phone’s physical location. Scenarios for location-aware apps include checking the user into a web service using the user’s instantaneous location, and tracking the user’s location as it changes over a period of time. The location data the phone provides comes from multiple sources, including GPS, Wi-Fi, and cellular. The visual studio phone emulator can be used to simulate these location changes. The location can be moved manually or simulate a sequence of location changes. For more info, see Location for Windows Phone 8.

Test scenario	Details
1. Prevent PositionChange events from firing too often and placing CPU processing load on the app.	Set the MovementThreshold property to the appropriate value for the app needs and make sure that events fire only outside of that threshold.
2. Handle the unknown location state.	Ensure the app can handle NA values. Test the app when it has no location state.
3. Test a large change in location data so that any internal calculations in the app do not fail in these cases.	For example, position changes greater than 1 degree. This can happen if the phone has had no valid location data for some period of time.
4. Test app in all hemispheres: North/South and West/East.	Ensure that your calculations on negative degree values are correct.
5. Test 0-degree and 180-degree location for longitude and latitude.	None.

Design considerations when using location:

Test scenario	Details
1. Use a lower level of accuracy to save on battery power if applicable.	None.
2. Check the Position.Location.IsUnknown and GeoPositionStatus properties to ensure that the location is valid.	None.

Resource usage and performance

Windows Phone apps need to be designed to efficiently use and preserve the limited resources of the phone platform. You want to design your apps to use the least possible CPU cycles, to access networks efficiently and purposefully, and to make the best use of visual components—graphics, bright colors, and themes use more power than a simpler UI.

Test areas to consider:

Test scenario	Details
1. Check for a nonresponsive or jerky UI.	This could be caused by long-running activity on the UI thread.
1. Check for Memory leaks – repeat scenarios multiple times to detect memory leaks during specific sequences.	Repeat page navigation could increase app memory.
2. Check for rapid battery drain.	Can be caused by using the app for long periods of time.

Tips and tools:

The Visual Studio app profiler, on the debug menu, is a key tool for looking into app memory and CPU usage.
Using the EnabledFrameRateCounter/EnableRedrawRegionscan be useful when app testing.
Use APIs in the DeviceStatus class track memory usage in the app, especially ApplicationCurrentMemoryUsage and ApplicationPeakMemoryUsage.
See App performance considerations for Windows Phone for an overview of app performance for Windows Phone.

In part 3 of this series, we’ll discuss additional areas and testing approaches to consider, including to network resources, device-specific tests for hardware variations, display resolution, app upgrade, common Store test cases, and real-world testing.