[meego-releases] MeeGo-handset-IA-Aava-Acceptance Test Report for Trunk:Testing-20100917

Fri Sep 17 00:46:22 PDT 2010

Test Objective
-----------------------
Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

Build Location
-----------------------

Kickstart file is at: gitorious.org:meego-os-base/image-configurations.git

Kickstart File: handset-ia32-aava-mtf.ks

We have modified the kickstart file a little bit to point the core and handset repo to Trunk:Testing repo (http://download.meego.com/testing ) to create the sanity test image against Trunk:Testing

The packages list of image, please see the detail report in http://wiki.meego.com/Quality/HandsetTestReport/AavaAcceptance20100917

Key Packages

kernel-mrst.i586 2.6.35.3-8.3
xorg-x11-server-Xorg-setuid.i586 1.9.0-2.1
mesa-libGL.i586 7.8.99.1~gitb018ea19a3-4.1
glibc.i686 2.11.90-19.7

Test Environment

-----------------------

l  Hardware: Aava Koski DV1/DV2

l  Test equipment (bluetooth devices, wlan devices, network simulators, sim simulators etc): wireless AP

Quality Summary
---------------------------

Highlights:
Lowlights:

n  Video player is unable to play the ogg files without audio streams embedded (5712<http://bugs.meego.com/show_bug.cgi?id=5712>)

n  Music player's theme and playing functions are still incorrect(6558<http://bugs.meego.com/show_bug.cgi?id=6558>, 6559<http://bugs.meego.com/show_bug.cgi?id=6559> ),  it will be fixed in 0.0.15 version.

n  Unable to display home screen automatically unless tapping the screen (5705<http://bugs.meego.com/show_bug.cgi?id=5705>).

n  SMS history fail to display(3670<http://bugs.meego.com/show_bug.cgi?id=3670>), this function is blocked by Bug 5953<http://bugs.meego.com/show_bug.cgi?id=5953> - [QMF] account creation does not work.

n  User may not find bluetooth device by about 2/3 possibility(5417<http://bugs.meego.com/show_bug.cgi?id=5417>)

n  VKB launches incorrectly(5706<http://bugs.meego.com/show_bug.cgi?id=5706>).

Performance data:
Boot time:57s
Browser start time: 8s
Audio player start time: 4.5s
Video player start time: 4s
The other performance data,  log and packages list, please see the attachment under http://wiki.meego.com/Quality/HandsetTestReport/AavaAcceptance20100917#Quality_Summary

 Test Result
-----------------------

Result Overview
-----------------------
Total TCs

Passed

Failed

Not testable

Executable rate

Pass rate of total

Pass rate of executed

36

23

10

3

91.67%

63.89%

69.7%

Detailed Test Results
---------------------------

Create Image

*  Create image with MIC2

Pass

System startup

*  Image Boot from SD card

Pass

Home screen

*  Check if core applications can be launched from app-launcher

Pass

*  switch between different applications

Pass

*  Check home screen theme and layout

Fail

Bug 3921<http://bugs.meego.com/show_bug.cgi?id=3921> - [REG] Some buttons (App-launcher, Home, Close) and app icons (Settings/Lock icons) are rendered as red blocks
Bug 5998<http://bugs.meego.com/show_bug.cgi?id=5998> - Launching indicator of applications in quick launch bar has calibration issue.

*  Display orientation detection

Pass

Fennec browser

Browsing website over wifi (WEP secured by default, WPA is WEP is not testable)

Pass

*  Browsing website over 3G (WCDMA)

N/A

Not support in meego 1.1 release.

*  Browser gesture support

Pass

Dialer

*  Dial one outgoing call, accept by other party, terminate this call (phonesim, GSM & WCDMA) (GSM and WCDMA cannot be covered until real modem supported,Will use phonesim to test before real modem support)

Pass

*  Display PhoneBook

Pass

*  Check call history

Pass

SMS

*  Send SMS ( phonesim, 2G & 3G ) (2G and 3G cannot be covered until real modem supported,Will use phonesim to test before real modem support)

Pass

*  Check message history (received, sent, draft)

Fail

Bug 3670<http://bugs.meego.com/show_bug.cgi?id=3670> - [SMS] The SMS history fail to show.

Connectivity

* scan wifi, check wifi connection and disconnection

Pass

*  Browse website and download some files through wifi

Pass

*  check bluetooth connection (pairing and connect to bluetooth handset)

Fail

Bug 5417<http://bugs.meego.com/show_bug.cgi?id=5417> - [Aava] when booting, user may not find bluetooth device by about 2/3 possibility

*  Connect device to computer with usb cable, check the usb mass storage device is detected by computer

Fail

Bug 5034<http://bugs.meego.com/show_bug.cgi?id=5034> - [MRST] rmmod g_file_storage leads to panic

*  Check sshd connection: connect board using usb gadget

Fail

Bug 5475<http://bugs.meego.com/show_bug.cgi?id=5475> - [AAVA] rmmod g_cdc leads to panic

Run time status/Settings/Control Panel

*  Check if runtime status are correct on toolbar

Pass

Virtual keyboard

*  Check if virtual keyboard can be launched automatically with apps which  need input text field (fennec, SMS)

Fail

Bug 5706<http://bugs.meego.com/show_bug.cgi?id=5706> - VKB launched incorrectly with apps which need input text field.

*  Support text input

Fail

Bug 5652<http://bugs.meego.com/show_bug.cgi?id=5652> - Some keys of VKB are rendered as red block

Photo viewer

*  Check if the UI is correct (top bar, menu, button, etc.)

Pass

*  Check if photo thumbnail index works

Pass

*  Check if photo could be played

Pass

Audio

*  Check if the UI is correct (top bar, menu, button, play queue etc.)

Fail

Bug 6558<http://bugs.meego.com/show_bug.cgi?id=6558> - [REG]Music Player doesn't have playing control buttons and progress bar

*  Check if music could be played (WAV, AAC & MP3 format ) on loudspeaker (play music by command line if audio player is not ready. aplay to play music file)

Fail

Bug 6559<http://bugs.meego.com/show_bug.cgi?id=6559> - [REG]Music Player crashed when tapping audio file list.

Video

*  Check if the UI is correct (top bar, menu, button, play queue etc.)

Pass

*  Check if video thumbnail index works

Pass

*  Check if video could be played (MP4, WMV &  3GP-H264)

Fail

Bug 5712<http://bugs.meego.com/show_bug.cgi?id=5712> - Video player is unable to play the ogg files without audio streams embedded

Contacts

*  Check if it works with dialer, SMS

Pass

Security & Power

*  Check PIN activation/deactivation (Need GUI support)

N/A

*  PIN requested at boot

N/A

*  Power Off / Power On

Pass

Touchscreen

*  Use touch screen to operate the system

Pass

Installation

*  Use zypper to install a package

Pass

Issue Summary
---------------------------
Top bugs:
Bug 5417<http://bugs.meego.com/show_bug.cgi?id=5417> - [Aava] when booting, user may not find bluetooth device by about 2/3 possibility
Bug 3670<http://bugs.meego.com/show_bug.cgi?id=3670> - [SMS] The SMS history fail to show.
Bug 5712<http://bugs.meego.com/show_bug.cgi?id=5712>- Video player is unable to play the ogg files without audio streams embedded
Bug 5706<http://bugs.meego.com/show_bug.cgi?id=5706> - VKB launched incorrectly with apps which need input text field.
Bug 6558<http://bugs.meego.com/show_bug.cgi?id=6558> - [REG]Music Player doesn't have playing control buttons and progress bar.
Bug 6559<http://bugs.meego.com/show_bug.cgi?id=6559> - [REG]Music Player crashed when tapping audio file list.

New bugs:

Acceptance test trends
---------------------------
[cid:image002.png at 01CB566B.45C52AF0]

Detailed Acceptance Test Report
---------------------------
http://wiki.meego.com/Quality/HandsetTestReport/AavaAcceptance20100917

Best Regards,
Yi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.meego.com/pipermail/meego-releases/attachments/20100917/26d682af/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 64631 bytes
Desc: image002.png
URL: <http://lists.meego.com/pipermail/meego-releases/attachments/20100917/26d682af/attachment-0001.png>