[Meego-qa] MTF unit tests to QA

Timo Härkönen timo.harkonen at digia.com
Thu Mar 31 01:57:45 PDT 2011 

On Thu, 2011-03-31 at 11:15 +0300, Miroslav.Safr at tieto.com wrote:
> But why don't you use eat packages (eat-selftest)?
> http://build.meego.com/package/files?package=eat&project=Trunk
> 
> here is already solved the user change:
> http://build.meego.com/package/view_file?file=eat.spec&package=eat&project=Trunk&srcmd5=377517912783c83790c1f626b0032faa
> 
> # %post selftest  
> # cp /etc/inittab /etc/inittab.back  
> # echo "ea:5:once:su - meego -c \"eat-selftest.sh 120\"" >> /etc/inittab  
> 
> so the testrunner is executed as meego user..
> 

I know. I wrote the eat-selftest thingy and maintain the eat-* packages.
The reason why I'm not using it for this is simply that the OTS (Open
Test system) [1] used with test automation doesn't work that way. If we
used the selftest thingy we would still have to write a separate test
xml and package to fetch the results from the device and would bump into
timing issues, etc.

If it's a problem to have su - meego -c calls in the test definition I
can maintain the de-nightly-* packages that have them.

For clarification the system is build so that a worker pc automatically
install an image to n900, fetches the test definition xml files for
named tests and executes them remotely through ssh in the device and
gets results, etc. from the tests run and uploads them into qa-reports.
This all is done fully automated without a need for human interaction
with the system.

We'll be making a video to illustrate the process during next week.
Hopefully that will illustrate it and get rid of some confusion around
test automation.

-Timo

[1] http://wiki.meego.com/Quality/QA-tools/OTS

> I would like to keep this really simple because the test xml definition file is changing with the library version so than when we update library we would have to update also this project..
> BR
> M.Safr
> ________________________________________
> From: Timo Härkönen [timo.harkonen at digia.com]
> Sent: Thursday, March 31, 2011 10:26 AM
> To: Safr Miroslav
> Cc: meego-qa-bounces at lists.meego.com
> Subject: RE: [Meego-qa] MTF unit tests to QA
> 
> On Thu, 2011-03-31 at 10:18 +0300, Miroslav.Safr at tieto.com wrote:
> > Thanks. This looks good.
> > I don't understand why there is that de-nightly package because for us it works normally
> > testrunner + eat-device and eat-selftest and than it executes all tests packages which are installed (/usr/lib/something-tests/tests.xml).
> >
> 
> I needed to do the package to have it call tests with su - meego -c
> since the automation system connects to the device as root. The selftest
> thing runs as the user meego.
> 
> -Timo
> 
> > The results are right.
> > => Executed 190 cases. Passed 183 Failed 7
> > in libmeegotouch 0.20.89 which is currently in trunk we did not adapted all tests to meego environment.
> > In newer version the pass rate will grow because we already solved few of those..
> >
> > BR
> > Mirek Safr
> > ________________________________________
> > From: meego-qa-bounces at lists.meego.com [meego-qa-bounces at lists.meego.com] On Behalf Of Timo Härkönen [timo.harkonen at digia.com]
> > Sent: Thursday, March 31, 2011 8:04 AM
> > To: Wang, Jing J
> > Cc: meego-qa at lists.meego.com
> > Subject: Re: [Meego-qa] MTF unit tests to QA
> >
> > Results from the first nightly:
> > http://qa-reports.meego.com/1.2/Core/Nightly-Automated/N900
> >
> > Includes: libmeegotouch-tests and tracker-tests
> >
> > I'm using modified versions of the tests.xml files to run those.
> > Basically I created new packages de-nightly-libmeegotouch-tests and
> > de-nightly-tracker-tests that just include a modified version of
> > tests.xml file. These can be found here:
> > https://build.pub.meego.com/package/files?package=de-nightly&project=home%3Atimoph
> >
> > You can also see the used test definition xmls as attachments to the
> > test run in qa-reports
> >
> > Related to how to run libmeegotouch-tests I did some experimentation
> >
> > 1) run them as root as they are
> > => Executed 190 cases. Passed 48 Failed 142
> > 2) set xhost + to enable root x access
> > => Executed 190 cases. Passed 77 Failed 113
> > 3) called the steps with su - meego -c
> > => Executed 190 cases. Passed 183 Failed 7
> >
> > Number 3 is currently used in the test run
> >
> > -Timo
> >
> > On Wed, 2011-03-30 at 10:42 +0300, Timo Härkönen wrote:
> > > On Wed, 2011-03-30 at 10:18 +0300, Wang, Jing J wrote:
> > > > >> I'd prefer test steps that need to connect to X to be wrapped with 'su - meego -c "/path/to/teststep"'
> > > > I can't agree with that as it looks kind of hack. Test case should not be assumed executing as root naturally. Actually I think it is better to assume test run as default user, and may change to root if required.
> > > >
> > >
> > > yes. it's a bit hackish but that's the current working way to do it in
> > > full automation. I've proposed a testdefinition level fix for it in
> > > https://bugs.meego.com/show_bug.cgi?id=11276
> > >
> > > -Timo
> > >
> > > > -----Original Message-----
> > > > From: meego-qa-bounces at lists.meego.com [mailto:meego-qa-bounces at lists.meego.com] On Behalf Of Timo H?rk?nen
> > > > Sent: Wednesday, March 30, 2011 3:07 PM
> > > > To: Miroslav.Safr at tieto.com
> > > > Cc: meego-qa at lists.meego.com
> > > > Subject: Re: [Meego-qa] MTF unit tests to QA
> > > >
> > > > Hi
> > > >
> > > > On Wed, 2011-03-30 at 09:40 +0300, Miroslav.Safr at tieto.com wrote:
> > > > > Hi,
> > > > > we would need to add our unit and low level funcitonal tests to QA
> > > > > asap:
> > > > >
> > > > > 1st phase (these are really important, you can start with them):
> > > > > libmeegotouch-tests
> > > > > meegotouch-demos-tests
> > > > >
> > > >
> > > > Working on this. setting dbus env is not enough. By default root cannot
> > > > access X in MeeGo and I'm a bit worried about the effect on results when
> > > > enabling it. I'd prefer test steps that need to connect to X to be
> > > > wrapped with 'su - meego -c "/path/to/teststep"'
> > > >
> > > > Tested two of of the packages listed below and ended up filing bugs
> > > > against both of them. please fix these issues. I'll continue checking
> > > > the other tests after I get libmeegotouch-tests running. To get them
> > > > running quickly I may have to fork the test package.
> > > >
> > > > > 2nd phase (these are tested and works ok):
> > > > > contextkit-tests
> > > >
> > > > https://bugs.meego.com/show_bug.cgi?id=15009
> > > >
> > > > > libcontentaction-tests
> > > > > tracker-tests
> > > >
> > > > https://bugs.meego.com/show_bug.cgi?id=15006
> > > >
> > > > > tumbler-tests
> > > > > libqtsparql-tests
> > > > > meegotouch-controlpanel-tests
> > > > > meegotouch-controlpanelapplets-tests
> > > > > meegotouch-feedback-tests
> > > > > meegotouch-feedbackreactionmaps-tests
> > > > > meegotouch-inputmethodengine-tests
> > > > > meegotouch-inputmethodframework-tests
> > > > > meegotouch-inputmethodkeyboard-tests
> > > > > meegotouchcp-datetime-tests
> > > > > meegotouchcp-language-tests
> > > > > meegotouchcp-regionformat-tests
> > > > >
> > > > >
> > > > > 3rd phase (these should work but can be added as last):
> > > > > meegotouch-benchmarks (comming soon)
> > > > > libprofile-qt-tests
> > > > > meegotouch-applauncherd-tests
> > > > > meegotouch-applifed-tests
> > > > > meegotouch-compositor-tests
> > > > > meegotouch-home-tests
> > > > > meegotouch-systemui-tests
> > > > > qmf-tests
> > > > >
> > > > >
> > > > > 4th phase:
> > > > > we have comming T-Driver functional tests soon..
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > execution:
> > > > > #we need to interact with X server
> > > > > export DISPLAY=:0
> > > > > #we also need dbus
> > > > > cat /proc/`pidof mcompositor`/environ |  tr '\000' '\n' | grep
> > > > > DBUS_SESSION_BUS_ADDRESS
> > > > > export DBUS_SESSION_BUS_ADDRESS=...
> > > >
> > > > eat-device package used in automation images sets up the dbus
> > > > environment among other things.
> > > >
> > > > > #for VKB tests we need this
> > > > > export QT_IM_MODULE=MInputContext
> > > > >
> > > >
> > > > This one I did not know about. Is it set for the user MeeGo by default?
> > > > if so I'd would call test steps related to it with su - meego -c "step"
> > > >
> > > > >
> > > > > #execution
> > > >
> > > > All things listed below are done automatically by ots except we don't
> > > > convert the results to html. Instead we send them to
> > > > qa-reports.meego.com. Results are not being sent there yet - I'll inform
> > > > this list when they are.
> > > >
> > > > -Timo
> > > >
> > > > > testrunner-lite -f /usr/share/<package name>/tests.xml -v
> > > > > -o /home/meego/test_result.xml
> > > > > #for example
> > > > > testrunner-lite -f /usr/share/libmeegotouch-tests/tests.xml -v
> > > > > -o /home/meego/test_result.xml
> > > > >
> > > > > #here can be also used -L parameter to redirect output stright to host
> > > > > as CITA does
> > > > >
> > > > > #get the results from device
> > > > > scp /home/meego/test_result.xml user at 192.168.2.14:/home/user/
> > > > > #or from host
> > > > > scp meego at 192.168.2.15:/home/meego/test_result.xml /home/user/
> > > > >
> > > > > #get nice html
> > > > > xsltproc /home/msafr/tools/testresults.xsl ./test_result.xml >
> > > > > test_result.html
> > > > >
> > > > > #for example:
> > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.xml
> > > > > http://af09.nrln.net/qt-mtf/2011_03_17_N900/qtmtf_n900_test_results.html
> > > > >
> > > > >
> > > > > Please let me know if some results are available or if you have some
> > > > > problems with execution.
> > > > >
> > > > > Thanks.
> > > > > BR
> > > > >
> > > > > Miroslav Safr
> > > > >
> > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > MeeGo-qa mailing list
> > > > MeeGo-qa at lists.meego.com
> > > > http://lists.meego.com/listinfo/meego-qa
> > >
> > >
> > > _______________________________________________
> > > MeeGo-qa mailing list
> > > MeeGo-qa at lists.meego.com
> > > http://lists.meego.com/listinfo/meego-qa
> >
> >
> > _______________________________________________
> > MeeGo-qa mailing list
> > MeeGo-qa at lists.meego.com
> > http://lists.meego.com/listinfo/meego-qa
> 
>

More information about the MeeGo-qa mailing list