Produced by Araxis Merge on 3/31/2017 1:06:36 PM Central Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.
| # | Location | File | Last Modified |
|---|---|---|---|
| 1 | EPIP_submissions.zip\EPIP_submissions\docs\XU_8.0_679 | EPIP_Test_Evaluation_(XU_8.0_679).docx | Fri Mar 31 16:50:28 2017 UTC |
| 2 | EPIP_submissions.zip\EPIP_submissions\docs\XU_8.0_679 | EPIP_Test_Evaluation_(XU_8.0_679).docx | Fri Mar 31 17:58:05 2017 UTC |
| Description | Between Files 1 and 2 |
|
|---|---|---|
| Text Blocks | Lines | |
| Unchanged | 4 | 552 |
| Changed | 0 | 0 |
| Inserted | 0 | 0 |
| Removed | 3 | 3 |
| Whitespace | |
|---|---|
| Character case | Differences in character case are significant |
| Line endings | Differences in line endings (CR and LF characters) are ignored |
| CR/LF characters | Not shown in the comparison detail |
No regular expressions were active.
| 1 | Existing P roduct Int ake Progra m (EPIP) | ||
| 2 | Patch XU*8 .0*679 | ||
| 3 | Test Evalu ation | ||
| 4 | |||
| 5 | |||
| 6 | |||
| 7 | Department of Vetera ns Affairs | ||
| 8 | March 2017 | ||
| 9 | Version 1. 0 | ||
| 10 | |||
| 11 | |||
| 12 | |||
| 13 | Revision H istory | ||
| 14 | Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Communi cations Pl an has bee n baseline d. | ||
| 15 | Date | ||
| 16 | Version | ||
| 17 | Descriptio n | ||
| 18 | Author | ||
| 19 | 03/21/2017 | ||
| 20 | 1.0 | ||
| 21 | Initial do cument. | ||
| 22 | EPIP Proje ct Team | ||
| 23 | |||
| 24 | Artifact R ationale | ||
| 25 | The test e valuation document i s the prim ary output of the te st and eva luation pr ocess, an integral p art of the systems e ngineering process, which iden tifies lev els of per formance a nd assists the devel oper in co rrecting d eficiencie s. | ||
| 26 | |||
| 27 | Table of C ontents | ||
| 28 | 1.Test Eva luation In troduction 1 | ||
| 29 | 1.1.Test E valuation Scope1 | ||
| 30 | 1.2.Test A rchitectur e1 | ||
| 31 | 1.3.Test E nvironment /Configura tion1 | ||
| 32 | 1.4.Instal lation Pro cess2 | ||
| 33 | 2.Test Dat a2 | ||
| 34 | 3.Issues2 | ||
| 35 | 4.Test Exe cution Log 2 | ||
| 36 | 5.Test Def ect Log3 | ||
| 37 | 6.Test Res ults Summa ry3 | ||
| 38 | 6.1.Defect Severity and Priori ty Levels3 | ||
| 39 | 6.2.Total Defects by Severity Level3 | ||
| 40 | 6.3.Breakd own of Tes t Results3 | ||
| 41 | 6.4.Perfor mance Test ing3 | ||
| 42 | 7.Test Cov erage4 | ||
| 43 | 7.1.Requir ements Cov ered4 | ||
| 44 | 7.2.Sectio n 508 Comp liance Cov erage4 | ||
| 45 | 8.Suggeste d Actions4 | ||
| 46 | 9.Defect S everity an d Priority Definitio ns4 | ||
| 47 | 9.1.Defect Severity Level5 | ||
| 48 | 9.1.1.Seve rity Level 1 – Criti cal5 | ||
| 49 | 9.1.2.Seve rity Level 2 - High5 | ||
| 50 | 9.1.3.Seve rity Level 3 - Mediu m6 | ||
| 51 | 9.1.4.Seve rity Level 4 - Low6 | ||
| 52 | 9.2.Priori ty Classif ications6 | ||
| 53 | 9.2.1.Prio rity 1 - R esolve Imm ediately6 | ||
| 54 | 9.2.2.Prio rity 2 - G ive High A ttention6 | ||
| 55 | 9.2.3.Prio rity 3 - N ormal Queu e6 | ||
| 56 | 9.2.4.Prio rity 4 - L ow Priorit y6 | ||
| 57 | 10.Optiona l Tables, Charts, an d Graphs7 | ||
| 58 | 11.Documen t Approval Signature s8 | ||
| 59 | Appendix A - Test Ex ecution Lo g9 | ||
| 60 | Appendix B – Defect Log10 | ||
| 61 | |||
| 62 | Test Evalu ation Intr oduction | ||
| 63 | The purpos e of this Test Evalu ation is t o: | ||
| 64 | Identify t he testing approach used. | ||
| 65 | Present a summary an alysis of the key te st results from the remediatio n of this intake for review an d assessme nt by desi gnated sta keholders. | ||
| 66 | Provide a general st atement of the quali ty of the system und er test. | ||
| 67 | Make recom mendations for futur e testing efforts. | ||
| 68 | Test Evalu ation Scop e | ||
| 69 | The scope of this Te st Evaluat ion is to verify the functiona lity of th e Patch XU *8.0*679 c ode modifi cation, as determine d by Funct ional, Com ponent Int egration/S ystem, and Regressio n testing. Testing a ctivities followed t he specifi cations ou tlined in the follow ing Master Test Plan : XU*8.0*6 79 Master Test Plan (included in Appendi x A). | ||
| 70 | Test Archi tecture | ||
| 71 | Following are the EP IP test ac counts use d by the L eidos Deve lopment an d SQA Test ing teams to test XU *8.0*679. | ||
| 72 | Developmen t Test Acc ounts | ||
| 73 | (For Unit Testing) | ||
| 74 | SQA Test A ccounts | ||
| 75 | (For Funct ional, Reg ression, a nd Compone nt Integra tion and S ystem Test ing) | ||
| 76 | VistAS1 ( alternate name: D1S1 ) | ||
| 77 | VistAG1 (a lternate n ame: D1G1) | ||
| 78 | VistAS2 (a lternate n ame: D1S2) –for CPRS GUI testi ng only | ||
| 79 | VistAG2 (a lternate n ame: D1G2) –for CPRS GUI testi ng only | ||
| 80 | Test Envir onment/Con figuration | ||
| 81 | The EPIP t est accoun ts are mai ntained by the EPIP System Adm inistrator , who inst alls all V A-released patches a s soon as they are n ationally released. All EPIP t est accoun ts are clo ned from e xisting VA Enterpris e Testing Services ( ETS) test accounts. The Comput erized Pat ient Recor d System ( CPRS) Grap hical User Interface (GUI) exe cutable is configure d for each VistA ins tance util izing a un ique Inter net Protoc ol (IP) ad dress to c onnect to the VistA applicatio ns. Any up dates to t he CPRS GU I executab le are han dled by th e EPIP Sys tem Admini strator. | ||
| 82 | All EPIP T est Engine ers and De velopers w ho have th e proper c redentials can acces s the test accounts. The VA Au stin Infor mation Tec hnology Ce nter (AITC ) support team reset s password s and sets up new ac cess crede ntials on an as-need ed basis. | ||
| 83 | Installati on Process | ||
| 84 | As soon as the remed iation pro cess is co mplete and the patch is availa ble for te sting, a K IDS build is created in the De velopment account an d then sen t to FORUM for final packaging . The patc h is then submitted to the VA SQA Lead’s Mailman a ccount for installat ion. | ||
| 85 | An EPIP De veloper or Test Engi neer utili zes the KI DS Install ation proc ess to ext ract the b uild from the patch and instal l the buil d into a t est accoun t. The ind ividual wh o installs the patch verifies the routin e checksum s and also checks fo r errors d uring the installati on process . If the p atch is su ccessfully installed without a ny errors, then the EPIP Test team proce eds with F unctional, Regressio n, and Com ponent Int egration a nd System testing. I f defects are found, then the Developmen t team wor ks to find a resolut ion and cr eates new versions o f the patc h until al l defects are resolv ed. | ||
| 86 | Test Data | ||
| 87 | The SQA Te sting team utilizes the test d ata in the designate d test acc ounts (D1G 1, D1G2). | ||
| 88 | The test d ata is enc rypted fol lowing the standards set forth by the VA Office of Informati on & Techn ology (OI& T). All Pe rsonally I dentifiabl e Informat ion (PII) and Protec ted Health Informati on (PHI) i s scrubbed and is no t availabl e to the T est Engine ers. | ||
| 89 | All testin g is execu ted using encrypted test patie nts availa ble from a ny of the EPIP test accounts. Examples o f encrypte d test pat ients: | ||
| 90 | AAAHURMMX, XPHY | ||
| 91 | BADHB, HAA DXS | ||
| 92 | FDHUX, YHI J | ||
| 93 | All tests were execu ted manual ly by EPIP Test Engi neers. | ||
| 94 | Issues | ||
| 95 | No issues were encou ntered dur ing testin g of XU*8. 0*679. | ||
| 96 | Title | ||
| 97 | Issue Desc ription | ||
| 98 | Type | ||
| 99 | Severity | ||
| 100 | N/A | ||
| 101 | N/A | ||
| 102 | N/A | ||
| 103 | N/A | ||
| 104 | Test Execu tion Log | ||
| 105 | The Test E xecution L og records the execu tion of te st scripts and docum ents the t est result s for each test scri pt. | ||
| 106 | The SQA Te sting team utilizes the Ration al Quality Manager ( RQM) tool for all te sting acti vities. Al l test doc uments are stored in the EPIP repository , includin g the Mast er Test Pl an, Test S uites, Tes t Cases, a nd Test Sc ripts. Tes t executio n is perfo rmed, and test resul ts recorde d, in RQM. The Test Engineer a dds the te st results to the Te st Executi on records to indica te whether testing a chieved Pa ss or Fail status. | ||
| 107 | The Test E xecution r ecords for XU*8.0*67 9 are incl uded in th e EPIP Pat ch XU*8.0* 679 Master Test Plan . The Mast er Test Pl an is avai lable in A ppendix A. | ||
| 108 | Test Defec t Log | ||
| 109 | The Test D efect Log is a tool for record ing, analy zing, trac king, and documentin g the clos ure of def ects. It s pecifies t he screen, field, be havior or result tha t occurred , and the IEEE-defin ed Severit y Level. I t includes enough in formation for the de veloper to find and re-create the defect . The Defe ct Log is available in Appendi x B. | ||
| 110 | Test Resul ts Summary | ||
| 111 | SQA testin g for this intake st arted in t he Dev1 Go ld1 test e nvironment on Februa ry 23, 201 7. Test ve rsion 1 wa s installe d in the t est enviro nment afte r Unit tes ting was c ompleted i n Dev1 Sil ver1. Upon completio n of Integ ration tes ting (Comp onent Inte gration an d System T esting, Fu nctional T esting, an d Regressi on Testing ), one (1) defect wa s found an d reported . | ||
| 112 | Test versi on 2 was i nstalled o n February 27, 2017 in the Dev 1 Gold1 te st environ ment. Due to a patch compiling error, th e EPIP dev elopment t eam immedi ately crea ted and in stalled Te st version 3. As a r esult, no testing wa s performe d on Test version 2. Upon comp letion of Test versi on 3 Integ ration tes ting (Comp onent Inte gration an d System T esting, Fu nctional T esting, an d Regressi on Testing ), zero (0 ) defects were found and repor ted. | ||
| 113 | Test versi on 4 was i nstalled o n March 9, 2017. The EPIP deve lopers mad e a minor code chang e required for the O n/Off swit ch, with n o impact t o patch fu nctionalit y. Upon co mpletion o f Integrat ion testin g (Compone nt Integra tion and S ystem Test ing, Funct ional Test ing, and R egression Testing) o n March 16 , 2017, ze ro (0) def ects were found and reported. | ||
| 114 | Defect Sev erity and Priority L evels | ||
| 115 | A defect i s defined as a flaw in a compo nent or sy stem that can cause the compon ent or sys tem to fai l to perfo rm its req uired func tion, e.g. , an incor rect state ment or da ta definit ion. A def ect, if en countered during exe cution, ma y cause a failure of the compo nent or sy stem. | ||
| 116 | Defects ar e categori zed accord ing to sev erity and priority l evels. The test anal yst assign s the seve rity, whil e the deve lopment ma nager assi gns the pr iority for repair. F or more in formation, see Defec t Severity and Prior ity Defini tion in th is Test Ev aluation. | ||
| 117 | Total Defe cts by Sev erity Leve l | ||
| 118 | The Defect Log in Ap pendix B d isplays th e defects encountere d while te sting this patch, an d the seve rity level of each. | ||
| 119 | Breakdown of Test Re sults | ||
| 120 | Testing wa s complete d on March 16, 2017. All test results we re recorde d in RQM. Detailed r esults are available in the EP IP Patch X U*8.0*679 Master Tes t Plan (se e Appendix A). | ||
| 121 | Performanc e Testing | ||
| 122 | Performanc e testing was not co nducted. | ||
| 123 | Test Cover age | ||
| 124 | The EPIP P atch XU*8. 0*679 Mast er Test Pl an contain s details on test co verage (se e Appendix A). | ||
| 125 | Requiremen ts Covered | ||
| 126 | The requir ements for XU*8.0*67 9 are stor ed in the Rational D ynamic Obj ect Orient ed Require ments Syst em (DOORS) . The test cases use d to valid ate that t he require ments have been addr essed are stored in RQM and ar e linked t o the requ irements, providing full trace ability. T he user st ories stor ed in the Rational C hange and Configurat ion Manage ment (CCM) applicati on (a.k.a. Rational Team Conce rt (RTC)) are linked to the re quirements and test cases. | ||
| 127 | The follow ing links provide ac cess to th e various EPIP repos itories in the Ratio nal toolki t. | ||
| 128 | EPIP (RM) – Go to Ar tifacts, t hen Browse Artifacts , and then search fo r the desi red patch number. Th e patch fo lder conta ins the re quirements for that patch. | ||
| 129 | EPIP (QM) – Go to Pl anning, th en Browse Test Plans , and then search fo r the desi red Master Test Plan . The Mast er Test Pl an and tes t cases ar e linked t o requirem ents. | ||
| 130 | EPIP (CM) – Go to Pl ans, then All Plans, and then search for the desir ed sprint Plan. The user stori es in each Plan are linked to requiremen ts and tes t cases. | ||
| 131 | Section 50 8 Complian ce Coverag e | ||
| 132 | Section 50 8 testing was not re quired for this patc h. | ||
| 133 | Suggested Actions | ||
| 134 | Leidos rec ommends mo ving this patch to I OC testing . | ||
| 135 | Defect Sev erity and Priority D efinitions | ||
| 136 | The classi fication o f defects within a s ystem exam ines both the severi ty and pri ority of t he defect. | ||
| 137 | Severity i s a measur e of how g reat the i mpact is o n the user ’s ability to comple te the doc umented ac tions with in the sys tem. | ||
| 138 | Priority d etermines the speed with which a given d efect must be repair ed. | ||
| 139 | Defect cla ssificatio n may be d etermined either bec ause testi ng is dela yed by a f ailure in the system or becaus e a cumber some worka round prev ents a use r from com pleting th e assigned tasks. Bo th severit y and prio rity measu res must b e recorded when sche duling def ect resolu tion tasks . | ||
| 140 | Defect Sev erity Leve l | ||
| 141 | The follow ing subsec tions iden tify the d efect seve rity level s. | ||
| 142 | Severity L evel 1 – C ritical | ||
| 143 | Institute of Electri cal and El ectronics Engineers (IEEE) def inition: T he defect results in the failu re of the complete s oftware sy stem, of a subsystem , or of a software u nit (progr am or modu le) within the syste m. | ||
| 144 | Any defect that comp romises pa tient safe ty or syst em securit y. Example s of syste m security defects i nclude bre ach of con fidentiali ty require ments of t he Privacy Act, the Health Ins urance Por tability a nd Account ability Ac t (HIPAA), or Federa l Tax Info rmation gu idelines. | ||
| 145 | Loss of sy stem funct ionality c ritical to user oper ations wit h no suita ble workar ound, i.e. , there is no way to achieve t he expecte d results using the applicatio n. | ||
| 146 | System cra sh or hang that prev ents furth er testing or operat ion of the complete applicatio n or a sec tion of th e applicat ion. | ||
| 147 | Any defect that caus es corrupt ion of dat a from a r esult of t he system (as oppose d to user error). | ||
| 148 | Any defect in which inappropri ate transm issions ar e consiste ntly gener ated or ap propriate transmissi ons of HL7 messages fail to be generated . | ||
| 149 | Loss of fu nctionalit y resultin g in erron eous eligi bility/enr ollment de terminatio ns or comm unications not being sent. | ||
| 150 | Severity L evel 2 - H igh | ||
| 151 | IEEE defin ition: The defect re sults in t he failure of the co mplete sof tware syst em, of a s ubsystem, or of a so ftware uni t (program or module ) within t he system. There is no way to make the f ailed comp onent(s) f unction. H owever, th ere are ac ceptable p rocessing alternativ es which w ill yield the desire d result. | ||
| 152 | A major de fect in th e function ality that does not result in corruption of data. | ||
| 153 | A major de fect in th e function ality resu lting in a failure o f all or p art of the applicati on, where: | ||
| 154 | The expect ed results can tempo rarily be achieved b y alternat e means. T he custome r indicate s the work around is acceptabl e for the short term . | ||
| 155 | Any defect that does not confo rm to Sect ion 508 st andards. | ||
| 156 | Any defect that resu lts in ina ccurate or missing r equirement s. | ||
| 157 | Any defect that resu lts in inv alid authe ntication or authent ication of an invali d end user . | ||
| 158 | Severity L evel 3 - M edium | ||
| 159 | IEEE defin ition: The defect do es not res ult in a f ailure, bu t causes t he system to produce incorrect , incomple te, or inc onsistent results, o r the defe ct impairs the syste ms usabili ty. | ||
| 160 | Minor func tionality is not wor king as in tended and a workaro und exists but is no t suitable for long term use | ||
| 161 | The inabil ity of a v alid user to access the system consisten t with gra nted privi leges | ||
| 162 | Typographi cal or gra mmatical e rrors in t he applica tion, incl uding inst allation g uides, use r guides, training m anuals, an d design d ocuments | ||
| 163 | Any defect producing cryptic, incorrect, or inappr opriate er ror messag es | ||
| 164 | Any defect that resu lts from t he use of non-standa rd data te rminology in the app lication o r document ation, as defined by the Depar tment of V eterans Af fairs | ||
| 165 | Cosmetic i ssues that are impor tant to th e integrit y of the p roduct, bu t do not r esult in d ata entry and or dat a quality problems. | ||
| 166 | Severity L evel 4 - L ow | ||
| 167 | IEEE defin ition: The defect do es not cau se a failu re, does n ot impair usability, and the d esired pro cessing re sults are easily obt ained by w orking aro und the de fect. | ||
| 168 | Minor loss of, or de fect in th e function ality wher e a long t erm use ex ists | ||
| 169 | Low-level cosmetic i ssues. | ||
| 170 | Priority C lassificat ions | ||
| 171 | The follow ing subsec tions iden tify the a ppropriate actions f or defects at each p riority le vel, per d efinitions of IEEE. | ||
| 172 | Priority 1 - Resolve Immediate ly | ||
| 173 | Further de velopment and/or tes ting canno t occur un til the de fect has b een repair ed. The sy stem canno t be used until the repair has been affe cted. | ||
| 174 | Priority 2 - Give Hi gh Attenti on | ||
| 175 | The defect must be r esolved as soon as p ossible be cause it i s impairin g developm ent and/or testing a ctivities. System us e will be severely a ffected un til the de fect is fi xed. | ||
| 176 | Priority 3 - Normal Queue | ||
| 177 | The defect should be resolved in the nor mal course of develo pment acti vities. It can wait until a ne w build or version i s created. | ||
| 178 | Priority 4 - Low Pri ority | ||
| 179 | The defect is an irr itant that should be repaired, but can b e repaired after mor e serious defects ha ve been fi xed. | ||
| 180 | Optional T ables, Cha rts, and G raphs | ||
| 181 | None. | ||
| 182 | Document A pproval Si gnatures | ||
| 183 | |||
| 184 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | ||
| 185 | Program/Pr oject Mana gerDate | ||
| 186 | |||
| 187 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | ||
| 188 | Business S ponsor Rep resentativ eDate | ||
| 189 | |||
| 190 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | ||
| 191 | Test LeadD ate | ||
| 192 | |||
| 193 | Appendix A - Test Ex ecution Lo g | ||
| 194 | The Test E xecution R ecords for XU*8.0*67 9 are incl uded in th e EPIP Pat ch XU*8.0* 679 Master Test Plan . | ||
| 195 | |||
| 196 | Appendix B – Defect Log | ||
| 197 | One defect was found during te sting of X U*8.0*679. The defec t has been resolved. | ||
| 198 | Note: Test version 2 was not t ested due to a patch compiling error. It was immed iately sup erseded by Test vers ion 3. | ||
| 199 | Defect ID | ||
| 200 | Affected S creen | ||
| 201 | Affected F ield | ||
| 202 | Observed B ehavior | ||
| 203 | Severity | ||
| 204 | Descriptio n | ||
| 205 | N/A | ||
| 206 | N/A | ||
| 207 | N/A | ||
| 208 | N/A | ||
| 209 | N/A | ||
| 210 | No defects were foun d during U nit Testin g of versi on 1.0. | ||
| 211 | N/A | ||
| 212 | N/A | ||
| 213 | N/A | ||
| 214 | N/A | ||
| 215 | N/A | ||
| 216 | No defects were foun d during C omponent I ntegration /Systems T esting of version 1. 0. | ||
| 217 | N/A | ||
| 218 | N/A | ||
| 219 | N/A | ||
| 220 | N/A | ||
| 221 | N/A | ||
| 222 | No defects were foun d during R egression testing of version 1 .0. | ||
| 223 | 472492 | ||
| 224 | VistA Drug Enter /Ed it | ||
| 225 | VistA Elec tronic Sig nature | ||
| 226 | N/A | ||
| 227 | Critical | ||
| 228 | One (1) de fect was f ound durin g Function al Testing of versio n 1.0. | ||
| 229 | N/A | ||
| 230 | N/A | ||
| 231 | N/A | ||
| 232 | N/A | ||
| 233 | N/A | ||
| 234 | No defects were foun d during U nit Testin g of versi on 3.0. | ||
| 235 | N/A | ||
| 236 | N/A | ||
| 237 | N/A | ||
| 238 | N/A | ||
| 239 | N/A | ||
| 240 | No defects were foun d during C omponent I ntegration /Systems T esting of version 3. 0. | ||
| 241 | N/A | ||
| 242 | N/A | ||
| 243 | N/A | ||
| 244 | N/A | ||
| 245 | N/A | ||
| 246 | No defects were foun d during R egression Testing of version 3 .0. | ||
| 247 | N/A | ||
| 248 | N/A | ||
| 249 | N/A | ||
| 250 | N/A | ||
| 251 | N/A | ||
| 252 | No defects were foun d during F unctional Testing of version 3 .0. | ||
| 253 | N/A | ||
| 254 | N/A | ||
| 255 | N/A | ||
| 256 | N/A | ||
| 257 | N/A | ||
| 258 | No defects were foun d during U nit Testin g of versi on 4.0. | ||
| 259 | N/A | ||
| 260 | N/A | ||
| 261 | N/A | ||
| 262 | N/A | ||
| 263 | N/A | ||
| 264 | No defects were foun d during C omponent I ntegration /Systems T esting of version 4. 0. | ||
| 265 | N/A | ||
| 266 | N/A | ||
| 267 | N/A | ||
| 268 | N/A | ||
| 269 | N/A | ||
| 270 | No defects were foun d during R egression Testing of version 4 .0. | ||
| 271 | N/A | ||
| 272 | N/A | ||
| 273 | N/A | ||
| 274 | N/A | ||
| 275 | N/A | ||
| 276 | No defects were foun d during F unctional Testing of version 4 .0. |
Araxis Merge (but not the data content of this report) is Copyright © 1993-2016 Araxis Ltd (www.araxis.com). All rights reserved.