Produced by Araxis Merge on 7/13/2017 3:41:31 PM Eastern Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.
| # | Location | File | Last Modified |
|---|---|---|---|
| 1 | Genisis_v3.zip\Release 3_Docs | Genisis2_Release 3_Test_Evaluation_Summary_06162017.docx | Wed Jun 28 20:38:22 2017 UTC |
| 2 | Genisis_v3.zip\Release 3_Docs | Genisis2_Release 3_Test_Evaluation_Summary_06162017.docx | Wed Jul 12 18:16:22 2017 UTC |
| Description | Between Files 1 and 2 |
|
|---|---|---|
| Text Blocks | Lines | |
| Unchanged | 2 | 834 |
| Changed | 1 | 2 |
| Inserted | 0 | 0 |
| Removed | 0 | 0 |
| Whitespace | |
|---|---|
| Character case | Differences in character case are significant |
| Line endings | Differences in line endings (CR and LF characters) are ignored |
| CR/LF characters | Not shown in the comparison detail |
No regular expressions were active.
| 1 | Test Evalu ation Summ ary | |
| 2 | Genomic In formation System for Integrate d Science 2 | |
| 3 | (Genisis2) Technical Services | |
| 4 | Release 3 | |
| 5 | ||
| 6 | ||
| 7 | ||
| 8 | June 2017 | |
| 9 | ||
| 10 | Document V ersion 1.0 | |
| 11 | ||
| 12 | Department of Vetera ns Affairs | |
| 13 | ||
| 14 | ||
| 15 | ||
| 16 | Revision H istory | |
| 17 | Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Communi cations Pl an has bee n baseline d. | |
| 18 | Date | |
| 19 | Version | |
| 20 | Descriptio n | |
| 21 | Author | |
| 22 | 06/16/2017 | |
| 23 | 1.0 | |
| 24 | Release 3 Test Evalu ation Summ ary Report | |
| 25 | Booz Allen Hamilton | |
| 26 | ||
| 27 | Artifact R ationale | |
| 28 | The test e valuation document i s the prim ary output of the te st and eva luation pr ocess, an integral p art of the systems e ngineering process, which iden tifies lev els of per formance a nd assists the devel oper in co rrecting d eficiencie s. | |
| 29 | The PMAS D irective c ites the S ervice Del ivery and Engineerin g (SDE) or ganization as having primary r esponsibil ities for system tes ting and c ertificati on. | |
| 30 | ||
| 31 | Table of C ontents | |
| 32 | 1.Test Eva luation In troduction 4 | |
| 33 | 1.1.Test E valuation Scope4 | |
| 34 | 1.2.Test A rchitectur e5 | |
| 35 | 1.3.Test E nvironment / Configur ation6 | |
| 36 | 1.4.Instal lation Pro cess8 | |
| 37 | 2.Test Dat a9 | |
| 38 | 3.Issues9 | |
| 39 | 4.Test Exe cution Log 9 | |
| 40 | 5.Test Res ults Summa ry12 | |
| 41 | 5.1.Defect Severity and Priori ty Levels1 2 | |
| 42 | 5.2.Total Defects by Severity Level12 | |
| 43 | 5.3.Defect s by Sever ity Level and Increm ent12 | |
| 44 | 5.4.Perfor mance Test ing12 | |
| 45 | 5.4.1.Test Event12 | |
| 46 | 5.4.2.Requ irements C overage St atus14 | |
| 47 | 5.4.3.Test Design14 | |
| 48 | 5.4.4.Perf ormance Te st Results 14 | |
| 49 | 5.4.5.Tran saction Re sponse Tim es14 | |
| 50 | 6.Test Cov erage14 | |
| 51 | 6.1.Requir ements Cov ered14 | |
| 52 | 6.2.Sectio n 508 Comp liance Cov erage14 | |
| 53 | 7.Suggeste d Actions1 5 | |
| 54 | 8.Defect S everity an d Priority Definitio ns15 | |
| 55 | 8.1.Defect Severity Level15 | |
| 56 | 8.1.1.Seve rity Level 1 – Criti cal15 | |
| 57 | 8.1.2.Seve rity Level 2 - High1 5 | |
| 58 | 8.1.3.Seve rity Level 3 - Mediu m16 | |
| 59 | 8.1.4.Seve rity Level 4 - Low16 | |
| 60 | 8.2.Priori ty Classif ications16 | |
| 61 | 8.2.1.Prio rity 1 - R esolve Imm ediately16 | |
| 62 | 8.2.2.Prio rity 2 - G ive High A ttention17 | |
| 63 | 8.2.3.Prio rity 3 - N ormal Queu e17 | |
| 64 | 8.2.4.Prio rity 4 - L ow Priorit y17 | |
| 65 | 9.Optional Tables, C harts, and Graphs18 | |
| 66 | 10.Documen t Approval Signature s19 | |
| 67 | ||
| 68 | Test Evalu ation Intr oduction | |
| 69 | The Genomi c Informat ion System for Integ rated Scie nce 2 (Gen isis2) pro vides addi tional fun ctionality to Genisi s 1.0 plat form throu gh automat ing data p rocessing, data requ est transa ctions, da ta request tracking for integr ating VA I nformatics and Compu ting Infra structure (VINCI) cl inical dat a into Gen isis2. In addition, to providi ng enhance d system a dministrat ion. The p urpose of Genisis2 t esting is to measure the funct ional qual ity of the software against th e specifie d user req uirements, prior to the system ’s release into the production environme nt. The te sting proc ess covers the major business functions for creati ons of dat a requests to suppor t human re search pro jects. It also provi des the ne cessary tr aceability to the de sign and r equirement s artifact s to ensur e that all component s have bee n fully de veloped an d can even tually ope rate in a production environme nt. | |
| 70 | The Test E valuation Summary co llects, or ganizes, a nd present s the test results a nd key mea sures of t esting to enable obj ective qua lity evalu ation and assessment . This rep ort provid es the sys tem’s stak eholders w ith an ass essment of the adequ acy of the managemen t, operati onal, and technical controls u sed to pro tect the c onfidentia lity, inte grity, and availabil ity of the system an d the data it stores , transmit s or proce sses. | |
| 71 | Test Evalu ation Scop e | |
| 72 | The Genisi s2 Test Re sult Repor t is for t he Departm ent of Vet erans Affa irs (VA) O ffice of R esearch an d Developm ent (ORD). The scope of the re port inclu des the as sessment o f the syst em level m anagement, operation al, and te chnical co ntrols as documented in the Sy stem Desig n Document (SDD) and Master Te st Plan th at directl y support the system . | |
| 73 | Testing pr ovides the chance to validate the functi onality of Genisis2 Release 3, which inc ludes the following high-level functiona l capabili ties: | |
| 74 | Role / Wor k Stream | |
| 75 | Features / Functiona lity | |
| 76 | Requestor (UI) | |
| 77 | Regression Create/Mo dify/Track Data Requ est | |
| 78 | Automatic email noti fication f or status changes | |
| 79 | Add Commen ts for Ret urn only | |
| 80 | Data Desti nation Man ager (UI) | |
| 81 | Regression Create/ M odify/Trac k/Approve/ Deny Retur n Data Req uest/Resul ts Accepte d/Results Not Accept ed | |
| 82 | Automatic email noti fication f or status changes | |
| 83 | Add Commen ts | |
| 84 | Data Sourc e Manager (UI) | |
| 85 | Regression Request A ccepted or Request N ot Accepte d (on beha lf of VINC I) | |
| 86 | Add Commen ts and Del iver Resul ts | |
| 87 | History (U I) | |
| 88 | Verify Hig h Level an d Detailed History i s captured for each transactio n related to each re quest | |
| 89 | Comments ( UI) | |
| 90 | Provide th e ability to add com ments thro ughout the lifecycle of a data request | |
| 91 | Backend | |
| 92 | Data File Assessment (file che cking) | |
| 93 | COPY Table Function (VINCI to Genisis2) | |
| 94 | Security | |
| 95 | Integratio n with VA’ s Active D irectory | |
| 96 | Role-based access wi th two-fac tor authen tication | |
| 97 | ||
| 98 | Test Archi tecture | |
| 99 | Genisis2 S ustainment testing i s conducte d in a Sof tware Qual ity Assura nce (SQA) environmen t which is located a t the Aust in Informa tion Techn ology Cent er (AITC). As shown in Figure 1, the tes ters conne ct to the Genisis2 S olution vi a standard Web Appli cation por ts over HT TPS. All c ommunicati on between the Genis is2 compon ents is th rough stan dard, VA a pproved co mmunicatio n protocol s. Please refer to S ection 1.3 for more details ab out the co mponents t hat are pa rt of the environmen t. | |
| 100 | ||
| 101 | VA Intrane t | |
| 102 | VA Intrane tFigure 1: Environme nt Archite cture | |
| 103 | ||
| 104 | ||
| 105 | ||
| 106 | ||
| 107 | ||
| 108 | The Test S erver Arch itecture c onsists of the follo wing compo nents: | |
| 109 | Angular Ap ache Webse rver runni ng on RHEL | |
| 110 | Wildfly Ap plication Server run ning on RH EL | |
| 111 | Microsoft SQL Server Database running on Microsoft Windows 2 008 R2 | |
| 112 | ||
| 113 | The Webser ver consis ts of Apac he running on Red Ha t Enterpri se Linux ( RHEL). It employs An gular JS a s the User Interface (UI) fram ework. | |
| 114 | The Applic ation Serv er runs on Red Hat E nterprise Linux (RHE L). It sup ports two main compo nents – Th e Applicat ion Code i n Java and the Java Business P rocess Man agement (J BPM) engin e. | |
| 115 | The Java c ode handle s the appl ication lo gic and se rves as th e backend engine tha t the webs erver call s using th e API inte rface. It also manag es the use r interact ions of th e workflow managemen t. The Jav a code als o uses the Spring fr amework an d manages the Object -Relationa l mapping needed for the appli cation. | |
| 116 | The Java B usiness Pr ocess Mana gement eng ine captur es the wor kflow logi c and exec utes it. T he JBPM en gine manag es its dat abase entr ies in a s eparate da tabase hos ted in the common da tabase ser ver instan ce of Micr osoft SQL server. Ch anges to t he workflo w can be m ade in thi s engine a nd deploye d. | |
| 117 | The Databa se Server is a Micro soft Windo ws 2008 R2 server an d hosts a Microsoft SQL server 2012 data base engin e. This da tabase hol ds and man ages all t he data re lated to t he applica tion. The JBPM engin e has a se parate dat abase here in the sa me instanc e and mana ges workfl ow related data in i t. | |
| 118 | Test Envir onment/ Co nfiguratio n | |
| 119 | The Genisi s2 Test en vironment is divided into the following conceptual layers an d the rela ted physic al layer c omponents depicted i n Figure 1 : | |
| 120 | ||
| 121 | Presentati on | |
| 122 | Applicatio n | |
| 123 | Data | |
| 124 | Authentica tion and A uthorizati on is not a separate layer of the applic ation, but is necess ary to des cribe it w hen outlin ing how th e applicat ion works. | |
| 125 | ||
| 126 | The Presen tation lay er is the top layer of the Gen isis2 Solu tion archi tecture an d represen ts the way a user in teracts wi th the Gen isis2 solu tion. It u ses Apache Web Serve r software and Angul ar JS as t he User In terface Fr amework. | |
| 127 | ||
| 128 | The Applic ation laye r uses a c ombination of Java C ode and Ja va Busines s Process Management engine to capture a n execute business p rocesses a nd rules. The Presen tation lay er calls t he Applica tion layer , which is the backe nd applica tion engin e using an Applicati on Program ming Inter face (API) . | |
| 129 | ||
| 130 | The securi ty model p rotects th e platform from unau thorized a ccess acro ss the web . Applicat ions that interface with the G enisis2 so lution pla tform use standard V A standard ized web s ervices to communica te. The Pl atform als o controls access to data thro ugh securi ty, contro ls access to the dat abase and raises eve nts for wo rkflow pro cesses and custom bu siness log ic impleme ntations. | |
| 131 | ||
| 132 | The Data l ayer, the lowest lay er in the architectu re, provid es the cen tralized d atabases f or storing the Genis is2 Soluti on data. T he Data la yer also c onsists of the Docum ent Reposi tory, whic h serves c ontent man agement an d dashboar d capabili ties, as w ell as, wo rkflow man agement us ed by the platform l ayer. | |
| 133 | Figure 2: Security A rchitectur e | |
| 134 | ||
| 135 | Genisis2 S ecurity Ar chitecture | |
| 136 | ||
| 137 | ||
| 138 | Authentica tion and A uthorizati on | |
| 139 | ||
| 140 | Figure 2 i llustrates the Genis is2 securi ty archite cture. Use r Names an d Password s are cont rolled by centralize d VA LDAP access con trol proce sses. Pass word Expir y and othe r administ rative pro cesses are controlle d by the V A LDAP gro up. | |
| 141 | ||
| 142 | Genisis2 u ses its ow n local da tabase ser ver to han dle Genisi s2 User Ro les. Genis is2 user t ypes are m anaged wit hin the lo cal databa se server to provide the user some funct ions and h ide others from them . For exam ple, a use r is recog nized as a Requestor , Data Des tination M anager or Genisis2 S ystem Admi nistrator and are af forded dif ferent lev els of fun ctionality within th e system. | |
| 143 | ||
| 144 | When a new user is c reated wit hin Genisi s2 it need s to authe nticate th at person’ s username and Resea rch groups they belo ng from th e VA LDAP server and update th e local da tabase. | |
| 145 | ||
| 146 | Genisis2 a uthenticat es any use r with onl y the loca l database server an d at the s ame time, informatio n about wh ich Resear ch Groups they belon g to, whic h projects they belo ng to, who their Dat a Destinat ion Manage r is all e xtracted f rom the se rver at th is time. | |
| 147 | ||
| 148 | Encryption is handle d by https protocol with ssl c ertificate s installe d on the s ervers and VA LDAP s ervers nat ively. | |
| 149 | ||
| 150 | Identity M anagement | |
| 151 | Genisis2 s upports th e followin g user typ es: | |
| 152 | Requestor | |
| 153 | Data Desti nation Man ager | |
| 154 | Data Sourc e Manager | |
| 155 | Genisis2 S ystem Admi nistrator (planned f or a futur e release) | |
| 156 | The Reques tor has th e minimum set of act ivities th ey can acc ess and pe rform in t he applica tion. For example, t he Request or can gen erate and track thei r own data requests. | |
| 157 | The Data D estination Manager h as all the capabilit ies of a R equestor, but can re view and a pprove req uests, rev iew and ap prove data results, and track requests m ore broadl y. | |
| 158 | The Data S ource Mana ger addres ses any qu estions ab out the da ta that th ey may hav e with the Requestor (through the Data D estination Manager a nd not dir ectly), pr epares and places th e data in a Source l anding zon e database , and noti fies the D ata Destin ation Mana ger of its location. | |
| 159 | The Data D estination Manager w ill then c opy the da ta over fr om the Sou rce landin g zone dat abase to t he Destina tion landi ng zone da tabase, ex tract the data to a flat file, and perfo rm any add itional cl eanup that may be re quired. Pe rsonally I dentifiabl e Informat ion (PII) and Protec ted Health Informati on (PHI) i nformation is then r emoved fro m this dat a and copi ed over to the speci fic Study Mart set u p for the Requestor. | |
| 160 | The Genisi s2 System Administra tor is a s uper user that has a ccess to a ll of the functions that the R equestor a nd the Dat a Destinat ion Manage r have; an d in addit ion, the G enisis2 Sy stem Admin istrator c an Create, Modify, o r Delete u sers. The Genisis2 S ystem Admi nistrator is respons ible for c reating a user accou nt within Genisis2 a nd assigni ng one of the roles. The Genis is2 applic ation then coordinat es with th e VA LDAP server to create and store the role info rmation. | |
| 161 | ||
| 162 | Access Con trol | |
| 163 | ||
| 164 | User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by th e VA LDAP group. | |
| 165 | ||
| 166 | Genisis2 c oordinates with the local VA L DAP server for usern ame and pa ssword aut henticatio n. Genisis 2 user typ es are man aged withi n a Genisi s2 databas e server t o provide the user s ome functi ons and hi de others from them. For examp le, a user is recogn ized as a Requestor, Data Dest ination Ma nager, Dat a Source M anager, or Genisis2 System Adm inistrator , and are afforded d ifferent l evels of f unctionali ty within the applic ation. | |
| 167 | ||
| 168 | Installati on Process | |
| 169 | The Genisi s2 Develop ment Team performed all requir ed install ations in the VA env ironment. For detail s, please see sectio ns 1.2 and 1.3. | |
| 170 | Test Data | |
| 171 | All the te st cases e xecuted fo r Genisis2 Release 3 testing a re designe d to creat e and mana ge data re quests as part of th e executio n of the t est case. There are also sever al scenari os that ex ecute the copying of data from one sourc e database to a dest ination da tabase wit h a verifi cation by accessing the databa se log. | |
| 172 | Issues | |
| 173 | There are no issues to report for this p roject. | |
| 174 | Test Execu tion Log | |
| 175 | The Test E xecution L og is used to record the manua l executio n of test scripts an d to docum ent the te st results for each test scrip t. The tab le below s hows the d ifferent t esting ite rations th at took pl ace during the softw are develo pment life cycle. Sof tware Qual ity Assura nce (SQA) testing an d User Acc eptance Te sting (UAT ) testing was perfor med with a ll of the test cases used by t he Genisis 2 Test Tea m. | |
| 176 | The Test E xecution L og include s the foll owing and is located in Ration al | |
| 177 | Item | |
| 178 | Definition | |
| 179 | Test Case | |
| 180 | The numeri cal identi fier for t he Test Ca se associa ted with t his Test C ase. | |
| 181 | Test ID | |
| 182 | The identi fication n umber assi gned to th is test ca se. | |
| 183 | Date Teste d | |
| 184 | The date t he test wa s executed . | |
| 185 | User Story | |
| 186 | The name a nd number of the use r story as sociated w ith the te st case. | |
| 187 | Requiremen t | |
| 188 | The name a nd number of the req uirement a ssociated with the t est case. | |
| 189 | Actual Res ult: | |
| 190 | Pass/Fail | |
| 191 | The test o utcome: P= Pass, F=Fa il. | |
| 192 | Tester Nam e | |
| 193 | The name o f the test er that pe rformed te st executi on. | |
| 194 | The table below prov ides a lis t of the t est cases executed t heir assoc iated requ irements a nd results for Genis is2 Releas e 3: | |
| 195 | Test Case # | |
| 196 | Requiremen t | |
| 197 | Pass/Fail | |
| 198 | Date | |
| 199 | TC0063 Dat a Source M anager_Rev iew Detail s_ of Appr oved_Data Request | |
| 200 | 4.4.1 The system sha ll provide the abili ty for a D ata Source Manager t o review t he approve d data req uest. | |
| 201 | P | |
| 202 | 6/5/2017 | |
| 203 | TC0064 Dat a Source M anager_2Fa ctor_Login | |
| 204 | 4.4.1.1: T he system shall requ ire the Da ta Source Manager to log in us ing 2-fact or authent ication. | |
| 205 | P | |
| 206 | 6/5/2017 | |
| 207 | TC0065Data Source Ma nager_Disp lay_Reques t View_Ico ns | |
| 208 | 4.4.1.2: T he system shall allo w the Data Source Ma nager to v iew a list of data r equests th at are pen ding accep tance. | |
| 209 | P | |
| 210 | 6/5/2017 | |
| 211 | TC0066 Dat a Source M anager_Rev iew_Action s and Hist ory View o n Requests | |
| 212 | 4.4.1.3: T he system shall allo w the Data Source Ma nager to s elect a da ta request and view the detail s. | |
| 213 | P | |
| 214 | 6/5/2017 | |
| 215 | TC0067 Dat a Source M anager _re ad only fi elds_excep t for comm ents | |
| 216 | 4.4.1.4: T he system shall disp lay the re quest as r ead-only. | |
| 217 | P | |
| 218 | 6/5/2017 | |
| 219 | TC0068 Dat a Source M anager_Man age_Data R equests | |
| 220 | 4.4.2: The system sh all provid e the abil ity for a Data Sourc e Manager to manage a data req uest. | |
| 221 | P | |
| 222 | 6/5/2017 | |
| 223 | TC0069 Dat a Source M anager_Add Comments_ Changing Status to Request Ca nnot be Fu lfilled | |
| 224 | 4.4.2.1: T he system shall allo w the Data Source Ma nager to r eject the data reque st. The sy stem sets the status to Reques t Cannot B e Fulfille d. | |
| 225 | P | |
| 226 | 6/5/2017 | |
| 227 | TC0070 Dat a Source M anager_Add Comments _Changing Status_to Accepted | |
| 228 | 4.4.2.2: T he system shall allo w the Data Source Ma nager to a ccept the data reque st. The sy stem shall set the s tatus to R equest Acc epted. | |
| 229 | P | |
| 230 | 6/5/2017 | |
| 231 | TC0071 Dat a Source M anager_Add Comments to Request _No Status Change | |
| 232 | 4.4.2.3: T he system shall allo w the Data Source Ma nager to a dd a Comme nt to the data reque st without changing the status of the re quest. | |
| 233 | P | |
| 234 | 6/5/2017 | |
| 235 | TC0072 Dat a Source M anager_Ver ify email notificati ons for Ac tion_Reque st Cannot Be Fulfill ed | |
| 236 | 4.4.2.4: T he system shall gene rate an em ail notifi cations to the Reque ster and D ata Destin ation Mana ger when t he request has been rejected b y the Data Source Ma nager. | |
| 237 | P | |
| 238 | 6/5/2017 | |
| 239 | TC0073 Dat a Source M anager_Ver ify Email notificati ons for Ac tion_Reque st Accepte d | |
| 240 | 4.4.2.5: T he system shall gene rate an em ail notifi cation to the Reques ter and Da ta Destina tion Manag er when th e request has been a ccepted by the Data Source Man ager. | |
| 241 | P | |
| 242 | 6/5/2017 | |
| 243 | TC0074 Dat a_Destinat ion_Manage r_has_abil ity_to_Cop yTable | |
| 244 | 4.5.1: The system sh all provid e the abil ity for a Data Desti nation Man ager to Co py tables from a Sou rce locati on to a De stination location. | |
| 245 | P | |
| 246 | 6/5/2017 | |
| 247 | TC0075 Sys tem_displa y_CopyTabl e_button_f or_Results _Delivered | |
| 248 | 4.5.1.1: T he system shall disp lay the Co py Table b utton on t he Manage Request sc reen for a request t hat has be en deliver ed (Status = Results Delivered ). | |
| 249 | P | |
| 250 | 6/5/2017 | |
| 251 | TC0076 DDM _enter_Tab leNames_fi eld | |
| 252 | 4.5.1.2: T he system shall disp lay a Tabl e Name dia log box to capture 1 table na me to be c opied. | |
| 253 | P | |
| 254 | 6/5/2017 | |
| 255 | TC0077 Sys tem_displa ys_CopyTab le_on_Mana ge_Request _Page | |
| 256 | 4.5.1.3: T he system shall disp lay the co py table f ields when the Copy Table butt on of the Manage Req uest scree n is execu ted. | |
| 257 | P | |
| 258 | 6/5/2017 | |
| 259 | TC0078 Sys tem_notify _Source_Sc hema_incor rect | |
| 260 | 4.5.1.3a: The system shall ret urn a noti fication w hen the So urce schem a name is incorrect. | |
| 261 | P | |
| 262 | 6/5/2017 | |
| 263 | TC0079 Sys tem_notify _Destinati on_Schema_ incorrect | |
| 264 | 4.5.1.3b: The system shall ret urn a noti fication w hen the De stination schema nam e is incor rect. | |
| 265 | P | |
| 266 | 6/5/2017 | |
| 267 | TC0080 Sys tem_notify _TableName _incorrect | |
| 268 | 4.5.1.3c: The system shall ret urn a noti fication w hen the Ta ble name i s incorrec t. | |
| 269 | P | |
| 270 | 6/5/2017 | |
| 271 | TC0081 Sys tem_notify _Copy_succ essfully | |
| 272 | 4.5.1.3d: The system notifies the Data D estination Manager t hat the co py process completed successfu lly. | |
| 273 | P | |
| 274 | 6/5/2017 | |
| 275 | TC0082 Sys tem_displa y_Copy_fie lds_when_e xecuted | |
| 276 | 4.5.1.4: T he system shall disp lay the co py fields when the C opy button is execut ed. | |
| 277 | P | |
| 278 | 6/5/2017 | |
| 279 | TC0083 - D isplay inf ormation r elated to deliver re sults | |
| 280 | 4.6.1. The system sh all displa y the abil ity for a Data Sourc e Manager to deliver the locat ion of the request d ata. | |
| 281 | P | |
| 282 | 6/5/2017 | |
| 283 | TC0084 - D eliver Res ults butto n availabl e for DSM role only | |
| 284 | 4.6.1.1. T he system shall disp lay the De liver Resu lts button on a data request i n the Requ est Accept ed status. | |
| 285 | P | |
| 286 | 6/5/2017 | |
| 287 | TC0085 - Comments r equired to Deliver R esults. | |
| 288 | 4.6.1.2. T he system shall requ ire the co mment when the Deliv er Request button is executed. | |
| 289 | P | |
| 290 | 6/5/2017 | |
| 291 | TC0086 - D SM reciept of email notificaio n that dat a results are ready. | |
| 292 | 4.6.1.3. T he system shall send an email notificati on to the assigned D ata Destin ation Mana ger that t he results are ready for revie w. | |
| 293 | P | |
| 294 | 6/5/2017 | |
| 295 | TC0087 - R equestor r eciept of email not ificaion t hat data r esults are ready. | |
| 296 | 4.6.1.4. T he system shall send an email notificati on to the Requester that the r esults wer e delivere d to the D ata Destin ation Mana ger. | |
| 297 | P | |
| 298 | 6/5/2017 | |
| 299 | Issues rep orted duri ng the tes t executio n process were enter ed into Ra tional Tea m Concert (CM), a wo rk item tr acking sys tem. Descr iption of each work item was p rovided al ong with t he test cy cle in whi ch it was found (Uni t vs. SQA vs. UAT). Defects we re either associated with a te st case th at failed or were ju dged to be a general defect of the appli cation (e. g., a GUI/ user inter face issue ). Severit y level an d Priority level wer e assigned to each d efect alon g with a u nique defe ct code ge nerated af ter enteri ng it into Rational. Severity levels and Priority levels wer e determin ed using t he standar d VA defin itions tha t were doc umented in a tab of the defect tracking spreadshee t and show n in Secti on 5 of th is report. The date of when ea ch defect was opened or closed was also provided, as well as the statu s of the d efect. | |
| 300 | Test Resul ts Summary | |
| 301 | Defect Sev erity and Priority L evels | |
| 302 | A defect i s defined as a flaw in a compo nent or sy stem that can cause the compon ent or sys tem to fai l to perfo rm its req uired func tion (e.g. , an incor rect state ment or da ta definit ion. A def ect, if en countered during exe cution, ma y cause a failure of the compo nent or sy stem). | |
| 303 | Defects ar e categori zed accord ing to sev erity and priority l evels. The test anal yst assign s the seve rity, whil e the deve lopment ma nager assi gns the pr iority for repair. F or more in formation, see Defec t Severity and Prior ity Defini tion in th is Test Ev aluation. | |
| 304 | Total Defe cts by Sev erity Leve l | |
| 305 | At the end of Genisi s2 Release 3 testing , includin g SQA and UAT, there was one ( 1) outstan ding minor defect re ported. Fe edback/com ments capt ured durin g the test effort we re categor ized. The open defec t will be corrected in Build/R elease 4, as approve d by the V A or durin g the sust ainment pe riod follo wing deplo yment to p roduction. | |
| 306 | Defects by Severity Level and Increment | |
| 307 | The defect identifie d during t his test e ffort was categorize d as minor and chang es to this defect wi ll be impl emented in Release 4 or in a f uture sust ainment re lease. | |
| 308 | Performanc e Testing | |
| 309 | This secti on is not applicable for Genis is2 Releas e 3 based on the lim ited numbe r of initi al users. Performanc e testing is a sched uled requi rement for Release 4 . | |
| 310 | Test Event | |
| 311 | Genisis2 R elease 3 t esting sta rted May 2 4, 2017. T he Genisis 2 Test Tea m created 25 test ca ses to sup port this effort. Du ring the i nternal te st, the te sters exec uted all t est cases within the spreadshe et applica ble to Rel ease 3 and executed regression tests to insure the re was no impact to Release 1 and Releas e 2 functi onality. | |
| 312 | Once the c ase was ex ecuted and results a nalyzed, t he spreads heet was m arked the step as ei ther Pass (P) or Fai l (F). | |
| 313 | The execut ed scripts were tabu lated by t est status : Passed, or Failed. | |
| 314 | The Genisi s2 Test Te am capture d test res ults and t riage was conducted internally with the developmen t team to validate a nd classif y the resu lts. | |
| 315 | The summar y test met rics are s hown below : | |
| 316 | 25 test ca ses were e xecuted | |
| 317 | 13 test ca ses Passed (by the e nd of Spri nt 4) | |
| 318 | 13 test ca ses had de fects file d against them | |
| 319 | 1 test def ect remain s to be co rrected in Release 4 or a futu re sustain ment relea se | |
| 320 | ||
| 321 | For a full list of d efects fou nd during developmen t and inte rnal testi ng, please see the a ttached JI RA Report. | |
| 322 | ||
| 323 | ||
| 324 | ||
| 325 | ||
| 326 | The Test C ases were submitted to the VA- SQA Team f or review and use as the basis for the S QA Test. T he Interna l Test Tea m schedule d a sessio n to perfo rm test ex ecution fo r all the test scena rios appli cable to R elease 3 a s well reg ression te st Release 1 and Rel ease 2 fun ctionality on May 30 , 2017. T he VA-SQA Team execu ted all sc enarios wi th the exc eption of the ‘Copy Table’ fun ction duri ng the fir st session . No issue s were fou nd and log ged in Rat ional Team Concert ( CM). A sec ond sessio n was held June 13, 2017 to de monstrate the copy t able scena rios once the change s were in place. | |
| 327 | The summar y test met rics are s hown below : | |
| 328 | 25 test ca ses were e xecuted | |
| 329 | 25 test ca ses Passed | |
| 330 | ||
| 331 | For detail s about th e three de fects, ple ase see th e Genisis2 Rational Instance ( CM). | |
| 332 | https:// DNS /ccm/web/p rojects/Ge nisis%20(C M)#action= com.ibm.te am.apt.vie wPlan&page =com.ibm.t eam.apt.we b.ui.plann edItems&id =_nEFWEGis EeaGZ_dHul LUfA&planM ode=com.ib m.team.apt .viewmodes .internal. iteration | |
| 333 | ||
| 334 | For more d etails abo ut the tes t that wer e executed for SQA, please see the attac hed spread sheet. | |
| 335 | ||
| 336 | ||
| 337 | For detail s about th e defect, please see the Genis is2 Ration al Instanc e (CM). | |
| 338 | The Busine ss Owner i dentified three UAT testers fo r Genisis2 Release 3 . A kickof f meeting was held t o obtain p reliminary user info rmation to and ident ify user r oles prior to the UA T. The Gen isis2 SME modified t he spreads heet to co nsolidate the scenar ios into o ne step-by -step scen ario and t he Interna l Test Tea m added co lumns and fields to include te ster infor mation, us er/role da ta and pas s/fail inf ormation. The Intern al Test Te am schedul ed one ses sion, June 12, 2017 to facilit ate test e xecution f or all of the test s cenarios a pplicable to Release 3 and com pleted the test with out any de fects or i ssues iden tified. | |
| 339 | The summar y test met rics are s hown below : | |
| 340 | 25 test ca ses were e xecuted | |
| 341 | 25 test ca ses Passed | |
| 342 | For more d etails abo ut the tes t that wer e executed for UAT, please see the attac hed spread sheet. | |
| 343 | ||
| 344 | Requiremen ts Coverag e Status | |
| 345 | The requir ements cov erage is e stablished within th e Rational Instance via linkag e between the requir ements in RM and the test case s in RQM. | |
| 346 | Test Desig n | |
| 347 | Test desig n is not a pplicable for Releas e 3. | |
| 348 | Performanc e Test Res ults | |
| 349 | Performanc e test res ults are n ot applica ble for Re lease 3. | |
| 350 | Transactio n Response Times | |
| 351 | Transactio n response times are not appli cable for Release 3. | |
| 352 | Test Cover age | |
| 353 | Requiremen ts Covered | |
| 354 | Please ref er to the Requiremen ts Traceab ility Matr ix (RTM) w ithin Rati onal for a detailed mapping be tween requ irements a nd test ca ses. The I nternal Te st Team es tablished coverage b y linking the requir ements in RM with th e test cas es in RQM. | |
| 355 | Section 50 8 Complian ce Coverag e | |
| 356 | The Genisi s2 Team ha s complete d internal verificat ion of the code to c onform to Section 50 8 Complian ce standar ds by usin g the VA-a pproved to ol, JAWS. Our intern al 508 SME evaluated the softw are result ing in an internal p ass for th e Release 3 code. Th e report f rom that e valuation has been i ncluded wi thin this document f or referen ce. The do cumentatio n required for self- certificat ion and re quest for audit was submitted to the VA- 508 Team o n Friday, June 9, 20 17. | |
| 357 | ||
| 358 | Suggested Actions | |
| 359 | All commen ts made by SQA and/o r UAT part icipants h ave been c ollated an d updated where requ ired withi n the test spreadshe ets. Pleas e see the documents attached i n Section 5.4.1. | |
| 360 | Defect Sev erity and Priority D efinitions | |
| 361 | The classi fication o f defects within a s ystem exam ines both the severi ty and pri ority of t he defect. | |
| 362 | Severity i s a measur e of how g reat the i mpact is o n the user ’s ability to comple te the doc umented ac tions with in the sys tem. | |
| 363 | Priority d etermines the speed with which a given d efect must be repair ed. | |
| 364 | Defect cla ssificatio n may be d etermined either bec ause testi ng is dela yed by a f ailure in the system or becaus e a cumber some worka round prev ents a use r from com pleting th e assigned tasks. Bo th severit y and prio rity measu res must b e recorded when sche duling def ect resolu tion tasks . | |
| 365 | Defect Sev erity Leve l | |
| 366 | The follow ing subsec tions iden tify the d efect seve rity level s. | |
| 367 | Severity L evel 1 – C ritical | |
| 368 | Institute of Electri cal and El ectronics Engineers (IEEE) def inition: T he defect results in the failu re of the complete s oftware sy stem, of a subsystem , or of a software u nit (progr am or modu le) within the syste m. | |
| 369 | Any defect that comp romises pa tient safe ty or syst em securit y. Example s of syste m security defects i nclude bre ach of con fidentiali ty require ments of t he Privacy Act, the Health Ins urance Por tability a nd Account ability Ac t (HIPAA), or Federa l Tax Info rmation gu idelines. | |
| 370 | Loss of sy stem funct ionality c ritical to user oper ations wit h no suita ble workar ound, i.e. , there is no way to achieve t he expecte d results using the applicatio n. | |
| 371 | System cra sh or hang that prev ents furth er testing or operat ion of the complete applicatio n or a sec tion of th e applicat ion. | |
| 372 | Any defect that caus es corrupt ion of dat a from a r esult of t he system (as oppose d to user error). | |
| 373 | Any defect in which inappropri ate transm issions ar e consiste ntly gener ated or ap propriate transmissi ons of HL7 messages fail to be generated . | |
| 374 | Loss of fu nctionalit y resultin g in erron eous eligi bility/enr ollment de terminatio ns or comm unications not being sent. | |
| 375 | Severity L evel 2 - H igh | |
| 376 | IEEE defin ition: The defect re sults in t he failure of the co mplete sof tware syst em, of a s ubsystem, or of a so ftware uni t (program or module ) within t he system. There is no way to make the f ailed comp onent(s) f unction. H owever, th ere are ac ceptable p rocessing alternativ es which w ill yield the desire d result. | |
| 377 | A major de fect in th e function ality that does not result in corruption of data. | |
| 378 | A major de fect in th e function ality resu lting in a failure o f all or p art of the applicati on, where: | |
| 379 | The expect ed results can tempo rarily be achieved b y alternat e means. T he custome r indicate s the work around is acceptabl e for the short term . | |
| 380 | Any defect that does not confo rm to Sect ion 508 st andards. | |
| 381 | Any defect that resu lts in ina ccurate or missing r equirement s. | |
| 382 | Any defect that resu lts in inv alid authe ntication or authent ication of an invali d end user . | |
| 383 | Severity L evel 3 - M edium | |
| 384 | IEEE defin ition: The defect do es not res ult in a f ailure, bu t causes t he system to produce incorrect , incomple te, or inc onsistent results, o r the defe ct impairs the syste ms usabili ty. | |
| 385 | Minor func tionality is not wor king as in tended and a workaro und exists but is no t suitable for long term use | |
| 386 | The inabil ity of a v alid user to access the system consisten t with gra nted privi leges | |
| 387 | Typographi cal or gra mmatical e rrors in t he applica tion, incl uding inst allation g uides, use r guides, training m anuals, an d design d ocuments | |
| 388 | Any defect producing cryptic, incorrect, or inappr opriate er ror messag es | |
| 389 | Any defect that resu lts from t he use of non-standa rd data te rminology in the app lication o r document ation, as defined by the Depar tment of V eterans Af fairs | |
| 390 | Cosmetic i ssues that are impor tant to th e integrit y of the p roduct, bu t do not r esult in d ata entry and or dat a quality problems. | |
| 391 | Severity L evel 4 - L ow | |
| 392 | IEEE defin ition: The defect do es not cau se a failu re, does n ot impair usability, and the d esired pro cessing re sults are easily obt ained by w orking aro und the de fect. | |
| 393 | Minor loss of, or de fect in th e function ality wher e a long-t erm use ex ists | |
| 394 | Low-level cosmetic i ssues. | |
| 395 | Priority C lassificat ions | |
| 396 | The follow ing subsec tions iden tify the a ppropriate actions f or defects at each p riority le vel, per d efinitions of IEEE. | |
| 397 | Priority 1 - Resolve Immediate ly | |
| 398 | Further de velopment and/or tes ting canno t occur un til the de fect has b een repair ed. The sy stem canno t be used until the repair has been affe cted. | |
| 399 | Priority 2 - Give Hi gh Attenti on | |
| 400 | The defect must be r esolved as soon as p ossible be cause it i s impairin g developm ent and/or testing a ctivities. System us e will be severely a ffected un til the de fect is fi xed. | |
| 401 | Priority 3 - Normal Queue | |
| 402 | The defect should be resolved in the nor mal course of develo pment acti vities. It can wait until a ne w build or version i s created. | |
| 403 | Priority 4 - Low Pri ority | |
| 404 | The defect is an irr itant that should be repaired, but can b e repaired after mor e serious defects ha ve been co rrected. | |
| 405 | Optional T ables, Cha rts, and G raphs | |
| 406 | Optional t ables, cha rts and gr aphs are n ot needed at this ti me. | |
| 407 | Document A pproval Si gnatures | |
| 408 | ||
| 409 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | |
| 410 | Program/P roject Man agerDate | |
| 411 | ||
| 412 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | |
| 413 | Business Sponsor Re presentati veDate | |
| 414 | ||
| 415 | Signed: __ __________ __________ __________ __________ __________ __________ _________ | |
| 416 | Lead Test erDate | |
| 417 | ||
| 418 |
Araxis Merge (but not the data content of this report) is Copyright © 1993-2016 Araxis Ltd (www.araxis.com). All rights reserved.