3. EPMO Open Source Coordination Office Redaction File Detail Report

Produced by Araxis Merge on 10/17/2017 10:40:44 AM Central Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.

3.1 Files compared

#	Location	File	Last Modified
1	FH_5.5_43.zip\FH_5.5_43\docs	EPIP_Test_Evaluation_(FH_5.5_43).doc	Tue Oct 17 14:47:06 2017 UTC
2	FH_5.5_43.zip\FH_5.5_43\docs	EPIP_Test_Evaluation_(FH_5.5_43).doc	Tue Oct 17 15:26:34 2017 UTC

3.2 Comparison summary

Description	Between Files 1 and 2
Description	Text Blocks	Lines
Unchanged	4	486
Changed	3	6
Inserted	0	0
Removed	0	0

3.3 Comparison options

Whitespace
Character case	Differences in character case are significant
Line endings	Differences in line endings (`CR` and `LF` characters) are ignored
CR/LF characters	Not shown in the comparison detail

3.4 Active regular expressions

No regular expressions were active.

3.5 Comparison detail

	1	Test Evalu ation Temp lateExisti ng Product Intake Pr ogram (EPI P)
	2	Patch FH5 .543
	3	Test Evalu ation
	4
	5	Department of Vetera ns Affairs
	6	September 2017
	7	Version 1. 0
	8	Revision H istory
	9	Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Communi cations Pl an has bee n baseline d.
	10	DateVersio nDescripti onAuthor09 /01/20171. 0Initial d ocument.EP IP Project TeamArtif act Ration ale
	11	The test e valuation document i s the prim ary output of the te st and eva luation pr ocess, an integral p art of the systems e ngineering process, which iden tifies lev els of per formance a nd assists the devel oper in co rrecting d eficiencie s.
	12	Table of C ontents
	13	1.
	14	Test Evalu ation Intr oduction
	15	1
	16	1.1.
	17	Test Evalu ation Scop e
	18	1
	19	1.2.
	20	Test Archi tecture
	21	1
	22	1.3.
	23	Test Envir onment/Con figuration
	24	1
	25	1.4.
	26	Installati on Process
	27	2
	28	2.
	29	Test Data
	30	2
	31	3.
	32	Issues
	33	2
	34	4.
	35	Test Execu tion Log
	36	2
	37	5.
	38	Test Defec t Log
	39	3
	40	6.
	41	Test Resul ts Summary
	42	3
	43	6.1.
	44	Defect Sev erity and Priority L evels
	45	3
	46	6.2.
	47	Total Defe cts by Sev erity Leve l
	48	4
	49	6.3.
	50	Breakdown of Test Re sults
	51	4
	52	6.4.
	53	Performanc e Testing
	54	4
	55	7.
	56	Test Cover age
	57	4
	58	7.1.
	59	Requiremen ts Covered
	60	4
	61	7.2.
	62	Section 50 8 Complian ce Coverag e
	63	5
	64	8.
	65	Suggested Actions
	66	5
	67	9.
	68	Defect Sev erity and Priority D efinitions
	69	5
	70	9.1.
	71	Defect Sev erity Leve l
	72	5
	73	9.1.1.
	74	Severity L evel 1 – C ritical
	75	5
	76	9.1.2.
	77	Severity L evel 2 - H igh
	78	6
	79	9.1.3.
	80	Severity L evel 3 - M edium
	81	6
	82	9.1.4.
	83	Severity L evel 4 - L ow
	84	6
	85	9.2.
	86	Priority C lassificat ions
	87	7
	88	9.2.1.
	89	Priority 1 - Resolve Immediate ly
	90	7
	91	9.2.2.
	92	Priority 2 - Give Hi gh Attenti on
	93	7
	94	9.2.3.
	95	Priority 3 - Normal Queue
	96	7
	97	9.2.4.
	98	Priority 4 - Low Pri ority
	99	7
	100	10.
	101	Optional T ables, Cha rts, and G raphs
	102	7
	103	11.
	104	Document A pproval Si gnatures
	105	8
	106	Appendix A - Test Ex ecution Lo g
	107	9
	108	Appendix B – Defect Log
	109	10
	110	Test Evalu ation Intr oduction
	111	The purpos e of this Test Evalu ation is t o:
	112	Identify t he testing approach used.
	113	Present a summary an alysis of the key te st results from the remediatio n of this intake for review an d assessme nt by desi gnated sta keholders.
	114	Provide a general st atement of the quali ty of the system und er test.
	115	Make recom mendations for futur e testing efforts.
	116	Test Evalu ation Scop e
	117	The scope of this Te st Evaluat ion is to verify the functiona lity of th e Patch FH 5.543 co de modific ation, as determined by Functi onal, Comp onent Inte gration/Sy stem, and Regression testing. Testing ac tivities f ollowed th e specific ations out lined in t he followi ng Master Test Plan: FH5.543 Master Te st Plan (i ncluded in Appendix A).
	118	Test Archi tecture
	119	Following are the EP IP test ac counts use d by the L eidos Deve lopment an d SQA Test ing teams to test FH 5.543.
	120	Developmen t Test Acc ounts
	121	(For Unit Testing)SQ A Test Acc ounts
	122	(For Funct ional, Reg ression, a nd Compone nt Integra tion and S ystem Test ing)VistAS 1 ( altern ate name: D1S1)VistA G1 (altern ate name: D1G1) Vist AS2 (alter nate name: D1S2) –
	123	for CPRS G UI testing onlyVistA G2 (altern ate name: D1G2) –
	124	for CPRS G UI testing onlyTest Environmen t/Configur ation
	125	The EPIP t est accoun ts are mai ntained by the EPIP System Adm inistrator , who inst alls all V A-released patches a s soon as they are n ationally released. All EPIP t est accoun ts are clo ned from e xisting VA Enterpris e Testing Services ( ETS) test accounts. The Comput erized Pat ient Recor d System ( CPRS) Grap hical User Interface (GUI) exe cutable is configure d for each VistA ins tance util izing a un ique Inter net Protoc ol (IP) ad dress to c onnect to the VistA applicatio ns. Any up dates to t he CPRS GU I executab le are han dled by th e EPIP Sys tem Admini strator.
	126	All EPIP T est Engine ers and De velopers w ho have th e proper c redentials can acces s the test accounts. The VA Au stin Infor mation Tec hnology Ce nter (AITC ) support team reset s password s and sets up new ac cess crede ntials on an as-need ed basis.
	127	Installati on Process
	128	As soon as the remed iation pro cess is co mplete and the patch is availa ble for te sting, a K IDS build is created in the De velopment account an d then sen t to FORUM for final packaging . The patc h is then submitted to the VA SQA Lead’s Mailman a ccount for installat ion.
	129	An EPIP De veloper or Test Engi neer utili zes the KI DS Install ation proc ess to ext ract the b uild from the patch and instal l the buil d into a t est accoun t. The ind ividual wh o installs the patch verifies the routin e checksum s and also checks fo r errors d uring the installati on process . If the p atch is su ccessfully installed without a ny errors, then the EPIP Test team proce eds with F unctional, Regressio n, and Com ponent Int egration a nd System testing. I f defects are found, then the Developmen t team wor ks to find a resolut ion and cr eates new versions o f the patc h until al l defects are resolv ed.
	130	Test Data
	131	The SQA Te sting team utilizes the test d ata in the designate d test acc ounts (D1G 1, D1G2).
	132	The test d ata is enc rypted fol lowing the standards set forth by the VA Office of Informati on & Techn ology (OI& T). All Pe rsonally I dentifiabl e Informat ion (PII) and Protec ted Health Informati on (PHI) i s scrubbed and is no t availabl e to the T est Engine ers.
	133	All testin g is execu ted using encrypted test patie nts availa ble from a ny of the EPIP test accounts. Examples o f encrypte d test pat ients:
	134	AAAHURMMX, XPHY
	135	BADHB, HAA DXS
	136	FDHUX, YHI J
	137	All tests were execu ted manual ly by EPIP Test Engi neers.
	138	Issues
	139	No issues were encou ntered dur ing testin g of FH5. 543.
	140	TitleIssue Descripti onTypeSeve rityN/AN/A N/AN/ATest Execution Log
	141	The Test E xecution L og records the execu tion of te st scripts and docum ents the t est result s for each test scri pt.
	142	The SQA Te sting team utilizes the Ration al Quality Managemen t (QM) too l for all testing ac tivities. All test d ocuments a re stored in the EPI P reposito ry, includ ing the Ma ster Test Plan, Test Suites, T est Cases, and Test Scripts. T est execut ion is per formed, an d test res ults recor ded, in Ra tional QM. The Test Engineer a dds the te st results to the Te st Executi on records to indica te whether testing a chieved Pa ss or Fail status.
	143	The Test E xecution r ecords for FH5.543 are inclu ded in the EPIP Patc h FH5.54 3 Master T est Plan. The Master Test Plan is availa ble in App endix A.
	144	Test Defec t Log
	145	The Test D efect Log is a tool for record ing, analy zing, trac king, and documentin g the clos ure of def ects. It s pecifies t he screen, field, be havior or result tha t occurred , and the IEEE-defin ed Severit y Level. I t includes enough in formation for the de veloper to find and re-create the defect . The Defe ct Log is available in Appendi x B.
	146	Test Resul ts Summary
	147	SQA testin g for this intake st arted in t he Dev1 Go ld1 test e nvironment on May 9, 2017 and ended on A ugust 7, 2 017.
	148	Test versi on 1 was i nstalled i n the Dev1 Gold1 tes t environm ent on May 9, 2017 a fter Unit testing in Dev1 Silv er1 was co mpleted. U pon comple tion of In tegration testing (C omponent I ntegration and Syste m Testing, Functiona l Testing, and Regre ssion Test ing), one (1) defect was found and repor ted.
	149	Test versi on 2 was i nstalled i n the Dev1 Gold1 tes t environm ent on May 11, 2017 after Unit testing i n Dev1 Sil ver1 was c ompleted. Upon compl etion of F unctional Testing, o ne (1) def ect was fo und and re ported.
	150	Test versi on 3 was i nstalled i n the Dev1 Gold1 tes t environm ent on May 12, 2017 after Unit testing i n Dev1 Sil ver1 was c ompleted. Upon compl etion of F unctional Testing, o ne (1) def ect was fo und and re ported.
	151	Test versi on 4 was i nstalled i n the Dev1 Gold1 tes t environm ent on May 19, 2017 after Unit testing i n Dev1 Sil ver1 was c ompleted. Upon compl etion of F unctional Testing, z ero (0) de fects were found and reported. Although no defects were foun d, a new t est versio n was crea ted to inc lude two a dditional NSRs (NSR2 0170510 an d NSR20170 519).
	152	Test versi on 5 was i nstalled i n the Dev1 Gold1 tes t environm ent on Jul y 20, 2017 after Uni t testing in Dev1 Si lver1 was completed. Upon comp letion of Integratio n testing (Component Integrati on and Sys tem Testin g, Functio nal Testin g, and Reg ression Te sting), on e (1) defe ct was fou nd and rep orted.
	153	Test versi on 6 was i nstalled i n the Dev1 Gold1 tes t environm ent on Aug ust 7, 201 7 after Un it testing in Dev1 S ilver1 was completed . Upon com pletion Re gression T esting and Functiona l Testing, zero (0) defects we re found a nd reporte d.
	154	Defect Sev erity and Priority L evels
	155	A defect i s defined as a flaw in a compo nent or sy stem that can cause the compon ent or sys tem to fai l to perfo rm its req uired func tion, e.g. , an incor rect state ment or da ta definit ion. A def ect, if en countered during exe cution, ma y cause a failure of the compo nent or sy stem.
	156	Defects ar e categori zed accord ing to sev erity and priority l evels. The test anal yst assign s the seve rity, whil e the deve lopment ma nager assi gns the pr iority for repair. F or more in formation, see Defec t Severity and Prior ity Defini tion in th is Test Ev aluation.
	157	Total Defe cts by Sev erity Leve l
	158	The Defect Log in Ap pendix B d isplays th e defects encountere d while te sting this patch, an d the seve rity level of each.
	159	Breakdown of Test Re sults
	160	Testing wa s complete d on Augus t 7, 2017. All test results we re recorde d in Ratio nal QM. De tailed res ults are a vailable i n the EPIP Patch FH* 5.5*43 Mas ter Test P lan (see A ppendix A) .
	161	Performanc e Testing
	162	Performanc e testing was not co nducted.
	163	Test Cover age
	164	The EPIP P atch FH5. 543 Maste r Test Pla n contains details o n test cov erage (see Appendix A).
	165	Requiremen ts Covered
	166	The requir ements for FH5.543 are store d in the R ational Re quirements Managemen t (RM) app lication. The test c ases store d in Ratio nal Qualit y Manageme nt (QM) ar e used to validate t hat the re quirements have been addressed , providin g full tra ceability. The user stories st ored in Ra tional Con figuration Managemen t (CM) are linked to the requi rements in RM and te st cases i n QM.
	167	The follow ing links provide ac cess to th e various Nutrition and Food S ervice rep ositories in the Rat ional tool kit. If li nk transla tion issue s prevent direct acc ess, copy and paste the URLs i nto your b rowser.
	168	NFS (RM) – – Go to A rtifacts, then Brows e Artifact s. Locate the EPIP f older on t he left si de of the page and e xpand it t o display patch fold ers. Each patch fold er contain s the requ irements f or the pat ch number shown in t he folder name.
	169	https://cl m.rational .oit. DNS /rm/web#ac tion=com.i bm.rdm.web .pages.sho wFoundatio nProjectDa shboard&pr ojectURI=h ttps%3A%2F %2Fclm.rat ional.oit. DNS %2Frm%2Fpr ocess%2Fpr oject-area s%2F_zpdEY FQiEea_0rJ Ulb_6xw&vv c.configur ation=http s%3A%2F%2F clm.ration al.oit. DNS %2Frm%2Fcm %2Fstream% 2F_0eWsEVQ iEea_0rJUl b_6xw
	170	NSF (QM) – Go to Pla nning, the n Browse T est Plans, and then search for the Maste r Test Pla n you need . The Mast er Test Pl an and tes t cases ar e linked t o requirem ents.
	171	https://cl m.rational .oit. DNS /qm/web/co nsole/NFS% 20(QM)#act ion=com.ib m.rqm.plan ning.home. actionDisp atcher&sub Action=vie wTestPlan& id=6658
	172	NFS (CM) – Go to Pla ns, then A ll Plans, and then s earch for the Sprint Plan you need. The user stori es in each Plan are linked to requiremen ts and tes t cases.
	173	https://cl m.rational .oit. DNS /ccm/web/p rojects/NF S%20(CM)#a ction=com. ibm.team.d ashboard.v iewDashboa rd
	174	Section 50 8 Complian ce Coverag e
	175	Section 50 8 test res ults were reported t o VA in th e followin g document s:
	176	EPIP_VASec tion508_Co mpliance_T est_Result s_(FH_5.5_ 43)
	177	EPIP_VASec tion508_Co nformance_ Validation _Statement _(FH_5.5_4 3)
	178	EPIP_VASec tion508_Se lf-Certifi cation_Doc ument_(FH_ 5.5_43)
	179	EPIP_VASec tion508_Ve rifiable_O bjective_E vidence_(F H_5.5_43)
	180	Suggested Actions
	181	Leidos rec ommends mo ving this patch to I OC testing .
	182	Defect Sev erity and Priority D efinitions
	183	The classi fication o f defects within a s ystem exam ines both the severi ty and pri ority of t he defect.
	184	Severity i s a measur e of how g reat the i mpact is o n the user ’s ability to comple te the doc umented ac tions with in the sys tem.
	185	Priority d etermines the speed with which a given d efect must be repair ed.
	186	Defect cla ssificatio n may be d etermined either bec ause testi ng is dela yed by a f ailure in the system or becaus e a cumber some worka round prev ents a use r from com pleting th e assigned tasks. Bo th severit y and prio rity measu res must b e recorded when sche duling def ect resolu tion tasks .
	187	Defect Sev erity Leve l
	188	The follow ing subsec tions iden tify the d efect seve rity level s.
	189	Severity L evel 1 – C ritical
	190	Institute of Electri cal and El ectronics Engineers (IEEE) def inition: T he defect results in the failu re of the complete s oftware sy stem, of a subsystem , or of a software u nit (progr am or modu le) within the syste m.
	191	Any defect that comp romises pa tient safe ty or syst em securit y. Example s of syste m security defects i nclude bre ach of con fidentiali ty require ments of t he Privacy Act, the Health Ins urance Por tability a nd Account ability Ac t (HIPAA), or Federa l Tax Info rmation gu idelines.
	192	Loss of sy stem funct ionality c ritical to user oper ations wit h no suita ble workar ound, i.e. , there is no way to achieve t he expecte d results using the applicatio n.
	193	System cra sh or hang that prev ents furth er testing or operat ion of the complete applicatio n or a sec tion of th e applicat ion.
	194	Any defect that caus es corrupt ion of dat a from a r esult of t he system (as oppose d to user error).
	195	Any defect in which inappropri ate transm issions ar e consiste ntly gener ated or ap propriate transmissi ons of HL7 messages fail to be generated .
	196	Loss of fu nctionalit y resultin g in erron eous eligi bility/enr ollment de terminatio ns or comm unications not being sent.
	197	Severity L evel 2 - H igh
	198	IEEE defin ition: The defect re sults in t he failure of the co mplete sof tware syst em, of a s ubsystem, or of a so ftware uni t (program or module ) within t he system. There is no way to make the f ailed comp onent(s) f unction. H owever, th ere are ac ceptable p rocessing alternativ es which w ill yield the desire d result.
	199	A major de fect in th e function ality that does not result in corruption of data.
	200	A major de fect in th e function ality resu lting in a failure o f all or p art of the applicati on, where:
	201	The expect ed results can tempo rarily be achieved b y alternat e means. T he custome r indicate s the work around is acceptabl e for the short term .
	202	Any defect that does not confo rm to Sect ion 508 st andards.
	203	Any defect that resu lts in ina ccurate or missing r equirement s.
	204	Any defect that resu lts in inv alid authe ntication or authent ication of an invali d end user .
	205	Severity L evel 3 - M edium
	206	IEEE defin ition: The defect do es not res ult in a f ailure, bu t causes t he system to produce incorrect , incomple te, or inc onsistent results, o r the defe ct impairs the syste ms usabili ty.
	207	Minor func tionality is not wor king as in tended and a workaro und exists but is no t suitable for long term use
	208	The inabil ity of a v alid user to access the system consisten t with gra nted privi leges
	209	Typographi cal or gra mmatical e rrors in t he applica tion, incl uding inst allation g uides, use r guides, training m anuals, an d design d ocuments
	210	Any defect producing cryptic, incorrect, or inappr opriate er ror messag es
	211	Any defect that resu lts from t he use of non-standa rd data te rminology in the app lication o r document ation, as defined by the Depar tment of V eterans Af fairs
	212	Cosmetic i ssues that are impor tant to th e integrit y of the p roduct, bu t do not r esult in d ata entry and or dat a quality problems.
	213	Severity L evel 4 - L ow
	214	IEEE defin ition: The defect do es not cau se a failu re, does n ot impair usability, and the d esired pro cessing re sults are easily obt ained by w orking aro und the de fect.
	215	Minor loss of, or de fect in th e function ality wher e a long t erm use ex ists
	216	Low-level cosmetic i ssues.
	217	Priority C lassificat ions
	218	The follow ing subsec tions iden tify the a ppropriate actions f or defects at each p riority le vel, per d efinitions of IEEE.
	219	Priority 1 - Resolve Immediate ly
	220	Further de velopment and/or tes ting canno t occur un til the de fect has b een repair ed. The sy stem canno t be used until the repair has been affe cted.
	221	Priority 2 - Give Hi gh Attenti on
	222	The defect must be r esolved as soon as p ossible be cause it i s impairin g developm ent and/or testing a ctivities. System us e will be severely a ffected un til the de fect is fi xed.
	223	Priority 3 - Normal Queue
	224	The defect should be resolved in the nor mal course of develo pment acti vities. It can wait until a ne w build or version i s created.
	225	Priority 4 - Low Pri ority
	226	The defect is an irr itant that should be repaired, but can b e repaired after mor e serious defects ha ve been fi xed.
	227	Optional T ables, Cha rts, and G raphs
	228	None.
	229	Document A pproval Si gnatures
	230	Signed: __ __________ __________ __________ __________ __________ __________ _________
	231	Program/Pr oject Mana ger
	232	Date
	233	Signed: __ __________ __________ __________ __________ __________ __________ _________
	234	Business S ponsor Rep resentativ e
	235	Date
	236	Signed: __ __________ __________ __________ __________ __________ __________ _________
	237	Test Lead
	238	Date
	239	Appendix A - Test Ex ecution Lo g
	240	The Test E xecution R ecords for FH5.543 are inclu ded in the EPIP Patc h FH5.54 3 Master T est Plan.
	241
	242	Appendix B – Defect Log
	243	Four (4) d efects wer e found du ring testi ng of FH5 .543. All defects h ave been r esolved.
	244	Defect IDA ffected Sc reenAffect ed FieldOb served Beh aviorSever ityDescrip tionN/AN/A N/AN/AN/AN o defects were found during Un it Testing of versio n 1.0.N/AN /AN/AN/AN/ ANo defect s were fou nd during Component Integratio n/Systems Testing of version 1 .0.N/AN/AN /AN/AN/ANo defects w ere found during Reg ression te sting of v ersion 1.0 .514658Vis tAPrint Re port Descr iptionsNo descriptio ns for the print rep orts for t he followi ng days: M onday/
	245	Wednesday/ Friday.Hig hOne (1) d efect was found duri ng Functio nal Testin g of versi on 1.0. N/ AN/AN/AN/A N/ANo defe cts were f ound durin g Unit Tes ting of ve rsion 2.0. 515446Vist APrint Rep ort Duplic atesDuplic ate report s were fou nd for the following days: Mon day/
	246	Wednesday/ FridayHigh One (1) de fect was f ound durin g Function al Testing of versio n 2.0. N/A N/AN/AN/AN /ANo defec ts were fo und during Unit Test ing of ver sion 3.0.5 24518VistA Corrupt Da ta Diction ary FilesC orrupted f iles withi n the Data Dictionar y that pre vent messa ge being q ueued to P -Message.H ighOne (1) defect wa s found du ring Funct ional Test ing of ver sion 3.0. N/AN/AN/AN /AN/ANo de fects were found dur ing Unit T esting of version 4. 0.N/AN/AN/ AN/AN/ANo defects we re found d uring Func tional Tes ting of ve rsion 4.0. Although n o defects were found during te sting of v ersion 4.0 , a new ve rsion was created to add two a dditional NSRs (NSR2 0170510 an d NSR20170 519).N/AN/ AN/AN/AN/A No defects were foun d during U nit Testin g of versi on 5.0.N/A N/AN/AN/AN /ANo defec ts were fo und during Component Integrati on/Systems Testing o f version 5.0.N/AN/A N/AN/AN/AN o defects were found during Re gression t esting of version 5. 0.568779Vi stATray Ti cket Flag- Patient Op tionError when a que ry is run for Tray T icket Flag Edit opti on - using Patient O ption.High One (1) de fect was f ound durin g Function al Testing of versio n 5.0. N/A N/AN/AN/AN /ANo defec ts were fo und during Unit Test ing of ver sion 6.0.N /AN/AN/AN/ AN/ANo def ects were found duri ng Regress ion testin g of versi on 6.0.N/A N/AN/AN/AN /ANo defec ts were fo und during Functiona l Testing of version 6.0.