19. EPMO Open Source Coordination Office Redaction File Detail Report

Produced by Araxis Merge on 7/13/2017 3:41:50 PM Eastern Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.

19.1 Files compared

#	Location	File	Last Modified
1	Genisis_v3.zip\Release 3_Docs	Genisis2_VIP_Release 3_Production_Operations_Manual_06162017.docx	Tue Jul 11 16:50:54 2017 UTC
2	Genisis_v3.zip\Release 3_Docs	Genisis2_VIP_Release 3_Production_Operations_Manual_06162017.docx	Thu Jul 13 13:27:21 2017 UTC

19.2 Comparison summary

Description	Between Files 1 and 2
Description	Text Blocks	Lines
Unchanged	1	1012
Changed	0	0
Inserted	0	0
Removed	0	0

19.3 Comparison options

Whitespace
Character case	Differences in character case are significant
Line endings	Differences in line endings (`CR` and `LF` characters) are ignored
CR/LF characters	Not shown in the comparison detail

19.4 Active regular expressions

No regular expressions were active.

19.5 Comparison detail

	1	Production Operation s Manual
	2
	3	Genomic In formation System for Integrate d Science 2 (Genisis 2) Technic al Service s
	4	Release 3
	5
	6
	7
	8
	9
	10
	11
	12	June 2017
	13
	14	Document V ersion 3.0
	15
	16	Department of Vetera ns Affairs
	17
	18	Document R evision Hi story
	19
	20
	21	Date
	22	Version
	23	Descriptio n
	24	Author
	25	02/10/2017
	26	1.0
	27	Release 1
	28	Booz Allen Hamilton
	29	05/12/2017
	30	2.0
	31	Release 2
	32	Booz Allen Hamilton
	33	06/16/2017
	34	3.0
	35	Release 3
	36	Booz Allen Hamilton
	37
	38
	39	Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Product ion Operat ions Manua l has been baselined .
	40
	41
	42
	43	Artifact R ationale
	44
	45	The Produc tion Opera tions Manu al provide s the info rmation ne eded by th e producti on operati ons team t o maintain and troub leshoot th e product. The Produ ction Oper ations Man ual must b e provided prior to release of the produ ct.
	46
	47	Table of C ontents
	48
	49	Introducti on5
	50	Routine Op erations5
	51	Administra tive Proce dures5
	52	System Sta rtup5
	53	System Sta rtup from Emergency Shutdown5
	54	System Shu tdown6
	55	Emergency System Shu tdown6
	56	Back-up & Restore6
	57	Back-up Pr ocedures7
	58	Restore Pr ocedures7
	59	Back-up Te sting8
	60	Storage an d Rotation 8
	61	Security / Identity Management 8
	62	Identity M anagement9
	63	Access Con trol9
	64	User Notif ications10
	65	User Notif ication Po ints of Co ntact10
	66	System Mon itoring, R eporting, and Tools1 0
	67	Dataflow D iagram10
	68	Availabili ty Monitor ing11
	69	Critical M etrics11
	70	Routine Up dates, Ext racts and Purges11
	71	Scheduled Maintenanc e11
	72	Capacity P lanning11
	73	Initial Ca pacity Pla n11
	74	Exception Handling11
	75	Routine Er rors12
	76	Security E rrors12
	77	Time-out E rrors12
	78	Concurrenc y12
	79	Significan t Errors12
	80	Applicatio n Error Lo gs12
	81	Applicatio n Error Co des and De scriptions 12
	82	Infrastruc ture Error s12
	83	Database12
	84	Web Server 13
	85	Applicatio n Server13
	86	Network13
	87	3.2.3.6.Lo gical and Physical D escription s13
	88	Dependent System(s)1 4
	89	Troublesho oting14
	90	System Rec overy14
	91	Restart af ter Non-Sc heduled Sy stem Inter ruption14
	92	Restart af ter Databa se Restore 14
	93	Back-out P rocedures1 4
	94	Rollback P rocedures1 4
	95	Operations and Maint enance Res ponsibilit ies ……………… ……..15 5.A pproval Si gnatures16
	96
	97
	98	Introducti on
	99	The Produc tion Opera tions Manu al describ es how to maintain t he compone nts of the Genomic I nformation System fo r Integrat ed Science 2 (Genisi s2), as we ll as how to trouble shoot prob lems that might occu r with thi s applicat ion in pro duction. T he intende d audience for this document a re the Inf ormation T echnology (IT) teams responsib le for hos ting and m aintaining the appli cation aft er product ion releas e. This do cument is normally f inalized j ust prior to product ion releas e and incl udes many updated el ements spe cific to t he hosting environme nt.
	100
	101	Routine Op erations
	102	Table 1 li sts the ro utine oper ations for Genisis2.
	103
	104	Table 1: R outine Ope rations
	105
	106	Routine Op eration
	107	Periodicit y
	108	Role perfo rming the Function
	109	Additional External Systems, i f Needed
	110	Additional Descripti on
	111	System Log in Availab ility from the VA Ne twork
	112	Daily
	113	Genisis2 S ystem Admi nistrator (Super Use r)
	114	None
	115	Verifies t hat the sy stem is up and runni ng.
	116	Systems Mo nitoring a nd Alertin g, Escalat ion
	117	Real time 24/7
	118	Genisis2 S ystem Admi nistrator (Super Use r)
	119	VA Systems Monitorin g and Aler ting Facil ities
	120	Monitors e ach of the Web, Appl ication an d Data Ser vers statu s. They ne ed to be u p and runn ing
	121
	122	Administra tive Proce dures
	123	System Sta rtup
	124	Table 2 pr ovides the system st artup proc edures for the Genes is2 server s.
	125
	126	Table 2: S ystem Star tup
	127
	128	Component
	129	Procedure
	130	Additional Comments
	131	Apache Ser ver
	132	sudo servi ce httpd s tart
	133	TBD
	134	Genisis2 A pplication Wildfly S erver
	135	nohup ./st andalone.s h &
	136	TBD
	137
	138	System Sta rtup from Emergency Shutdown
	139	In the eve nt of an e mergency s hutdown, T able 3 lis ts the sys tem startu p procedur es for the Genesis2 servers.
	140
	141	Table 3: S ystem Star tup from E mergency S hutdown
	142
	143	Component
	144	Procedure
	145	Additional Comments
	146	Apache Ser ver
	147	sudo servi ce httpd s tart
	148	TBD
	149	Genisis2 A pplication Wildfly S erver
	150	cd /opt/wi ldfly
	151	nohup ./st andalone.s h &
	152	TBD
	153
	154
	155	System Shu tdown
	156	Table 4 pr ovides the system sh utdown pro cedures fo r the Gene sis2 serve rs.
	157
	158	Table 4: S ystem Shut down
	159
	160	Component
	161	Procedure
	162	Additional Comments
	163	Apache Ser ver
	164	sudo servi ce httpd s top
	165	TBD
	166	Genisis2 A pplication Wildfly S erver
	167	ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
	168	TBD
	169
	170
	171	Emergency System Shu tdown
	172	For an eme rgency sys tem shutdo wn, Table 5 lists th e system s hutdown pr ocedures f or the Gen esis2 serv ers.
	173
	174	Table 5: E mergency S ystem Shut down
	175
	176	Component
	177	Procedure
	178	Additional Comments
	179	Apache Ser ver
	180	sudo servi ce httpd s top
	181	TBD
	182	Genisis2 A pplication Wildfly S erver
	183	ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
	184	TBD
	185
	186
	187	Back-up & Restore
	188	Figure 1 r epresents the Genisi s2 server architectu re. It con sists of t he followi ng compone nts that n eed to be backed up:
	189	Apache Web server run ning Red H at Enterpr ise Linux (RHEL)
	190	Apache Web server2/Wi ldfly Appl ication Se rver runni ng RHEL
	191	Database S erver runn ing Micros oft Window s 2008 R2, SQL Serve r 2012
	192
	193
	194
	195
	196	Figure 1: Genisis2 S erver Arch itecture
	197
	198	Back-up Pr ocedures
	199	Table 6 pr ovides fre quency rec ommendatio ns for per forming ba ck-ups. Ve terans Aff airs (VA) Enterprise Operation s (EO) fol lows their Standard Operating Procedures (SOPs) fo r completi ng the bac k-ups.
	200
	201	Table 6: C omponent B ack-up Fre quency
	202
	203	Category
	204	Component
	205	Frequency
	206	Applicatio ns
	207	Applicatio ns
	208	Daily back -ups
	209	Applicatio n Servers
	210	Apache Web Server Ap plication Server Bui ld Server
	211	Weekly ful l back-ups
	212	Database
	213	Database
	214	Hourly bac k-ups
	215	Database S erver
	216	Windows Da ta Base Se rver
	217	Weekly ful l back-ups
	218
	219
	220	Restore Pr ocedures
	221	Restore pr ocedures f rom back-u ps may be necessary. Depending upon what needs to be backed up, VA EO follows th eir existi ng SOPs fo r completi ng the res tores.
	222	Table 7 li sts the ap plicable b ack-up fre quency fro m which re stores can be perfor med.
	223
	224	Table 7: C omponent R estore Fre quency
	225
	226	Category
	227	Component
	228	Frequency
	229	Applicatio ns
	230	Applicatio ns
	231	Daily back -ups
	232	Applicatio n Servers
	233	Apache Web Server Ap plication Server Bui ld Server
	234	Weekly ful l back-ups
	235	Database
	236	Database
	237	Hourly bac k-ups
	238	Database S erver
	239	Windows Da tabase Ser ver
	240	Weekly ful l back-ups
	241
	242	Back-up Te sting
	243	Recommenda tions for back-up te sting incl ude perfor ming a res tore opera tion. Spec ifically:
	244	For applic ations – f ollow the standard E O restore procedures for Linux servers.
	245	For databa se – follo w the stan dard EO re store proc edures for the Micro soft SQL s erver 2012 and the W indows ser ver.
	246
	247	Storage an d Rotation
	248	Recommenda tions for storage an d rotation are that EO follows their sta ndard disa ster recov ery proced ures curre ntly in pl ace.
	249
	250	Security / Identity Management
	251	Figure 2 r epresents the Genisi s2 securit y architec ture.
	252
	253	Genisis Se curity Arc hitecture
	254
	255
	256
	257
	258	Applicatio n
	259	Server
	260
	261	Genisis2 S ystem Auth enticating Roles
	262	Genisis2 A pplication
	263	Roles
	264	Authentica tion
	265
	266	VA LDAP Se rver
	267
	268
	269
	270
	271
	272
	273
	274
	275
	276
	277	Username/P assword Au thenticati on
	278
	279	Figure 2: Genisis2 S ecurity Ar chitecture
	280	User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by VA LDAP grou p.
	281
	282	Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis 2 Roles to allow acc ess to par ts of the Genisis2 a pplication . If a use r does not have a ro le within Genisis2, they canno t proceed beyond the login pag e. Genisis 2 user typ es are man aged by th e Genisis2 applicati on to prov ide each u ser with c ertain fun ctionality , dependin g upon the ir role. F or example , a user i s recogniz ed as a Re questor, D ata Destin ation Mana ger, Data Source Man ager, or G enisis2 Sy stem Admin istrator, and are af forded dif ferent lev els of fun ctionality within th e applicat ion.
	283
	284	Identity M anagement
	285	Genisis2 s upports th e followin g user typ es:
	286	Requestor
	287	Data Desti nation Man ager
	288	Data Sourc e Manager
	289	Genisis2 S ystem Admi nistrator (planned f or a futur e release)
	290	The Reques tor has th e minimum set of act ivities th ey can acc ess and pe rform in t he applica tion. For example, t he Request or can gen erate and track thei r own data requests.
	291	The Data D estination Manager h as all the capabilit ies of a R equestor, but can re view and a pprove req uests, rev iew and ap prove data results, and track requests m ore broadl y.
	292	The Data S ource Mana ger addres ses any qu estions ab out the da ta that th ey may hav e with the Requestor (through the Data D estination Manager a nd not dir ectly), pr epares and places th e data in a Source l anding zon e database , and noti fies the D ata Destin ation Mana ger of its location.
	293	The Data D estination Manager w ill then c opy the da ta over fr om the Sou rce landin g zone dat abase to t he Destina tion landi ng zone da tabase, ex tract the data to a flat file, and perfo rm any add itional cl eanup that may be re quired. Pe rsonally I dentifiabl e Informat ion (PII) and Protec ted Health Informati on (PHI) i nformation is then r emoved fro m this dat a and copi ed over to the speci fic Study Mart set u p for the Requestor.
	294	The Genisi s2 System Administra tor is a s uper user that has a ccess to a ll of the functions that the R equestor a nd the Dat a Destinat ion Manage r have; an d in addit ion, the G enisis2 Sy stem Admin istrator c an Create, Modify, o r Delete u sers. The Genisis2 S ystem Admi nistrator is respons ible for c reating a user accou nt within Genisis2 a nd assigni ng one of the roles. The Genis is2 applic ation then coordinat es with th e VA LDAP server to create and store the role info rmation.
	295
	296	Access Con trol
	297	As indicat ed in Sect ion 2.2, U ser Names and Passwo rds are co ntrolled b y centrali zed VA LDA P access c ontrol pro cesses. Pa ssword Exp iry and ot her admini strative p rocesses a re control led by the VA LDAP g roup.
	298
	299	Genisis2 u ses browse r-based Wi ndows auth entication for usern ame and pa ssword aut henticatio n. Genisis 2 manages user types and provi des the us er with ac cess to sp ecific fun ctionality as descri bed in Sec tion 2.2.1 .
	300
	301	User Notif ications
	302	EO may hav e specific user noti fication a nd escalat ion protoc ols for at tending to systems t hat are do wn with ot her applic ations and servers o n which th ey are hos ted.
	303	Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s for user notificat ions/escal ations cur rently in place.
	304
	305	User Notif ication Po ints of Co ntact
	306	Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s regardin g specific notificat ion points of contac t (POCs) c urrently i n place.
	307
	308	System Mon itoring, R eporting, and Tools
	309	Recommenda tions for system mon itoring, r eporting, and tools are that E O follows their stan dard proce dures for such activ ities curr ently in p lace. Thes e same too ls and pro cesses can be used f or with Ge nisis2 ser vers.
	310
	311	Dataflow D iagram
	312	The Genesi s2 Request flow is d epicted in Figure 4.
	313
	314
	315	Smart orch estration and automa tion of
	316	workflows and ETL pr ocesses wi ll allow G enISIS 2.0 to preser ve data in tegrity an d scale to meet esca lating dem ands of a national r esource
	317
	318	Figure 4: Genisis2 R equest Flo w
	319
	320	Availabili ty Monitor ing
	321	Recommenda tions are that for G enisis2, E O follows their stan dard proce dures for monitoring the avail ability of other app lications, and monit oring the performanc e of appli cations fo r capacity planning purposes.
	322
	323	Critical M etrics
	324	There are no metrics specific to the Gen isis2 appl ication re garding up time or do wntime. EO may have standard u ptime metr ics in pla ce for mon itoring th e availabi lity of ap plications . Our reco mmendation s are to u se the sam e ones for the Genis is2 applic ation.
	325
	326	Routine Up dates, Ext racts and Purges
	327	Genisis2 d oes not an d should n ot require any routi ne data pu rges. The applicatio n needs to maintain a history of request s from Day 1 and kee p them for posterity . The data stored is not so la rge as to require pu rges for t he near fu ture.
	328
	329	Scheduled Maintenanc e
	330	Genisis2 s cheduled m aintenance requires routine pa tches for these soft ware compo nents:
	331	Red Hat Li nux
	332	Microsoft Windows Se rver 2008 R2
	333	Microsoft SQL Server 2012
	334	EO can use their sta ndard oper ating proc edures to schedule a maintenan ce window at a time that is co nvenient f or and in coordinati on with th e user bas e.
	335
	336	Capacity P lanning
	337	Genisis2 i s not such a perform ance inten sive appli cation tha t it requi res period ic capacit y planning reviews. The curren t capacity planned i n terms of number of servers, memory, an d disk spa ce assigne d is more than suffi cient to h andle over 50 times the curren t user bas e expected .
	338
	339	2.7.1.Init ial Capaci ty Plan
	340	The initia l number o f users ex pected ove r the next two years is a maxi mum of 100 , with no more than 50 users e xpected to be logged in concur rently. Th e current capacity p lanned in terms of n umber of s ervers, me mory, and disk space assigned is more th an suffici ent for 3+ years of operation.
	341
	342	Exception Handling
	343	Recommenda tions are that EO fo llows the standard T ier1 and T ier2 suppo rt process es current ly in plac e for Geni sis2 excep tion handl ing that i t uses for other VA applicatio ns.
	344
	345	Routine Er rors
	346	Security E rrors
	347	Authentica tion and A uthorizati on errors can be exp ected. Sin ce Genesis 2 uses the VA LDAP s ystem, sta ndard VA T ier 1 and Tier 2 sup port proce sses curre ntly in pl ace for th e inabilit y to log i n because of a wrong password, wrong use rname, etc . are reco mmended fo r Genisis2 .
	348
	349	Time-out E rrors
	350	Genisis2 t ime-out er rors may o ccur when the webser ver is not available . Standard VA Tier 1 and Tier 2 support processes currently in place f or unavail able appli cations du e to the w ebserver o r network being down are recom mended for Genisis2.
	351
	352	Concurrenc y
	353	Genisis2 c oncurrency errors ar e not expe cted or id entifiable separate from appli cation err ors. Stand ard VA Tie r 1 and Ti er 2 suppo rt process es current ly in plac e for unav ailable ap plications are recom mended for Genisis2.
	354
	355	Significan t Errors
	356	Significan t errors c an be defi ned as err ors or con ditions th at affect the system stability , availabi lity, perf ormance, o r otherwis e make the system un available to its use r base. Th e followin g subsecti ons contai n informat ion to aid administr ators, ope rators, an d other su pport pers onnel in t he resolut ion of sig nificant e rrors, con ditions, o r other is sues.
	357
	358	Applicatio n Error Lo gs
	359	The applic ation erro r logs are located a s follows: - /opt/ge nisis-appl ication-wi ldfly- 10. 0.0/standa lone/log/s erver.log
	360
	361	Applicatio n Error Co des and De scriptions
	362	No applica tion speci fic error codes exis t for the Genisis2 a pplication .
	363
	364	Infrastruc ture Error s
	365	Database
	366	Microsoft SQL Server 2012 is a component of the Ge nisis2 app lication. Generic da tabase err ors troubl eshooting can be fou nd in the Microsoft web pages located at https:// msdn.micro soft.com/e n-us/libra ry/aa95208 1.aspx. Th e site dis cusses typ ical conne ctivity, p ermissions , and data base sizin g problems . Standard VA Tier 1 and Tier 2 support processes currently in place a re recomme nded for G enisis2.
	367	Genisis2 a pplication errors ar e manifest ed as appl ication er rors and a re covered in the Ge nisis2 Mai ntenance a nd Trouble shooting G uide.
	368
	369	Web Server
	370	The troubl eshooting guide for the Apache Web Serve r is locat ed at htt ps://httpd .apache.or g/docs/2.4 /custom-er ror.html.
	371
	372	Applicatio n Server
	373	The troubl eshooting guide for the JBOSS Wildfly Ap plication Server is located at https:// docs.jboss .org/autho r/display/ WFLY10/Tro ubleshooti ng+Common+ Issues.
	374
	375	Network
	376	EO manages the netwo rk where t he servers are hoste d for the Genisis2 a pplication . Standard VA help d esk operat ional proc esses curr ently in p lace for a ddressing network er rors are r ecommended for Genis is2. The G enisis2 ap plication has been c onfigured and tested with a st andard set of networ k paramete rs. Due to security protocols, any chang es in thes e paramete rs should be address ed by EO.
	377
	378	Authentica tion and A uthorizati on
	379	User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by th at group. (Refer to Figure 2 f or the Gen isis Secur ity Archit ecture dia gram.)
	380	Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis Roles to allow acce ss to part s of the G enisis2 ap plication. If a user does not have a rol e within G enisis2, t hey cannot proceed b eyond the login page . Genisis2 user type s are mana ged by the Genisis2 applicatio n to provi de each us er with ce rtain func tionality, depending upon thei r role. Fo r example, a user is recognize d as a Req uestor, Da ta Destina tion Manag er, or Gen isis2 Syst em Adminis trator and are affor ded differ ent levels of functi onality wi thin the a pplication .
	381	Errors rel ating to A uthenticat ion and Au thorizatio n may lead a user to the VA LD AP, since that group manages t he Usernam e/Password authentic ations for Genisis2. EO or the appropria te VA LDAP help desk would be the author ities that address t hese error s through the use of Tier 1 an d Tier 2 s upport pro cedures cu rrently in place.
	382
	383	Logical an d Physical Descripti ons
	384
	385	Refer to F igure 1 fo r the Geni sis2 serve r architec ture diagr am.
	386	The Webser ver consis ts of Apac he running on RHEL. It employs Angular J S as the U ser Interf ace (UI) f ramework.
	387	The Wildfl y Applicat ion Server runs on R HEL. It su pports two main comp onents: th e Applicat ion Code i n Java and the Java Business P rocess Man agement (J BPM) engin e.
	388	The Databa se Server runs Micro soft Windo ws 2008 R2 and hosts a Microso ft SQL ser ver 2012. The Genisi s2 Databas es are hos ted here.
	389
	390	Dependent System(s)
	391	Genisis2 a cquires th e users’ u sernames/p asswords a nd other u ser detail s from the VA LDAP s ystem. Thi s is the o nly system Genisis2 is depende nt upon an d in progr ammatic co mmunicatio n with aut omatically . Errors/t roubleshoo ting are h andled by the approp riate VA L DAP help d esk.
	392
	393	Troublesho oting
	394	The step b y step pro cess for t roubleshoo ting when the system is not up and the u sers not a ble to log in:
	395	Ensure tha t the Apac he and Wil dfly serve rs are up and runnin g.
	396	Ensure tha t the corr ect config uration fi les are pr esent in e ach server and are p ointing to the corre ct end poi nts.
	397	Ensure tha t the fire wall ports are open and the se rvers can communicat e with eac h other.
	398	Ensure tha t the SQL server is up and run ning and c orrect acc ess creden tials are present (e dited).
	399	Ensure tha t SQL serv er user ha s adequate database permission s for CRUD operation s.
	400
	401	System Rec overy
	402	EO manages the serve rs for the Genisis2 applicatio n. Standar d system r ecovery pr ocesses cu rrently in place sho uld be use d for brin ging the s ystems bac k online. If all ser vers are b ack online , then Gen isis2 serv ices are a lso restor ed. Genisi s2 applica tion compo nents comm unicate us ing REST c alls and w ere config ured and t ested with these cal ls.
	403
	404	Restart af ter Non-Sc heduled Sy stem Inter ruption
	405	Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a n on-schedul ed system interrupti on.
	406
	407	Restart af ter Databa se Restore
	408	Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a d atabase re store oper ation.
	409
	410	Back-out P rocedures
	411	The back-o ut procedu res for th e Applicat ion Server and the W ebserver a re as foll ows:
	412	Wildfly Ap plication Server: De lete the W ildfly War file from deploymen t director y.
	413	Apache Web server: De lete all c ontents un der /var/w ww/html/.
	414
	415	Rollback P rocedures
	416	Release 1 will be ru nning on t he PRE-PRO D environm ent with t he databas e frozen a s of the d ate of Rel ease 2. Fo llow the i nstallatio n procedur es for Rel ease 1 on the PROD e nvironment . Copy
	417
	418	the Releas e 1 Databa se from th e PRE-PROD database server to the PROD s erver. Thi s will rol lback Rele ase 2 to R elease 1 o n the PROD productio n servers. Restart s ervices on the PROD machines.
	419
	420	Operations and Maint enance Res ponsibilit ies
	421
	422	Service
	423	Group
	424	Applicatio n Administ ration
	425	Genisis2 S ustainment Team
	426	Applicatio n Testing
	427	Performed by the Gen isis2 Deve lopment Te am
	428	After the Warranty P eriod, Gen isis2 Sust ainment Te am
	429	Applicatio n Installa tion
	430	Performed by the Gen isis2 Deve lopment Te am
	431	After the Warranty P eriod, Gen isis2 Sust ainment Te am
	432	Applicatio n Architec ture
	433	During dev elopment, the Genisi s2 Applica tion Archi tect
	434	After the Warranty P eriod, Gen isis2 Sust ainment Te am
	435	Asset Mana gement
	436	Genisis2 S A
	437	Desktop/Cl ient Suppo rt
	438	Dr. Saiju Pyarajan, Business O wner Paul Hsieh, Gen isis2 Sust ainment PM
	439	Linux Syst ems Admini stration
	440	Genisis2 L inux SA Te am
	441	Windows Sy stems Admi nistration
	442	Genisis2 W indows SA Team
	443	Capacity M anagement
	444	Genisis2 L inux SA Te am Genisis 2 Windows Team
	445	Release Ma nagement
	446	Genisis2 D evelopment Team
	447	After the Warranty P eriod, Gen isis2 Sust ainment Te am
	448	Project Ma nagement
	449	Kim Nix, G enisis2 Pr ogram Mana ger Paul H sieh, Geni sis2 Susta inment PM
	450	Storage Ma nagement
	451	Genisis2 S A
	452	Security M anagement
	453	Genisis2 S ustainment Team
	454	Database A dministrat ion
	455	Genisis2 S A/Database Administr ation Team
	456	Network/In frastructu re Adminis tration
	457	Pittsburgh Infrastru cture/Netw orking Tea m
	458
	459	Approval S ignatures
	460
	461	REVIEW DAT E: SCRIBE:
	462
	463	Signed: Program Ma nager, Ken RikardDat e
	464
	465	Signed: Product Ow ner, Saiju PyarajanD ate
	466
	467	Signed: Receiving Organizati on, Karen LeakeDate
	468
	469	Signed: Product Su pport, Pau l HsiehDat e
	470
	471	Template R evision Hi story
	472
	473
	474	Date
	475	Version
	476	Descriptio n
	477	Author
	478	March 2016
	479	1.6
	480	Updated to remove PM AS referen ces and to include V IP referen ces.
	481	Eliminated unnecessa ry text an d most ins tances of passive vo ice.
	482	Wichita VI P Release Process Te am
	483	June 2015
	484	1.5
	485	Updated co ver and ed ited for S ection 508 conforman ce and rem ediated wi th Common Look Offic e tool
	486	Process Ma nagement
	487	May 2015
	488	1.4
	489	Revised co ntent by P MAS Proces s Improvem ent Lockdo wn and reo rdered cov er sheet t o enhance SharePoint search re sults
	490	Process Ma nagement
	491	November 2 014
	492	1.3
	493	Updated Se ction 4 fo r url chan ge to the Operations and Maint enance Res ponsibilit y Matrix
	494	Process Ma nagement
	495	December 2 013
	496	1.2
	497	Correction to headin gs
	498	Process Ma nagement
	499	March 2013
	500	1.1
	501	Formatted to documen tation sta ndards and edited fo r Section 508 confor mance
	502	Process Ma nagement
	503	January 20 13
	504	1.0
	505	Initial Do cument
	506	PMAS Busin ess Office