15. EPMO Open Source Coordination Office Redaction File Detail Report

Produced by Araxis Merge on 10/5/2017 9:14:12 AM Eastern Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.

15.1 Files compared

#	Location	File	Last Modified
1	GenISIS_Build_4.zip\Build 4	Genisis2_VIP_Build 4_Production_Operations_Manual_09152017.docx	Tue Aug 29 18:23:30 2017 UTC
2	GenISIS_Build_4.zip\Build 4	Genisis2_VIP_Build 4_Production_Operations_Manual_09152017.docx	Wed Oct 4 17:07:52 2017 UTC

15.2 Comparison summary

Description	Between Files 1 and 2
Description	Text Blocks	Lines
Unchanged	1	766
Changed	0	0
Inserted	0	0
Removed	0	0

15.3 Comparison options

Whitespace
Character case	Differences in character case are significant
Line endings	Differences in line endings (`CR` and `LF` characters) are ignored
CR/LF characters	Not shown in the comparison detail

15.4 Active regular expressions

No regular expressions were active.

15.5 Comparison detail

	1	Production Operation s Manual
	2	Genomic In formation System for Integrate d Science 2
	3	(Genisis2) Technical Services
	4	Release 3
	5
	6
	7
	8	June 2017
	9
	10	Document V ersion 3.0
	11
	12	Department of Vetera ns Affairs
	13
	14	Document R evision Hi story
	15	Date
	16	Version
	17	Descriptio n
	18	Author
	19	02/10/2017
	20	1.0
	21	Release 1
	22	Booz Allen Hamilton
	23	05/12/2017
	24	2.0
	25	Release 2
	26	Booz Allen Hamilton
	27	06/16/2017
	28	3.0
	29	Release 3
	30	Booz Allen Hamilton
	31
	32	Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Product ion Operat ions Manua l has been baselined .
	33
	34	Artifact R ationale
	35	The Produc tion Opera tions Manu al provide s the info rmation ne eded by th e producti on operati ons team t o maintain and troub leshoot th e product. The Produ ction Oper ations Man ual must b e provided prior to release of the produ ct.
	36
	37	Table of C ontents
	38	1.Introduc tion1
	39	2.Routine Operations 1
	40	2.1.Admini strative P rocedures1
	41	2.1.1.Syst em Startup 1
	42	2.1.1.1.Sy stem Start up from Em ergency Sh utdown1
	43	2.1.2.Syst em Shutdow n2
	44	2.1.2.1.Em ergency Sy stem Shutd own2
	45	2.1.3.Back -up & Rest ore2
	46	2.1.3.1.Ba ck-up Proc edures3
	47	2.1.3.2.Re store Proc edures3
	48	2.1.3.3.Ba ck-up Test ing4
	49	2.1.3.4.St orage and Rotation4
	50	2.2.Securi ty / Ident ity Manage ment4
	51	2.2.1.Iden tity Manag ement5
	52	2.2.2.Acce ss Control 5
	53	2.3.User N otificatio ns6
	54	2.3.1.User Notificat ion Points of Contac t6
	55	2.4.System Monitorin g, Reporti ng, and To ols6
	56	2.4.1.Data flow Diagr am6
	57	2.4.2.Avai lability M onitoring7
	58	2.4.3.Crit ical Metri cs7
	59	2.5.Routin e Updates, Extracts and Purges 7
	60	2.6.Schedu led Mainte nance7
	61	2.7.Capaci ty Plannin g7
	62	2.7.1.Init ial Capaci ty Plan7
	63	3.Exceptio n Handling 7
	64	3.1.Routin e Errors8
	65	3.1.1.Secu rity Error s8
	66	3.1.2.Time -out Error s8
	67	3.1.3.Conc urrency8
	68	3.2.Signif icant Erro rs8
	69	3.2.1.Appl ication Er ror Logs8
	70	3.2.2.Appl ication Er ror Codes and Descri ptions8
	71	3.2.3.Infr astructure Errors8
	72	3.2.3.1.Da tabase8
	73	3.2.3.2.We b Server9
	74	3.2.3.3.Ap plication Server9
	75	3.2.3.4.Ne twork9
	76	3.2.3.6.Lo gical and Physical D escription s9
	77	3.3.Depend ent System (s)10
	78	3.4.Troubl eshooting1 0
	79	3.5.System Recovery1 0
	80	3.5.1.Rest art after Non-Schedu led System Interrupt ion10
	81	3.5.2.Rest art after Database R estore10
	82	3.5.3.Back -out Proce dures10
	83	3.5.4.Roll back Proce dures10
	84	4.Operatio ns and Mai ntenance R esponsibil ities11
	85	5.Approval Signature s12
	86
	87	Introducti on
	88	The Produc tion Opera tions Manu al describ es how to maintain t he compone nts of the Genomic I nformation System fo r Integrat ed Science 2 (Genisi s2), as we ll as how to trouble shoot prob lems that might occu r with thi s applicat ion in pro duction. T he intende d audience for this document a re the Inf ormation T echnology (IT) teams responsib le for hos ting and m aintaining the appli cation aft er product ion releas e. This do cument is normally f inalized j ust prior to product ion releas e and incl udes many updated el ements spe cific to t he hosting environme nt.
	89	Routine Op erations
	90	Table 1 li sts the ro utine oper ations for Genisis2.
	91	Table 1: R outine Ope rations
	92	Routine Op eration
	93	Periodicit y
	94	Role perfo rming the Function
	95	Additional External Systems, i f Needed
	96	Additional Descripti on
	97	System Log in Availab ility from the VA Ne twork
	98	Daily
	99	Genisis2 S ystem Admi nistrator (Super Use r)
	100	None
	101	Verifies t hat the sy stem is up and runni ng.
	102	Systems Mo nitoring a nd Alertin g, Escalat ion
	103	Real time 24/7
	104	Genisis2 S ystem Admi nistrator (Super Use r)
	105	VA Systems Monitorin g and Aler ting Facil ities
	106	Monitors e ach of the Web, Appl ication an d Data Ser vers statu s. They ne ed to be u p and runn ing
	107	Administra tive Proce dures
	108	System Sta rtup
	109	Table 2 pr ovides the system st artup proc edures for the Genes is2 server s.
	110	Table 2: S ystem Star tup
	111	Component
	112	Procedure
	113	Additional Comments
	114	Apache Ser ver
	115	sudo servi ce httpd s tart
	116	TBD
	117	Genisis2 A pplication Wildfly S erver
	118	nohup ./st andalone.s h &
	119	TBD
	120	System Sta rtup from Emergency Shutdown
	121	In the eve nt of an e mergency s hutdown, T able 3 lis ts the sys tem startu p procedur es for the Genesis2 servers.
	122	Table 3: S ystem Star tup from E mergency S hutdown
	123	Component
	124	Procedure
	125	Additional Comments
	126	Apache Ser ver
	127	sudo servi ce httpd s tart
	128	TBD
	129	Genisis2 A pplication Wildfly S erver
	130	cd /opt/wi ldfly
	131	nohup ./st andalone.s h &
	132	TBD
	133
	134	System Shu tdown
	135	Table 4 pr ovides the system sh utdown pro cedures fo r the Gene sis2 serve rs.
	136	Table 4: System Shu tdown
	137	Component
	138	Procedure
	139	Additional Comments
	140	Apache Ser ver
	141	sudo servi ce httpd s top
	142	TBD
	143	Genisis2 A pplication Wildfly S erver
	144	ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
	145	TBD
	146
	147	Emergency System Shu tdown
	148	For an eme rgency sys tem shutdo wn, Table 5 lists th e system s hutdown pr ocedures f or the Gen esis2 serv ers.
	149	Table 5: Emergency System Shu tdown
	150	Component
	151	Procedure
	152	Additional Comments
	153	Apache Ser ver
	154	sudo servi ce httpd s top
	155	TBD
	156	Genisis2 A pplication Wildfly S erver
	157	ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
	158	TBD
	159
	160	Back-up & Restore
	161	Figure 1 r epresents the Genisi s2 server architectu re. It con sists of t he followi ng compone nts that n eed to be backed up:
	162	Apache Web server run ning Red H at Enterpr ise Linux (RHEL)
	163	Apache Web server2/Wi ldfly Appl ication Se rver runni ng RHEL
	164	Database S erver runn ing Micros oft Window s 2008 R2, SQL Serve r 2012
	165
	166	Figure 1: Genisis2 S erver Arch itecture
	167	Back-up Pr ocedures
	168	Table 6 pr ovides fre quency rec ommendatio ns for per forming ba ck-ups. Ve terans Aff airs (VA) Enterprise Operation s (EO) fol lows their Standard Operating Procedures (SOPs) fo r completi ng the bac k-ups.
	169	Table 6: C omponent B ack-up Fre quency
	170	Category
	171	Component
	172	Frequency
	173	Applicatio ns
	174	Applicatio ns
	175	Daily back -ups
	176	Applicatio n Servers
	177	Apache Web Server
	178	Applicatio n Server
	179	Build Serv er
	180	Weekly ful l back-ups
	181	Database
	182	Database
	183	Hourly bac k-ups
	184	Database S erver
	185	Windows Da ta Base Se rver
	186	Weekly ful l back-ups
	187
	188	Restore Pr ocedures
	189	Restore pr ocedures f rom back-u ps may be necessary. Depending upon what needs to be backed up, VA EO follows th eir existi ng SOPs fo r completi ng the res tores.
	190	Table 7 li sts the ap plicable b ack-up fre quency fro m which re stores can be perfor med.
	191
	192	Table 7: C omponent R estore Fre quency
	193	Category
	194	Component
	195	Frequency
	196	Applicatio ns
	197	Applicatio ns
	198	Daily back -ups
	199	Applicatio n Servers
	200	Apache Web Server
	201	Applicatio n Server
	202	Build Serv er
	203	Weekly ful l back-ups
	204	Database
	205	Database
	206	Hourly bac k-ups
	207	Database S erver
	208	Windows Da tabase Ser ver
	209	Weekly ful l back-ups
	210	Back-up Te sting
	211	Recommenda tions for back-up te sting incl ude perfor ming a res tore opera tion. Spec ifically:
	212	For applic ations – f ollow the standard E O restore procedures for Linux servers.
	213	For databa se – follo w the stan dard EO re store proc edures for the Micro soft SQL s erver 2012 and the W indows ser ver.
	214	Storage an d Rotation
	215	Recommenda tions for storage an d rotation are that EO follows their sta ndard disa ster recov ery proced ures curre ntly in pl ace.
	216	Security / Identity Management
	217	Figure 2 r epresents the Genisi s2 securit y architec ture.
	218
	219	Figure 2: Genisis2 S ecurity Ar chitecture
	220	User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by VA LDAP grou p.
	221	Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis 2 Roles to allow acc ess to par ts of the Genisis2 a pplication . If a use r does not have a ro le within Genisis2, they canno t proceed beyond the login pag e. Genisis 2 user typ es are man aged by th e Genisis2 applicati on to prov ide each u ser with c ertain fun ctionality , dependin g upon the ir role. F or example , a user i s recogniz ed as a Re questor, D ata Destin ation Mana ger, Data Source Man ager, or G enisis2 Sy stem Admin istrator, and are af forded dif ferent lev els of fun ctionality within th e applicat ion.
	222	Identity M anagement
	223	Genisis2 s upports th e followin g user typ es:
	224	Requestor
	225	Data Desti nation Man ager
	226	Data Sourc e Manager
	227	Genisis2 S ystem Admi nistrator (planned f or a futur e release)
	228	The Reques tor has th e minimum set of act ivities th ey can acc ess and pe rform in t he applica tion. For example, t he Request or can gen erate and track thei r own data requests.
	229	The Data D estination Manager h as all the capabilit ies of a R equestor, but can re view and a pprove req uests, rev iew and ap prove data results, and track requests m ore broadl y.
	230	The Data S ource Mana ger addres ses any qu estions ab out the da ta that th ey may hav e with the Requestor (through the Data D estination Manager a nd not dir ectly), pr epares and places th e data in a Source l anding zon e database , and noti fies the D ata Destin ation Mana ger of its location.
	231	The Data D estination Manager w ill then c opy the da ta over fr om the Sou rce landin g zone dat abase to t he Destina tion landi ng zone da tabase, ex tract the data to a flat file, and perfo rm any add itional cl eanup that may be re quired. Pe rsonally I dentifiabl e Informat ion (PII) and Protec ted Health Informati on (PHI) i nformation is then r emoved fro m this dat a and copi ed over to the speci fic Study Mart set u p for the Requestor.
	232	The Genisi s2 System Administra tor is a s uper user that has a ccess to a ll of the functions that the R equestor a nd the Dat a Destinat ion Manage r have; an d in addit ion, the G enisis2 Sy stem Admin istrator c an Create, Modify, o r Delete u sers. The Genisis2 S ystem Admi nistrator is respons ible for c reating a user accou nt within Genisis2 a nd assigni ng one of the roles. The Genis is2 applic ation then coordinat es with th e VA LDAP server to create and store the role info rmation.
	233	Access Con trol
	234	As indicat ed in Sect ion 2.2, U ser Names and Passwo rds are co ntrolled b y centrali zed VA LDA P access c ontrol pro cesses. Pa ssword Exp iry and ot her admini strative p rocesses a re control led by the VA LDAP g roup.
	235	Genisis2 u ses browse r-based Wi ndows auth entication for usern ame and pa ssword aut henticatio n. Genisis 2 manages user types and provi des the us er with ac cess to sp ecific fun ctionality as descri bed in Sec tion 2.2.1 .
	236	User Notif ications
	237	EO may hav e specific user noti fication a nd escalat ion protoc ols for at tending to systems t hat are do wn with ot her applic ations and servers o n which th ey are hos ted.
	238	Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s for user notificat ions/escal ations cur rently in place.
	239	User Notif ication Po ints of Co ntact
	240	Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s regardin g specific notificat ion points of contac t (POCs) c urrently i n place.
	241	System Mon itoring, R eporting, and Tools
	242	Recommenda tions for system mon itoring, r eporting, and tools are that E O follows their stan dard proce dures for such activ ities curr ently in p lace. Thes e same too ls and pro cesses can be used f or with Ge nisis2 ser vers.
	243	Dataflow D iagram
	244	The Genesi s2 Request flow is d epicted in Figure 4.
	245
	246	Figure 4: Genisis2 R equest Flo w
	247
	248	Availabili ty Monitor ing
	249	Recommenda tions are that for G enisis2, E O follows their stan dard proce dures for monitoring the avail ability of other app lications, and monit oring the performanc e of appli cations fo r capacity planning purposes.
	250	Critical M etrics
	251	There are no metrics specific to the Gen isis2 appl ication re garding up time or do wntime. EO may have standard u ptime metr ics in pla ce for mon itoring th e availabi lity of ap plications . Our reco mmendation s are to u se the sam e ones for the Genis is2 applic ation.
	252	Routine Up dates, Ext racts and Purges
	253	Genisis2 d oes not an d should n ot require any routi ne data pu rges. The applicatio n needs to maintain a history of request s from Day 1 and kee p them for posterity . The data stored is not so la rge as to require pu rges for t he near fu ture.
	254	Scheduled Maintenanc e
	255	Genisis2 s cheduled m aintenance requires routine pa tches for these soft ware compo nents:
	256	Red Hat Li nux
	257	Microsoft Windows Se rver 2008 R2
	258	Microsoft SQL Server 2012
	259	EO can use their sta ndard oper ating proc edures to schedule a maintenan ce window at a time that is co nvenient f or and in coordinati on with th e user bas e.
	260	Capacity P lanning
	261	Genisis2 i s not such a perform ance inten sive appli cation tha t it requi res period ic capacit y planning reviews. The curren t capacity planned i n terms of number of servers, memory, an d disk spa ce assigne d is more than suffi cient to h andle over 50 times the curren t user bas e expected .
	262	Initial Ca pacity Pla n
	263	The initia l number o f users ex pected ove r the next two years is a maxi mum of 100 , with no more than 50 users e xpected to be logged in concur rently. Th e current capacity p lanned in terms of n umber of s ervers, me mory, and disk space assigned is more th an suffici ent for 3+ years of operation.
	264	Exception Handling
	265	Recommenda tions are that EO fo llows the standard T ier1 and T ier2 suppo rt process es current ly in plac e for Geni sis2 excep tion handl ing that i t uses for other VA applicatio ns.
	266	Routine Er rors
	267	Security E rrors
	268	Authentica tion and A uthorizati on errors can be exp ected. Sin ce Genesis 2 uses the VA LDAP s ystem, sta ndard VA T ier 1 and Tier 2 sup port proce sses curre ntly in pl ace for th e inabilit y to log i n because of a wrong password, wrong use rname, etc . are reco mmended fo r Genisis2 .
	269	Time-out E rrors
	270	Genisis2 t ime-out er rors may o ccur when the webser ver is not available . Standard VA Tier 1 and Tier 2 support processes currently in place f or unavail able appli cations du e to the w ebserver o r network being down are recom mended for Genisis2.
	271	Concurrenc y
	272	Genisis2 c oncurrency errors ar e not expe cted or id entifiable separate from appli cation err ors. Stand ard VA Tie r 1 and Ti er 2 suppo rt process es current ly in plac e for unav ailable ap plications are recom mended for Genisis2.
	273	Significan t Errors
	274	Significan t errors c an be defi ned as err ors or con ditions th at affect the system stability , availabi lity, perf ormance, o r otherwis e make the system un available to its use r base. Th e followin g subsecti ons contai n informat ion to aid administr ators, ope rators, an d other su pport pers onnel in t he resolut ion of sig nificant e rrors, con ditions, o r other is sues.
	275	Applicatio n Error Lo gs
	276	The applic ation erro r logs are located a s follows: - /opt/ge nisis-appl ication-wi ldfly-10.0 .0/standal one/log/se rver.log
	277	Applicatio n Error Co des and De scriptions
	278	No applica tion speci fic error codes exis t for the Genisis2 a pplication .
	279	Infrastruc ture Error s
	280	Database
	281	Microsoft SQL Server 2012 is a component of the Ge nisis2 app lication. Generic da tabase err ors troubl eshooting can be fou nd in the Microsoft web pages located at https://m sdn.micros oft.com/en -us/librar y/aa952081 .aspx. The site disc usses typi cal connec tivity, pe rmissions, and datab ase sizing problems. Standard VA Tier 1 and Tier 2 support p rocesses c urrently i n place ar e recommen ded for Ge nisis2.
	282	Genisis2 a pplication errors ar e manifest ed as appl ication er rors and a re covered in the Ge nisis2 Mai ntenance a nd Trouble shooting G uide.
	283	Web Server
	284	The troubl eshooting guide for the Apache Web Serve r is locat ed at http s://httpd. apache.org /docs/2.4/ custom-err or.html.
	285	Applicatio n Server
	286	The troubl eshooting guide for the JBOSS Wildfly Ap plication Server is located at https://d ocs.jboss. org/author /display/W FLY10/Trou bleshootin g+Common+I ssues.
	287	Network
	288	EO manages the netwo rk where t he servers are hoste d for the Genisis2 a pplication . Standard VA help d esk operat ional proc esses curr ently in p lace for a ddressing network er rors are r ecommended for Genis is2. The G enisis2 ap plication has been c onfigured and tested with a st andard set of networ k paramete rs. Due to security protocols, any chang es in thes e paramete rs should be address ed by EO.
	289	Authentica tion and A uthorizati on
	290	User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by th at group. (Refer to Figure 2 f or the Gen isis Secur ity Archit ecture dia gram.)
	291	Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis Roles to allow acce ss to part s of the G enisis2 ap plication. If a user does not have a rol e within G enisis2, t hey cannot proceed b eyond the login page . Genisis2 user type s are mana ged by the Genisis2 applicatio n to provi de each us er with ce rtain func tionality, depending upon thei r role. Fo r example, a user is recognize d as a Req uestor, Da ta Destina tion Manag er, or Gen isis2 Syst em Adminis trator and are affor ded differ ent levels of functi onality wi thin the a pplication .
	292	Errors rel ating to A uthenticat ion and Au thorizatio n may lead a user to the VA LD AP, since that group manages t he Usernam e/Password authentic ations for Genisis2. EO or the appropria te VA LDAP help desk would be the author ities that address t hese error s through the use of Tier 1 an d Tier 2 s upport pro cedures cu rrently in place.
	293	Logical an d Physical Descripti ons
	294	Refer to F igure 1 fo r the Geni sis2 serve r architec ture diagr am.
	295	The Webser ver consis ts of Apac he running on RHEL. It employs Angular J S as the U ser Interf ace (UI) f ramework.
	296	The Wildfl y Applicat ion Server runs on R HEL. It su pports two main comp onents: th e Applicat ion Code i n Java and the Java Business P rocess Man agement (J BPM) engin e.
	297	The Databa se Server runs Micro soft Windo ws 2008 R2 and hosts a Microso ft SQL ser ver 2012. The Genisi s2 Databas es are hos ted here.
	298
	299	Dependent System(s)
	300	Genisis2 a cquires th e users’ u sernames/p asswords a nd other u ser detail s from the VA LDAP s ystem. Thi s is the o nly system Genisis2 is depende nt upon an d in progr ammatic co mmunicatio n with aut omatically . Errors/t roubleshoo ting are h andled by the approp riate VA L DAP help d esk.
	301	Troublesho oting
	302	The step b y step pro cess for t roubleshoo ting when the system is not up and the u sers not a ble to log in:
	303	Ensure tha t the Apac he and Wil dfly serve rs are up and runnin g.
	304	Ensure tha t the corr ect config uration fi les are pr esent in e ach server and are p ointing to the corre ct end poi nts.
	305	Ensure tha t the fire wall ports are open and the se rvers can communicat e with eac h other.
	306	Ensure tha t the SQL server is up and run ning and c orrect acc ess creden tials are present (e dited).
	307	Ensure tha t SQL serv er user ha s adequate database permission s for CRUD operation s.
	308	System Rec overy
	309	EO manages the serve rs for the Genisis2 applicatio n. Standar d system r ecovery pr ocesses cu rrently in place sho uld be use d for brin ging the s ystems bac k online. If all ser vers are b ack online , then Gen isis2 serv ices are a lso restor ed. Genisi s2 applica tion compo nents comm unicate us ing REST c alls and w ere config ured and t ested with these cal ls.
	310	Restart af ter Non-Sc heduled Sy stem Inter ruption
	311	Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a n on-schedul ed system interrupti on.
	312	Restart af ter Databa se Restore
	313	Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a d atabase re store oper ation.
	314	Back-out P rocedures
	315	The back-o ut procedu res for th e Applicat ion Server and the W ebserver a re as foll ows:
	316	Wildfly Ap plication Server: De lete the W ildfly War file from deploymen t director y.
	317	Apache Web server: De lete all c ontents un der /var/w ww/html/.
	318	Rollback P rocedures
	319	Release 1 will be ru nning on t he PRE-PRO D environm ent with t he databas e frozen a s of the d ate of Rel ease 2. Fo llow the i nstallatio n procedur es for Rel ease 1 on the PROD e nvironment . Copy the Release 1 Database from the P RE-PROD da tabase ser ver to the PROD serv er. This w ill rollba ck Release 2 to Rele ase 1 on t he PROD pr oduction s ervers. Re start serv ices on th e PROD mac hines.
	320	Operations and Maint enance Res ponsibilit ies
	321	EO manages the netwo rk where t he servers for the G enisis2 ap plication are hosted . Standard help desk operation al process es current ly in plac e should b e used for addressin g operatio ns and mai ntenance r esponsibil ities at t he data ce nter where the appli cation is hosted. Us ually, the se include patches f or securit y and rout ine minor version up grades for security and perfor mance reas ons. If ma jor upgrad es are nee ded, the a pplication may need to undergo a proper version re lease proc ess with a ppropriate testing b eforehand.
	322
	323
	324	Approval S ignatures
	325
	326	REVIEW DAT E:
	327	SCRIBE:
	328
	329	Signed: __ __________ __________ __________ __________ __________ __________ _________
	330	Program Ma nager, Ken Rikkard D ate
	331
	332	Signed: __ __________ __________ __________ __________ __________ __________ _________
	333	Product Ow ner, Saiju Pyarajan Date
	334
	335	Signed: __ __________ __________ __________ __________ __________ __________ _________
	336	Receiving Organizati on, Karen Leake Dat e
	337
	338	Signed: __ __________ __________ __________ __________ __________ __________ _________
	339	Product Su pport, Pau l Hsieh Date
	340
	341
	342
	343
	344
	345
	346
	347
	348
	349
	350	Template R evision Hi story
	351	Date
	352	Version
	353	Descriptio n
	354	Author
	355	March 2016
	356	1.6
	357	Updated to remove PM AS referen ces and to include V IP referen ces. Elimi nated unne cessary te xt and mos t instance s of passi ve voice.
	358	Wichita VI P Release Process Te am
	359	June 2015
	360	1.5
	361	Updated co ver and ed ited for S ection 508 conforman ce and rem ediated wi th Common Look Offic e tool
	362	Process Ma nagement
	363	May 2015
	364	1.4
	365	Revised co ntent by P MAS Proces s Improvem ent Lockdo wn and reo rdered cov er sheet t o enhance SharePoint search re sults
	366	Process Ma nagement
	367	November 2 014
	368	1.3
	369	Updated Se ction 4 fo r url chan ge to the Operations and Maint enance Res ponsibilit y Matrix
	370	Process Ma nagement
	371	December 2 013
	372	1.2
	373	Correction to headin gs
	374	Process Ma nagement
	375	March 2013
	376	1.1
	377	Formatted to documen tation sta ndards and edited fo r Section 508 confor mance
	378	Process Ma nagement
	379	January 20 13
	380	1.0
	381	Initial Do cument
	382	PMAS Busin ess Office
	383