3. EPMO Open Source Coordination Office Redaction File Detail Report

Produced by Araxis Merge on 12/21/2017 6:15:08 PM Eastern Standard Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.

3.1 Files compared

#	Location	File	Last Modified
1			Thu Dec 21 23:15:07 2017 UTC
2	Genisis_2.0_v7_bld7.zip\Build 7	Genisis2_VIP_Build 7_Production_Operations_Manual_12152017.docx	Thu Dec 21 22:38:23 2017 UTC

3.2 Comparison summary

Description	Between Files 1 and 2
Description	Text Blocks	Lines
Unchanged	0	0
Changed	0	0
Inserted	1	627
Removed	0	0

3.3 Comparison options

Whitespace
Character case	Differences in character case are significant
Line endings	Differences in line endings (`CR` and `LF` characters) are ignored
CR/LF characters	Not shown in the comparison detail

3.4 Active regular expressions

No regular expressions were active.

3.5 Comparison detail

				1		Production Operation s Manual
				2
				3		Genomic In formation System for Integrate d Science 2 (Genisis 2) Technic al Service s
				4		Build 7
				5
				6
				7
				8
				9
				10
				11
				12		December 2 017
				13
				14		Document V ersion 6.0
				15
				16		Department of Vetera ns Affairs
				17
				18		Document R evision Hi story
				19
				20
				21		Date
				22		Version
				23		Descriptio n
				24		Author
				25		02/10/2017
				26		1.0
				27		Build 1 up dates.
				28		Booz Allen Hamilton
				29		05/12/2017
				30		2.0
				31		Build 2 up dates.
				32		Booz Allen Hamilton
				33		06/16/2017
				34		3.0
				35		Build 3 up dates.
				36		Booz Allen Hamilton
				37		09/15/2017
				38		4.0
				39		Build 4 up dates.
				40		Booz Allen Hamilton
				41		11/03/2017
				42		5.0
				43		Build 5 up dates.
				44		Booz Allen Hamilton
				45		11/24/2017
				46		6.0
				47		Build 7 up dates.
				48		Booz Allen Hamilton
				49
				50
				51		Note: The revision h istory cyc le begins once chang es or enha ncements a re request ed after t he Product ion Operat ions Manua l has been baselined .
				52
				53
				54
				55		Artifact R ationale
				56
				57		The Produc tion Opera tions Manu al provide s the info rmation ne eded by th e producti on operati ons team t o maintain and troub leshoot th e product. The Produ ction Oper ations Man ual must b e provided prior to release of the produ ct.
				58
				59		Table of C ontents
				60
				61		Introducti on5
				62		Routine Op erations5
				63		Administra tive Proce dures5
				64		System Sta rtup5
				65		System Sta rtup from Emergency Shutdown6
				66		System Shu tdown6
				67		Emergency System Shu tdown7
				68		Back-up & Restore7
				69		Back-up Pr ocedures9
				70		Restore Pr ocedures9
				71		Back-up Te sting9
				72		Storage an d Rotation 9
				73		Security / Identity Management 10
				74		Identity M anagement1 0
				75		Access Con trol11
				76		User Notif ications11
				77		User Notif ication Po ints of Co ntact12
				78		System Mon itoring, R eporting, and Tools1 2
				79		Dataflow D iagram for Genisis2 Data Reque st Workflo ws12
				80		Data Flow Diagram fo r Genisis2 Terminolo gy Service 12
				81		Availabili ty Monitor ing13
				82		Critical M etrics13
				83		Routine Up dates, Ext racts and Purges13
				84		Scheduled Maintenanc e13
				85		Capacity P lanning14
				86		Initial Ca pacity Pla n14
				87		Exception Handling14
				88		Routine Er rors14
				89		Security E rrors14
				90		Time-out E rrors14
				91		Concurrenc y14
				92		Significan t Errors14
				93		Applicatio n Error Lo gs15
				94		Applicatio n Error Co des and De scriptions 15
				95		Infrastruc ture Error s15
				96		Database15
				97		Web Server 15
				98		Applicatio n Server15
				99		Network16
				100		3.2.3.6.Lo gical and Physical D escription s16
				101		Dependent System(s)1 7
				102		Troublesho oting17
				103		System Rec overy17
				104		Restart af ter Non-Sc heduled Sy stem Inter ruption18
				105		Restart af ter Databa se Restore 18
				106		Back-out P rocedures1 8
				107		Rollback P rocedures1 8
				108		Operations and Maint enance Res ponsibilit ies19
				109		Approval S ignatures2 0
				110
				111
				112		Introducti on
				113		The Produc tion Opera tions Manu al describ es how to maintain t he compone nts of the Genomic I nformation System fo r Integrat ed Science 2 (Genisi s2), as we ll as how to trouble shoot prob lems that might occu r with thi s applicat ion in pro duction. T he intende d audience for this document a re the Inf ormation T echnology (IT) teams responsib le for hos ting and m aintaining the appli cation aft er product ion releas e. This do cument is normally f inalized j ust prior to product ion releas e and incl udes many updated el ements spe cific to t he hosting environme nt.
				114
				115		Routine Op erations
				116		Table 1 li sts the ro utine oper ations for Genisis2.
				117
				118		Table 1: R outine Ope rations
				119
				120		Routine Op eration
				121		Periodicit y
				122		Role perfo rming the Function
				123		Additional External Systems, i f Needed
				124		Additional Descripti on
				125		System Log in Availab ility from the VA Ne twork
				126		Daily
				127		Genisis2 S ystem Admi nistrator (Super Use r)
				128		None
				129		Verifies t hat the sy stem is up and runni ng.
				130		Systems Mo nitoring a nd Alertin g, Escalat ion
				131		Real time 24/7
				132		Genisis2 S ystem Admi nistrator (Super Use r)
				133		VA Systems Monitorin g and Aler ting Facil ities
				134		Monitors e ach of the Web, Appl ication an d Data Ser vers statu s. They ne ed to be u p and runn ing
				135
				136		Administra tive Proce dures
				137		System Sta rtup
				138		Table 2 pr ovides the system st artup proc edures for the Genes is2 server s.
				139
				140		Table 2: S ystem Star tup
				141
				142		Component
				143		Procedure
				144		Additional Comments
				145		Apache Ser ver
				146		sudo servi ce httpd s tart
				147		For Genisi s2 Workflo ws
				148		Genisis2 A pplication Wildfly S erver
				149		nohup ./st andalone.s h &
				150		For Genisi s2 Workflo ws
				151		Genisis2 A pplication Tomcat Se rver
				152		sudo servi ce tomcat start
				153		For Genisi s2 Termino logy Servi ce
				154		Apache Sol r Server
				155		cd <<SOLR_ HOME>>
				156		cd /u06/ts /solr-6.0. 0
				157
				158		bin/solr s tart
				159		For Genisi s2 Termino logy Servi ce - assum ing SOLR_H OME is /u0 6/ts/solr- 6.0.0
				160
				161
				162		Component
				163		Procedure
				164		Additional Comments
				165		Apache Jen a/Fuseki s erver
				166		cd <<FUSEK I_HOME>>
				167		cd /u06/ts /fuseki
				168
				169		./fuseki s tart
				170		For Genisi s2 Termino logy Servi ce -assumi ng FUSEKI_ HOME is
				171		/u06/ts/fu seki
				172
				173		System Sta rtup from Emergency Shutdown
				174		In the eve nt of an e mergency s hutdown, T able 3 lis ts the sys tem startu p procedur es for the Genesis2 servers.
				175
				176		Table 3: S ystem Star tup from E mergency S hutdown
				177
				178		Component
				179		Procedure
				180		Additional Comments
				181		Apache Ser ver
				182		sudo servi ce httpd s tart
				183		For Genisi s2 Workflo ws
				184		Genisis2 A pplication Wildfly S erver
				185		cd /opt/wi ldfly
				186		nohup ./st andalone.s h &
				187		For Genisi s2 Workflo ws
				188		Genisis2 A pplication Tomcat Se rver
				189		sudo servi ce tomcat start
				190		For Genisi s2 Termino logy Servi ce
				191		Apache Sol r Server
				192		cd <<SOLR_ HOME>>
				193		cd /u06/ts /solr-6.0. 0
				194
				195		bin/solr s tart
				196		For Genisi s2 Termino logy Servi ce - assum ing SOLR_H OME is /u0 6/ts/solr- 6.0.0
				197		Apache Jen a/Fuseki s erver
				198		cd <<FUSEK I_HOME>>
				199		cd /u06/ts /fuseki
				200
				201		./fuseki s tart
				202		For Genisi s2 Termino logy Servi ce - assu ming FUSEK I_HOME is
				203		/u06/ts/fu seki
				204
				205
				206		System Shu tdown
				207		Table 4 pr ovides the system sh utdown pro cedures fo r the Gene sis2 serve rs.
				208
				209		Table 4: S ystem Shut down
				210
				211		Component
				212		Procedure
				213		Additional Comments
				214		Apache Ser ver
				215		sudo servi ce httpd s top
				216		For Genisi s2 Workflo ws
				217		Genisis2 A pplication Wildfly S erver
				218		ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
				219		For Genisi s2 Workflo ws
				220		Genisis2 A pplication Tomcat Se rver
				221		sudo servi ce tomcat stop
				222		For Genisi s2 Termino logy Servi ce
				223
				224
				225		Apache Sol r Server
				226		ps -ef \| g rep solr \| awk '{pri nt
				227		$2}' \| xar gs kill -9
				228		For Genisi s2 Termino logy Servi ce
				229		Apache Jen a/Fuseki s erver
				230		ps -ef \| g rep fuseki \| awk '{p rint $2}' \| xargs ki ll -9
				231		For Genisi s2 Termino logy Servi ce
				232
				233
				234		Emergency System Shu tdown
				235		For an eme rgency sys tem shutdo wn, Table 5 lists th e system s hutdown pr ocedures f or the Gen esis2 serv ers.
				236
				237		Table 5: E mergency S ystem Shut down
				238
				239		Component
				240		Procedure
				241		Additional Comments
				242		Apache Ser ver
				243		sudo servi ce httpd s top
				244		For Genisi s2 Workflo ws
				245		Genisis2 A pplication Wildfly S erver
				246		ps -ef \| g rep jboss \| awk '{pr int $2}' \| xargs kil l -9
				247		For Genisi s2 Workflo ws
				248		Genisis2 A pplication Tomcat Se rver
				249		sudo servi ce tomcat stop
				250		For Genisi s2 Termino logy Servi ce
				251		Apache Sol r Server
				252		ps -ef \| g rep solr \| awk '{pri nt
				253		$2}' \| xar gs kill -9
				254		For Genisi s2 Termino logy Servi ce
				255		Apache Jen a/Fuseki s erver
				256		ps -ef \| g rep fuseki \| awk '{p rint $2}' \| xargs ki ll -9
				257		For Genisi s2 Termino logy Servi ce
				258
				259
				260		Back-up & Restore
				261		Figure 1 r epresents the Genisi s2 server architectu re. It con sists of t he followi ng compone nts that n eed to be backed up:
				262		Apache Web server run ning Red H at Enterpr ise Linux (RHEL)
				263		Apache Web server2/Wi ldfly Appl ication Se rver runni ng RHEL
				264		Database S erver runn ing Micros oft Window s 2008 R2, SQL Serve r 2012
				265
				266
				267
				268
				269		Figure 1: Genisis2 S erver Arch itecture f or Data Re quest Work flows
				270
				271
				272		Figure 2 r epresents the Genisi s2 server architectu re for Ter minology S ervice. It consists of the fol lowing com ponents th at need to be backed up:
				273		Apache Tom cat Applic ation Serv er running Red Hat E nterprise Linux (RHE L)
				274		Apache SOL R running RHEL
				275		Database S erver runn ing Micros oft Window s 2008 R2, SQL Serve r 2012
				276
				277
				278
				279		Port
				280		Port 80Cen tos/RedHat Linux
				281
				282
				283
				284
				285
				286
				287
				288
				289
				290
				291
				292		Web Server / Applicat ion Server Running
				293		Apache HTT P
				294		Tomcat
				295		Jena/Fusek i
				296		SOLR
				297		Database S erver
				298		Windows Se rver 2008 R2
				299		Microsoft SQL Server 2012
				300
				301
				302
				303
				304		Figure 2: Genisis2 S erver Arch itecture f or Termino logy Servi ce
				305
				306		Back-up Pr ocedures
				307		Table 6 pr ovides fre quency rec ommendatio ns for per forming ba ck-ups. Ve terans Aff airs (VA) Enterprise Operation s (EO) fol lows their Standard Operating Procedures (SOPs) fo r completi ng the bac k-ups.
				308
				309		Table 6: C omponent B ack-up Fre quency
				310
				311		Category
				312		Component
				313		Frequency
				314		Applicatio ns
				315		Applicatio ns
				316		Daily back -ups
				317		Applicatio n Servers
				318		Apache Web Server Ap plication Server Bui ld Server
				319		Weekly ful l back-ups
				320		Database
				321		Database
				322		Hourly bac k-ups
				323		Database S erver
				324		Windows Da ta Base Se rver
				325		Weekly ful l back-ups
				326
				327		Restore Pr ocedures
				328		Restore pr ocedures f rom back-u ps may be necessary. Depending upon what needs to be backed up, VA EO follows th eir existi ng SOPs fo r completi ng the res tores.
				329		Table 7 li sts the ap plicable b ack-up fre quency fro m which re stores can be perfor med.
				330
				331		Table 7: C omponent R estore Fre quency
				332
				333		Category
				334		Component
				335		Frequency
				336		Applicatio ns
				337		Applicatio ns
				338		Daily back -ups
				339		Applicatio n Servers
				340		Apache Web Server Ap plication Server Bui ld Server
				341		Weekly ful l back-ups
				342		Database
				343		Database
				344		Hourly bac k-ups
				345		Database S erver
				346		Windows Da tabase Ser ver
				347		Weekly ful l back-ups
				348
				349		Back-up Te sting
				350		Recommenda tions for back-up te sting incl ude perfor ming a res tore opera tion. Spec ifically:
				351		For applic ations – f ollow the standard E O restore procedures for Linux servers.
				352		For databa se – follo w the stan dard EO re store proc edures for the Micro soft SQL s erver 2012 and the W indows ser ver.
				353
				354		Storage an d Rotation
				355		Recommenda tions for storage an d rotation are that EO follows their sta ndard disa ster recov ery proced ures curre ntly in pl ace.
				356
				357		Security / Identity Management
				358		Figure 2 r epresents the Genisi s2 securit y architec ture.
				359
				360
				361
				362		Figure 2: Genisis2 S ecurity Ar chitecture
				363		User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by VA LDAP grou p.
				364		Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis 2 Roles to allow acc ess to par ts of the Genisis2 a pplication . User Rol es are man aged in a separate G enisis2 Op en LDAP se rver like other Mill ion Vetera n Program (MVP) appl ications. If a user does not h ave a role within Ge nisis2, th ey cannot proceed be yond the l ogin page. Genisis2 user types are utili zed by the Genisis2 applicatio n to provi de each us er with ce rtain func tionality, depending upon thei r role. Fo r example, a user is recognize d as a Res earcher, G enisis2 Ma nager, VIN CI Manager , or Genis is2 System Administr ator, and are afford ed differe nt levels of functio nality wit hin the ap plication.
				365
				366		Identity M anagement
				367		Genisis2 s upports th e followin g user typ es for Gen isis2 Data Request W orkflows:
				368		Researcher
				369		Genisis2 M anager
				370		VINCI Mana ger
				371		Genisis2 S ystem Admi nistrator
				372		Please not e that Gen isis2 Buil d 7 is a R esearcher ONLY relea se. The on ly role re cognized i n Release 7 is the R esearcher role in th e set of r oles above .
				373
				374		The Resear cher has t he minimum set of ac tivities t hey can ac cess and p erform in the applic ation. For example, the Resear cher can g enerate an d track th eir own da ta request s.
				375		The Genisi s2 Manager has all t he capabil ities of a Researche r, but can review an d approve requests, review and approve d ata result s, and tra ck request s more bro adly.
				376		The VINCI Manager ad dresses an y question s about th e data tha t they may have with the Resea rcher (thr ough the G enisis2 Ma nager and not direct ly), prepa res and pl aces the d ata in a S ource land ing zone d atabase, a nd notifie s the Geni sis2 Manag er of its location.
				377		The Genisi s2 Manager will then copy the data over from the S ource land ing zone d atabase to the Desti nation lan ding zone database, extract th e data to a flat fil e, and per form any a dditional cleanup th at may be required. Personally Identifia ble Inform ation (PII ) and Prot ected Heal th Informa tion (PHI) informati on is then removed f rom this d ata and co pied over to the spe cific Stud y Mart set up for th e Research er.
				378		The Genisi s2 System Administra tor is a s uper user that can p erform all the funct ions that the Resear cher and t he Genisis 2 Manager can. In ad dition, th e Genisis2 System Ad ministrato r can sync hronize wi th the Ope n LDAP ser ver on-dem and or per iodically on schedul e to retri eve and st ore the la test set o f users an d their ro les.
				379		Genisis2 s upports th e Research er user ty pe for Ter minology S ervice. Th e Research er uses Te rminology Service to locate th e data ele ments they need by n avigating a set of C oncepts fr om many On tologies l ike SNOMED -CT, HP, N CIT, etc. The goal o f Terminol ogy Servic e is to le t Research ers start their sear ches with the clinic al and bio -medical l anguage th ey are fam iliar with , and arri ve at conc epts in th ese Ontolo gies. The concepts a re attache d to speci fic data e lements in Genomic, Survey and Clinical data. Term inology Se rvice prov ides a gui ded search capabilit y to data elements o f interest for resea rch use.
				380
				381		Access Con trol
				382		As indicat ed in Sect ion 2.2, U ser Names and Passwo rds are co ntrolled b y centrali zed VA LDA P access c ontrol pro cesses. Pa ssword Exp iry and ot her admini strative p rocesses a re control led by the VA LDAP g roup.
				383		Genisis2 u ses browse r-based Wi ndows auth entication for usern ame and pa ssword aut henticatio n. Genisis 2 manages user types and provi des the us er with ac cess to sp ecific fun ctionality as descri bed in Sec tion 2.2.1 .
				384
				385		User Notif ications
				386		EO may hav e specific user noti fication a nd escalat ion protoc ols for at tending to systems t hat are do wn with ot her applic ations and servers o n which th ey are hos ted.
				387		Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s for user notificat ions/escal ations cur rently in place.
				388
				389		User Notif ication Po ints of Co ntact
				390		Recommenda tions for user notif ications f or any sch eduled or unschedule d changes in the sys tem state such as pl anned outa ges, patch upgrades, etc. are that EO fo llows thei r standard procedure s regardin g specific notificat ion points of contac t (POCs) c urrently i n place.
				391
				392		System Mon itoring, R eporting, and Tools
				393		Recommenda tions for system mon itoring, r eporting, and tools are that E O follows their stan dard proce dures for such activ ities curr ently in p lace. Thes e same too ls and pro cesses can be used f or with Ge nisis2 ser vers.
				394
				395		Dataflow D iagram for Genisis2 Data Reque st Workflo ws
				396		The Genesi s2 Request flow is d epicted in Figure 4.
				397
				398
				399		Smart orch estration and automa tion of
				400		workflows and ETL pr ocesses wi ll allow G enISIS 2.0 to preser ve data in tegrity an d scale to meet esca lating dem ands of a national r esource
				401
				402		Figure 4: Genisis2 R equest Flo w
				403
				404
				405		Data Flow Diagram fo r Genisis2 Terminolo gy Service
				406
				407		The Genesi s2 Termino logy Servi ce flow is depicted in Figure 5.
				408
				409
				410
				411
				412		Figure 5: Genisis2 T erminology Service F low
				413
				414
				415		Availabili ty Monitor ing
				416		Recommenda tions are that for G enisis2, E O follows their stan dard proce dures for monitoring the avail ability of other app lications, and monit oring the performanc e of appli cations fo r capacity planning purposes.
				417
				418		Critical M etrics
				419		There are no metrics specific to the Gen isis2 appl ication re garding up time or do wntime. EO may have standard u ptime metr ics in pla ce for mon itoring th e availabi lity of ap plications . Our reco mmendation s are to u se the sam e ones for the Genis is2 applic ation.
				420
				421		Routine Up dates, Ext racts and Purges
				422		Genisis2 d oes not an d should n ot require any routi ne data pu rges. The applicatio n needs to maintain a history of request s from Day 1 and kee p them for posterity . The data stored is not so la rge as to require pu rges for t he near fu ture.
				423
				424		Scheduled Maintenanc e
				425		Genisis2 s cheduled m aintenance requires routine pa tches for these soft ware compo nents:
				426		Red Hat Li nux
				427		Microsoft Windows Se rver 2008 R2
				428
				429		Microsoft SQL Server 2012
				430		EO can use their sta ndard oper ating proc edures to schedule a maintenan ce window at a time that is co nvenient f or and in coordinati on with th e user bas e.
				431
				432		Capacity P lanning
				433		Genisis2 i s not such a perform ance inten sive appli cation tha t it requi res period ic capacit y planning reviews. The curren t capacity planned i n terms of number of servers, memory, an d disk spa ce assigne d is more than suffi cient to h andle over 50 times the curren t user bas e expected .
				434
				435		Initial Ca pacity Pla n
				436		The initia l number o f users ex pected ove r the next two years is a maxi mum of 100 , with no more than 50 users e xpected to be logged in concur rently. Th e current capacity p lanned in terms of n umber of s ervers, me mory, and disk space assigned is more th an suffici ent for 3+ years of operation.
				437
				438		Exception Handling
				439		Recommenda tions are that EO fo llows the standard T ier1 and T ier2 suppo rt process es current ly in plac e for Geni sis2 excep tion handl ing that i t uses for other VA applicatio ns.
				440
				441		Routine Er rors
				442		Security E rrors
				443		Authentica tion and A uthorizati on errors can be exp ected. Sin ce Genesis 2 uses the VA LDAP s ystem, sta ndard VA T ier 1 and Tier 2 sup port proce sses curre ntly in pl ace for th e inabilit y to log i n because of a wrong password, wrong use rname, etc . are reco mmended fo r Genisis2 .
				444
				445		Time-out E rrors
				446		Genisis2 t ime-out er rors may o ccur when the webser ver is not available . Standard VA Tier 1 and Tier 2 support processes currently in place f or unavail able appli cations du e to the w ebserver o r network being down are recom mended for Genisis2.
				447
				448		Concurrenc y
				449		Genisis2 c oncurrency errors ar e not expe cted or id entifiable separate from appli cation err ors. Stand ard VA Tie r 1 and Ti er 2 suppo rt process es current ly in plac e for unav ailable ap plications are recom mended for Genisis2.
				450
				451		Significan t Errors
				452		Significan t errors c an be defi ned as err ors or con ditions th at affect the system stability , availabi lity, perf ormance, o r otherwis e make the system un available to its use r base. Th e followin g subsecti ons contai n informat ion to aid administr ators, ope rators, an d other su pport pers onnel in t he resolut ion of sig nificant e rrors, con ditions, o r other is sues.
				453
				454		Applicatio n Error Lo gs
				455		The applic ation erro r logs are located a s follows: - /opt/ge nisis-appl ication-wi ldfly- 10. 0.0/standa lone/log/s erver.log
				456
				457		Applicatio n Error Co des and De scriptions
				458		No applica tion speci fic error codes exis t for the Genisis2 a pplication .
				459
				460		Infrastruc ture Error s
				461		Database
				462		Microsoft SQL Server 2012 is a component of the Ge nisis2 app lication. Generic da tabase err ors troubl eshooting can be fou nd in the Microsoft web pages located at https:// msdn.micro soft.com/e n-us/libra ry/aa95208 1.aspx. Th e site dis cusses typ ical conne ctivity, p ermissions , and data base sizin g problems . Standard VA Tier 1 and Tier 2 support processes currently in place a re recomme nded for G enisis2.
				463		Genisis2 a pplication errors ar e manifest ed as appl ication er rors and a re covered in the Ge nisis2 Mai ntenance a nd Trouble shooting G uide.
				464
				465		Web Server
				466		The troubl eshooting guide for the Apache Web Serve r is locat ed at htt ps://httpd .apache.or g/docs/2.4 /custom-er ror.html.
				467
				468		Applicatio n Server
				469		The troubl eshooting guide for the JBOSS Wildfly Ap plication Server is located at https:// docs.jboss .org/autho r/display/ WFLY10/Tro ubleshooti ng+Common+ Issues.
				470
				471
				472		The troubl eshooting guide for Apache Tom cat Applic ation Serv er is loca ted at ht tps://wiki .apache.or g/tomcat/F AQ/Trouble shooting_a nd_Diagnos tics
				473
				474		The troubl eshooting resource f or Apache Jena/Fusek i is locat ed at htt ps://jena. apache.org /help_and_ support/#e mail-suppo rt-lists
				475
				476		The troubl eshooting resources for Apache SOLR are located at https:// lucene.apa che.org/so lr/communi ty.html
				477
				478		Network
				479		EO manages the netwo rk where t he servers are hoste d for the Genisis2 a pplication . Standard VA help d esk operat ional proc esses curr ently in p lace for a ddressing network er rors are r ecommended for Genis is2. The G enisis2 ap plication has been c onfigured and tested with a st andard set of networ k paramete rs. Due to security protocols, any chang es in thes e paramete rs should be address ed by EO.
				480
				481		Authentica tion and A uthorizati on
				482		User Names and Passw ords are c ontrolled by central ized VA LD AP access control pr ocesses. P assword Ex piry and o ther admin istrative processes are contro lled by th at group. (Refer to Figure 2 f or the Gen isis Secur ity Archit ecture dia gram.)
				483		Genisis2 u ses integr ated PIV/W indows Aut henticatio n that the VA LDAP s erver supp orts. When a user lo gs into th e VA Netwo rk using t heir PIV c ard, they are authen ticated in itially. G enisis2 us es browser -based Win dows Authe ntication to authent icate the login of t his user a nd then us es Genisis 2 Roles to allow acc ess to par ts of the Genisis2 a pplication . User Rol es are man aged in a separate G enisis2 Op en LDAP se rver like other Mill ion Vetera n Program (MVP) appl ications. If a user does not h ave a role within Ge nisis2, th ey cannot proceed be yond the l ogin page. Genisis2 user types are utili zed by the Genisis2 applicatio n to provi de each us er with ce rtain func tionality, depending upon thei r role. Fo r example, a user is recognize d as a Res earcher, G enisis2 Ma nager, VIN CI Manager , or Genis is2 System Administr ator, and are afford ed differe nt levels of functio nality wit hin the ap plication.
				484		Errors rel ating to A uthenticat ion and Au thorizatio n may lead a user to the VA LD AP, since that group manages t he Usernam e/Password authentic ations for Genisis2. EO or the appropria te VA LDAP help desk would be the author ities that address t hese error s through the use of Tier 1 an d Tier 2 s upport pro cedures cu rrently in place.
				485
				486		Logical an d Physical Descripti ons
				487
				488		Genisis2 D ata Reques t Workflow s: Refer t o Figure 1 for the G enisis2 se rver archi tecture di agram.
				489		The Webser ver consis ts of Apac he running on RHEL. It employs Angular J S as the U ser Interf ace (UI) f ramework.
				490		The Wildfl y Applicat ion Server runs on R HEL. It su pports two main comp onents: th e Applicat ion Code i n Java and the Java Business P rocess Man agement (J BPM) engin e.
				491		The Databa se Server runs Micro soft Windo ws 2008 R2 and hosts a Microso ft SQL ser ver 2012. The Genisi s2 Databas es are hos ted here.
				492
				493		Genisis2 T erminology Service: Refer to F igure 2 fo r the Geni sis2 Termi nology Ser vice serve r architec ture diagr am.
				494		The Webser ver consis ts of Apac he running on RHEL. It employs Angular J S as the U ser Interf ace (UI) f ramework.
				495		The Tom Ap plication Server run s on RHEL. It suppor ts multipl e main com ponents: t he Applica tion Code in Java, J ena/Fuseki , and SOLR servers.
				496
				497		The Databa se Server runs Micro soft Windo ws 2008 R2 and hosts a Microso ft SQL ser ver 2012. The Genisi s2 Termino logy Servi ce databas es are hos ted here.
				498
				499		Dependent System(s)
				500		Genisis2 a cquires th e username s/password s and othe r user det ails from the VA LDA P system. Genisis ac quires the user role s from the Genisis2 Open LDAP server. Ge nisis2 is dependent upon the V A LDAP and is in pro grammatic communicat ion with i t automati cally.
				501		Errors/tro ubleshooti ng are han dled by th e appropri ate VA LDA P and Geni sis2 help desks.
				502
				503		Troublesho oting
				504		The step b y step pro cess for t roubleshoo ting when the system is not up and the u sers not a ble to log in:
				505		Genisis2 D ata Reques t Workflow s:
				506		Ensure tha t the Apac he and Wil dfly serve rs are up and runnin g.
				507		Ensure tha t the corr ect config uration fi les are pr esent in e ach server and are p ointing to the corre ct end poi nts.
				508		Ensure tha t the fire wall ports are open and the se rvers can communicat e with eac h other.
				509		Ensure tha t the SQL server is up and run ning and c orrect acc ess creden tials are present (e dited).
				510		Ensure tha t SQL serv er user ha s adequate database permission s for CRUD operation s.
				511
				512		Genisis2 T erminology Service:
				513		Ensure tha t the Apac he and Tom cat server s are up a nd running .
				514		Ensure tha t the corr ect config uration fi les are pr esent in e ach server and are p ointing to the corre ct end poi nts.
				515		Ensure tha t the fire wall ports are open and the se rvers can communicat e with eac h other.
				516		Ensure tha t the SQL server is up and run ning and c orrect acc ess creden tials are present (e dited).
				517		Ensure tha t SQL serv er user ha s adequate database permission s for CRUD operation s.
				518
				519
				520		System Rec overy
				521		EO manages the serve rs for the Genisis2 applicatio n. Standar d system r ecovery pr ocesses cu rrently in place sho uld be use d for brin ging the s ystems bac k online. If all ser vers are b ack online , then Gen isis2 serv ices are a lso restor ed. Genisi s2 applica tion compo nents comm unicate us ing REST c alls and w ere config ured and t ested with these cal ls.
				522
				523		Restart af ter Non-Sc heduled Sy stem Inter ruption
				524		Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a n on-schedul ed system interrupti on.
				525
				526		Restart af ter Databa se Restore
				527		Since EO m anages the servers f or the Gen isis2 appl ication, s tandard sy stem recov ery proces ses curren tly in pla ce should be used fo r bringing systems b ack online after a d atabase re store oper ation.
				528
				529		Back-out P rocedures
				530		The back-o ut procedu res for th e Applicat ion Server and the W ebserver a re as foll ows:
				531		Genisis 2 Data Reque st Workflo ws:
				532		Wildfly Ap plication Server: De lete the W ildfly War file from deploymen t director y.
				533		Apache Web server: De lete all c ontents un der /var/w ww/html/.
				534
				535		Genisis 2 Terminolog y Service:
				536		Wildfly Ap plication Server: De lete the T omcat file directory from depl oyment dir ectory.
				537		Apache Web server: De lete all c ontents un der /var/w ww/html/.
				538
				539
				540		Rollback P rocedures
				541		Release 1 will be ru nning on t he PRE-PRO D environm ent with t he databas e frozen a s of the d ate of Rel ease 2. Fo llow the i nstallatio n procedur es for Rel ease 1 on the PROD e nvironment . Copy the Release 1 Database from the P RE-PROD da tabase ser ver to the PROD serv er. This w ill rollba ck Release 2 to Rele ase 1 on t he PROD pr oduction s ervers. Re start serv ices on th e PROD mac hines. Rel ease numbe rs 1 and 2 are used here to il lustrate h ow roll ba ck is done . In gener al, it wil l be Relea se N and R elease N-1 , where N is the cur rent relea se.
				542
				543
				544		Operations and Maint enance Res ponsibilit ies
				545
				546		Service
				547		Group
				548		Applicatio n Administ ration
				549		Genisis2 S ustainment Team
				550		Applicatio n Testing
				551		Genisis2 D evelopment Team; aft er the war ranty peri od, Genisi s2 Sustain ment Team
				552		Applicatio n Installa tion
				553		Genisis2 D evelopment Team; aft er the war ranty peri od, Genisi s2 Sustain ment Team
				554		Applicatio n Architec ture
				555		During dev elopment, the Genisi s2 Applica tion Archi tect; afte r the Warr anty Perio d, Genisis 2 Sustainm ent Team
				556		Asset Mana gement
				557		Genisis2 S ystem Admi nistrator (SA)
				558		Desktop/Cl ient Suppo rt
				559		Dr. Saiju Pyarajan, Business O wner Paul Hsieh, Gen isis2 Sust ainment PM
				560		Linux Syst ems Admini stration
				561		Genisis2 L inux SA Te am
				562		Windows Sy stems Admi nistration
				563		Genisis2 W indows SA Team
				564		Capacity M anagement
				565		Genisis2 L inux SA Te am Genisis 2 Windows Team
				566		Release Ma nagement
				567		Genisis2 D evelopment Team; aft er the war ranty peri od, Genisi s2 Sustain ment Team
				568		Project Ma nagement
				569		Kim Nix, G enisis2 PM
				570		Paul Hsieh , Genisis2 Sustainme nt PM
				571		Storage Ma nagement
				572		Genisis2 S A
				573		Security M anagement
				574		Genisis2 S ustainment Team
				575		Database A dministrat ion
				576		Genisis2 S A/Database Administr ation Team
				577		Network/In frastructu re Adminis tration
				578		Pittsburgh Infrastru cture/Netw orking Tea m
				579
				580		Approval S ignatures
				581
				582		REVIEW DAT E: SCRIBE:
				583
				584		Signed: Program Ma nager, Ken RikkardDa te
				585
				586		Signed: Product Ow ner, Saiju PyarajanD ate
				587
				588		Signed: Receiving Organizati on, Karen LeakeDate
				589
				590		Signed: Product Su pport, Pau l HsiehDat e
				591
				592		Template R evision Hi story
				593
				594
				595		Date
				596		Version
				597		Descriptio n
				598		Author
				599		March 2016
				600		1.6
				601		Updated to remove PM AS referen ces and to include V IP referen ces.
				602		Eliminated unnecessa ry text an d most ins tances of passive vo ice.
				603		Wichita VI P Release Process Te am
				604		June 2015
				605		1.5
				606		Updated co ver and ed ited for S ection 508 conforman ce and rem ediated wi th Common Look Offic e tool
				607		Process Ma nagement
				608		May 2015
				609		1.4
				610		Revised co ntent by P MAS Proces s Improvem ent Lockdo wn and reo rdered cov er sheet t o enhance SharePoint search re sults
				611		Process Ma nagement
				612		November 2 014
				613		1.3
				614		Updated Se ction 4 fo r url chan ge to the Operations and Maint enance Res ponsibilit y Matrix
				615		Process Ma nagement
				616		December 2 013
				617		1.2
				618		Correction to headin gs
				619		Process Ma nagement
				620		March 2013
				621		1.1
				622		Formatted to documen tation sta ndards and edited fo r Section 508 confor mance
				623		Process Ma nagement
				624		January 20 13
				625		1.0
				626		Initial Do cument
				627		PMAS Busin ess Office