Produced by Araxis Merge on 5/14/2018 1:18:38 PM Eastern Daylight Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.
| # | Location | File | Last Modified |
|---|---|---|---|
| 1 | MCCF_EDI_TAS_Infrastructure.zip\MCCF_EDI_TAS_Infrastructure\mag_sys_build\playbooks | systemd_monitor.yml | Tue Mar 27 23:51:08 2018 UTC |
| 2 | MCCF_EDI_TAS_Infrastructure.zip\MCCF_EDI_TAS_Infrastructure\mag_sys_build\playbooks | systemd_monitor.yml | Tue May 8 03:19:11 2018 UTC |
| Description | Between Files 1 and 2 |
|
|---|---|---|
| Text Blocks | Lines | |
| Unchanged | 6 | 506 |
| Changed | 5 | 10 |
| Inserted | 0 | 0 |
| Removed | 0 | 0 |
| Whitespace | |
|---|---|
| Character case | Differences in character case are significant |
| Line endings | Differences in line endings (CR and LF characters) are ignored |
| CR/LF characters | Not shown in the comparison detail |
No regular expressions were active.
| 1 | # Ansible Playbook | |
| 2 | # David Ho lton - Hal faker PII 8 Jan 2018 | |
| 3 | # | |
| 4 | # Install a systemd- based proc ess monito r. It requ ires its o wn systemd | |
| 5 | # process. Each pro cess that needs to b e monitore d requires a OnFailu re | |
| 6 | # line in its system d configur ation file . When the process b eing monit ored | |
| 7 | # fails it will trig ger the sy stemd moni toring pro cess which will call | |
| 8 | # its scri pt file to try to re start the prcoess a default nu mber of ti mes | |
| 9 | # and emai l pre-dete rmined ema il address es with th e incident and the | |
| 10 | # restart results. | |
| 11 | # | |
| 12 | # The emai l addresse s are held in a sepa rate file in /var/tm p named | |
| 13 | # <service name>.stop file The service co nfig file should cre ate this f ile | |
| 14 | # when the service s tarts. | |
| 15 | # | |
| 16 | # examples of the On Failure li ne and ser vices conf ig file fo r sinopia and jenkin s | |
| 17 | # are incl uded at th e end of t his file. | |
| 18 | # | |
| 19 | --- | |
| 20 | - name: In stall and setup Syst emd monito ring and n otificatio n system | |
| 21 | hosts: r hel7 | |
| 22 | become: yes | |
| 23 | vars: | |
| 24 | FileLi st: | |
| 25 | - /u sr/local/s bin/unit-s tatus-mail .sh | |
| 26 | - /e tc/systemd /system/un it-status- mail@.serv ice | |
| 27 | tasks: | |
| 28 | - name: Install th e restart and notifi cation scr ipt. | |
| 29 | blocki nfile: | |
| 30 | dest : /usr/loc al/sbin/un it-status- mail.sh | |
| 31 | crea te: yes | |
| 32 | owne r: root | |
| 33 | grou p: root | |
| 34 | mode : 0750 | |
| 35 | cont ent: | | |
| 36 | #! /bin/bash | |
| 37 | # | |
| 38 | # unit-sta tus-mail.s h v1.0 201 7-12-29 PII David.Holt on@halfake r.com) | |
| 39 | # | |
| 40 | # Purpose: | |
| 41 | # To restart a failed service (p roperly co nfigured), and send email to n otify abou t the fail ure. | |
| 42 | # Specifics: | |
| 43 | # Any servic e that wou ld make us e of this script nee ds the fol lowing lin e in its c onfigurati on | |
| 44 | # file norma lly found in /usr/li b/systemd/ system (or sometimes in /etc/s ystemd/sys tem). | |
| 45 | # OnFailure= unit-statu s-mail@%n. service | |
| 46 | # This line goes in th e [Unit] s ection of the file. | |
| 47 | # ...Obvious ly without the pound sign at t he beginni ng of the line. | |
| 48 | # When the s ervice fai ls it will call the unit-statu s-mail@.se rvice. | |
| 49 | # The unit-s tatus-mail @.service triggers t his script from its ExecStart line. | |
| 50 | # | |
| 51 | # A log file is writte n in /tmp, the name is unit-st atus-${MYD ATE}.log | |
| 52 | # | |
| 53 | # Parse inpu t from the commandli ne | |
| 54 | # The UNIT s hould give us the na me of the service be ing report ed to the script. | |
| 55 | UN IT=$1 | |
| 56 | # The EXTRAs may conta in informa tion that could be h elpful. | |
| 57 | EX TRA="" | |
| 58 | fo r e in "${ @:2}"; do EXTRA+="$e "$'\n'; do ne | |
| 59 | # | |
| 60 | # Git a snap shot of th e time to use in the log file name. | |
| 61 | # The format of the ti me is 2017 1231.13001 1UTC | |
| 62 | # | |
| 63 | MY DATE=$(dat e +%Y%m%d. %H%M%SUTC) | |
| 64 | # Set up the log file now that w e have the date. | |
| 65 | LO GFILE=/tmp /unit-stat us-${MYDAT E}.log | |
| 66 | to uch ${LOGF ILE} | |
| 67 | # Now get th e status o f the serv ice we wil l try to r estart. Th e status " should" be failed. | |
| 68 | UN ITSTATUS=$ (systemctl status $U NIT --full --lines=1 00) | |
| 69 | # These vari ables work with the restart fu nction. | |
| 70 | # The maximu m number o f attempts that will be made t o restart the servic e. It is a separate | |
| 71 | # value used to hold t he max val ue for rep orting in the email. | |
| 72 | MA XATTEMPTS= 5 | |
| 73 | # The burn d own counte r will be decremente d each res tart round until it reaches 0. At 0 | |
| 74 | # the script will stop trying to restart t he service . | |
| 75 | BU RNDOWNCNTR =${MAXATTE MPTS} | |
| 76 | # A counter to keep tr ack of how many time s we have tried to r estart the service. Used for | |
| 77 | # reporting in the ema il in cont rast to th e MAXATTEM PTS variab le value. | |
| 78 | ST ARTATTEMPT =0 | |
| 79 | # Capture th e current state of t he service s, just ge tting the word that follows th e line wit h | |
| 80 | # Active on it. Norma lly the va lues are a ctive or f ailed. Thi s value is used in t he restart TheService | |
| 81 | # function t o give us something to evaluat e in the l oop. | |
| 82 | IN ITIAL_STAT US=$(syste mctl statu s ${UNIT} | awk '/Ac tive/ {gsu b(/[)(]/," "); print $2 $3;}') | |
| 83 | # Recording these star ting value s to the l og file. | |
| 84 | ec ho "Burndo wn count s tarts at: ${BURNDOWN CNTR}" >> ${LOGFILE} | |
| 85 | ec ho "Start attempt co unt starts at: ${STA RTATTEMPT} " >> ${LOG FILE} | |
| 86 | ec ho "The in itial stat us is: ${I NITIAL_STA TUS}" >> $ {LOGFILE} | |
| 87 | se ndTheEmail (){ | |
| 88 | # This funct ion writes and sends the email about the failed an d restart function. | |
| 89 | # | |
| 90 | # List the r ecipients for the em ail. | |
| 91 | #M AILTO=" PII " | |
| 92 | MA ILTO=$(< / var/tmp/${ UNIT}.stop file) | |
| 93 | # Who the em ail is fro m. Not a r eal user. | |
| 94 | MA ILFROM="un it-status- mailer" | |
| 95 | # The main e mail comma nd using m ailx, and using a he re documen t for the body of th e email. | |
| 96 | # The UNIT v ariable is the name of the ser vice being reported. | |
| 97 | # The "-S se ndwait" sw itch is re quired to make the m ail system wait long enough to send the message. | |
| 98 | # during manual tes ting, the email was always sen t successf ully. In the automa ted, syste md | |
| 99 | # environ ment the e mail repor ted as bei ng sent, b ut never r eached its destinati on. This | |
| 100 | # switch fixed the problem. | |
| 101 | ma ilx -S sen dwait -s "Status ma il for uni t: ${UNIT} " -r "${MA ILFROM}" $ {MAILTO} < <EOF | |
| 102 | St atus repor t for unit : ${UNIT} | |
| 103 | ${ EXTRA} | |
| 104 | _ | |
| 105 | ER ROR OUTPUT FROM syst emctl | |
| 106 | == === | |
| 107 | ${ UNITSTATUS } | |
| 108 | _ | |
| 109 | _ | |
| 110 | SE RVICE REST ART REPORT | |
| 111 | == ===== | |
| 112 | Th ere have b een ${STAR TATTEMPT} attempts t o start th e services . Max atte mpts is ${ MAXATTEMPT S} by defa ult. | |
| 113 | _ | |
| 114 | _ | |
| 115 | Se rvice stat us is now: | |
| 116 | ${ FULLCURREN TSTATUS} | |
| 117 | _ | |
| 118 | _ | |
| 119 | EO F | |
| 120 | # Log this a ction to t he log fil e and sysl og. | |
| 121 | ec ho "Status mail sent to: ${MAI LTO} for u nit: ${UNI T}" >> ${L OGFILE} | |
| 122 | ec ho -e "Sta tus mail s ent to: ${ MAILTO} fo r unit: ${ UNIT}" | |
| 123 | } | |
| 124 | re startTheSe rvice(){ | |
| 125 | # CURRENTSTA TUS is fir st set to INTIAL_STA TUS, but m ay change value as t he functio n progress es. | |
| 126 | CU RRENTSTATU S=${INITIA L_STATUS} | |
| 127 | # | |
| 128 | # Until the CURRENTSTA TUS is act ive OR the BURNDOWNC NTR reache s 0 we kee p going th rough the loop. | |
| 129 | # | |
| 130 | # (The "lo gic" of th e until lo op and the if statem ent evalua ting the s ame condit ions seems redundant | |
| 131 | # to m e, but I w as unable to make th e script b ehave as d esired wit hout it. A nyone is w elcome to fix it.) | |
| 132 | un til [[ "${ CURRENTSTA TUS}" == " activerunn ing" ]] || [[ "${BUR NDOWNCNTR} " -eq "0" ]]; do | |
| 133 | if [ [ "${CURRE NTSTATUS}" == "activ erunning" ]] || [[ " ${BURNDOWN CNTR}" -eq "0" ]]; t hen | |
| 134 | # | |
| 135 | # If the CURRE NTSTATUS i s active O R the BURN DOWNCNTR r eached 0 g et out of here. | |
| 136 | CURR ENTSTATUS= $(systemct l status $ {UNIT} | a wk '/Activ e/ {gsub(/ [)(]/,""); print $2 $3; }') | |
| 137 | FULL CURRENTSTA TUS=$(syst emctl stat us ${UNIT} --full -- lines=100) | |
| 138 | echo "" >> ${L OGFILE} | |
| 139 | echo "Current status is: ${CURRENT STATUS}" > > ${LOGFIL E} | |
| 140 | echo "" >> ${L OGFILE} | |
| 141 | echo "Full sta tus is: ${ FULLCURREN TSTATUS}" >> ${LOGFI LE} | |
| 142 | brea k 2 | |
| 143 | else | |
| 144 | # | |
| 145 | # In crement th e STARTATT EMPT count er which i s reported in the em ail as par t of the r eport. | |
| 146 | STAR TATTEMPT=$ ((STARTATT EMPT + 1)) | |
| 147 | # Lo g the valu e of the S TARTATTEPM T varuable | |
| 148 | echo "Start at tempt coun t is: ${ST ARTATTEMPT }" >> ${LO GFILE} | |
| 149 | # | |
| 150 | # Th is next co mmand rest arts the s ervice spe cified by the variab le $UNIT. | |
| 151 | syst emctl star t ${UNIT} | |
| 152 | # De crement th e BURNDOWN CNTR by 1 | |
| 153 | BURN DOWNCNTR=$ ((BURNDOWN CNTR - 1)) | |
| 154 | # Wa iting 2 se conds whil e the serv ice starts up. | |
| 155 | slee p 2 | |
| 156 | CURR ENTSTATUS= $(systemct l status $ {UNIT} | a wk '/Activ e/ {gsub(/ [)(]/,""); print $2 $3; }') | |
| 157 | # gr ab the ful l status t o be sent in the ema il. | |
| 158 | FULL CURRENTSTA TUS=$(syst emctl stat us ${UNIT} --full -- lines=100) | |
| 159 | # wr ite the cu rrent stat us to the log file. | |
| 160 | echo "" >> ${L OGFILE} | |
| 161 | echo "Current status is: ${CURRENT STATUS}" > > ${LOGFIL E} | |
| 162 | echo "" >> ${L OGFILE} | |
| 163 | echo "Full sta tus is: ${ FULLCURREN TSTATUS}" >> ${LOGFI LE} | |
| 164 | fi | |
| 165 | do ne | |
| 166 | } | |
| 167 | # Send the f ull status to the lo g file at the BEGINN ING of the script. | |
| 168 | sy stemctl st atus ${UNI T} >> ${LO GFILE} | |
| 169 | # Main. The main porti on of the script to call the f unctions a bove. | |
| 170 | re startTheSe rvice | |
| 171 | se ndTheEmail | |
| 172 | # Send the f ull status to the lo g file at the END of the scrip t. | |
| 173 | sy stemctl st atus ${UNI T} >> ${LO GFILE} | |
| 174 | ||
| 175 | - name: Install th e SystemD monitoring service f ile. | |
| 176 | blocki nfile: | |
| 177 | dest : /etc/sys temd/syste m/unit-sta tus-mail@. service | |
| 178 | crea te: yes | |
| 179 | owne r: root | |
| 180 | grou p: root | |
| 181 | mode : 0644 | |
| 182 | cont ent: | | |
| 183 | [U nit] | |
| 184 | De scription= Unit Statu s Mailer S ervice | |
| 185 | Af ter=networ k.target | |
| 186 | Ki llMode=pro cess | |
| 187 | [S ervice] | |
| 188 | Ty pe=Simple | |
| 189 | Ex ecStart=/u sr/local/s bin/unit-s tatus-mail .sh %I "Ho stname: %H " "Machine ID: %m" " Boot ID: % b" | |
| 190 | ||
| 191 | - name: Remove Ans ible block markers f rom the sc ript and s ystemd fil es. | |
| 192 | linein file: | |
| 193 | rege xp: 'ANSIB LE MANAGED BLOCK' | |
| 194 | stat e: absent | |
| 195 | dest : "{{ item }}" | |
| 196 | with_i tems: "{{ FileList } }" | |
| 197 | ||
| 198 | - name: Reload Sys temD | |
| 199 | become : true | |
| 200 | comman d: systemc tl daemon- reload | |
| 201 | ||
| 202 | # Example service co nfig files for two s ervices. | |
| 203 | # | |
| 204 | # Sinopia is a nativ e systemd service. | |
| 205 | # [Unit] | |
| 206 | # Descript ion=Sinopi a NPM cach e and priv ate repo s erver | |
| 207 | # After=ne twork.targ et | |
| 208 | # OnFailur e=unit-sta tus-mail@% n.service | |
| 209 | # [Service ] | |
| 210 | # Type=sim ple | |
| 211 | # User=sin opia | |
| 212 | # Group=si nopia | |
| 213 | # ExecStar tPre=/usr/ bin/bash - c '/bin/ec ho PII ,root@loca lhost > /v ar/tmp/%n. stopfile' | |
| 214 | # ExecStar t="/usr/bi n/sinopia" -c "/etc/ sinopia/co nfig.yaml" | |
| 215 | # Restart= on-abort | |
| 216 | # WorkingD irectory=/ var/www/ht ml/pub/rep o/npm/sino pia | |
| 217 | # [Install ] | |
| 218 | # WantedBy =multi-use r.target | |
| 219 | # | |
| 220 | # Jenkins is still a systemV s erver, but systemD a uto-genera tes a syst emD servic e config f ile if | |
| 221 | # one does not alrea dy exist. | |
| 222 | # Using th e example below the values can be used t o make it work with the system d monitori ng service . | |
| 223 | # | |
| 224 | #[Unit] | |
| 225 | #Documenta tion=man:s ystemd-sys v-generato r(8) | |
| 226 | #SourcePat h=/etc/rc. d/init.d/j enkins | |
| 227 | #Descripti on=LSB Jen kins Autom ation Serv er | |
| 228 | #Before=ru nlevel3.ta rget | |
| 229 | #Before=ru nlevel5.ta rget | |
| 230 | #Before=sh utdown.tar get | |
| 231 | #Before=en case.servi ce | |
| 232 | #Before=be sclient.se rvice | |
| 233 | #After=rem ote-fs.tar get | |
| 234 | #After=net work-onlin e.target | |
| 235 | #After=tim e-sync.tar get | |
| 236 | #After=nss -lookup.ta rget | |
| 237 | #After=tim e-sync.tar get | |
| 238 | #After=sen dmail.serv ice | |
| 239 | #After=net work-onlin e.target | |
| 240 | #Wants=net work-onlin e.target | |
| 241 | #Conflicts =shutdown. target | |
| 242 | #StartLimi tIntervalS ec=0 | |
| 243 | #OnFailure =unit-stat us-mail@%n .service | |
| 244 | #[Service] | |
| 245 | #Type=fork ing | |
| 246 | #Restart=n o | |
| 247 | #TimeoutSe c=5min | |
| 248 | #IgnoreSIG PIPE=no | |
| 249 | #KillMode= process | |
| 250 | #GuessMain PID=no | |
| 251 | #RemainAft erExit=yes | |
| 252 | #ExecStart Pre=/usr/b in/bash -c '/bin/ech o PII ,root@loca lhost > /v ar/tmp/%n. stopfile' | |
| 253 | #ExecStart =/etc/rc.d /init.d/je nkins star t | |
| 254 | #ExecStop= /etc/rc.d/ init.d/jen kins stop | |
| 255 | #ExecReloa d=/etc/rc. d/init.d/j enkins rel oad | |
| 256 | #[Install] | |
| 257 | #WantedBy= multi-user .target | |
| 258 | # |
Araxis Merge (but not the data content of this report) is Copyright © 1993-2016 Araxis Ltd (www.araxis.com). All rights reserved.