Produced by Araxis Merge on 12/14/2017 1:28:07 PM Eastern Standard Time. See www.araxis.com for information about Merge. This report uses XHTML and CSS2, and is best viewed with a modern standards-compliant browser. For optimum results when printing this report, use landscape orientation and enable printing of background images and colours in your browser.
| # | Location | File | Last Modified |
|---|---|---|---|
| 1 | CUI-CPP-v2.3.2-source.zip\doc | trouble_shooting_environment_variables.md | Mon Nov 20 23:19:07 2017 UTC |
| 2 | CUI-CPP-v2.3.2-source.zip\doc | trouble_shooting_environment_variables.md | Thu Dec 14 15:25:24 2017 UTC |
| Description | Between Files 1 and 2 |
|
|---|---|---|
| Text Blocks | Lines | |
| Unchanged | 5 | 316 |
| Changed | 4 | 8 |
| Inserted | 0 | 0 |
| Removed | 0 | 0 |
| Whitespace | |
|---|---|
| Character case | Differences in character case are significant |
| Line endings | Differences in line endings (CR and LF characters) are ignored |
| CR/LF characters | Not shown in the comparison detail |
No regular expressions were active.
| 1 | # Trouble Shooting E nvironment Variables | |
| 2 | ||
| 3 | The doc/sy stem_envir onment_var iables.md file attem pts to doc ument all of the env ironment v ariables u sed by thi s web appl ication an d its many pieces. | |
| 4 | ||
| 5 | This docum ent's goal is to hel p the soft ware devel oper or De vOps engin eer focus on resolvi ng a probl em which m ay be a re sult of en vironment variable v alues not being as e xpected. | |
| 6 | ||
| 7 | ## Context | |
| 8 | ||
| 9 | We have at least thr ee major c ontexts in which we develop an d operate this syste m. Those contexts c an generic ally be cl assified a s AWS/EC2, docker co ntainers/i mages and native hos t machines . The goa l of this project is to have a system th at operate s in all c ontexts th e same way . The way we have c hosen to c onfigure o ur system is through the use o f system e nvironment variables . | |
| 10 | ||
| 11 | The system environme nt variabl es can cha nge from c ontext to context in order to accomplish the same behavior o f the syst em. Somet imes the v alues of t he variabl es for a g iven conte xt will ca use a prob lem which may not be easy to t rack down. | |
| 12 | ||
| 13 | The values of system environme nt variabl es are set in three different ways: 1) ` .env*` fil es - some configurat ion manage d and some not; 2) ` config/doc ker/*_serv ice.yml` f iles which are all c onfigurati on managed ; and, 3) manually s et within a terminal /console s ession. | |
| 14 | ||
| 15 | The follow sections discuss th e common p roblems th at may occ ur with th e value of variables being set via these methods. | |
| 16 | ||
| 17 | ### `.env* ` Files. | |
| 18 | ||
| 19 | As explain ed in `doc /system_en vironment_ variables. md` the `. env*` file s are proc essed in t wo differe nt ways de pending up on whether they are being used within a `bash` she ll script or within the Rails applicatio n of `rake ` task. | |
| 20 | ||
| 21 | The `.env* .local` fi les are no t configur ation mana ged. They are inten ded as loc al over-ri ders for u se by the software d eveloper o n their in dividual w orkstation s. Its po ssible tha t the DevO ps enginee r may also file thes e files us eful in pr oviding so me host-sp ecific con figuration . This us e case is not recomm ended. Th e better p ractice fo r host-spe cific conf iguration is to make use of th e `DEPLOY_ TYPE` syst em environ ment varia ble to dir ect the lo ading of a host-spec ific `.env .#{DEPLOY_ TYPE}` fil e. | |
| 22 | ||
| 23 | It is poss ible that most probl ems caused by enviro nment vari ables bein g wrong ar e a result of a sett ing in a ` .env*.loca l` file. | |
| 24 | ||
| 25 | The next m ost likely cause of problems i s an envir onment var ialbe bein g set in t he wrong f ile. Ther e are thre e basic la yers of `. env*` file s. The "t op" later is the bas e (aka. de fault) lay er impleme nted with the `.env` file. In this file should on ly go valu e assignme nts that a re common to all sub sequent la yers - env ironment a nd deploym ent type. | |
| 26 | ||
| 27 | Setting `A PP_URL` to a host-sp ecific val ue is an i nvalid use of the `. env` file. | |
| 28 | ||
| 29 | The next l ayer is th e environm ent layer directed b y the `RAI LS_ENV` va lue. The values in this layer adds new or makes c hanges to environmen t variable s specific to a beha vioural en vironment. So far t he only en vironments that have been defi ned are th e common R ails envir onments of developme nt, test a nd product ion. | |
| 30 | ||
| 31 | The final layer is t he deploym ent (host- specific) layer dire cted by th e `DEPLOY_ TYPE` envi ronment va riable val ue. This layer adds new or ch anges the value of e nvironment variables that have have come before in the other two layer s. This i s the prop er layer f or setting the value s of host- specific v ariables. | |
| 32 | ||
| 33 | The follow ing sectio ns discuss how these different layers ar e processe d by the d ifferent c ontexts of the syste m. | |
| 34 | ||
| 35 | #### Rails and `rake ` Tasks | |
| 36 | ||
| 37 | The good n ews is tha t `rake` t asks and t he Rails a pplication process t he `.env*` the same way - the first valu e set is t he officia l value - as describ ed by the `dotenv` g em documen tation. I n fact `ra ke` tasks use `confi g/applicat ion.rb` to process t he `.env*` the same way as the Rails app lication. | |
| 38 | ||
| 39 | Refer to t he diagram in `doc/e nv_file_re lationship .dot.pdf` where you will notic e that the Rails app lication s tarts at t he bottom and goes u p in terms of the or der of whi ch `.env*` is proces sed. | |
| 40 | ||
| 41 | A common p roblem occ urs when t here is an environme nt variabl e that has a depende ncy on som e other en vironment variable. Consider this commo n configur ation: | |
| 42 | ||
| 43 | ``` | |
| 44 | # File 1 | |
| 45 | export PRO TOCOL="htt p://" | |
| 46 | export HOS T="localho st" | |
| 47 | export POR T=" PORT " | |
| 48 | export URL =$PROTOCOL $HOST:$POR T | |
| 49 | ||
| 50 | # File 2 | |
| 51 | export POR T=" PORT " | |
| 52 | ``` | |
| 53 | ||
| 54 | When these two files are proce ssed top-t o-bottom ( like is do ne in `bas h` shell s cripts) th e value of `url` wit h be `http ://localho st: PORT ` which is different when the files are processed bottom-to- top as is done by th e Rails ap plication and `rake` tasks. That proce ssing orde r will gen erate a va lue for `U RL` as `ht tp://local host: PORT ` because the `doten v` behavio r is that the *first * value se t wins. | |
| 55 | ||
| 56 | In the bot tom-to-top order `PO RT` was in itially se t o 4321. The subseq uent assig nment of ` PORT` to PORT is ignore d by `dote nv` proces sing. | |
| 57 | ||
| 58 | ||
| 59 | ### Manual ly Set Wit hin a Term inal/Conso le Session | |
| 60 | ||
| 61 | As a devel oper it is common pr actice to have multi ple termin al/console windows o pen on you r workstat ion. You may be wor king in th e same fea ture branc h in each window. | |
| 62 | ||
| 63 | There are three syst em environ ment varia bles that must be ma nually set within a terminal/c onsole win dow: `RAIL S_ENV`, `R ACK_ENV` a nd `DEPLOY _ENV`. Th e value of these thr ee environ ment varia bles direc t which `. env*` file s are proc essed. If their val ues are no t the same in each t erminal/co nsole wind ow you may get diffe rent resul ts when ru nning the applicatio n. The ap plication may work i n one wind ow and not in anothe r. | |
| 64 | ||
| 65 | That is a condition that can d rive any d eveloper c razy. | |
| 66 | ||
| 67 | ||
| 68 | ### `confi g/docker/* _service.y ml` Files | |
| 69 | ||
| 70 | The final context is docker. We use the `docker-c ompose` ut ility to b uild docke r containe rs from do cker image s. We get three off icially ma intained i mages from the docke r hub: Rub y version 2.3.3, Pos tgreSQL ve rsion 9.3 and Redis version 3. 2.8. From these ima ges `docke r-compose` builds th e containe rs which w e use. | |
| 71 | ||
| 72 | The script `bin/dock er_cui_cre ate.sh` is responsib le for set ting the e nvironment within wh ich the `d ocker-comp ose` utili ty is exec uted. The script ma kes use of `bin/setu p_environm ent` to en sure that the `.env* ` files ar e processe d (sourced ) in the p roper orde r. This i s the top- to-bottom order that is shown in the `do c/env_file _relations hip.dot.pd f` diagram . | |
| 73 | ||
| 74 | We use ver sion 3 of the `docke r-compose. yml` file format. T he version allows fo r the use of environ ment varia bles withi n the `doc ker-compos e.yml` fil e. | |
| 75 | ||
| 76 | Docker use s the term `service` to refere nce "thing s" which m ay or may not really be consid er "contai ners." Th e docker v ocabulary is confusi ng. Let u se agree t hat "servi ce" and "c ontainer" mean the s ame thing. If we ar e wrong so meone more educated will enlig hten us. | |
| 77 | ||
| 78 | There are three serv ices defin ed for thi s system: db, redis and web. Each serv er/contain er can be executed o n a develo per's work station or through s ome magic means yet unexplaine d be execu ted on an AWS/EC2 in stance. | |
| 79 | ||
| 80 | Each servi ce has an associated `config/d ocker/#{se rvice}_doc ker-compos e.yml` fil e which us es an `m4` text macr o `include ()` to imp ort a corr esponding `config/do cker/#{ser vice}_serv ice.yml` f ile. | |
| 81 | ||
| 82 | > `m4` is a common * NIX comman d line uti lity. If your works tation doe s not have this util ity as par t of its n ormal dist ribution y ou can get it throug h you prim ary packag e manager. For exam ple on the Macintosh OSX platf orm you ca n use `bre w install m4` to get the utili ty. The ` awk` commo n *NIX com mand line utility is also used . If your workstati on does no t have it you can ge t it the s ame way th at you use d to get t he `m4` ut ility. | |
| 83 | ||
| 84 | There are two places within th e `*_servi ce.yml` fi le in whic h environm ent variab les are us ed. | |
| 85 | ||
| 86 | #### ports Element | |
| 87 | ||
| 88 | The `ports ` element of the `*_ service.ym l` file is used to d efine the mapping of a port in ternal to the contai ner (servi ce) to the host mach ine - a de veloper's workstatio n or a dep loyed AWS/ EC2 instan ce. The f ollowing t able shows the envir onment nam es used by each serv ice to imp lement thi s mapping. | |
| 89 | ||
| 90 | | Servuce | Variable Name for Host Port | Variable Name for Container Port | | |
| 91 | | ------- | -------- ---------- --------- | -------- ---------- ---------- ---- | | |
| 92 | | db | DOCKER_H OST_DB_POR T | DBPORT | | |
| 93 | | redis | DOCKER_H OST_REDIS_ PORT | REDIS_PO RT | | |
| 94 | | web | DOCKER_H OST_WEB_PO RT | PUMA_POR T | | |
| 95 | ||
| 96 | As a devel oper if yo u have a P ostgreSQL or Redis s erver runn ing native ly on your workstati on AND you attempt t o start up a similar docker co ntainer, y ou may hav e port con flicts. T he service may refus e to start in its do cker conta iner becau se the por t is alrea dy in use. In this case you h ave two ch oices: 1) terminate your nativ e applicat ions; or, 2) change the port m apping usi ng an appr opriate `. env*.local ` file. | |
| 97 | ||
| 98 | If you cho se to chan ge the por t mapping, the commo n choice f or the `DO CKER_HOST_ *_PORT` is 10000 + t he value o f `*PORT` like so: | |
| 99 | ||
| 100 | ``` | |
| 101 | # .env.doc ker.local file | |
| 102 | export DOC KER_HOST_D B_PORT="1$ DBPORT" | |
| 103 | ``` | |
| 104 | ||
| 105 | > In order to get th e `.env.do cker.local ` file to be process ed the val ue of `DEP LOY_TYPE` must be `d ocker`. | |
| 106 | ||
| 107 | PostgreSQL has some special en vironment variables that are u sed within the `db_s ervice.yml ` file and by the pr ogram itse lf. These are expla ined in mo re detail in the fol lowing sec tions. | |
| 108 | ||
| 109 | ||
| 110 | #### envir onment Ele ment | |
| 111 | ||
| 112 | The format of the `d ocker-comp ose.yml` f iles allow s for vari ous ways i n which th e values o f environm ent variab les can be fed into the contai ner. The way in whi ch our sys tem uses i s the `env ironment` element of the YAML file forma t. | |
| 113 | ||
| 114 | The format of the `e nvironment ` YAML ele ment is di scussed at [https:// docs.docke r.com/comp ose/compos e-file/#en vironment] (https://d ocs.docker .com/compo se/compose -file/#env ironment) | |
| 115 | ||
| 116 | We use the array syn tax so tha t we can p ass existi ng environ ment varia bles as we ll as assi gne values to other environmen t variable s. | |
| 117 | ||
| 118 | The [docum entation f or the Pos tgreSQL v9 .3 docker image](htt ps://hub.d ocker.com/ _/postgres /) explain s how the variables `POSTGRES_ USER`, `PO STGRES_PAS SWORD` and `POSTGRES _DB` are u sed during initializ ation of t he contain er to crea te the rol e `POSTGRE S_USER` an d to creat e the data base `POST GRES_DB` a utomatical ly. | |
| 119 | ||
| 120 | These thre e special environmen t are not used anywh ere else i n our syst em so we h ave use of the abili ty of `doc ker-compos e` to pass in variab les with a ssignments using exi sting vari ables with in our exi stin `.env *` files. | |
| 121 | ||
| 122 | ``` | |
| 123 | - POSTGRES _USER=$DBU SER | |
| 124 | - POSTGRES _PASSWORD= $DBPASS | |
| 125 | - POSTGRES _DB=${DBNA ME}_${RAIL S_ENV} | |
| 126 | ``` | |
| 127 | ||
| 128 | Notice how the `POST GRES_DB` v alue is co mposed of both the ` DBNAME` an d `RAILS_E NV` the sa me way tha t it is wi thin the ` config/dat abase.yml` file. | |
| 129 | ||
| 130 | PostgreSQL (and othe r programs ) gave spe cial envir onment var iables whi ch are use d in vario us stages of that in itializati on and sta rtup. The se environ ment varia bles can b e thought os as bein g invisibl e because you don't necessary know that they exist unless yo u have a g ood uderst anding of the progra m/utility that they support. | |
| 131 | ||
| 132 | The next s ection dis cusses the environme nt variabl es used by PostgreSQ L. | |
| 133 | ||
| 134 | ||
| 135 | ## Invisib le Variabl es | |
| 136 | ||
| 137 | Most *NIX programs/u tilities m ake use of command l ine parame ters, conf iguration files and/ or environ ment varia bles for t heir confi guration. Some use all three. In the c ase of Pos tgreSQL th ere are se veral envi ronment va riables th at define its config uration wh ich may re sult in pr oblems wit hin our ap plication. These en vironment varibles f or Postgre SQL are do cumented a t [https:/ /www.postg resql.org/ docs/9.3/s tatic/libp q-envars.h tml](https ://www.pos tgresql.or g/docs/9.3 /static/li bpq-envars .html). | |
| 138 | ||
| 139 | As a devel oper it is likely th at you are running P ostgreSQL natively a nd may hav e some of these envi ronment va riables se t in one o f your `~/ .bashrc` f iles. Som e of the m ore common ones are listed in the table below. No tice how t he variabl e names ar e "namespa ced" using `PG` as a prefix. The "PG" o f course r efers to P ostgreSQL. | |
| 140 | ||
| 141 | | Variable Name | Ho w Used by PostgreSQL | | |
| 142 | | -------- ----- | -- ---------- ---------- | | |
| 143 | | PGHOST | behaves t he same as the host connection parameter . | | |
| 144 | | PGHOSTAD DR | behav es the sam e as the h ostaddr co nnection p arameter. This can b e set inst ead of or in additio n to PGHOS T to avoid DNS looku p overhead . | | |
| 145 | | PGPORT | behaves t he same as the port connection parameter .| | |
| 146 | | PGDATABA SE | behav es the sam e as the d bname conn ection par ameter. | | |
| 147 | | PGUSER | behaves t he same as the user connection parameter . | | |
| 148 | | PGPASSWO RD | behav es the sam e as the p assword co nnection p arameter. Use of thi s environm ent variab le is not recommende d for secu rity reaso ns, as som e operatin g systems allow non- root users to see pr ocess envi ronment va riables vi a ps; inst ead consid er using t he ~/.pgpa ss file (s ee Section 31.15). | | |
| 149 | | PGPASSFI LE | speci fies the n ame of the password file to us e for look ups. If no t set, it defaults t o ~/.pgpas s (see Sec tion 31.15 ). | | |
| 150 | ||
| 151 | If you hav e any of t hese `PG*` environme nt variabl es defined in your t erminal/co nsole wind ow you may have prob lems. | |
| 152 | ||
| 153 | Our system has chose n to use " DB" as a p refix for environmen t variable s associat ed with th e database service. This is c onsistent with the R ails/Activ eRecord gn osticism w ith regard to the da tabase man agement sy stem. It is possibl e to creat e a Rails- based appl ication th at can mak e use of a ny databas e manageme nt system. We chose PostgreSQ L as our i nitial com ponent. W e may chan ge our sle ction in t he future to some ot her databa se managem ent system . | |
| 154 | ||
| 155 | The `.env* ` files on ly define `DB*` envi ronment va riables. Having `PG PORT` defi ned in the terminal/ console wi ndow and h aving `DBP ORT` defin ed in the `.env` fil e will res ult in con fusion. T o mitigate this conf usion we m ake use of the abili ty of the `docker-co mpose` uti lity to se t the valu e of some environmen t variable s. | |
| 156 | ||
| 157 | In the `co nfig/docke r/db_servi ce.yml` fi le within the `envir onment` YA ML element we have t his config uration de fined: | |
| 158 | ||
| 159 | ``` | |
| 160 | - PGPORT=$ DBPORT | |
| 161 | ``` | |
| 162 |
Araxis Merge (but not the data content of this report) is Copyright © 1993-2016 Araxis Ltd (www.araxis.com). All rights reserved.