PETL and other data pipeline scripts for CES / PIH Mexico
This lives at /opt/ces-etl on ces-laguna.pih-emr.org. pull-db.sh and source-db.sh run on the root crontab.
To "deploy," do git pull in that directory.
Cron jobs are monitored in the Mexico EMR job of the PIH EMR Healthchecks service.
The PETL jobs and datasource configurations are in this directory.
The PETL installation, located in the usual directory at
/home/petl/bin/, refers to /opt/ces-etl/petl for these configurations.
The reason these are in this repository and not in
config-ces is
because at the time of this writing (Oct 2021), CES ETL is colocated
with the CES Laguna EMR (a production system), which means that deploying
config-ces causes EMR downtime. The config files in this directory
can be "deployed" independently of config-ces and the EMR.
partnersinhealth/ces-etl layers this project's datasources/, jobs/ and application-docker.yml
(as its application.yml) on the PETL base image,
partnersinhealth/petl. CI builds and pushes it (Dockerfile, build context target/docker/,
populated by mvn package) on every push to master and on every release, tagged latest and the
version. To build it locally: ./build-runtime-docker-image.sh, which layers on a locally built
partnersinhealth/petl:local. It builds on partnersinhealth/petl:latest (the Dockerfile's PETL_BASE_IMAGE default).
It runs as openmrs-contrib-distro-tools'
petl service (see its docs/services.md), e.g. for ces-ci:
PETL_IMAGE_NAME=partnersinhealth/ces-etl
PETL_FULL_REFRESH_JOBS="create-partitions.yml refresh-cesci-data.yml"
PETL_SQLSERVER_DATABASE=openmrs_ces_ci
application-docker.yml maps the cesci OpenMRS datasource and the warehouse SQL Server
datasource onto distro-tools' PETL_MYSQL_* and PETL_SQLSERVER_* variables. The other sites'
datasources aren't set; set one by its Spring environment-variable name where it's needed (e.g.
DATASOURCES_OPENMRS_CAPITAN_HOST).