NUTRINTG Set Analyzer [WORK IN PROGRESS]
- Sebastian Baltruszewicz
Simple explanation
Set Analyzer is responsible for coordinating whole process. It is mostly checking if rest of services can start processing specific batch of files. It needs to make sure that all files needed, to process one of batches (identified with data and strategyCode), are present on s3. If that is the case then Set Analyzer sends message to set_analyzer_to_processor queue on SQS. In this message are information's specifying this batch (data, strategy code and some additional metadata needed in further parts of process). Additionally set analyzer adds entry in blockedStrategyCodes collection to avoid problem with non - chronological upload of data.
Infrastructure
All infrastructure are placed in AWS cloud.
Bucket on s3 - set analyzer searches for files here
NoSQL DB - MongoDB / DocumentDB - used to save information about blockades, found deltas, checking metadata of files uploaded by fetcher
Output queue - SQS - queue where set analyzer sends messages to trigger processors work
Most important process properties
Property | Description | Example |
|---|---|---|
| set to decide delay between checking for new deltas | 0 0/1 * * * ? |
| Source folder on s3 bucket | fetched |
| Collection in mongo where metadata are saved by fetcher service | sfmcToCdpFileFetcher_fileMetaDataStore |