/
NUTRINTG File Encrypter (DEPRACATED/OUTDATED DOCUMENTATION)

NUTRINTG File Encrypter (DEPRACATED/OUTDATED DOCUMENTATION)

Owned by Sebastian Baltruszewicz
Aug 16, 2023
4 min read

Infrastructure

All infrastructure are placed in AWS cloud.

Input queue - is the entry point to the service

ACK Timeout = 30 min - reserved time to return a response to the queue. If the response will not be sent, the message will be available to be consumed once again

Message retention period = 7 days - reserved time to pop out a message. If not happen, the message will be removed automatically without processing

Max redelivery counter = 1 - max number of retry to process the message. If the message will not be consumed successfully after the last try it will be placed into Dead-letter queue
Dead-letter queue - queue with messages processed without any result within a given timeframe (ACK Timeout) - the dead-letter-queue is common for all services

Output queue - is the exit point from the service

NoSQL DB - MongoDB / DocumentDB - used to store ProcessedDelta and EncryptedFile file-configuration

Service configuration

InputQueueName = cdp-sfmc-to-cdp-file-processor-to-file-encrypter-${environment}.fifo

OutputQueueName = cdp-sfmc-to-cdp-file-encrypter-to-file-sender-${environment}.fifo

InputBucketName = cdp-sfmc-to-cdp-${environment}

OutputBucketName = cdp-sfmc-to-cdp-${environment}

where environment can have those values:

  • dev

  • feature

  • regression

  • production

Most important process properties

Property

Description

Example

Property

Description

Example

process.fixed-delay

Polling interval in mills

10000

process.delta-interval-in-days

Interval in which we expect to receive full delta. It is required to preserve order of delta messages, and it should be set to the same value as in file-set-analyzer service.

1

process.checksumFileSuffix

Checksum file suffix

.md5

process.checksumAlgorithm

checksum algorithm which should be used for generating chesum file. Available: MD5,SHA-1, SHA-256

MD5

process.encryptedFileSuffix

Encrypted file suffix

.pgp

process.encryption-recipient-email

Encryption email recipient for which we have public key.pem

 

process.base64EncodedEncryptionKey

Encryption key in base64 format:

cat key.pem | bash64 -w 0

 

process.encryption-thread-pool-size

Number of parallel encryption threads

2

Processing messages

Flow chart

Flow chart description for processing messages

Receive a message on the input queue

The process is triggered by an input message sent to the input SQS queue.

Message format:

{​​​​​​​ "files": [ "someFolder/output/MJNPOL_PUSH/20200505/C_APP-PUSH_MJNPOL_20200506142356.dat", "someFolder/output/MJNPOL_PUSH/20200505/C_CMP-PUSH_MJNPOL_20200506142356.dat" ], "strategyCode": "MJNPOL_REGULAR", "date": "20210601" }​​​​​​​

strategyCode - it is required by downstream process listening outbound queue

date - it is required by downstream process listening outbound queue

files - list of files which should be encrypted and for which md5 checksum should be calculated.

Validate the input message

The input message must contain fields:

  • strategyCode

  • date

  • files

If any of the fields are missing or blank, the error message is sent to the error queue.

Send Error Mail

Error mail body is prepared based on template and sent via Amazon Mail Service. No message removal from the input queue is required.

Encrypt All Files

Split message for different processing threads configured by process.encryption-thread-pool-size

Encrypt File

Encrypt file and generate checksum file simultaneously. It's worth to notice that checksum file is calculated for encrypted file.

Send a message to the output queue

Once the message is sent to the output queue, the message from the input queue is removed.

Message format:

{​​​​​​​ "files": [ "someFolder/output/MJNPOL_PUSH/20200505/C_APP-PUSH_MJNPOL_20200506142356.dat.pgp", "someFolder/output/MJNPOL_PUSH/20200505/C_APP-PUSH_MJNPOL_20200506142356.dat.pgp.md5", "someFolder/output/MJNPOL_PUSH/20200505/C_CMP-PUSH_MJNPOL_20200506142356.dat.pgp", "someFolder/output/MJNPOL_PUSH/20200505/C_CMP-PUSH_MJNPOL_20200506142356.dat.pgp.md5" ], "strategyCode": "MJNPOL_REGULAR", "date": "20210601" }​​​​​​​

files - list of all files generated by file-encrypter (checksum files and encrypted files)

Spring integration channels graph and error handling

Below there is presented a channel graph of the file-encrypter:

 

Processing the message

  1. The received message is split (for each file listed in the inbound message, the new one is created)

  2. All messages are processed in Processing Channel (each brand has only one output file)

  3. All messages are aggregated in AggregatorChannel and sent as one aggregated result to the result channel where the ProcessingDelta is saved, Inbound queue message is removed, and successful email is sent.

Error handling

It is worth to notice that there are two more channels in the message channel graph:

  • General Error Channel - responsible for handling exceptions in the main flow, it sends email message with the details of the exception (ex. JSON format exception after receiving the message)

  • Aggregation Compensation Channel - handles all errors occurred in processing the one split message. The channel is responsible for sending the compensation message to the aggregation channel. It is required because all messages that were produced in the 1) Splitter have to be received all in aggregation channel to proceed the aggregation operation (ex. one upload failed). It is because split step produces messages which contains correlationId, sequenceNumber, sequenceSize headers and aggregator handler triggers only if all sequence messages are collected. The compensation channel gets the original message headers and adds it to the error message