/
NUTRINTG Shopify to CDP Integration

NUTRINTG Shopify to CDP Integration

Preconditions:

Brand-Org-Code is unique globally (particular Brand-Org-Code will not be present within two different Markets / Countries).

Order data

 Click here to expand...

Shopify → Middleware → CDP

Epsilon CDP spec: RB_E-Commerce_Functional_Specification V_1.8.docx

Steps

SERVICE  EP-275 - Getting issue details... STATUS

  1. Receive Shopify webhook type: orders/create - HTTP request with JSON payload.
    RB endpoint should have valid SSL certificate, since Shopify verifies SSL certificates when delivering payloads to HTTPS webhook addresses.
  2. Acknowledge receiving data by sending 200 OK responds within 5 seconds. If there is no response, or an error is returned, then Shopify retries the connection 19 times over the next 48 hours. A webhook is deleted if there are 19 consecutive failures. Any response outside of the 200 range, including 3XX HTTP redirection codes, indicates that you did not receive the webhook.

  3. Validate json payload:
    1. whether X-Shopify-Topic header field contains "orders/create" or "order/updated"value
    2. whether X-Shopify-Shop-Domain header field is in the integrated stores list on MongoDB (source: Shopify URL Mapping.xlsx)
    3. whether X-Shopify-Hmac-SHA256 header field against correct HMAC (list was shared with Tristan)  EP-230 - Shopify Error handling and webhook verification ON HOLD
    4. whether JSON schema is correct. 

  4. IF any of the validations returned false → store json to the Error queue and return HTTP code 400.
    ELSE → continue flow.
  5. Change customer-id field to: BrandOrgCode (from Shopify URL Mapping.xlsx) & "SHPF" & customer_id. Example: FRADURSHPF207119551
  6. Change value of "shipping_address-country" and "billing_address-country_code" with value from "Market code" column from in the Shopify URL Mapping.xlsx (URL Mapping tab).
  7. Change "created_at" field:
    name: to "order_date"
    value: adjust to format YYYY-MM-DDTHH:MM:SS and UTC timezone

  8. Add to json fields from "Additional fields table" below
  9. List of required fields and order (different from original json) is provided in the Shopify URL Mapping.xlsx
  10. Sends the record to the Middleware Router queue. Router decide if the message should go to cdp-shopify-order-adapter or cds-middleware-message-adapter
  11. IF any of the SERVICE steps failed return HTTP code 500 and send email - titled "Shopify Order Service Flow Failure" with order_id and exception description in the content.
    EP-348 - Getting issue details... STATUS
    EP-351 - Getting issue details... STATUS
    EP-350 - Getting issue details... STATUS

    cdp-shopify-order-adapter 


  1. Read messages from the Shopify Order queue
  2. Save record as  CSV(.txt) file at EC2 catalogue (one order / json = one file):
    <yyy_mm_dd_hh> catalogue in UTC timezone
    → "<X-Shopify-Shop-Domain>" catalogue (field in the json header, example: "https://durexfrance.myshopify.com")
  3. Create file for each "X-Shopify-Shop-Domain" containing records from last 24h, by merging appropriate files with bash script. The process executes once per day at a specific hour (configurable) and contains all records from that hour the day before up to the time of execution (e.g.: if that hour is specified at 3 p.m. and today is 12 March, then all records from 11 March 3:00:00 p.m. inclusive to 12 March 3:00:00 p.m. exclusive).
  4. Name files according to File naming table (v1).
  5. Create an empty file with headers for each store without orders.
  6. Add headers according to Shopify URL Mapping.xlsx
  7. Convert to CSV (.txt)
  8. PGP encrypt files. Production key: frms_prod_key_pub.asc . Tests key:  public_key-narnia.epsilon.com.asc

  9. Create MD5

  10. Transfer files to incoming folder of the sFTP locations for the respective country (market code) before 19:30 UTC every day (the uploading should be finished by this time.)
    The folder names will be the same as the 3-character ISO country codes ("Market code" column in the Shopify URL Mapping.xlsx) for the respective countries.
    Path: Incoming/Shopify/<3 letter ISO Country_Name>. sFTP credentials are provided in the table below.

  11. Move transferred (encrypted) files to "Archive" folder for backup purposes. For this folder data retentions is set for 21 days.  EP-296 - Getting issue details... STATUS
  12. IF any of the ADAPTER steps failed email is sent - titled "Shopify Order Adapter Flow Failure" with folder name and exception description in the content.

cds-middleware-message-adapter

  1. Read message from CDS Adapter Queue
  2. Check configuration for AccountSource code.
  3. If the confgiuration does not exists then drops the message (IMPORTANT! in given situation the client does not know the request has been dropped since it is an async communication)
  4. Otherwise CDS Adapter retrieve configuration from the database (bucketId, url, brand, market) for given ASC
  5. Sends request to CDS. 
  6. In case of any failure: the same as indicates the point no 5.


# in the final fileField nameSource (column in the Shopify URL Mapping.xlsx)
45brand_org_codeBrand-Org-Code
46language_codeLanguage code
47vendorvendor
48txn_src_codetxn_src_code
49acct_src_code_custAccount_Source_Code Customer data
50acct_src_code_shipAccount_Source_Code Shipping data


Behavior when sent header X-Shopify-Topic="orders/updated"

In case of request is sent to CDS - the new version of record is created. What is important the email address must not change, otherwise, two separate records will be created. The new version of the records cuases, that when one consumes search API to CDS, the latest version of row is returned. In case if there is an integration with SFMC, it depends on the DE settings, proper primary key must be set. For this topic please reach out to SF Team.

In case of request is sent to CDP - the most important, orderId must be the same, which is pretty obvious. Middleware sends orders to CDP in batches every 24 hrs. If the order is created and then updated in the same period of 24 hrs, then only the updated order is sent to CDP. Otherwise, one day the created order data is sent to CDP, and another day its updated version is sent to CDP. Then it is CDP reposnbility to handle the situation properly. For this scenario, the requirements are given below (Epsilon statement regards order updates)

Epsilon statement regards order updates
In the Order, the number of line items should not be increased or decreased. In fact, there should be no change in the line items, otherwise, data corruption might happen.
 
The FS document has the following assumptions.
AS009
We will not be receiving any updates for orders that change the number of lines since we receive only the final order.
AS011
Epsilon will accept all the financial status values and update the transaction if the financial status change. Currently No logic exists to delete items or reorder item numbers if the content of the items changes between versions. RB will not make changes to the number of items in the order.


File naming table

X-Shopify-Shop-DomainFilenameFilename v2Filename v3
https://durexfrance.myshopify.comRB_FR_ORDER_RBFRADUR_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://durexgermany.myshopify.comRB_DE_ORDER_RBDEUDUR_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://durexspain.myshopify.comRB_ES_ORDER_RBESPDUR_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://durexuk.myshopify.comRB_EN_ORDER_RBGBRDUR_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://enfashop-es.myshopify.comRB_ES_ORDER_MJNESP_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://enfashop-pl.myshopify.comRB_PL_ORDER_MJNPOL_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://schollfrance.myshopify.comRB_FR_ORDER_RBFRASCO_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://schollgermany.myshopify.comRB_DE_ORDER_RBDEUSCO_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://scholluk.myshopify.comRB_EN_ORDER_RBGBRSCO_SHOPIFY_<timestamp>.json->.txt->.txt.pgp
https://durex-uk-test-store.myshopify.comRB_EN_ORDER_RBGBRDUR_SHOPIFY_<timestamp>.json->.txt->txt.pgp

<timestamp>  = File creatation datetime. Only digits in UTC timezone, example: "20191207040627".
Examplary filename: RB_EN_ORDER_RBGBRDURX_SHOPIFY_20191207040627.txt.pgp

File specification

FormatCSV (.txt)
Delimitercomma, Additionally all values should be enclosed by double quotes.
TerminatorCRLF
Headersrequired (names according to Shopify URL Mapping.xlsx) The case of the headers does not matter. It can be sent as “eventtype” or “EVENTTYPE”.


sFTP credentials

HOST

sftp.skynet.epsilon.com

Port

22

UAT Username

rbnk_q_ecom

PROD Username

rbnk_p_ecom

EP-333 - Getting issue details... STATUS

Additional documentation

Shopify API: https://help.shopify.com/en/api/guides
Shopify webhooks API: https://help.shopify.com/en/api/reference/events/webhook
Mule ObjectStore: https://docs.mulesoft.com/object-store/



Customer data

 Click here to expand...

Shopify → Middleware → CDP

Steps


SERVICE  EP-280 - Getting issue details... STATUS

  1. REST RB endpoint receives Shopify webhook (types: customers/create, customers/update)
  2. Validate json payload:
    1. whether X-Shopify-Topic header field contains " customers/create" or "customers/update" values
    2. whether X-Shopify-Shop-Domain header field is in the integrated stores list on MongoDB (source: Shopify URL Mapping.xlsx)
    3. whether X-Shopify-Hmac-SHA256 header field against correct HMAC (list was shared with Tristan)  EP-230 - Shopify Error handling and webhook verification ON HOLD
    4. whether JSON schema is correct. shopify-customer-schema.json

  3. IF payload didn't pass validation move it to Error queue (secured by another queue - Dead letter queue) and return HTTP code 400.
    ELSE → continue flow.
  4. Add LDS agreements data to the json body according to Body table below.
  5. Store data as json in the queue (Anypoint MQ)
  6. IF processing payload failed return HTTP code 500.

    ADAPTER  EP-273 - Getting issue details... STATUS
  7. Read queue
  8. Create json header and body according to below table including
  9. IF webhook type = customers/update 
  10. Send json to Epsilon's API secured by oauth2:
    1. Add Profile API Post call -  page 47 of MJN_Epsilon Integration API Document v1.36.pdf IF webhook type = customers/create
    2. Update Profile by ID API Post call -  page 93 of MJN_Epsilon Integration API Document v1.36.pdf IF webhook type = customers/update. Adding ProfileId as URL parameter - api/v1/profiles/{profileId}.
      Get ProfileId by "Get Profiles (Search)" CDP API method - page 162, providing SourceAccountNumber according to Body table below.
  11. CDP responds with Http 201 code in the header and sent data in the body

SERVICE to ADAPTER data:

FIELDTYPEVALUE
operationstring"CREATE" - if webhook type = customers/create
"UPDATE" - if webhook type = customers/update

acceptLanguage
string"en-US"
programCode
string→ "Brand-Org-Code" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx
brandOrgCode
string→ "Brand-Org-Code" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx
accountSource
string→ "Account_Source_Code Customer data" column in the "URL Mapping" tab) Shopify URL Mapping.xlsx
sourceAccountNumber
string→ "sourceAccountNumber" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx
profileData
object

→ "Customer" tab Shopify URL Mapping.xlsx
If "Addresses" array from Shopify is empty, then send empty "Addresses" array. Since "CountryCode" value is required for each instance of the Address. 

If "addresses → phone" value in the Shopify webhook is NULL or empty, then send empty  "Phones" array. Since PhoneNumber" value is required for each instance of the Phones.  EP-427 - Getting issue details... STATUS



ADAPTER to CDP API data:

Body = profileData object content (as above)
Header json fields sent by ADAPTER to CDP API:

Header fieldValue
AuthorizationToken - should be refreshed every 48h
Accept-Language Provided by SERVICE
Program-CodeProvided by SERVICE
Brand-Org-CodeProvided by SERVICE
Account-sourceProvided by SERVICE
Source-Account-NumberProvided by SERVICE ???
Content-Type

Token Calls - application/x-www-form- urlencoded
All Other Calls - application/json or text/json

External-Correlation-IdUID


UPDATING profile

EP-369 - Getting issue details... STATUS

In order to update customer profile in CDP request to "Update Profile by ID" of CDP API is sent. Method is described on page 93 of MJN_Epsilon Integration API Document v1.36.pdf 
Below there are additional requirements for the content of the updated objects in the request comparing to profile creating.

Object to update"Update Profile by ID" CDP API method (page 93)
Profile
  1. Retrieve "ProfileId" value by "Get Profiles (Search)" CDP API method - page 162 of MJN_Epsilon Integration API Document v1.36.pdf , providing appropriate "SourceAccountNumber".
  2. Add ProfileId as URL parameter - api/v1/profiles/{profileId}.

Addresses

Provide the same "AddressId" value - retrieve it from CDP, take one where "SourceAccountNumber" match.
IF "Addresses" array from Shopify is empty, then add status: "I" to the "Addresses" object. In order to deactivate this address object in CDP.

Emails

Provide the same "EmailId" value - retrieve it from CDP, take one where "SourceAccountNumber" match.
IF "email" field value from Shopify is empty or Null, then add "status: "I" to the "Emails" object. In order to deactivate this email object in CDP.

Phones

Provide the same "PhoneId" value - retrieve it from CDP, take one where "SourceAccountNumber" match.
IF "Addresses" (where "default":true) ->"phone" field value from Shopify is empty or Null, then add "status: "I" to the Phones object. In order to deactivate this phone object in CDP.

SocialAccounts

Provide the same "ProfileSocialAccountId" value - retrieve it from CDP, take one where "SourceAccountNumber" match.

ChildrenProvide the same "ProfileChildId" value - retrieve it from CDP, take one where "BirthDate" (if still there are two objects than + "FirstName" and "LastName") match. *Because FirstName and LastName are not obligatory.

ProfileSubscriptions

Provide the same "SubscriptionId" and "ChannelCode" values.
AgreeementsProvide the same "BusinessId" value - take it from "LDS agreements" tab Shopify URL Mapping.xlsx
UnmappedAttributesIs overwritten each time with any provided value.

ProfileItemList

Provide the same key (property) name.



Abandoned checkout directly to SFMC

 Click here to expand...

I as Middleware want to obtain checkouts data from Shopify by API and send abandoned ones data directly to SFMC, so that SFMC is able to send the email reminder to the customer even 1 hour after the abandonment. Since Shopify provides API only for all checkouts, I as Middleware want to consider as abandoned, checkouts which haven't been finished within hour after last update.
Since reminder email contains images, CDP Middleware requests also product images links and then combines data from both sources to one request sent to SMFC API.

Tasks in Jira:
EP-309 - Getting issue details... STATUS
EP-334 - Getting issue details... STATUS

Shopify API documentation

API Leak rate limit: 4 requests per second

REQUEST AUTHORISATION

  • Basic HTTP authentication by using their Admin API key and password as a username and password OR
  • by including to the request header X-Shopify-Access-Token: {access_token}, where {access_token} is replaced by this store Admin API password.

More details are available in the Shopify API Authorisation Documentation

Get abandoned checkout data (getAbandonedCheckout) 

ENDPOINT
endpoint = {storeDomain}/admin/api/2019-07/checkouts.json?limit=250&updated_at_min={ startDate }&updated_at_max={ endDate }
method = GET

  • storeDomain = domain of the store (ended with myshopify.com) = X-Shopify-Shop-Domain column in the "URL Mapping" tab of Shopify URL Mapping.xlsx
    Each store in the scope (available in the "URL Mapping" tab) is requested
  • "2019-07" - version of the Shopify API
  • "limit=250" - number of returned by Shopify API checkouts
  • "updated_at_min=..." - query to restrain results to those updated at after the provided date.
  • "updated_at_max=..." - query to restrain results to those updated at before the provided date.
  • startDate = now - 65 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:42+00:00.
  • endDate = now - 60 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:47+00:00.


HOW OFTEN request is send to each store
Each 5 min.


EXPECTED ANSWER

  • HTTP/1.1 200 OK
  • Json with up to 250 checkouts, not older than 3 months and ordered from the oldest to the newest one
Sample response
{
  "checkouts": [
    {
      "id": 26371164,
      "token": "zs9ru89kuqcdagk8bz4r9hnxt22wwd42",
      "cart_token": "68778783ad298f1c80c3bafcddeea02f",
      "email": "bob.norman@hostmail.com",
      "gateway": "bogus",
      "buyer_accepts_marketing": false,
      "created_at": "2012-10-12T07:05:27-04:00",
      "updated_at": "2012-10-12T07:05:27-04:00",
      "landing_site": null,
      "note": null,
      "note_attributes": [
        {
          "name": "custom engraving",
          "value": "Happy Birthday"
        },
        {
          "name": "colour",
          "value": "green"
        }
      ],
      "referring_site": null,
      "shipping_lines": [
        {
          "title": "Free Shipping",
          "price": "0.00",
          "code": "Free Shipping",
          "source": "shopify",
          "applied_discounts": [],
          "id": "5da41c1738454765"
        }
      ],
      "taxes_included": false,
      "total_weight": 400,
      "currency": "USD",
      "completed_at": null,
      "closed_at": null,
      "user_id": null,
      "location_id": null,
      "source_identifier": null,
      "source_url": null,
      "device_id": null,
      "phone": null,
      "customer_locale": null,
      "line_items": [
        {
          "applied_discounts": [],
          "key": 49148385,
          "destination_location_id": null,
          "fulfillment_service": "manual",
          "gift_card": false,
          "grams": 200,
          "origin_location_id": null,
          "product_id": 632910392,
          "properties": null,
          "quantity": 10,
          "requires_shipping": true,
          "sku": "IPOD2008RED",
          "tax_lines": [],
          "taxable": true,
          "title": "IPod Nano - 8GB",
          "variant_id": 49148385,
          "variant_title": "Red",
          "variant_price": null,
          "vendor": "Apple",
          "user_id": null,
          "unit_price_measurement": null,
          "country_hs_codes": [],
          "country_code_of_origin": null,
          "province_code_of_origin": null,
          "harmonized_system_code": null,
          "compare_at_price": null,
          "line_price": "1990.00",
          "price": "199.00"
        },
        {
          "applied_discounts": [],
          "key": 808950810,
          "destination_location_id": null,
          "fulfillment_service": "manual",
          "gift_card": false,
          "grams": 200,
          "origin_location_id": null,
          "product_id": 632910392,
          "properties": null,
          "quantity": 10,
          "requires_shipping": true,
          "sku": "IPOD2008PINK",
          "tax_lines": [],
          "taxable": true,
          "title": "IPod Nano - 8GB",
          "variant_id": 808950810,
          "variant_title": "Pink",
          "variant_price": null,
          "vendor": "Apple",
          "user_id": null,
          "unit_price_measurement": null,
          "country_hs_codes": [],
          "country_code_of_origin": null,
          "province_code_of_origin": null,
          "harmonized_system_code": null,
          "compare_at_price": null,
          "line_price": "1990.00",
          "price": "199.00"
        }
      ],
      "name": "#26371164",
      "source": null,
      "abandoned_checkout_url": "https://checkout.local/690933842/checkouts/zs9ru89kuqcdagk8bz4r9hnxt22wwd42/recover",
      "discount_codes": [],
      "tax_lines": [
        {
          "price": "11.94",
          "rate": 0.06,
          "title": "State Tax"
        }
      ],
      "source_name": "web",
      "presentment_currency": "USD",
      "total_discounts": "0.00",
      "total_line_items_price": "3980.00",
      "total_price": "3991.94",
      "total_tax": "11.94",
      "subtotal_price": "3980.00",
      "billing_address": {
        "first_name": "Bob",
        "address1": "Chestnut Street 92",
        "phone": "555-625-1199",
        "city": "Louisville",
        "zip": "40202",
        "province": "Kentucky",
        "country": "United States",
        "last_name": "Norman",
        "address2": "",
        "company": null,
        "latitude": 45.41634,
        "longitude": -75.6868,
        "name": "Bob Norman",
        "country_code": "US",
        "province_code": "KY"
      },
      "shipping_address": {
        "first_name": "Bob",
        "address1": "Chestnut Street 92",
        "phone": "555-625-1199",
        "city": "Louisville",
        "zip": "40202",
        "province": "Kentucky",
        "country": "United States",
        "last_name": "Norman",
        "address2": "",
        "company": null,
        "latitude": 45.41634,
        "longitude": -75.6868,
        "name": "Bob Norman",
        "country_code": "US",
        "province_code": "KY"
      },
      "customer": {
        "id": 207119551,
        "email": "bob.norman@hostmail.com",
        "accepts_marketing": false,
        "created_at": "2019-06-28T15:06:47-04:00",
        "updated_at": "2019-06-28T15:06:47-04:00",
        "first_name": "Bob",
        "last_name": "Norman",
        "orders_count": 1,
        "state": "disabled",
        "total_spent": "199.65",
        "last_order_id": 450789469,
        "note": null,
        "verified_email": true,
        "multipass_identifier": null,
        "tax_exempt": false,
        "phone": null,
        "tags": "",
        "last_order_name": "#1001",
        "currency": "USD",
        "accepts_marketing_updated_at": "2005-06-12T11:57:11-04:00",
        "marketing_opt_in_level": null,
        "admin_graphql_api_id": "gid://shopify/Customer/207119551",
        "default_address": {
          "id": 207119551,
          "customer_id": 207119551,
          "first_name": null,
          "last_name": null,
          "company": null,
          "address1": "Chestnut Street 92",
          "address2": "",
          "city": "Louisville",
          "province": "Kentucky",
          "country": "United States",
          "zip": "40202",
          "phone": "555-625-1199",
          "name": "",
          "province_code": "KY",
          "country_code": "US",
          "country_name": "United States",
          "default": true
        }
      }
    }
  ]
}


EXCEPTION
IF Shopify returned 250 checkouts THEN

1. Count number of checkouts for this period of time (5 min), sending request to endpoint:
endpoint = {storeDomain}/admin/api/2019-07/checkouts/count.json?updated_at_min={ startDate }&updated_at_max={ endDate }
method = GET

  • exception.StartDate = (datetime of the request which received 250 checkouts)  - 65 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:42+00:00.
  • exception.EndDate.Exception = (datetime of the request which received 250 checkouts) datetime now - 60 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:47+00:00.
Sample response
// 20190807130008
// https://durexuk.myshopify.com/admin/api/2019-04/checkouts/count.json?updated_at_min=2019-07-22T00:09:47-01:00&updated_at_max=2019-07-22T09:18:52-01:00

{
  "count": 32
}


2. Repeat requests for "count" times, until all checkouts for these five minutes are retrieved.

3. Go back to normal process


HOW checkout data is STORED
Abandoned checkouts (where "completed_at": null) data is saved to AnypointMQ, where it waits for consolidation with product images data.

Get product images data (getProductImage)

As Middleware I want to have a database of product Images links of every integrated Shopify store, so that SFMC receives full abandoned checkout data and able to send email reminders containing product images.

ENDPOINT
endpoint = {storeDomain}/admin/api/2019-07/products.json?limit=250&fields=image
method = GET

  • storeDomain = domain of the store (ended with myshopify.com) = X-Shopify-Shop-Domain column in the "URL Mapping" tab of Shopify URL Mapping.xlsx
    Each store in the scope (available in the "URL Mapping" tab) is requested.
  • "2019-07" - version of the Shopify API
  • "limit=250" - number of returned by Shopify API checkouts. 250 is above max number of products one RB store is offering - Scholl UK 201 product, but it can increase once a year, so it should be monitored.
  • "fields=image" - constraining response to only image related fields


Sample response
// 20190726155851
// https://durex-uk-test-store.myshopify.com/admin/api/2019-04/products.json?fields=image

{
  "products": [
    {
      "image": {
        "id": 5430477062202,
        "product_id": 628184809530,
        "position": 1,
        "created_at": "2018-08-22T13:36:51+01:00",
        "updated_at": "2018-08-22T13:36:51+01:00",
        "alt": "7 Nights Of Fun",
        "width": 1024,
        "height": 1024,
        "src": "https://cdn.shopify.com/s/files/1/0070/7513/5546/products/7-nights-of-fun-col-condoms-lubes-pleasure-sets-sex-toys-relationship-advice-durex-uk-test-store_674.jpg?v=1534941411",
        "variant_ids": [
          
        ]
      }
    },
    {
      "image": {
        "id": 6569550577722,
        "product_id": 2120385560634,
        "position": 1,
        "created_at": "2018-11-09T14:34:17+00:00",
        "updated_at": "2018-11-09T14:34:17+00:00",
        "alt": "Carry On Travelling, front",
        "width": 1500,
        "height": 1500,
        "src": "https://cdn.shopify.com/s/files/1/0070/7513/5546/products/carry-on-travelling-bundle-front-durex-uk_49fbca5b-eeff-49f3-a792-2b814c77408d.jpg?v=1541774057",
        "variant_ids": [
          
        ]
      }
    }
]
}


HOW OFTEN image request is sent to each store
Every 4 hours


HOW images are STORED and UPDATED
IF any of the objects has "updated_at" field with value > lastProductImageRequestDate THEN productImagedatabase for this specific store is overwrited.

Id est below values from "products" [ "image": {} ] object overwrite existing values.

  • "product_id" field value in the "productID" column / field.
  • "id" field value in the "imageId" column / field
  • "src" field value in the "productImage" column / field.

productImagedatabase = DynamoDB. Database contain images for all integrated stores, since product_id is unique across entire Shopify platform. Productimage can be accessed by dedicated function with productID as input.

Get Individual_ID (getIndividualId) 

EP-463 - Getting issue details... STATUS

  1. IndividualId value is retrieved from CDP by "Get Profiles (Search)" CDP API method - page 162 of MJN_Epsilon Integration API Document v1.36.pdf , providing appropriate "SourceAccountNumber".
  2. SourceAccountNumber is constructed in Middleware in the below way:

    • BrandOrcCode & “_” & customer_id from Shopify API received within 374145073. Path in Json: checkouts → customer → id


  3. If no account is found, the data has to be sent to Salesforce with other event ID and with email as ContactKey.

Consolidate checkout and product images data (ConsolidateData)

As Middleware I want to combine abandoned checkout and product images data, so that I am able to send a request to SFCM containing the required set of information.
Consolidation and transfer to SFMC is happening realtime, as soon as there is abandoned checkout data, it is enriched with "productImage" image field, Individual_ID and send to SFMC.

  • Abandoned checkout data is taken form AnypointMQ
  • product image is taken from productImagedatabase
  • Individual_ID is taken from 374145073

  • "productImage" field of this store productImagedatabase WHERE productImagedatabase.productID = shopifywebhook.line_items.product_Id is added it to checkout data.
  • Individual_ID is added

Mapping of the fields sent to SFMC (available also in the "Abandoned checkout" tab of Shopify URL Mapping.xlsx):

 Click here to expand...
#Shopify / productImagedatabase Field nameFile and SFMC field NameField Data TypeMax LengthReq (Y/N)Description/Notes
1idcheckout-idlong
YId of the checkout. Should be added on CDP side, before sending to CDP as batchfile
2individual_idindividual_idstring255YIndividual_ID - ID of the person in CDP. Should be added on CDP side, before sending to CDP as batchfile
3customer -> first_namecheckout-customer-first_namestring60YFirst Name – required
4customer -> last_namecheckout-customer-last_namestring60YLast Name – required
5abandoned_checkout_urlabandoned_checkout_urlstring1024YThe recovery URL to recover the cart
6buyer_accepts_marketingbuyer_accepts_marketingboolean
YIndicates whether the person accepts marketing (TRUE=Yes; False=No)
7cart_tokencart_tokenstring32YThe unique ID for the cart
8created_atcreated_atstring19YThe date/time when the order was created (YYYY-MM-DDTHH:MM:SS)
9emailemailstring254YThe customer's email address
10referring_sitereferring_sitestring500NThe website that the customer clicked on to come to the shop
11subtotal_pricesubtotal_pricestring38YPrice of the order before shipping and taxes
12taxes_includedtaxes_includedstring10YIndicates whether taxes are included ( (TRUE=Yes; False=No)
13total_discountstotal_discountsstring38YThe total of the discounts applied to the order
14total_line_items_pricetotal_line_items_pricestring38YThe sum of the prices of all line items in the checkout in presentment currency.
15total_pricetotal_pricestring38YThe sum of all prices of items in the order including taxes and discounts
16total_taxtotal_taxstring38YThe sum of all the taxes applied to the checkout in presentment currency.
17total_weighttotal_weightnumber
YThe sum of the weights of all items in the order
18updated_atupdated_atstring19YThe date/time when the checkout was last modified (YYYY-MM-DDTHH:MM:SS) format
19namecheckout_namestring255NThe checkout name as represented by a number. E.g. #981820079255243537
20tokencheckout_tokenstring255NA unique ID for a checkout. E.g. b1946ac92492d2347c6235b4d2611184
21shipping_address -> first_nameshipping_address-first_namestring60NHeader level. First name associated to the Shipping address 
22shipping_address -> last_nameshipping_address-last_namestring60NHeader level. Last name associated to the Shipping address 
23shipping_lines -> titleshipping_lines-titlestring40NHeader level. The title of the shipping method, e.g. Small Packet International Air.
24discount_codes -> discount_code -> codediscount_codestring20NHeader level. Discount code. Only one code will be available in the "discount_codes" object.
25{counting line items starting from 1}line_items-line_numberinteger
NDetails level. Position of the line item in the order
26line_items -> skuline_items-skustring30NDetails level. Product SKU
27line_items -> titleline_items-titlestring1000NDetails level. Product name
28line_items -> priceline_items-pricestring38NDetails level. Product price
29line_items -> quantityline_items-quantityinteger
NDetails level. Quantity purchased
30"productImage" field of this store productImagedatabase,
WHERE
productImagedatabase.productID = shopifywebhook.line_items.product_Id
line-items-image_urlstring1024NDetails level. URL to item image

Json sent to SMFC

Abandoned checkout JSON
{  
   "checkout-id": 7877338071086,
   "checkout-customer-first_name": "Tomasz",
   "checkout-customer-last_name":"Lem",
   "abandoned_checkout_url": "https://durex-uk-husky-test.myshopify.com/3456172078/checkouts/1d4505800ad62fb354a225f3b1dcc22c/recover?key=a2253df3135f9a18df6307d3ca7d9d5d",
   "buyer_accepts_marketing": true,
   "cart_token": "17dc8320f2d9f0fb49ca94ce11f79180",
   "created_at": "2019-06-13T08:10:33-04:00",
   "email": "tomasz.lem@mail.net",
   "referring_site": "",
   "subtotal_price": "170.39",
   "taxes_included": true,
   "total_discounts": "0.00",
   "total_line_items_price": "170.39",
   "total_price": "180.39",
   "total_tax": "31.86",
   "total_weight": 968,
   "updated_at": "2019-08-12T06:03:57-04:00",
   "checkout_name": "#7877338071086",
   "checkout_token": "1d4505800ad62fb354a225f3b1dcc22c",
   "shipping_address-first_name": "Tomasz",
   "shipping_address-last_name": "Lem",
   "shipping_lines-title": "Small Packet International Air",
   "discount_code": "QDFKWEJ",
   "line_items":[  
      {  
         "line_items-line_number": 1,
         "line_items-sku": "3030446",
         "line_items-title": "Durex Thin Feel Condoms 30 Pack",
         "line_items-price": "15.49",
         "line_items-quantity": 11,
         "line-items-image_url": "https://cdn.shopify.com/s/files/1/2992/1628/products/durex-uk-condoms-durex-surprise-me-condoms-40-pack-7178711957586_540x.jpg?v=1561979155"
      }
   ],
   "individual_id": "4020008919656"
}


IF productImage" field of this store productImagedatabase is not available, getProductImage method is forced and combining for this product is repeated.


Send to SFMC (sendToSFMC)

Data is sent to SFMC Journey Builder API.

Appendix Item 1 - Journey Builder API

The API Entry Event gives you two things:

  1. Inserts contact data into a data extension (aka Journey Entry Event data extension)
  2. Fire an entry event to trigger the contact into the Journey.

To set up an API entry Event see https://help.salesforce.com/articleView?id=mc_jb_admit_contacts_via_api.htm&type=5. Once you have set up an Entry Event you will be provided with an Event Definition Key.
 
You will need to pass this key with the JB Fire Entry Event API call. This is the API the Mulesoft team will call.

Sample call
Host: https://YOUR_SUBDOMAIN.rest.marketingcloudapis.com
POST /interaction/v1/events
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
    "ContactKey": "ID601",
    "EventDefinitionKey":"AcmeBank-AccountAccessed",
    "Data": {
        "accountNumber":"123456",
        "patronName":"John Smith" }
}


Consolidation and transfer to SFMC is happening realtime, as soon as there is abandoned checkout data, it is enriched with "productImage" field and send to SFMC.
Nothing is sent to SFMC if there were no abandoned checkouts for this specific store.


Shopify URL Mapping.xlsx


Shopify webhooks spec: https://help.shopify.com/en/api/reference/events/webhook
List of stores: /wiki/spaces/RES/pages/134119897


CDS docs: /wiki/spaces/GUA/pages/326238486


Archive - Objectstore description

 Click here to expand...

ObjectStore - service provided by MuleSoft as a part of CloudHub. It is distributed storage engine - in memory or persistent. Data can be accesed by ObjectStore Mule adapter or by REST API.
Example how to retrievie data from OS by API. To use object store from outside of Mule cloud hub it needs to go through this steps:

1. Get token:

curl --request POST \
  --url https://anypoint.mulesoft.com/accounts/login \
  --header 'content-type: application/json' \
  --data '{
	"username": "your login",
	"password": "your password"
}'


2. Get list of environments:

curl --request GET \
  --url https://anypoint.mulesoft.com/accounts/api/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments \
  --header 'authorization: Bearer token-from-step-1' 

3. Get list of stores:

curl --request GET \
  --url https://anypoint.mulesoft.com/accounts/api/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments \
  --header 'authorization: Bearer token-from-step-1’

4. Get list of partitions:

curl --request GET \
  --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions \
  --header 'authorization: Bearer token-from-step-1'

5. Get list of keys:

curl --request GET \
  --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions/DEFAULT_PARTITION/keys \
  --header 'authorization: Bearer token-from-step-1’

6. Get values by key:

curl --request GET \
  --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions/DEFAULT_PARTITION/keys/key-name \
  --header 'authorization: Bearer token-from-step-1'

Result will be simillar to this:

{
  "stringValue": "value",
  "keyId": "key-name",
  "valueType": "STRING"
}

Development information:

Please install lombok plugin at Anypoint Studio to get support for auto-generated code(Intellij support through plugin on the market):

  1. Copy lombok.jar file onto AnypointStudio\plugins
  2. Add at end of AnypointStudio\AnypointStudio.ini 
    -javaagent:lombok.jar
  3. Restart IDE
  4. At info panel "About Anypoint Studio" you shall sth similar to see:

Additional fields table

lombok.jar