@Rich_Learner  To resolve the issue where the ttype column returns NULL and subclasses'  values aren't being parsed correctly, follow these steps: 1. Case Sensitivity in JSON Paths The JSON keys Type and ProductCOde use uppercase letters. Correct the paths in get_json_object to match the exact case: Use '$.Type' instead of '$.type'. Ensure '$.ProductCOde'  matches the case in the JSON (if it's a typo, adjust accordingly). 2. Properly Handle Nested Arrays Avoid flattening both the outer products array and inner Offer array into the same lateral view. Use separate lateral views for each array level. Corrected Code   WITH product AS ( SELECT xml_data, GET_JSON_OBJECT(xml_data, '$.Customer.CustomerId') AS applicationid, -- Extract products array as a JSON string GET_JSON_OBJECT(xml_data, '$.Customer.products') AS products_json FROM Custome_info ), -- Explode the products array exploded_products AS ( SELECT applicationid, product_item FROM product LATERAL VIEW OUTER EXPLODE( SPLIT( REGEXP_REPLACE( REGEXP_REPLACE(products_json, '^\\[|\\]$', ''), -- Remove outer brackets '\\}\\,\\{', '\\}\\;\\{' -- Split products ), ';' ) ) p AS product_item ), -- Explode the Offer array within each product exploded_offers AS ( SELECT applicationid, GET_JSON_OBJECT(product_item, '$.ProductCOde') AS ProductCode, GET_JSON_OBJECT(product_item, '$.Type') AS ttype, offer_item FROM exploded_products LATERAL VIEW OUTER EXPLODE( SPLIT( REGEXP_REPLACE( REGEXP_REPLACE( GET_JSON_OBJECT(product_item, '$.Offer'), '^\\[|\\]$', '' -- Remove inner Offer brackets ), '\\}\\,\\{', '\\}\\;\\{' -- Split Offer items ), ';' ) ) o AS offer_item ) -- Combine results (products and their offers) SELECT applicationid, COALESCE( GET_JSON_OBJECT(offer_item, '$.ProductCOde'), -- Offer-level ProductCode ProductCode -- Product-level ProductCode ) AS ProductCode, COALESCE( GET_JSON_OBJECT(offer_item, '$.Type'), -- Offer-level Type ttype -- Product-level Type ) AS ttype FROM exploded_offers;   Explanation Case Sensitivity Fix: Using '$.Type' ensures the correct extraction of the Type field from the JSON. Nested Array Handling: First, explode the products array to get individual product objects. Then, explode the Offer array within each product to access sub-offer details. Combined Results: Use COALESCE to merge product-level and offer-level values into a single output, ensuring both levels are represented. Expected Output ApplicationID | ProductCode | ttype 19900101      | ABC  | C1 19900101      | A1       | CBA 19900101      | A2      | DBA 19900101      | A3      | BBA 19900101      | XYZ    | C2 19900101      | B1       | BBA 19900101      | B2      | BBA Happy hadooping  ... View more

@rj27  Some clarification on the git setup  This is the author name that will appear in commit messages Set the global Git username to "NiFi Registry"  This is the email address associated with commits Set the global Git email to "nifi-registry@example.com"   Values to be passed  <flowPersistenceProvider>         <property name="Flow Storage Directory">./flow_storage</property>        <property name="Git Remote To Push">origin</property>        <property name="Git Remote Access User">username</property>       <property name="Git Remote Access Password">password</property>       <property name="Remote Clone Repository">https://git-repo-url/your-flow-repo.git</property> </flowPersistenceProvider> ... View more

@rj27  To set up Git integration for Apache NiFi Registry using SSH authentication, you need to configure the NiFi Registry to use a Git-based flow persistence provider. Analysis of Current Setup You have Apache NiFi 1.28 running on AWS ECS Fargate You have Apache NiFi Registry 1.28 running on AWS ECS Fargate Both applications are communicating with each other successfully You need to integrate NiFi Registry with Git using SSH authentication Below are the detailed steps to achieve this on an AWS ECS instance running on Fargate with NiFi and NiFi Registry 1.28. Detailed Steps for Git Integration Step 1: Update NiFi Registry Configuration Modify the nifi-registry.properties file in your container Add the following properties to configure the Git flow persistence provider # Git Configuration nifi.registry.db.git.remote=true nifi.registry.db.git.remote.to.push=true nifi.registry.db.git.repository=/opt/nifi-registry/git-repository nifi.registry.db.git.flow.storage.directory=/opt/nifi-registry/flow-storage nifi.registry.db.git.remote.url=ssh://git@your-git-server:port/your-repo.git nifi.registry.db.git.remote.branch=master Step 2: Set Up SSH Keys for Authentication 1. Generate an SSH key pair inside your container mkdir -p /opt/nifi-registry/.ssh ssh-keygen -t rsa -b 4096 -C "nifi-registry@example.com" -f /opt/nifi-registry/.ssh/id_rsa -N "" 2. Add your public key to your Git repository's authorized keys (in GitHub, GitLab, etc.) Copy the contents of /opt/nifi-registry/.ssh/id_rsa.pub Add it to your Git provider as a deploy key or authentication key 3. Configure SSH client in the container cat > /opt/nifi-registry/.ssh/config << EOF Host your-git-server      IdentityFile /opt/nifi-registry/.ssh/id_rsa      StrictHostKeyChecking no      UserKnownHostsFile /dev/null EOF 4. Set proper permissions chmod 700 /opt/nifi-registry/.ssh chmod 600 /opt/nifi-registry/.ssh/id_rsa chmod 644 /opt/nifi-registry/.ssh/id_rsa.pub chmod 600 /opt/nifi-registry/.ssh/config Step 3: Update ECS Task Definition for Persistence 1. Update your ECS task definition to include a volume for SSH keys and Git repository  validate the JSON's "volumes": [     {    "name": "nifi-registry-git",    "dockerVolumeConfiguration": {       "scope": "task",       "driver": "local",       "labels": null,       "autoprovision": true      }   } ] 2. Mount this volume in your container definition "mountPoints": [      {         "sourceVolume": "nifi-registry-git",         "containerPath": "/opt/nifi-registry/.ssh",         "readOnly": false       },      { "sourceVolume": "nifi-registry-git",         "containerPath": "/opt/nifi-registry/git-repository",         "readOnly": false      } ] Step 4: Configure Git User Information Set Git user configuration git config --global user.name "NiFi Registry" git config --global user.email "nifi-registry@example.com" Step 5: Initialize the Git Repository Initialize the local Git repository mkdir -p /opt/nifi-registry/git-repository cd /opt/nifi-registry/git-repository git init git remote add origin ssh://git@your-git-server:port/your-repository.git 2. Test the connection ssh -T git@your-git-server Step 6: Configure NiFi to Connect to NiFi Registry In NiFi UI, configure the Registry Client: Click on the hamburger menu (≡) in the top-right corner Select "Controller Settings" Go to the "Registry Clients" tab Add a new Registry Client with: Name: Git-Backed Registry URL: http://your-nifi-registry:18080 Step 7: Restart NiFi Registry Restart the NiFi Registry service to apply change # If using systemd systemctl restart nifi-registry # If using the command line ./bin/nifi-registry.sh restart # In AWS ECS, update the service to force new deployment aws ecs update-service --cluster your-cluster --service your-nifi-registry-service --force-new-deployment Troubleshooting 1. Check NiFi Registry logs for Git-related errors: tail -f /opt/nifi-registry/logs/nifi-registry-app.log 2. Verify SSH connectivity ssh -vT git@your-git-server 3. Common issues: Permission problems: Ensure the NiFi Registry user has appropriate permissions Known hosts: If StrictHostKeyChecking is on, you need to accept the host key first Firewall: Ensure outbound connections to the Git server are allowed from the ECS task                                                 Important precautions  Security: Ensure the private key is stored securely and not exposed in the container image or logs. Automation: Consider using AWS Secrets Manager or Parameter Store to manage the SSH key and passphrase securely. Backup: Regularly back up your Git repository to avoid data loss. Happy hadooping           ... View more

@0tto  Please could you share your NiFi logs. Happy hadooping ... View more

@ose_gold  The SFTP issues appear to stem from incorrect permissions and ownership in your Docker setup. Here's the analysis and solution: Key Issues: Root ownership instead of user 'foo' for /home/foo Incorrect chroot directory permissions Docker volume mount permissions You have a volume nifi-conf:/opt/nifi/nifi-current/conf for the NiFi container, but it's not declared in the volumes section at the bottom check the addition below Docker Compose has newer versions 3.8 It might be a good idea to update to a more recent version depending on the features you need.   version: '3' #Docker Compose has newer versions 3.8 services: nifi: image: apache/nifi:latest # Consider specifying a version container_name: nifi ports: - "8089:8443" - "5656:5656" volumes: - nifi-conf:/opt/nifi/nifi-current/conf environment: NIFI_WEB_PROXY_HOST: localhost:8089 SINGLE_USER_CREDENTIALS_USERNAME: admin SINGLE_USER_CREDENTIALS_PASSWORD: {xxyourpasswdxx} sftp: image: atmoz/sftp volumes: - ./sftp/upload:/home/foo/upload ports: - "2222:22" command: foo:pass:1001 # Add these permissions user: "1001:1001" environment: - CHOWN_USERS=foo - CHOWN_DIRS=/home/foo volumes: nifi-conf:   Before starting containers   # Set correct permissions on host mkdir -p ./sftp/upload chown -R 1001:1001 ./sftp/upload chmod 755 ./sftp/upload   This configuration: Sets proper user/group ownership Maintains correct chroot permissions Ensures volume mount permissions are preserved Prevents permission conflicts between host and container Happy Hadooping ... View more

@amru  Can you share the logs ... View more

@rsurti  If SAML authentication works for LAN users but not for users on Wi-Fi, even when both are on the same network, it suggests differences in how the network or devices are configured for each connection type. Here’s how you can troubleshoot and resolve this issue: DNS Resolution: Check if Wi-Fi users can resolve the identity provider (IdP) and service provider (SP) URLs correctly.     nslookup idp.example.com   Capture SAML requests/responses using browser developer tools (Network tab). Look for differences in: Redirect URLs Assertions Error codes Common issues include: Misconfigured callback URLs. Device Configuration Check if device firewalls or VPNs are interfering with SAML traffic over Wi-Fi. Ensure browser settings (e.g., cookie policies) do not block SAML cookies. Connect a user to LAN and Wi-Fi simultaneously (if possible) to identify differences in routing or access. Please revert Happy hadooping ... View more

@spiker  According to the below zk parameter you have set it to false meaning you are using external and not embedded zk is that the case? # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server   nifi.state.management.embedded.zookeeper.start=false   Yet the below zk config seems contradictory # zookeeper properties, used for cluster management #   nifi.zookeeper.connect.string=zookeeper:2181 # Zookeeper should resolve to correct host(s) for the Zookeeper ensemble   Check this documentation for setting external zookeepers If you are using embedded zk the adjust the following entries in your nifi.properties   nifi.state.management.embedded.zookeeper.start=true nifi.zookeeper.connect.string=IP01:2181,IP02:2181,IP03:2181 nifi.zookeeper.auth.type=default nifi.remote.input.host=IP01 # Localhost ip nifi.remote.input.secure=false nifi.remote.input.socket.port=9998 nifi.remote.input.http.enabled=true # set true if you want http nifi.cluster.is.node=true nifi.cluster.node.address=IP01 # Localhost ip nifi.cluster.node.protocol.port=7474 nifi.web.http.host=IP01 # Localhost ip. use either https or http nifi.web.http.port=8443 nifi.cluster.load.balance.port=6342   zookeeper.properties This file contains additional info to be used by zookeeper to know about the servers.   server.1=IP01:2888:3888 server.2=IP02:2888:3888 server.3=IP03:2888:3888 clientPort=2181   In order to maintain the nifi state across instances,you need to modify the state-management.xml and provide a new state provider pointing to zookeeper.   <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">ip1:2181,ip2:2181,ip3:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider>   Here the Access Control has been set to open to be able to login without an username/pass but you should configure your to use your oidc-provider I would assume . OpenId Connect SSO Properties  Does the user email exist in the OIDC token and is accessible check the Azure AD and confirm the token contains the expected email and upn   nifi.security.user.oidc.claim.identifying.user={email} nifi.security.user.oidc.fallback.claims.identifying.user=upn   Can you ensure the OPENID discovery URL is reachable from the NiFi nodes and resolves correctly run the below curl to confirm connectivity   curl -v https://login.microsoftonline.com/XXXXXXXXXXXXXXXXXXXXXXX/v2.0/.well-known/openid-configuration   Validate HTTPS and OIDC   curl -vk https://<nifi-node>:8443/nifi-api/ curl -vk https://<nifi-node>:8443/nifi-api/access/oidc/callback   Clear Cache Stop NiFi on all nodes and clear state directory ./conf/state-management.xml and restart the cluster Hope that helps Happy hadooping ... View more