introduction
Laissez-moi vous expliquer mon accordéon tout de suite. Oui, Internet regorge de manuels. Oui, plein de solutions au tour par tour. Oui, vous pouvez dire que tout est mâché mâché. Mais spécifiquement dans mon cas, comme cela arrive toujours, il y avait une poignée de «mais»:
Il existe des manuels expliquant comment configurer un ensemble de registres NiFi et NiFi avec l'authentification et l'autorisation activées. Mais ... des certificats auto-signés sont utilisés.
Il existe des manuels séparés sur la façon de visser un certificat commercial pour NiFi; respectivement pour le registre NiFi "kagbe is the same". Mais l'authentification et l'autorisation mutuelles se produiront avec l'utilisation du SSL bidirectionnel ... et nous avons LDAP ... et alors vous ne pourrez pas assurer la cohérence du couple doux en utilisant uniquement l'annuaire externe sur votre simple intuition.
Il existe des manuels de connexion avec LDAP pour les registres NiFi et NiFi. Nooon ... comme dans le "mais" précédent, des questions se posent, comment se débrouiller avec seulement LDAP, car nous avons toujours NiFi Cli, mais il ne sait pas comment faire LDAP.
En d'autres termes, tous les manuels ont une petite nuance: ils ne couvrent que les scénarios les plus simples. Il n'y a tout simplement pas de documentation pour les bundles complexes. De plus, au cours de la mise en place du bundle, j'ai rencontré des difficultés qui ne se rencontrent que quelques fois dans la toile bourgeoise et elles sont toutes soit sans réponses, soit les réponses ne sont pas pertinentes.
Ce que nous avons et ce que nous voulons obtenir
On a:
Certificat SSL / TLS générique commercial;
Un serveur d'annuaire qui peut faire LDAP (dans mon cas, Apache DS);
Le serveur NiFi sur lequel se déroule le développement;
Serveur NiFi dans un environnement d'épicerie;
Serveur de registre NiFi.
Nous voulons:
Authentification, autorisation et HTTPS. De plus, NiFi ne vous permet pas d'activer les mécanismes d'authentification intégrés sans HTTPS;
Gestion centralisée des utilisateurs sur un serveur d'annuaire;
Flux de version et groupe de processus d'une manière similaire à gith;
.
:
Apache DS;
Java Keystore Java Truststore, HTTPS;
LDAP NiFi;
LDAP NiFi Registry;
;
NiFi NiFi Registry;
NiFi NiFi Cli.
wildcard : *.love.you;
FQDN Apache DS: auth.love.you;
Apache DS: uid=admin,ou=system; : secret;
Apache DS love.you: dc=love,dc=you;
Apache DS nifiAdmin nifiUser NiFi NiFi Registry;
Apache DS nifiAdmins nifiUsers NiFi NiFi Registry;
FQDN NiFi, : nifi-deve.love.you;
FQDN NiFi Registry: nifi-re.love.you;
FQDN () NiFi: nifi.love.you;
, NiFi: $NIFI_HOME;
, NiFi Registry: $NIFI_REGISTRY_HOME;
, NiFi Toolkit: $NIFI_TOOLKIT_HOME;
, : /love/conf/cert;
PKCS12, Java Keystore Java Truststore:
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
Apache DS
, , Apache DS . . Azure AD, " ", NiFi, AD' . Apache DS , , Apache.
10389 . Apache Directory Studio. , , .
New connection:
:
: uid=admin,ou=system secret. Check Authentication:
.
. - member objectclass inetOrgPerson. LDAP Browser' ou=schema Reload Entry:
cn=inetorgperson ou=objectClasses. m-oid=2.16.840.1.113730.3.2.2. Open With -> LDIF Entry Editor:
m-may: member Ctrl+S:
Open Configuration:
Advanced Partition configuration...
Add, ID Suffix. Ctrl+S:
: . . . :
. - LDIF . - import.ldif .
import.ldif
dn: ou=people,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people
dn: ou=groups,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups
dn: cn=nifiUsers,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiUsers
uniqueMember: cn=nifiUser,ou=people,dc=love,dc=you
dn: cn=nifiAdmins,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiAdmins
uniqueMember: cn=nifiAdmin,ou=people,dc=love,dc=you
dn: cn=nifiUser,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiUser
description: A nifiUser user
sn: nifiUser
uid: nifiUser
mail: nifiUser@love.you
userpassword: password
member: cn=nifiUsers,ou=groups,dc=love,dc=you
dn: cn=nifiAdmin,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiAdmin
description: A nifiAdmin user
sn: nifiAdmin
uid: nifiAdmin
mail: nifiAdmin@love.you
userpassword: password
member: cn=nifiAdmins,ou=groups,dc=love,dc=you
dn: cn=*.love.you,ou=people,dc=love,dc=you
objectClass: person
objectClass: top
cn: *.love.you
sn: *.love.you
LDIF Import:
:
:
cn=*.love.you
, ou=people cn=*.love.you, objectClass: person:
, : , *.love.you. NiFi Registry.
Java Keystore Java Truststore
:
PEM-: fullchain.crt;
PEM-: private.key;
certificate authority (CA) PEM-: cacert.pem;
, . , .
. :
-----BEGIN CERTIFICATE----- ... ... ... -----END CERTIFICATE-----
.
, .
, , , ( ) . CA .
Java Keystore . PKCS12:
openssl pkcs12 -export -out nifi.p12 -inkey private.key -in fullchain.crt -name nifi-key
PKCS12 . .
Java Keystore PKCS12 :
keytool -importkeystore -srckeystore nifi.p12 -srcstoretype pkcs12 -srcalias nifi-key -destkeystore keystore.jks -deststoretype jks -destalias nifi-key
, . Java Keystore, .
Java Truststore:
keytool -importcert -alias nifi-cert -file cacert.pem -keystore truststore.jks
Java Truststore. . Keystore Truststore. . .
JKS NiFi, , NiFi NiFi Registry, /love/conf/cert/;
NiFi , , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi-deve.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi-deve.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
: nifi.web.http.host nifi.web.http.port. , , NiFi .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
# Identity Mapping Properties #
nifi.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
: . .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi Registry, $NIFI_REGISTRY_HOME/config/nifi-registry.properties, .
nifi-registry.properties
# web properties #
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=nifi-re.love.you
nifi.registry.web.https.port=8443
# security properties #
nifi.registry.security.keystore=/abc/conf/cert/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.truststore=/abc/conf/cert/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.needClientAuth=false
nifi.registry.security.identity.provider=ldap-identity-provider
# Identity Mapping Properties #
nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE
: . .
16 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
LDAP NiFi
LDAP : $NIFI_HOME/conf/login-identity-providers.xml $NIFI_HOME/conf/authorizers.xml
login-identity-providers.xml . LDAP- . .
login-identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</loginIdentityProviders>
, 27 29. 27 , , LDAP-. {0} - , User NiFi; uid - , , . , nifiUser.
, . ApacheDS.
29 - : (USE_USERNAME), DN (USE_DN). , , . DN .
authorizers.xml , NiFi "" . .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
:
31 - objectClass ApacheDS, ;
35 - , ;
39 - objectClass, ;
50 - , ( identifier: <identifier>ldap-user-group-provider</identifier> <userGroupProvider>);
52 - , - "" .
NiFi nifiAdmin , ldif-.
LDAP NiFi Registry
NiFi Registry NiFi. $NIFI_REGISTRY_HOME/conf/identity-providers.xml .
identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
<provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</identityProviders>
$NIFI_REGISTRY_HOME/conf/authorizers.xml .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="NiFi Group Name"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
NiFi Registry nifiAdmin , ldif-.
Identity Mapping Properties nifi.properties nifi-registry.properties.
, NiFi NiFi Registry.
, "Proxy user requests". , , . , " " " ", , . , , . , . . .
? , NiFi NiFi Registry , . NiFi Registry NiFi , NiFi Java Keystore , , NiFi Registry. , wildcard , CN=*.love.you.
!
, NiFi . , , CN . . , CN cn. NiFi .
. ApacheDS cn . . . : . NiFi , : nifiUser, nifiAdmin *.love.you. cn=....
, , , , .
, , ....mapping.pattern.dn capture group, ....mapping.value.dn . CN=*.love.you *.love.you, ApacheDS.
NiFi NiFi Registry
NiFi . , , NiFi . , .
Operate:
Policies , :
, *.love.you proxy user requests:
Controller Settings:
REGISTRY CLIENTS:
, + , , :
NiFi Registry, , :
USERS :
:
Bucket Test
NiFi. process group Test. process group Version -> Start version control:
:
SAVE process group :
process group .
NiFi NiFi Cli
, process group flow Registry, () NiFi.
NiFi Toolkit NiFi .
: $NIFI_TOOLKIT_HOME/config/reg.properties $NIFI_TOOLKIT_HOME/config/nifi-prod.properties:
NiFi Registry:
reg.properties
baseUrl=https://nifi-re.love.you:8443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
() NiFi:
nifi-prod.properties
baseUrl=https://nifi.love.you:9443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
:
NiFi Registry ID , flow.
flow ID ID flow, () NiFi.
flow;
flow () NiFi.
$NIFI_TOOLKIT_HOME/bin :
:
./cli.sh registry list-buckets -p ../conf/reg.properties
:
# Name Id Description
- ---- ------------------------------------ -----------
1 Test 84744b22-0e2e-4342-b6ab-6658ff0e54ec (empty)
flow 84744b22-0e2e-4342-b6ab-6658ff0e54ec:
./cli.sh registry list-flows -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -p ../conf/reg.properties
:
# Name Id Description
- ------------------ ------------------------------------ ------------------
1 PG versioning test bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 PG versioning test
flow:
./cli.sh registry list-flow-versions -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -p ../conf/reg.properties
:
Ver Date Author Message
--- -------------------------- --------- ------------------
1 Thu, Apr 22 2021 09:28 UTC nifiAdmin PG versioning test
flow () NiFi:
./cli.sh nifi pg-import -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -fv 1 -p ../conf/nifi-prod.properties
:
f90df705-0178-1000-04fe-45ba27e76f99
- ID process group () NiFi:
, HowTo . , , (, NiFi Registry), "TRACE" NiFi NiFi Registry . , , .
En ce qui concerne CI / CD, je n'ai pas dessiné toute la chaîne. Chacun a ses propres schémas et scénarios, et le pipeline lui-même serait hors sujet. Et l'exemple ci-dessus NiFi Cli, je pense, est un bon point de départ pour mettre en œuvre votre propre schéma de livraison.
Merci à tous ceux qui l'ont lu et j'espère que le texte a été utile.