{"id":99,"date":"2019-08-29T06:00:29","date_gmt":"2019-08-29T06:00:29","guid":{"rendered":"http:\/\/virtorbis.virtcompute.com\/?p=99"},"modified":"2019-08-29T07:05:21","modified_gmt":"2019-08-29T07:05:21","slug":"lab-setup-red-hat-openshift-4-1-on-vmware-vsphere-6-7","status":"publish","type":"post","link":"https:\/\/virtorbis.virtcompute.com\/?p=99","title":{"rendered":"Lab-Setup: Red Hat OpenShift 4.1 on VMware vSphere 6.7"},"content":{"rendered":"\n
\"\"
<\/figcaption><\/figure>\n\n\n\n

This is a home lab setup to test deployment of Red Hat OpenShift 4.1 on VMware vSphere 6.7 using Intel NUC.<\/p>\n\n\n\n

Below is a high level view of the hardware setup.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

NUC 1 is my jump-host to access all the other hosts.
Ctrl 1-3 are the OpenShift master nodes, Cmpt 1-3 are the worker nodes and BS-1 is the bootstrap node.

This deployment will be using User Provisioned Infrastructure approach (UPI) on baremetal. All the OpenShift nodes require internet connection to download images. Hence using basecamp to route all the traffic from VM’s to internet.<\/p>\n\n\n\n

Virtual Machine resource assignment as below (trying not to burn my NUC’s)<\/p>\n\n\n\n
VM<\/td>Operating System<\/td>vCPU<\/td>RAM<\/td>Storage (THIN)<\/td><\/tr>
Bootstrap<\/td>RHCOS<\/td>4<\/td>8 GB<\/td>120 GB<\/td><\/tr>
Control plane<\/td>RHCOS<\/td>4<\/td>8 GB<\/td>120 GB<\/td><\/tr>
Compute<\/td>RHCOS<\/td>4<\/td>4 GB<\/td>120 GB<\/td><\/tr><\/tbody><\/table>\n\n\n\n

IP scheme :<\/p>\n\n\n\n
VM<\/td>Hostname<\/td>domain<\/td>IP address<\/td>DHCP\/Static<\/td><\/tr>
basecamp<\/td>basecamp<\/td>demo.com<\/a><\/td>192.168.100.10<\/td>static<\/td><\/tr>
bs-1<\/td>bootstrap-0<\/td>demo.com<\/a><\/td>192.168.100.70<\/td>DHCP<\/td><\/tr>
ctrl-1<\/td>control-plane-0<\/td>demo.com<\/a><\/td>192.168.100.71<\/td>DHCP<\/td><\/tr>
ctrl-2<\/td>control-plane-1<\/td>demo.com<\/a><\/td>192.168.100.72<\/td>DHCP<\/td><\/tr>
ctrl-3<\/td>control-plane-2<\/td>demo.com<\/a><\/td>192.168.100.73<\/td>DHCP<\/td><\/tr>
cmpt-1<\/td>compute-0<\/td>demo.com<\/a><\/td>192.168.100.74<\/td>DHCP<\/td><\/tr>
cmpt-2<\/td>compute-1<\/td>demo.com<\/a><\/td>192.168.100.75<\/td>DHCP<\/td><\/tr>
cmpt-3<\/td>compute-2<\/td>demo.com<\/a><\/td>192.168.100.76<\/td>DHCP<\/td><\/tr><\/tbody><\/table>\n\n\n\n

Download all the required images, installer, client tools and image Pull Secret from : https:\/\/cloud.redhat.com\/openshift\/install\/vsphere\/user-provisioned<\/a>

I’m using rhcos-4.1.0-x86_64-metal-bios.raw.gz as all the vm’s created on the esxi nodes are with BIOS.<\/p>\n\n\n\n

Now lets prepare the infrastructure:

1> Configure the DHCP on the basecamp to assign IP for all the VM on bootup, I’m also trying to map IP to specific nodes using MAC address.

Below is the config used for DHCP on the basecamp:<\/p>\n\n\n\n

# dhcpd.conf\n#\n# Sample configuration file for ISC dhcpd\n#\n\n# option definitions common to all supported networks...\noption domain-name \"demo.com\";\noption domain-name-servers basecamp.demo.com;\n\ndefault-lease-time 600;\nmax-lease-time 7200;\n\n# Use this to enble \/ disable dynamic dns updates globally.\n#ddns-update-style none;\n\n# If this DHCP server is the official DHCP server for the local\n# network, the authoritative directive should be uncommented.\n#authoritative;\n\n# Use this to send dhcp log messages to a different log file (you also\n# have to hack syslog.conf to complete the redirection).\nlog-facility local7;\n\n# No service will be given on this subnet, but declaring it helps the \n# DHCP server to understand the network topology.\n\n# This is a very basic subnet declaration\n\nsubnet 192.168.100.0 netmask 255.255.255.0 {\n  range 192.168.100.80 192.168.100.100;\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n  option domain-name \"demo.com\";\n}\n\n# This declaration allows BOOTP clients to get dynamic addresses,\n# which we don't really recommend.\n\n#subnet 10.254.239.32 netmask 255.255.255.224 {\n#  range dynamic-bootp 10.254.239.40 10.254.239.60;\n#  option broadcast-address 10.254.239.31;\n#  option routers rtr-239-32-1.example.org;\n#}\n\n# A slightly different configuration for an internal subnet.\n#subnet 10.5.5.0 netmask 255.255.255.224 {\n#  range 10.5.5.26 10.5.5.30;\n#  option domain-name-servers ns1.internal.example.org;\n#  option domain-name \"internal.example.org\";\n#  option routers 10.5.5.1;\n#  option broadcast-address 10.5.5.31;\n#  default-lease-time 600;\n#  max-lease-time 7200;\n#}\n\n# Hosts which require special configuration options can be listed in\n# host statements.   If no address is specified, the address will be\n# allocated dynamically (if possible), but the host-specific information\n# will still come from the host declaration.\n\n#host passacaglia {\n#  hardware ethernet 0:0:c0:5d:bd:95;\n#  filename \"vmunix.passacaglia\";\n#  server-name \"toccata.fugue.com\";\n#}\n\n# Fixed IP addresses can also be specified for hosts.   These addresses\n# should not also be listed as being available for dynamic assignment.\n# Hosts for which fixed IP addresses have been specified can boot using\n# BOOTP or DHCP.   Hosts for which no fixed address is specified can only\n# be booted with DHCP, unless there is an address range on the subnet\n# to which a BOOTP client is connected which has the dynamic-bootp flag\n# set.\nhost bootstrap-0 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.70;\n  option host-name \"bootstrap-0.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost compute-0 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.74;\n  option host-name \"compute-0.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost compute-1 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.75;\n  option host-name \"compute-1.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost compute-2 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.76;\n  option host-name \"compute-2.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost control-plane-0 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.71;\n  option host-name \"control-plane-0.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost control-plane-1 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.72;\n  option host-name \"control-plane-1.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}\n\nhost control-plane-2 {\n  hardware ethernet <<MAC Address of VM>>;\n  fixed-address 192.168.100.73;\n  option host-name \"control-plane-2.upi.demo.com\";\n  option routers 192.168.100.10;\n  option domain-name-servers 192.168.100.10;\n}<\/pre>\n\n\n\n
2> DNS configuration
As this is a lab setup, and I dont have a LB, we will use DNS with round robin.<\/p>\n\n\n\n
[root@basecamp ~]# cat \/etc\/named\/zones\/db.demo.com \n$TTL 86400\n@       IN      SOA     basecamp.demo.com. root.basecamp.demo.com. (\n              3         ; Serial\n             604800     ; Refresh\n              86400     ; Retry\n            2419200     ; Expire\n             604800 )   ; Negative Cache TTL\n; name servers - NS records\n@     IN      NS      basecamp.demo.com.\n; name servers - A records\nbasecamp IN A 192.168.100.10\n; 192.168.100.0\/16 - A records\nvcenter        IN A       192.168.100.60\nesxi01         IN A       192.168.100.51\nesxi02         IN A       192.168.100.52\nesxi03         IN A       192.168.100.53\n\n$ORIGIN apps.upi.demo.com.\n* A 192.168.100.74\n* A 192.168.100.75\n* A 192.168.100.76\n\n$ORIGIN upi.demo.com.\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-0\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-1\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-2\nbootstrap-0     A 192.168.100.70\ncontrol-plane-0 A 192.168.100.71\ncontrol-plane-1 A 192.168.100.72\ncontrol-plane-2 A 192.168.100.73\napi     IN      A 192.168.100.71\n        IN      A 192.168.100.72\n        IN      A 192.168.100.73\n        IN      A 192.168.100.70\napi-int IN      A 192.168.100.71\n        IN      A 192.168.100.72\n        IN      A 192.168.100.73\n        IN      A 192.168.100.70\netcd-0          A 192.168.100.71\netcd-1          A 192.168.100.72\netcd-2          A 192.168.100.73\ncompute-0       A 192.168.100.74\ncompute-1       A 192.168.100.75\ncompute-2       A 192.168.100.76<\/pre>\n\n\n\n
Below is the iptables setting to create portrouting for the VM to connect to external network (internet)<\/p>\n\n\n\n
[root@basecamp tmp]# cat \/etc\/sysconfig\/iptables\n*mangle\n:PREROUTING ACCEPT [2603:265754]\n:INPUT ACCEPT [2594:264998]\n:FORWARD ACCEPT [9:756]\n:OUTPUT ACCEPT [1676:336014]\n:POSTROUTING ACCEPT [1685:336770]\nCOMMIT\n*nat\n:PREROUTING ACCEPT [17:2820]\n:INPUT ACCEPT [16:2736]\n:OUTPUT ACCEPT [3:209]\n:POSTROUTING ACCEPT [0:0]\n-A POSTROUTING -o wlp58s0 -j MASQUERADE\nCOMMIT\n*filter\n:INPUT ACCEPT [2568:262171]\n:FORWARD ACCEPT [9:756]\n:OUTPUT ACCEPT [1657:326928]\nCOMMIT<\/pre>\n\n\n\n
Now, lets prepare the installation configuration file.
create a new directory to store all the install configs and create a new file as below “install-config.yaml”<\/p>\n\n\n\n
[root@basecamp tmp]# cat install-config.yaml \napiVersion: v1\nbaseDomain: demo.com\ncompute:\n- hyperthreading: Enabled \n  name: worker\n  replicas: 0\ncontrolPlane:\n  hyperthreading: Enabled\n  name: master \n  replicas: 3 \nmetadata:\n  name: upi\nnetworking:\n  clusterNetworks:\n  - cidr: 10.128.0.0\/14\n    hostPrefix: 23\n  networkType: OpenShiftSDN\n  serviceNetwork:\n  - 172.30.0.0\/16\nplatform:\n  none: {}\npullSecret: \u2018get the secret from https:\/\/cloud.redhat.com\/openshift\/install\/metal\/user-provisioned ' \nsshKey: \u2019ssh public key to access the OCP nodes with core user\u2019 <\/pre>\n\n\n\n
Create the ignition and Kubeconfig files:<\/p>\n\n\n\n
# .\/openshift-install create ignition-configs --dir=\/tmp\/ocp<\/strong>_41\/install-dir<\/pre>\n\n\n\n
This will create the ignition files for bootstrap, master and compute. And also generates kubeconfig and kubeadin-password files in the same location under auth directory<\/p>\n\n\n\n
Install web server on the basecamp and move the ignition files to the webserver directory. These files will be fetched by RHCOS when bootstrapping.

All the ignition files and rhcos image is placed in the web server<\/p>\n\n\n\n
[root@basecamp ~]# ls -l \/var\/www\/html\/\ntotal 1408020\n-rw-r--r--. 1 root root    278269 Aug 14 14:50 bootstrap.ign\n-rw-r--r--. 1 root root      1817 Aug 14 14:50 master.ign\n-rw-r--r--. 1 root root 719954527 Aug  6 13:48 rhcos-4.1.0-x86_64-metal-bios.raw.gz\n-rw-r--r--. 1 root root 721567045 Jul 15 10:24 rhcos-4.1.0-x86_64-metal-uefi.raw.gz\nlrwxrwxrwx. 1 root root        36 Aug  6 13:49 rhcos.raw.gz -> rhcos-4.1.0-x86_64-metal-bios.raw.gz\n-rw-r--r--. 1 root root      1817 Aug 14 14:50 worker.ign<\/pre>\n\n\n\n
Copy the RHCPS installer ISO to the shared datastore on the esxi nodes<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Attach the RHCOS iso to all the VM’s and power on<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Select the disk, and provide the image URL and Ignition file URL details to start the boot strapping and installation.<\/p>\n\n\n\n
Image URL : http:\/\/<<IP address of the Webserver>>\/rhcos.raw.gz \n\nIgnition file URL : \nhttp:\/\/<<IP address of the Webserver>>\/bootstrap.ign\nhttp:\/\/<<IP address of the Webserver>>\/master.ign\nhttp:\/\/<<IP address of the webserver>>\/worker.ign<\/pre>\n\n\n\n
Lets wait for the bootstrap VM, you can monitor this using the below command:<\/p>\n\n\n\n
[root@basecamp ocp_41]# .\/openshift-install --dir=\/tmp\/ocp_41\/install-dir wait-for bootstrap-complete --log-level debug<\/strong>\n\nDEBUG OpenShift Installer v4.1.8-201907241243-dirty \nDEBUG Built from commit e8d7e37ea7655522ac8f6ede471fd1d3ebd1bcba \nINFO Waiting up to 30m0s for the Kubernetes API at https:\/\/api.upi.demo.com:6443... \nDEBUG Still waiting for the Kubernetes API: the server could not find the requested resource \nDEBUG Still waiting for the Kubernetes API: the server could not find the requested resource \nINFO API v1.13.4+4b86cc5 up                       \nINFO Waiting up to 30m0s for bootstrapping to complete... \nDEBUG Bootstrap status: complete                   \nINFO It is now safe to remove the bootstrap resources <\/pre>\n\n\n\n
Now power off the bootstrap VM and remove the entry from the DNS file:<\/p>\n\n\n\n
[root@basecamp ~]# cat \/etc\/named\/zones\/db.demo.com \n$TTL 86400\n@       IN      SOA     basecamp.demo.com. root.basecamp.demo.com. (\n              3         ; Serial\n             604800     ; Refresh\n              86400     ; Retry\n            2419200     ; Expire\n             604800 )   ; Negative Cache TTL\n; name servers - NS records\n@     IN      NS      basecamp.demo.com.\n; name servers - A records\nbasecamp IN A 192.168.100.10\n; 192.168.100.0\/16 - A records\nvcenter        IN A       192.168.100.60\nesxi01         IN A       192.168.100.51\nesxi02         IN A       192.168.100.52\nesxi03         IN A       192.168.100.53\n\n$ORIGIN apps.upi.demo.com.\n* A 192.168.100.74\n* A 192.168.100.75\n* A 192.168.100.76\n\n$ORIGIN upi.demo.com.\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-0\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-1\n_etcd-server-ssl._tcp SRV 0 10 2380 etcd-2\nbootstrap-0     A 192.168.100.70\ncontrol-plane-0 A 192.168.100.71\ncontrol-plane-1 A 192.168.100.72\ncontrol-plane-2 A 192.168.100.73\napi     IN      A 192.168.100.71\n        IN      A 192.168.100.72\n        IN      A 192.168.100.73\napi-int IN      A 192.168.100.71\n        IN      A 192.168.100.72\n        IN      A 192.168.100.73\netcd-0          A 192.168.100.71\netcd-1          A 192.168.100.72\netcd-2          A 192.168.100.73\ncompute-0       A 192.168.100.74\ncompute-1       A 192.168.100.75\ncompute-2       A 192.168.100.76<\/pre>\n\n\n\n
From the downloaded client tools (openshift-client-linux-4.1.11.tar.gz), extract the oc utility. <\/p>\n\n\n\n
[root@basecamp ocp_41]# tar -xvf openshift-client-linux-4.1.11.tar.gz \n README.md\n oc\n kubectl\n [root@basecamp ocp_41]# ls\n kubectl  oc  openshift-client-linux-4.1.11.tar.gz  README.md <\/pre>\n\n\n\n
export KUBECONFIG=<installation_directory>\/auth\/kubeconfig <\/pre>\n\n\n\n
Use oc command and the auth file to check the nodes<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Check for any pending CSR on the nodes and approve is any pending<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now lets configure the storage for image repository
As this is a lab setup setting the image registry to empty directory<\/p>\n\n\n\n
#oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{\"spec\":{\"storage\":{\"emptyDir\":{}}}}'<\/pre>\n\n\n\n
Before finishing the installation, check if all the cluster operators are online.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Check for the cluster installation completion using the below command<\/p>\n\n\n\n
[root@basecamp ocp_41]# .\/openshift-install --dir=\/tmp\/ocp_41\/install-dir wait-for install-complete <\/strong>\n\nINFO Waiting up to 30m0s for the cluster at https:\/\/api.upi.demo.com:6443 to initialize... \nINFO Waiting up to 10m0s for the openshift-console route to be created... \nINFO Install complete!                            \nINFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=\/tmp\/ocp_41\/install-dir\/auth\/kubeconfig' \nINFO Access the OpenShift web-console here: https:\/\/console-openshift-console.apps.upi.demo.com \nINFO Login to the console with user: kubeadmin, password: XXXXX-XXXXX-XXXXX-XXXXX<\/pre>\n\n\n\n
Now try to login to the console using the above mentioned user and password<\/p>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
This is a home lab setup to test deployment of Red Hat OpenShift 4.1 on VMware vSphere 6.7 using Intel NUC. Below is a high level view of the hardware setup. NUC 1 is my jump-host to access all the other hosts.Ctrl 1-3 are the OpenShift master nodes, Cmpt 1-3 are the worker nodes and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51],"tags":[46,47,45,44,49,48,43],"class_list":["post-99","post","type-post","status-publish","format-standard","hentry","category-openshift","tag-ocp","tag-ocp-on-vmware","tag-ocp-upi","tag-ocp4-1","tag-openshift-4-1","tag-openshift-on-vmware","tag-red-hat-openshift"],"_links":{"self":[{"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/posts\/99","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=99"}],"version-history":[{"count":4,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/posts\/99\/revisions"}],"predecessor-version":[{"id":113,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=\/wp\/v2\/posts\/99\/revisions\/113"}],"wp:attachment":[{"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=99"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=99"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtorbis.virtcompute.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=99"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}