The clue is that There are 1 datanode(s) running and 1 node(s) are excluded in this operation. This is a single node CDH, so we would expect only one node to be running, but why is it excluded? By default, the CDH Quickstart VM runs in NAT mode – it has its own private IP address, in my case 10.0.2.15, with port forwarding configured so that Hadoop FS, Hive etc are available from the host machine. I have quickstart.cloudera set to 127.0.0.1 (localhost) in my /etc/hosts so I can use that hostname to access services in the VM. What's happening is that SDC's request to the Hadoop name node is forwarded correctly, and the name node returns the data node's location. By default, the Hadoop client library tries to connect to the name node's IP address, 10.0.2.15, on port 50075, but this fails, since that address is inside the VM NAT, and not accessible from the host.
How do we resolve the problem? One way would be to reconfigure the VM to use bridged networking, so the VM's IP address is directly accessible from the host, but I chose a quicker, easier fix. Setting dfs.client.use.datanode.hostname to true in the Hadoop FS destination configuration tells the Hadoop client to connect using the hostname, quickstart.cloudera, rather than the IP address.
Now requests to the name node are forwarded to the VM, and my pipeline runs.
